Archived listing. The details below describe a past opening.
Required Skills & Experience
- 5+ years of Data Engineering experience.
- Strong hands-on experience with Google BigQuery and Databricks.
- Proven experience building and supporting BigQuery → Databricks replication or migration pipelines.
- Strong proficiency in SQL and data modeling concepts.
- Advanced experience with PySpark and Databricks notebook development.
- Experience with large-scale data movement and distributed data processing.
- Experience implementing incremental load and replication strategies.
- Strong understanding of performance tuning and optimization techniques.
- Experience working in cloud-based data environments.
Excellent analytical, troubleshooting, and problem-solving skills.
Nice to Have Skills & Experience
- Experience with Apache Iceberg, especially Databricks-managed Iceberg tables.
- Experience with Unity Catalog and enterprise data governance.
- Experience with BigQuery authorized views and federated connections.
- Experience orchestrating and productionizing Databricks jobs and workflows.
- Knowledge of CI/CD pipelines for data engineering solutions.
- Experience with data lakehouse architectures.
- Prior experience supporting enterprise-scale cloud migration projects. Retail industry experience.
Job Description
Overview:
Home Depot is seeking experienced Data Engineers to support a high-priority initiative focused on replicating enterprise data from Google BigQuery into Databricks. This role will be responsible for designing, building, and optimizing large-scale data replication pipelines while ensuring performance, scalability, and reliability across the data platform.
The ideal candidate will have hands-on experience with BigQuery-to-Databricks migrations and a strong background in modern data engineering, distributed processing, and cloud-based data platforms.
Key Responsibilities:
-Design, develop, and maintain data replication pipelines from BigQuery to Databricks. -Build scalable ingestion frameworks supporting large-scale enterprise datasets. -Develop and optimize Databricks notebooks using PySpark. -Implement incremental data replication and change data capture (CDC) strategies. -Optimize query performance, data movement, and processing efficiency. -Work with BigQuery authorized views and federated connections to support enterprise data access patterns. -Utilize Apache Iceberg and Unity Catalog for data governance and management. -Collaborate with architects, engineers, and business stakeholders to define migration requirements and solutions. -Productionize and monitor Databricks workflows and jobs. -Troubleshoot data quality, performance, and replication issues across environments.