Data Engineer III | 1177

About This Opportunity

CTI Staffing is partnering with a leading retail organization to find a Senior Databricks Data Engineer for their enterprise data team based in the Greater Boston, MA area.

The team is building out a modern Azure data platform that brings together data from many retail systems. This role matters because reliable, incremental pipelines are what let the business trust its data and act on it faster.

What You’ll Do

• Design, build, and optimize ETL/ELT pipelines using Databricks and PySpark.
• Build Change Data Capture pipelines for incremental and near real-time processing.
• Develop scalable data solutions on Azure using Delta Lake and Kafka streaming.
• Improve existing pipelines by quickly learning current codebases and delivering targeted upgrades.
• Ingest and process data from flat files, XML, JSON, SQL Server, JDBC sources, APIs, message queues, and Kafka topics.
• Implement data quality rules and quarantine error records.
• Apply data modeling techniques including SCD Type 1 and Type 2.
• Automate workflows with Databricks Workflows and GitHub-based CI/CD, and work with developers, architects, and product owners in an Agile team.

Requirements

What You Bring

Must-Have:

• Extensive hands-on Databricks experience, able to contribute from day one.
• Proven experience building end-to-end data pipelines with CDC.
• Strong PySpark and SQL skills.
• Experience working in Microsoft Azure.
• Working knowledge of Delta Lake and distributed data processing.
• Experience with Kafka or comparable streaming technologies.
• Hands-on experience with SCD Type 1 and Type 2 data modeling.
• Familiarity with GitHub and CI/CD for version control and deployment.

Nice-to-Have:

• Python experience beyond PySpark.
• Databricks Workflows and Unity Catalog experience.
• Experience with message queues and API-based ingestion.
• Retail, CPG, or supply chain data background.
• Databricks certification.

Technical Environment: Azure, Databricks, PySpark, SQL, Delta Lake, Kafka, Unity Catalog, GitHub, CI/CD, SQL Server, JSON, XML, REST APIs.