Lead Data Engineer - dbt, Python, SQL, Databricks

Required Skills & Experience

• Advanced SQL and strong data modeling expertise.
• Strong knowledge of data warehousing and ETL/ELT architecture.
• Experience with Databricks, Snowflake, AWS, or similar cloud data platforms.
• Experience with CI/CD, GitHub, and production data pipelines.
• 8+ years of data engineering experience.
• dbt Core or dbt Cloud

Nice to Have Skills & Experience

• Spark SQL and Python
• Streaming ingestion and schema evolution
• Parquet and JSON processing

Job Description

Seeking a Senior/Lead Data Engineer to design, build, and optimize scalable data pipelines supporting analytics, AI, and machine learning initiatives. This role will own pipeline development from ingestion through transformation and delivery while driving data quality, reliability, and engineering best practices.

Key Responsibilities

• Design and build scalable, fault-tolerant data pipelines using modern cloud data platforms.
• Develop complex SQL transformations and data models leveraging medallion architecture.
• Build and maintain DBT testing, data quality, and validation frameworks.
• Support batch and streaming ingestion, including schema evolution and Parquet/JSON processing.
• Optimize pipeline performance, reliability, and scalability.
• Manage CI/CD and deployment workflows through GitHub.
• Partner with ML engineers and data scientists to support AI/ML data and feature pipelines.
• Drive strong documentation, lineage, and engineering standards.