Sr Data Engineer (Azure Databricks)

Required Skills & Experience

• 5-7+ years of Data Engineering experience building large-scale data platforms
• Strong hands-on experience with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS)
• Advanced PySpark development experience, including DataFrames, transformations, joins, aggregations, and performance tuning
• Experience developing and maintaining Delta Live Tables (DLT) pipelines
• Strong SQL background with experience supporting legacy data warehouse environments
• Experience migrating SQL-based pipelines to modern Python/PySpark architectures
• Experience with Infrastructure as Code (IaC) and Pipeline as Code methodologies
• Proficiency with YAML, JSON, and configuration-driven development patterns
• Experience automating CI/CD deployments using DevOps pipelines
• Strong understanding of Medallion Architecture (Bronze, Silver, Gold)
• Experience building ingestion, transformation, and curation pipelines across Bronze, Silver, and Gold layers
• Experience troubleshooting, monitoring, and supporting production data pipelines
• Familiarity with Databricks Asset Bundles, Unity Catalog, and Delta Lake preferred
• Healthcare industry experience preferred
• Medicare, payer, member enrollment, or claims data experience strongly preferred Understanding of healthcare data quality, PHI/PII handling, and regulatory requirements preferred

Job Description

• Design, develop, and deploy Azure Databricks data pipelines using PySpark and Delta Live Tables
• Modernize legacy on-prem SQL pipelines into scalable cloud-native Databricks solutions
• Build configuration-driven data pipelines using YAML, JSON, and reusable framework patterns
• Develop and maintain CI/CD deployment processes for data engineering workflows
• Implement and support Medallion Architecture pipelines across Bronze, Silver, and Gold layers
• Partner with upstream and downstream teams to ensure data quality, reliability, and scalability
• Troubleshoot and resolve production pipeline issues across Databricks and legacy SQL environments
• Enhance monitoring, observability, and operational support processes
• Support Medicare and payer-focused initiatives involving member, claims, and operational datasets
• Follow established architecture standards and engineering best practices to ensure platform consistency and maintainability