Job Description
- Design, build, and maintain end-to-end IoT data pipelines from edge devices into Databricks using the medallion architecture (Bronze/Silver/Gold)
- Lead the development and optimization of streaming ingestion pipelines using Apache Kafka and Databricks Structured Streaming
- Define and enforce data engineering best practices, including pipeline design patterns, data quality frameworks, and schema governance
- Architect and manage Delta Lake tables, including partitioning strategies, Z-ordering, and vacuuming for performance and cost efficiency
- Collaborate with infrastructure and cloud teams to configure and manage Databricks clusters, Unity Catalog, and Delta Live Tables
- Mentor junior and mid-level data engineers through code reviews, pair programming, and technical guidance
- Partner with data scientists and analysts to ensure processed data meets downstream analytical and ML requirements
- Troubleshoot production pipeline failures and drive root cause analysis and resolution
- Evaluate and introduce new tools and technologies to continuously improve the data platform
Skills and Requirements
- 5+ years of data engineering experience, with at least 2 years working extensively with Databricks
- Strong proficiency in Python and SQL, with experience writing production-grade Spark code
- Hands-on experience building and managing Kafka-based streaming pipelines
- Deep familiarity with Delta Lake, Delta Live Tables, and the Databricks ecosystem (Unity Catalog, Workflows, MLflow)
- Experience with IoT data patterns - high-frequency telemetry ingestion, device state management, time-series data
- Solid understanding of cloud infrastructure (Azure preferred) and infrastructure-as-code concepts
- Strong communication skills with the ability to lead technical discussions and translate requirements into architecture
- Experience with Geotab or similar fleet telematics platforms
- Databricks Certified Data Engineer Associate or Professional certification
- Familiarity with Azure Key Vault, Event Hubs, or Azure Data Factory
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to HR@insightglobal.com.