IoT Data Engineer

Required Skills & Experience

• 5+ years of data engineering experience, with at least 2 years working extensively with Databricks
• Strong proficiency in Python and SQL, with experience writing production-grade Spark code
• Hands-on experience building and managing Kafka-based streaming pipelines
• Deep familiarity with Delta Lake, Delta Live Tables, and the Databricks ecosystem (Unity Catalog, Workflows, MLflow)
• Experience with IoT data patterns - high-frequency telemetry ingestion, device state management, time-series data
• Solid understanding of cloud infrastructure (Azure preferred) and infrastructure-as-code concepts
• Strong communication skills with the ability to lead technical discussions and translate requirements into architecture

Nice to Have Skills & Experience

• Experience with Geotab or similar fleet telematics platforms
• Databricks Certified Data Engineer Associate or Professional certification
• Familiarity with Azure Key Vault, Event Hubs, or Azure Data Factory

Job Description

• Design, build, and maintain end-to-end IoT data pipelines from edge devices into Databricks using the medallion architecture (Bronze/Silver/Gold)
• Lead the development and optimization of streaming ingestion pipelines using Apache Kafka and Databricks Structured Streaming
• Define and enforce data engineering best practices, including pipeline design patterns, data quality frameworks, and schema governance
• Architect and manage Delta Lake tables, including partitioning strategies, Z-ordering, and vacuuming for performance and cost efficiency
• Collaborate with infrastructure and cloud teams to configure and manage Databricks clusters, Unity Catalog, and Delta Live Tables
• Mentor junior and mid-level data engineers through code reviews, pair programming, and technical guidance
• Partner with data scientists and analysts to ensure processed data meets downstream analytical and ML requirements
• Troubleshoot production pipeline failures and drive root cause analysis and resolution
• Evaluate and introduce new tools and technologies to continuously improve the data platform