Robert Half is seeking a Senior Data Engineer to design, develop, and maintain scalable cloud-based data solutions supporting enterprise reporting, analytics, and business operations. This position will focus heavily on Azure Databricks, building reliable data pipelines, integrating data from multiple enterprise systems, and improving data quality and performance.
The ideal candidate has strong hands-on data engineering experience and can independently develop pipelines and data models while collaborating with architects, analysts, and business stakeholders on solution design. Experience with manufacturing environments or ERP data is highly valued.
Residency Requirement: Candidates must currently reside in Wisconsin to be considered for this position. Flexible hybrid/remote work schedule, with occasional onsite for meetings.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks, PySpark, Python, and SQL.
- Build and optimize data ingestion processes across ERP, manufacturing, supply chain, inventory, finance, and other enterprise systems.
- Develop and maintain cloud-based lakehouse solutions using Delta Lake and Medallion Architecture.
- Create and optimize data models supporting reporting, business intelligence, analytics, and operational decision-making.
- Develop reusable data engineering components and frameworks to improve efficiency, reliability, and maintainability.
- Integrate Azure Databricks with Azure Data Factory, Azure Data Lake Storage, and other cloud services.
- Monitor and troubleshoot data pipelines, workflows, and Databricks environments to improve performance and resolve production issues.
- Implement data validation, quality controls, security, and governance standards.
- Collaborate with data architects, analysts, and business stakeholders to translate requirements into scalable technical solutions.
- Participate in code reviews, technical design discussions, and Agile development activities.
- 3–5+ years of experience in data engineering, cloud data development, or a related field.
- At least 2 years of hands-on experience developing solutions in Azure Databricks.
- Strong experience with Python, PySpark, SQL, ETL/ELT pipelines, and data modeling.
- Experience with Azure Data Factory, Azure Data Lake Storage, and cloud-based data warehousing or lakehouse environments.
- Understanding of Delta Lake, Apache Spark, and modern data architecture concepts.
- Experience developing, troubleshooting, and optimizing production data pipelines.
- Familiarity with Git, Azure DevOps, CI/CD, and data engineering best practices.
- Ability to independently manage development tasks and collaborate with technical and business teams.
- Bachelor's degree in a related field or equivalent combination of education and experience.
Preferred Qualifications
- Experience supporting data engineering initiatives in manufacturing or industrial environments.
- Familiarity with ERP systems, particularly Infor M3, and manufacturing data related to production, inventory, supply chain, or finance.
- Experience with Snowflake, Microsoft Fabric, Power BI, or other cloud data platforms.
- Knowledge of Unity Catalog, data governance, and cloud security practices.
- Experience with streaming data, real-time analytics, or AI/ML data preparation.
- Familiarity with Infrastructure as Code tools such as Terraform or Bicep.
- Azure or Databricks certifications.
Work Environment
- Flexible remote/hybrid arrangement for candidates in Wisconsin, with occasional onsite meetings based on business or project needs.
- Onsite expectations are generally limited to project needs, team meetings, and occasional collaboration.