Azure Databricks Engineer

Required Skills & Experience

• 5+ years as a data engineer with focus on Azure cloud services
• Experience working in financial services or corporate enterprise environment
• Hands on experience with Databricks, including Spark and distributed data processing
• SQL experience
• Experience building data pipoelines using Azure Data Factory or similar tools
• Azure AQL Database experience
• Experience with Python or Scala
• Experience with C# and PowerShell for automation/integration tasks

Nice to Have Skills & Experience

• Experience with Azure DevOps and CI/CD pipelines
• Experience with Delta Lake and modern data lakehouse architectures
• Familiarity with data governance and data catalog tools
• Experience working in Agile or Scrum environments
• Microsoft or Databricks certifications

Job Description

Insight Global is looking for an Azure Databricks Engineer to join one of their Investment Banking clients in New York City. The individual will operate across multiple lines of business within corporate technology including: Real Estate Strategy & Operations, Human Capital Group, Compliance, and Finance. This role focuses on building scalable data solutions, driving data integration, and enabling analytics across the firm. The ideal candidate brings strong data engineering experience with Azure and Databricks, and works effectively across teams to deliver shared data platforms and pipelines. This is a hybrid role requiring 3 days a week onsite in Manhattan.

Responsibilities include:

• Design, build, and maintain scalable data pipelines using Azure and Databricks
• Develop and optimize ETL and ELT workflows for structured and unstructured data
• Collaborate with business and technical teams across corporate functions to define data requirements and deliver solutions
• Implement data models and data architectures to support analytics and reporting use cases
• Integrate enterprise systems such as ServiceNow, Dataverse, and Workday into centralized data platforms
• Ensure data quality, governance, and security standards are met across all solutions
• Monitor and troubleshoot data workflows, ensuring reliability and performance
• Contribute to the evolution of enterprise data platforms and recommend improvements based on emerging technologies