Data Engineer

Required Skills & Experience

• Extensive experience designing and building enterprise-scale data engineering solutions.
• Strong hands-on experience with Python, PySpark, SQL, Apache Spark and distributed data processing.
• Experience designing and operating large-scale batch and real-time data pipelines and modern lakehouse platforms like databricks.
• Experience building cloud-native data solutions using AWS, Azure, or comparable enterprise cloud platforms.
• Experience with modern data architecture patterns including data lakes, lakehouses, data products, and event-driven architectures.
• Experience with orchestration technologies such as Apache Airflow, Databricks Workflows equivalent.
• Strong understanding of data governance, data quality, lineage, metadata management, security, privacy, and access controls.
• Ability to diagnose and resolve complex performance, scalability, reliability, and operational issues.
• Strong problem-solving, communication, and cross-functional collaboration skills.
• Experience integrating structured, semi-structured, unstructured, and streaming data.
• Ability to work effectively in an R&D environment with evolving requirements and rapid experimentation.

Nice to Have Skills & Experience

• Experience supporting machine learning, generative AI, or agentic AI applications.
• Experience with CI/CD, automated testing, version control, Infrastructure as Code, observability, and platform automation.
• Experience with MLflow, Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric.
• Experience with Kafka or other event-streaming technologies.
• Experience in payments, financial services, or another regulated industry.

Job Description

This is a highly hands-on engineering role focused on translating evolving R&D requirements into secure, reliable, scalable, and production-ready data platforms and pipelines. You will collaborate with AI Engineers, Data Scientists, Researchers, Software Engineers, and Product Managers to accelerate experimentation and production delivery.

• Design and build scalable batch, streaming, and event-driven data pipelines and platforms.
• Develop data ingestion, transformation, enrichment, processing, and consumption capabilities using modern lakehouse and cloud-native architectures.
• Build reusable data products, APIs, services, and pipelines supporting analytics, machine learning, generative AI, and agentic AI workloads.
• Optimize solutions for performance, scalability, reliability, resiliency, observability, and cost efficiency.
• Implement engineering practices including data modeling, automated testing, CI/CD, Infrastructure as Code, and deployment automation.
• Implement data quality, lineage, metadata, cataloging, security, privacy, governance, and access controls.
• Evaluate emerging data and AI technologies through prototypes and proof-of-concepts and transition successful solutions toward production.
• Troubleshoot complex data, performance, scalability, and production issues.
• Translate evolving product and R&D requirements into practical technical designs and incremental delivery plans.
• Collaborate with cross-functional teams and communicate technical decisions, dependencies, risks, and trade-offs.