Required Skills & Experience
- Extensive experience designing and building enterprise-scale data engineering solutions.
- Strong hands-on experience with Python, PySpark, SQL, Apache Spark and distributed data processing.
- Experience designing and operating large-scale batch and real-time data pipelines and modern lakehouse platforms like databricks.
- Experience building cloud-native data solutions using AWS, Azure, or comparable enterprise cloud platforms.
- Experience with modern data architecture patterns including data lakes, lakehouses, data products, and event-driven architectures.
- Experience with orchestration technologies such as Apache Airflow, Databricks Workflows equivalent.
- Strong understanding of data governance, data quality, lineage, metadata management, security, privacy, and access controls.
- Ability to diagnose and resolve complex performance, scalability, reliability, and operational issues.
- Strong problem-solving, communication, and cross-functional collaboration skills.
- Experience integrating structured, semi-structured, unstructured, and streaming data.
- Ability to work effectively in an R&D environment with evolving requirements and rapid experimentation.
Nice to Have Skills & Experience
- Experience supporting machine learning, generative AI, or agentic AI applications.
- Experience with CI/CD, automated testing, version control, Infrastructure as Code, observability, and platform automation.
- Experience with MLflow, Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric.
- Experience with Kafka or other event-streaming technologies.
- Experience in payments, financial services, or another regulated industry.
Job Description
This is a highly hands-on engineering role focused on translating evolving R&D requirements into secure, reliable, scalable, and production-ready data platforms and pipelines. You will collaborate with AI Engineers, Data Scientists, Researchers, Software Engineers, and Product Managers to accelerate experimentation and production delivery.
- Design and build scalable batch, streaming, and event-driven data pipelines and platforms.
- Develop data ingestion, transformation, enrichment, processing, and consumption capabilities using modern lakehouse and cloud-native architectures.
- Build reusable data products, APIs, services, and pipelines supporting analytics, machine learning, generative AI, and agentic AI workloads.
- Optimize solutions for performance, scalability, reliability, resiliency, observability, and cost efficiency.
- Implement engineering practices including data modeling, automated testing, CI/CD, Infrastructure as Code, and deployment automation.
- Implement data quality, lineage, metadata, cataloging, security, privacy, governance, and access controls.
- Evaluate emerging data and AI technologies through prototypes and proof-of-concepts and transition successful solutions toward production.
- Troubleshoot complex data, performance, scalability, and production issues.
- Translate evolving product and R&D requirements into practical technical designs and incremental delivery plans.
- Collaborate with cross-functional teams and communicate technical decisions, dependencies, risks, and trade-offs.