Required Skills & Experience
- 7+ years of professional Data Engineering experience, including designing and supporting enterprise-scale data platforms.
- Advanced Python development experience, including production data pipelines, reusable frameworks, automation, and data processing.
- Advanced SQL expertise, including complex transformations, joins, window functions, query optimization, and large-scale data processing.
- Strong experience designing, developing, and maintaining production ETL/ELT pipelines.
- Hands-on experience with cloud-native data platforms, preferably GCP; strong AWS experience is also accepted.
- Experience with data modeling, dimensional modeling, data warehousing, and scalable pipeline architecture.
- Experience with batch processing and workflow orchestration using tools such as Airflow/Cloud Composer or comparable technologies.
- Experience implementing data quality, validation, governance, monitoring, and troubleshooting for production pipelines.
- Proficiency with Git/GitHub, source control, testing, and modern software engineering practices.
- Strong understanding of distributed data processing, pipeline reliability, and scalable system design.
- Demonstrated ability to independently lead technical solutions, evaluate architectural trade-offs, and take ownership of production systems.
Nice to Have Skills & Experience
Google Cloud Platform experience, including BigQuery, Vertex AI, Cloud Run, Cloud Composer, and Pub/Sub. Experience with dbt. Experience with machine learning deployment and MLOps practices. Experience supporting recommendation systems, customer analytics, pricing solutions, experimentation platforms, or optimization models. Experience handling large-scale transactional, customer, web, or geospatial datasets. Exposure to LLMs, RAG frameworks, AI agents, vector databases, or generative AI applications. Experience with graph technologies such as Neo4j or similar platforms. Experience building internal platforms, reusable frameworks, or developer tooling. Experience working in highly collaborative environments where engineers and Data Scientists partner closely.
Job Description
We are seeking a highly technical, hands-on Senior Data Engineer with 7+ years of experience designing, building, and scaling enterprise data platforms and production data pipelines. This role sits within our Data Science and Engineering organization, supporting high-impact initiatives across dynamic pricing, customer analytics, website recommendations, forecasting, and operational decision-making. The ideal candidate brings advanced Python and SQL expertise, strong data transformation skills, cloud engineering experience, and the ability to architect scalable batch and real-time data solutions. This is primarily a Data Engineering role, with opportunities to support machine learning infrastructure, real-time model serving, and emerging AI initiatives. Direct experience building or training ML models is not required.
Key Responsibilities
Data Engineering & Pipeline Development (70%)
- Design, develop, and maintain scalable batch and real-time data pipelines using Python, SQL, PySpark, dbt, and cloud-native technologies.
- Build and optimize ETL/ELT frameworks that ingest, clean, transform, and integrate data from multiple enterprise sources.
- Develop robust data pipelines leveraging GCP technologies including BigQuery, Dataflow, Cloud Composer, and Pub/Sub.
- Design reusable data models, dimensional models, and transformation frameworks supporting enterprise analytics and downstream applications.
- Work extensively with website, first-party customer, transactional, behavioral, and operational data to enable business insights and decision-making.
- Implement and optimize streaming data pipelines using frameworks such as Apache Beam, Apache Flink, Kafka Streams, or comparable technologies.
- Build event-driven architectures supporting low-latency data ingestion, processing, and delivery.
- Establish best practices for data quality, data governance, lineage, validation, and pipeline observability.
- Optimize data processing and cloud warehouse performance for scalability, reliability, and cost efficiency. Cloud & Platform Engineering
- Design and maintain scalable cloud-based data infrastructure, primarily within Google Cloud Platform (GCP).
- Develop containerized applications and data services using Docker, Kubernetes, and Cloud Run.
- Implement CI/CD pipelines, automated testing, version control, and deployment processes using GitHub and modern software engineering practices.
- Build reusable Python frameworks, shared libraries, and engineering tools to accelerate development.
- Monitor and troubleshoot production pipelines, ensuring high availability, fault tolerance, and efficient recovery.
- Evaluate technical architecture, identify opportunities for improvement, and implement scalable solutions. Architecture, Collaboration & Technical Leadership
- Take ownership of technical solutions from initial design through production deployment and ongoing support.
- Partner with Data Scientists, analysts, engineers, and business stakeholders to translate complex requirements into scalable data solutions.
- Lead technical design discussions and make sound architectural decisions around performance, scalability, and maintainability.
- Establish engineering best practices across coding standards, documentation, testing, and production reliability.
- Participate in code reviews, mentor engineers, and contribute to a collaborative engineering environment.
- Identify opportunities to modernize existing data platforms and improve overall engineering efficiency.