Senior Data Engineer, Data Platform
Location : San Francisco, CA
About the Role
We are seeking a hands-on with senior Big Data Engineer to design, build, and operate mission critical ETL pipelines ingesting petabyte scale database into our Hadoop Ecosystem. You will partner with engineering, analytics and business intelligence teams to improve data delivery speed, pipeline reliability and processing efficiency using Java, Spark, and AWS. This role requires deep technical expertise in distributed systems, production operation mindset and ability to troubleshoot complex data issues on a scale.
Key Responsibilities
- Design and develop ETL workflows using Java to ingest data from databases, APIs , and file feeds into Hadoop/Hive
- Build and maintain reusable framework components(Actions, UDFs, Serds) for the ETL pipelines
- Configure and optimize data ingestion using Sqoop, Spark, and Oozie with incremental load, partitioning and CDC strategies
- Monitor and troubleshoot production pipelines ,resolve data quality issues, investigate failures , and optimize performance bottleneck.
- Implement data security controls – Encryption/decryption of PII data, AWS Secrets Manager integration, and compliance requirements.
- Collaborate with upstream/downstream teams – sources system owners , data analyst and BI teams to ensure data availability and quality.
- Participate in on-call rotation for critical production data feeds and SLA compliance.
Must-Have Skills
- Programming and Development
- Java – Expert -level proficiency in building production grade ETL framework, customers actions, and reusable libraries. Must-Have have experience with Maven multi-model projects, dependency management with shaded JARs, and handling Hadoop class path conflicts.
- Groovy – Strong scripting skills for configuration DSLs, row processing, and partition logic in declarative ETL framework.
- Git and version controls - Proficient with branching strategies, pull request workflows, code reviews and Jira -integrated commit discipline.
- Scala – Functional programming experience for spark development with ability to write concise idiomatic Scala for data processing at scale.
- SQL – Ability to write complex SQL queries across multiple databases (MySQL, PG, Oracle, SQL server etc.)
- Data Engineering
- ETL Pipeline Development - Proven track record building production ETL framework with reusable components. Deep understanding of ETL concepts, load patterns, CDC processing, and data lineage.
- Apache Spark – Deep expertise with spark on production scale datasets. Must-Have have hands on experience optimizing spark jobs, tuning memory/shuffle and working with DataFrame/RDDs in Scala or Java.
- Kafka/ Golden Gate streaming – Real/Near real-time data processing with Kafka, Golden gate or spark streaming.
- Airflow – Experience migrating from Oozie to modern workflow orchestration
- Databricks – Exposure to unified analytics platforms, Delta Lake and notebook driven development.
- Big Data and Distributed systems
- Hadoop Ecosystem - Strong production scale experience with HDFS operations, Hive query optimization and managing portion tables at scale. Understanding of Data locality, replication factors and cluster resource management.
- Hive/HQL - Experience in complex analytics queries on multi – million row tables.
- Oozie – Proficient in designing coordinator/workflow orchestration with dataset dependency, SLA monitoring and handling late – data scenarios.
- Sqoop – Hands-on on experience Configuring bulk imports with parallel rappers, split by strategies, incremental modes and handling large scale database exports
- Cloud and Infrastructure
- AWS – Production experience with S3 data lakes (lifecycle policies, versioning, encryption), EMR cluster operations, IAM role-based access, and Secrets Manager for credentials management. Understanding of AWS networking, VPC and security groups.
- Data Security and compliance - Hands-on on experience implementing PII encryption/ decryption, SafeNet, Data masking, and working on regulated environment.
Good-to-Have Skills
- Banking, lending or other regulated financial services experience
- Data modeling
- Observability and monitoring frameworks for data pipelines
- Databricks Certified Data Engineer Professional
- Presto/Trino
- NoSQL databases
Soft Skills
- Strong problem solving and communication skills with ability to explain data engineering concepts to non-technical stakeholders.
- Experience working on client facing roles.
Location and Work Arrangements
- San Francisco Bay Area
- This position requires onshore presence in our San Francisco office.
Manage data engineering projects. Provide strategic guidance. Oversee data engineering team. Ensure continuous improvement of data processes. Graduate in Computer Science, Data Science, or related field. 3-4 years of experience in data engineering or a related field.