We are looking for a Data Automation Engineer to design, develop, and maintain scalable data automation solutions leveraging AWS cloud services, selected Azure technologies, and emerging Generative AI capabilities.
Key Responsibilities:
· Design and implement scalable data automation workflows using AWS services with integration to Azure platforms. · Develop and maintain ETL/ELT processes to ingest, transform, and move data across DynamoDB, SQL Server, Azure SQL, and other enterprise data sources. · Build and support batch and near real-time data pipelines using Apache Spark and streaming technologies such as Kafka or Flume. · Develop scalable data processing solutions utilizing Amazon EMR, AWS Fargate, Kubernetes, and related cloud technologies. · Integrate REST APIs, external services, and Amazon Connect customer interaction data into analytics platforms. · Evaluate and prototype Generative AI solutions using Amazon Bedrock, Azure OpenAI, Hugging Face, LangChain, and related technologies. · Implement source control, CI/CD, and automation practices using GitHub, Azure DevOps, Jenkins, Docker, and Agile methodologies. · Ensure data solutions meet security and compliance requirements, including IAM, encryption, RBAC, VPC isolation, and firewall policies.
Core Qualifications:
· Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related technical field. · 5+ years of experience in data engineering, data automation, software engineering, or a related discipline. · Strong hands-on experience developing Python- and SQL-based data pipelines in AWS environments. · Experience with Apache Spark and at least one streaming/ingestion technology such as Apache Kafka or Apache Flume. · Hands-on experience with AWS services including several of the following: S3, AWS Glue, Lambda, EMR, Step Functions, and AWS database services. · Experience integrating REST APIs and external services into automated data workflows. · Experience working with CI/CD and source control platforms such as GitHub, Azure DevOps, or Jenkins. · Strong troubleshooting, performance tuning, and production support experience across data platforms and pipelines. · Excellent communication and collaboration skills. · U.S. Citizenship and ability to obtain a Federal Public Trust clearance.
Preferred Qualifications:
· Experience integrating Large Language Models (LLMs) such as Amazon Bedrock, Azure OpenAI, Hugging Face, or similar AI services into production workflows. · Experience with Retrieval-Augmented Generation (RAG), embeddings, vector databases, Apache Solr, OpenSearch, pgvector, or similar technologies. · Multi-cloud experience supporting both AWS and Azure environments. · Experience with Docker, Kubernetes, and containerized application deployments. · Experience operationalizing Generative AI solutions, including monitoring, evaluation, observability, and lifecycle management. · Familiarity with data governance and cataloging solutions such as Microsoft Purview, Unity Catalog, or AWS Glue Catalog. · Experience with Infrastructure as Code tools including Terraform, AWS CloudFormation, or Bicep. · Knowledge of FedRAMP, HIPAA, PCI-DSS, or other regulated compliance environments. · Relevant cloud, AI, or data engineering certifications.