AWS Databricks Engineer

Overview

Join a team where innovation meets mission. Our AI, cloud, cyber, and modernization solutions save agencies thousands of hours, safeguard national security, and strengthen health and humanitarian missions worldwide. With 1,700+ team members, 1,500+ AI/data experts, and 100+ prime contracts, we deliver at scale and with purpose. We’ve been recognized as a Top Workplace by the Washington Post for six straight years and named to the Inc. 5000 Fastest Growing Private Companies 13 of the past 14 years. Credence is a welcoming home for those looking to grow and contribute to positive change. We encourage all employees to expand beyond their boundaries and dive into important, world-changing federal challenges. Position Summary Credence has an immediate need for an AWS Databricks Engineer to join our growing AI and Automation practice. You will serve as a key technical contributor responsible for designing, building, and optimizing scalable cloud-based data solutions using AWS and Databricks. You’ll develop robust data pipelines, implement modern data lakehouse architectures, and collaborate across engineering, data science, and stakeholder teams to deliver secure, high-performance data capabilities that advance federal missions.

Responsibilities include, but are not limited to, the duties listed below:

• Databricks Engineering Design, build, and optimize scalable data pipelines and processing workflows using Databricks, Apache Spark, PySpark, and Delta Lake.
• Data Lakehouse Architecture Design and implement modern lakehouse solutions using Databricks and AWS services. Apply medallion architecture (Bronze/Silver/Gold layers) to improve data quality, governance, lineage, performance, and accessibility.
• AWS Data Integration Build and maintain ETL/ELT pipelines integrating structured, semi-structured, and unstructured data across AWS services such as S3, Glue, RDS, Redshift, Lambda, and related cloud data platforms.
• Data Transformation & Optimization Develop efficient data transformation workflows using SQL, Python, and PySpark. Optimize Spark workloads, queries, clusters, and storage configurations for performance, reliability, and cost efficiency.
• Data Governance & Quality Implement data quality, cataloging, lineage, governance, and access-control practices across Databricks and AWS environments, including the use of Unity Catalog where applicable.
• Cloud Automation & DevOps Automate data platform deployments and workflows using Infrastructure as Code (IaC), CI/CD pipelines, Git-based development practices, and AWS-native deployment tools.
• Collaborative Engineering Work closely with data engineers, software engineers, data scientists, cloud engineers, and mission stakeholders to translate business and technical requirements into production-ready data solutions.
• Technical Rigor & Documentation Develop clean, maintainable, and well-documented code while following engineering, security, and federal development standards. Support code reviews, testing, troubleshooting, and technical design documentation.

Requirements

• U.S. citizenship with the ability to obtain and maintain a Secret clearance is required.
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related technical field.
• Hands-on experience in data engineering, cloud engineering, or related technical roles.
• Hands-on experience building and maintaining production data solutions using Databricks, Apache Spark, and PySpark.
• Strong experience with AWS data services such as S3, Glue, RDS, Redshift, Lambda, IAM, and CloudWatch.
• Strong understanding of ETL/ELT processes, data lake and lakehouse architectures, data modeling, and large-scale data processing.
• Hands-on experience with Delta Lake, SQL, Python, and Spark-based data processing.
• Experience developing, scheduling, monitoring, and troubleshooting production data pipelines and workflows.
• Experience with Git and CI/CD platforms such as GitLab, GitHub Actions, Jenkins, or AWS CodePipeline.
• Understanding of cloud security, identity and access management, encryption, and secure data-handling practices.
• Strong communication skills and a client-oriented mindset with the ability to collaborate across technical and non-technical teams.

Preferred

• Experience implementing Databricks Lakehouse Architecture and medallion architecture using Bronze, Silver, and Gold data layers.
• Experience with Databricks Unity Catalog, Delta Live Tables, Databricks Workflows, or similar data orchestration and governance capabilities.
• Strong Python proficiency and experience with data engineering libraries and frameworks such as PySpark, Pandas, and Pydantic.
• Experience with Infrastructure as Code tools such as Terraform, OpenTofu, AWS CDK, or CloudFormation.
• AWS and/or Databricks certifications, such as AWS Certified Data Engineer - Associate, AWS Certified Solutions Architect, or Databricks Certified Data Engineer credentials.
• Knowledge of federal cybersecurity, RMF, FedRAMP, or other government regulatory and compliance frameworks.
• Experience delivering cloud-native data solutions within federal government or other highly regulated environments.

Salary Range: $155,000 - $185,000 annually. Actual compensation will be determined based on the selected candidate's experience, education, certifications, skills, and overall qualifications.

Benefits

• Health Care Plan (Medical, Dental & Vision)
• Retirement Plan (401k, IRA)
• Life Insurance (Basic, Voluntary & AD&D)
• Paid Time Off (Vacation, Sick & Public Holidays)
• Family Leave (Maternity, Paternity)
• Short Term & Long Term Disability
• Training & Development
• Wellness Resources