Staff AI & Data Engineer - Databricks

Summary

Staff AI & Data Engineer - Databricks

Anywhere

Type: Contract

Category: Data

Industry: Technology

Workplace Type: Remote

Standard Hours: Open

Reference ID: JN -112026-109353

Date Posted: 11/03/2026

Shortcut: http://careers.eliassen.com/SeR7jt

+ Description

+ Recommended Jobs

Description:

Remote

Our client is seeking a highly experienced Staff AI & Data Engineer to design and build enterprise-grade data and AI solutions on Databricks. This hands-on role focuses on strengthening the Databricks platform, onboarding new data sources, building reusable ingestion and transformation frameworks, creating trusted Gold-layer data products, enabling cross-domain analytics, and developing semantic and agent-ready capabilities. The ideal candidate brings strong data engineering expertise with modern AI and GenAI engineering skills and works across pipelines, modeling, data quality, governance, semantic layers, natural-language analytics, and AI agents.

_Due to client requirements, applicants must be willing and able to work on a w2 basis. For our w2 consultants, we offer a great benefits package that includes Medical, Dental, and Vision benefits, 401k with company matching, and life insurance._

Rate: $80.00 to $90.00/hr. w2

JN -112026-109353

Responsibilities:

+ Design and build production-grade data pipelines on Databricks.

+ Ingest data from enterprise applications, databases, APIs, SaaS platforms, files, and other source systems.

+ Build and maintain Bronze, Silver, and Gold medallion architectures.

+ Implement incremental processing and change data capture patterns where appropriate.

+ Standardize types, codes, units, keys, and business definitions across source systems.

+ Implement deduplication, historical tracking, schema evolution, and data lineage.

+ Optimize pipelines for reliability, performance, scalability, and cost.

+ Build a configuration-driven source onboarding framework and Lakeflow pipeline templates.

+ Automate source registration, deployment, monitoring, lineage, and operational controls.

+ Implement data quality expectations, anomaly detection, monitoring, and quarantine processes.

+ Implement job and pipeline monitoring, alerting, audit logging, and operational dashboards.

+ Support automated deployment and promotion using Databricks Asset Bundles and CI/CD.

+ Develop conformed dimensions and shared-key strategies across domains.

+ Build integrated Gold-layer products and reusable data contracts.

+ Build AI-ready Gold data products with ownership, lineage, definitions, and quality information.

+ Develop semantic context, business glossary, ontology, and certified KPIs in Unity Catalog.

+ Build governed Unity Catalog functions and agent-ready tools and endpoints.

+ Implement tracing, evaluation, testing, monitoring, and governance for AI and agent workflows.

+ Support development of AI agents that use trusted enterprise data.

Experience Requirements:

+ Advanced hands-on experience with Databricks, Apache Spark, Delta Lake, and SQL.

+ Hands-on experience with AWS services related to data analytics and AI, including EMR and Bedrock.

+ Strong Python and SQL development skills.

+ Experience building scalable batch and/or streaming data pipelines.

+ Deep understanding of Bronze/Silver/Gold medallion architectures.

+ Experience with Databricks Lakeflow or comparable orchestration technologies.

+ Experience implementing enterprise data quality, lineage, observability, and monitoring.

+ Experience with Unity Catalog and governed enterprise data environments.

+ Strong understanding of dimensional modeling, data products, semantic models, and cross-domain integration.

+ Experience with Git-based development, CI/CD, automated testing, and production deployment.

+ Ability to troubleshoot complex distributed data and pipeline issues.

+ Experience building AI and GenAI solutions against enterprise data.

+ Understanding of semantic layers and natural-language-to-data architectures.

+ Experience with Databricks Genie or comparable natural-language analytics technology.

+ Experience building tools, functions, APIs, or services consumed by AI agents.

+ Understanding of MCP, agent tool calling, tracing, evaluation, and AI governance.

+ Experience developing production AI solutions with security, observability, and governance controls.

+ Preferred: Databricks Data Engineer Professional or equivalent experience.

+ Preferred: Experience with Databricks AI/BI, Genie, MLflow, and MCP-based integrations.

+ Preferred: Experience in Real Estate, Finance, People/Workplace, Operations, or similar domains.

+ Preferred: Experience building metadata-driven or configuration-driven ingestion frameworks.

_Recruitment Transparency Notice_

_Eliassen Group values transparency in our recruitment practices. Please be advised that Eliassen Group utilizes artificial intelligence (AI) tools as part of its initial application screening_ _and hiring_ _process. You may receive email and SMS notifications from the Eliassen Virtual Recruiting Team (_ _noreply@eliassen.com_ _, 781-808-2924) inviting you to complete a brief voice screening as part of your application process. These tools assist our hiring teams in different ways, including but not limited to, assistance in reviewing application materials to help identify candidates whose qualifications most closely match the requirements of the position. All AI-assisted evaluations and responses are reviewed by human recruiters before any hiring decisions are made. The use of AI in our process is intended to support fairness, efficiency, and consistency, and Eliassen Group takes measures to prevent bias or discrimination in connection with its hiring practices. By proceeding, you acknowledge, agree, and consent to Eliassen Group's use of these tools, including AI tools, as part of the application and hiring process._

_Skills, experience, and other compensable factors will be considered when determining pay rate. The pay range provided in this posting reflects a W2 hourly rate; other employment options may be available that may result in pay outside of the provided range._

_W2 employees of Eliassen Group who are regularly scheduled to work 30 or more hours per week are eligible for the following benefits: medical (choice of 3 plans), dental, vision, pre-tax accounts, other voluntary benefits including life and disability insurance, 401(k) with match, and sick time if required by law in the worked-in state/locality._

_If anyone reaches out to you about an open position connected with Eliassen Group, please ensure that you are working directly with us by confirming the following:_

_· When you work with Eliassen Group, all email communication will come...