Data Engineer

Data Engineer

Department: Technology

Reports To: Senior Manager, Data Operations

Location: Remote

FLSA Status: Exempt

People Leader: No

Travel: Less than 10%

Summary

Company Summary

Accuity partners with hospitals and health systems through a technology-enabled, physician-led model that improves clinical documentation integrity, coding accuracy, reimbursement optimization, and quality outcomes.

Job Summary

The Data Engineer designs, builds, tests, documents, and operates data pipelines and models on Accuity's Azure Databricks platform. Reporting to the Senior Manager, Data Operations, this role owns assigned assets from source discovery and ingestion through Bronze, Silver, and Gold, delivering reliable, reusable data for reporting, analytics, applications, and data science.

The Data Engineer independently delivers assigned work within the architecture, engineering standards, and operating procedures established by the Data Architect and supported by the Senior Data Engineer. Delivery includes models, quality controls, API-based integrations, documentation, deployment, and ongoing support.

As Accuity migrates from SQL Server and SSIS to Databricks and Delta Lake, the role recovers existing business rules, implements modern solutions, validates results, and supports controlled retirement of replaced workloads.

Responsibilities

Pipeline Development and Delivery

• Own assigned assets through requirements clarification, design, development, testing, deployment, and production support.
• Build ingestion, transformation, and publishing pipelines using SQL, Python, PySpark, Databricks, and Delta Lake; integrate databases, files, APIs, and other approved sources.
• Implement orchestration, scheduling, dependencies, error handling, retries, and recovery; support safe reruns and backfills without duplicate or inconsistent results.
• Apply incremental processing, change data capture, schema evolution, and late-arriving data handling where appropriate.
• Use version control, peer review, automated testing, and controlled deployment practices to deliver maintainable solutions.

Medallion Architecture and Data Modeling

• Design and implement Bronze, Silver, and Gold datasets within established architecture standards.
• Preserve source data and ingestion metadata in Bronze; build standardized, validated, deduplicated, and reusable Silver datasets.
• Develop Gold datasets and dimensional models that meet agreed business and analytical requirements.
• Define and document grain, keys, relationships, facts, dimensions, and historical behavior; implement slowly changing dimensions where needed.
• Work with the Data Architect and Senior Data Engineer on complex models, shared entities, cross-domain dependencies, and architectural exceptions.

API Integration and Data Exchange

• Support API-based access and standardized file exchange using approved patterns appropriate to the source, consumer, security requirements, and business need.
• Use approved low-code or no-code tools to configure API connections and simple data services within established architecture and security standards.
• Apply established authentication, request and response mapping, validation, error handling, and monitoring practices; escalate custom development or complex API design needs.

Data Quality and Validation

• Translate agreed business rules and acceptance criteria into automated checks for completeness, accuracy, uniqueness, consistency, referential integrity, and freshness.
• Reconcile outputs against sources and investigate discrepancies; test expected behavior, edge cases, failures, and recovery.
• Work with analysts and business stakeholders to validate intended use; maintain test evidence and document limitations, unresolved issues, and approved differences.

Documentation and Knowledge Management

• Create and maintain source-to-target mappings, transformation specifications, data models, lineage, and dataset and column definitions.
• Document business rules, ownership, refresh schedules, dependencies, intended uses, and API interfaces where applicable.
• Maintain runbooks for monitoring, troubleshooting, recovery, reruns, and backfills; record implementation decisions, release changes, and migration findings.
• Update documentation with each material change so another engineer can understand, operate, and extend the deployed asset.

Production Operations and Improvement

• Monitor assigned workloads and respond to failures, delayed data, and quality alerts under the team's support and escalation procedures.
• Use logs, metrics, lineage, and query analysis to restore service, determine root causes, and implement durable corrective actions.
• Tune queries and processing for performance, scalability, and compute efficiency; automate manual work and reduce technical debt.
• Assess downstream impact and coordinate changes and releases with affected teams.

Migration and Modernization

• Analyze SQL Server objects, stored procedures, SSIS packages, scheduled jobs, and dependencies; recover undocumented business rules and exceptions.
• Reimplement approved logic in Databricks; identify opportunities to simplify or consolidate processing with technical and business owners.
• Build reconciliation and parallel-run tests; execute assigned cutover, rollback, and stabilization activities under approved plans.
• Support legacy workloads that remain in service and retire replaced components after validation, dependency review, and approval.

Governance and Security

• Apply established Unity Catalog standards for organization, metadata, ownership, classification, lineage, and approved access controls.
• Follow Accuity security, privacy, and PHI-handling policies and applicable compliance requirements; apply least privilege within delegated responsibilities.
• Maintain evidence of testing, changes, and operating controls for reviews and audits.

Collaboration and Accountability

• Work with the manager on priorities and delivery commitments; clarify requirements and source behavior with analysts, application teams, and business stakeholders.
• Seek guidance on shared definitions and architecture exceptions; coordinate identity, infrastructure, connectivity, and platform dependencies with the Azure team.
• Participate in design and code reviews, incorporate feedback, and validate and document solutions transferred from external delivery partners.
• Communicate progress, risks, dependencies, defects, and unresolved questions early.

Other Duties as Assigned

• Perform additional responsibilities as needed to support data engineering, technology, and organizational goals.

Qualifications

Education and Credentials

• Bachelor's degree in computer science, information systems, engineering, or a related technical field preferred; equivalent professional experience may be considered.
• Relevant Databricks, Azure, or data engineering certifications preferred.

Required Experience and Skills

• 3+ years of data engineering, database development, analytics engineering, or ETL development experience, including production workload responsibility.
• Strong SQL proficiency and working Python proficiency for processing, automation, and troubleshooting.
• Hands-on pipeline development and support using Databricks, Spark, Delta Lake, or a comparable modern data platform.
• Practical understanding of medallion architecture and relational and dimensional modeling, including grain, keys, relationships, and history.
• Experience with data validation, reconciliation, monitoring, production troubleshooting, version control, testing, and controlled deployment.
• Working understanding of API endpoints, HTTP methods, authentication, request and response structures, pagination, rate limits, and error handling.
• Familiarity with data exchange formats and specifications such as JSON, XML, CSV, and EDI, including schema validation and field mapping.
• Ability to configure and troubleshoot integrations using low-code or no-code tools and recognize when custom development or architectural guidance is needed.
• Ability to analyze existing processing, produce clear technical documentation, and independently deliver assigned work within established standards.

Preferred Experience

• Production experience with Azure Databricks, Delta Lake, PySpark, and Unity Catalog.
• Migration experience involving SQL Server, stored procedures, SSIS, or other legacy ETL platforms; familiarity with Azure Data Factory, Azure storage, or Azure Functions.
• Experience configuring API integrations or publishing simple data services using low-code or no-code platforms.
• Healthcare, clinical, encounter, claims, or revenue cycle data experience; familiarity with HL7, FHIR, or X12 EDI and handling PHI.

Core Competencies

• Delivery Ownership: Carries assigned assets from discovery through deployment, documentation, and ongoing support.
• Modeling and Correctness: Creates understandable structures, validates results, and investigates discrepancies.
• Operational Judgment: Considers recovery, security, cost, and downstream impact when building solutions.
• Documentation and Communication: Maintains usable documentation and communicates decisions, risks, and uncertainty clearly.
• Collaboration and Adaptability: Applies feedback, develops modern engineering skills, and works proactively in a remote team.

Additional Requirements

1. Physical Requirements: The requirements described here are representative of those that must be met by an employee to successfully perform the essential functions of this job with or without reasonable accommodations. Unless otherwise indicated, Accuity positions require interaction with people and technology while either sitting or standing. Employees must be able to communicate via phone, email, etc. and sit for extended periods of time, with or without reasonable accommodations. Physical effort and exposure to physical risk are limited to that of an office role / environment.
2. Position and Employment Statement: While this job description is intended to be an accurate reflection of the job requirements, management reserves the right to modify, add or remove duties from a job and to assign other duties as necessary and at any time. All positions at Accuity Delivery Systems, LLC, are at-will employment, and a position description is not a guarantee of a job or of job responsibilities.

While this job description is intended to be an accurate reflection of the job requirements, management reserves the right to modify, add or remove duties from a job and to assign other duties as necessary and at any time. All positions at Accuity are at-will employment, and a position description is not a guarantee of a job or of job responsibilities.