Mid-level Data Engineer

Required Skills & Experience

• Strong SQL skills and practical dbt experience.
• Practical Python skills: read, troubleshoot, modify, and port existing code; build straightforward custom integrations.
• Experience building and migrating data integrations and ETL/ELT pipelines.
• Understanding of REST API authentication, pagination, rate limits, retries, and incremental extraction.
• Understanding of incremental loading, CDC, idempotency, deduplication, schema evolution, and backfills.
• Understanding of SCD types and handling current versus historical data.
• Practical git-based development experience, including merge requests, code reviews, CI/CD, and automated testing.
• Basic Airflow experience covering DAGs, schedules, dependencies, retries, and monitoring.
• Attention to detail, particularly in data validation and safe migration.
• Ability to investigate routine issues independently and recognise when to escalate.
• Clear communication about progress, assumptions, blockers, and risks.
• Ability to follow platform standards, work with review feedback, and maintain useful documentation.

Nice to Have Skills & Experience

• BigQuery experience is preferred.
• Hands-on experience with Fivetran or Airbyte.
• Experience with migration reconciliation, cutovers, and legacy pipeline decommissioning. Scope and ownership

Job Description

JOB DESCRIPTION

Insight Global is looking for a mid-level Data Engineer to join the Data Integrations team of a Marketing Software client. This person will migrate existing integrations to the Unified Data Platform, deliver new integrations using established patterns, and safely retire legacy pipelines. This hands-on mid-level engineer should be able to understand existing implementations, deliver scoped integration work, validate the results, and communicate clearly with the internal team.

KEY RESPONSIBILITIES

• Understand existing dbt and legacy Python integrations, including their source data, transformation logic, schedules, and downstream dependencies.
• Migrate integrations to the Unified Data Platform using patterns and guidance provided by the internal team.
• Configure Fivetran or Airbyte integrations where suitable; implement straightforward Python integrations when standard connectors are insufficient.
• Integrate with internal and external REST APIs, handling authentication, pagination, rate limits, retries, and incremental extraction.
• Configure transformations, schedules, dependencies, tests, and monitoring; support incremental loads, deduplication, and backfills.
• Validate migrated data against the source or legacy implementation and resolve routine discrepancies and failures.
• Submit changes through merge requests, address code review feedback, and use existing CI/CD and automated testing processes.
• Document implementation and operational details, support handover, and safely remove legacy pipelines after successful migration.