Data Engineer

SV IT, Inc. is looking for Data Engineers to work in Plain View, NY and various unanticipated locations throughout the U.S. Must possess Bachelors degree or foreign equiv. in Computer Science or related field and exp. in Apache Spark, Azure Databricks, Scala, Snowflake, Azure Data Factory, Apache Airflow, Big Data Processing, and PySpark. Send resumes to info@svitinc.net, REF# 1984.074. Job Duties:Lead the migration of enterprise reporting pipelines from legacy Snowflake-based processing and Excel macros to a scalable Azure Databricks Lakehouse architecture, modernizing the platform for improved performance and maintainability. Design and implement a multi-layer Delta Lake architecture in Azure Databricks to standardize ingestion, enforce schema validation, and enable reliable downstream reporting, while building a BV ingestion framework that refreshes Business Table data incrementally based on configurable refresh keys to efficiently process only changed records. Build ingestion frameworks using Scala, Spark SQL, and Databricks Auto Loader to process daily inbound files from Azure Blob Storage (via SFTP), ensuring incremental and fault-tolerant data loading. Engineer 18+ report-specific ETL pipelines in Azure Databricks using Scala and Spark SQL with standardized ingestion and transformation frameworks, optimizing execution through partition pruning, column pruning, predicate pushdown, and incremental processing patterns to ensure scalable and reusable data processing across multiple reporting workloads. Optimize large-scale Spark workloads by tuning cluster configurations, autoscaling policies, executor memory/cores while leveraging dynamic resource allocation, caching strategies, and efficient Data Frame persistence to significantly improve job throughput and reduce compute costs. Improve distributed query performance by applying Adaptive Query Execution (AQE), broadcast joins, repartitioning strategies, skew handling techniques, and optimized join planning, minimizing shuffle overhead and balancing executor workloads for large-scale datasets. Integrate event-driven ingestion workflows using Azure Event Grid Topics and Azure Queue, enabling reliable notification, where we send notification for every feed either it is succeeded or rejected and send notifications for the data quality check failures. Develop and optimize analytical queries using Spark SQL and Snowflake, enabling business users to generate financial reports such as MTD, QTD, YTD, and PTD insights. Apache Spark, Azure Databricks, Scala, Snowflake, Azure Data Factory, Apache Airflow, Big Data Processing, and PySpark. Position may require travel. Minimum Requirements: Education: Bachelors degree or foreign equiv. in Computer Science or related field. Experience: 24 months of experience in the position offered or related position. Special Requirements: Six months of the aforementioned 24 months of experience must include the following skills: Apache Spark, Azure Databricks, Scala, Snowflake, Azure Data Factory, Apache Airflow, Big Data Processing, and PySpark.