Hybrid Lead Azure Data Engineer

at Rackspace Technology

• Job title: Lead Azure Data Engineer
• Job type: Full Time
• Experience level: Senior
• Salary: Not specified
• Degree requirement: No Education Requirement
• Tech skills: Azure, Cloud, ETL, Kafka, PySpark, Spark, SQL, Unity
• Location requirements: Hybrid, Gurgaon, India

About the role

• Lead Azure Data Engineer designing and optimizing data ecosystems on Microsoft Cloud. Responsible for building scalable data platforms and pipelines for analytics and reporting.

Responsibilities

• Lead end-to-end development of scalable data pipelines and orchestration frameworks using Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks, and Microsoft Fabric.
• Build robust real-time and batch data pipelines, including integration with streaming sources (e.g., Event Hubs, Kafka) and structured streaming engines.
• Design and implement Structured Streaming applications in Spark for near-real-time processing of streaming data.
• Develop and maintain ETL/ELT pipelines and transformations leveraging Spark, PySpark, SQL, and fabric orchestration capabilities.
• Architect and implement data solutions using Microsoft Fabric, including OneLake, Dataflows, warehouses, and Fabric capacity planning to support enterprise analytics.
• Collaborate on data governance, cataloging, and asset organization using Unity Catalog within Databricks and Fabric environments.
• Manage Microsoft Fabric capacity and resource utilization to optimize performance and cost efficiency for analytics workloads.
• Design, deploy, and optimize Databricks dashboards and reporting artifacts for business stakeholders.
• Apply best practices for data modelling, caching, file sizing, and performance tuning of Spark and Delta Lake jobs (e.g., Z-ORDER, broadcast joins, adaptive query execution).
• Oversee governance, access controls, metadata management, and lineage using Unity Catalog.
• Lead and mentor a team of data engineers, fostering best practices in development, operations, documentation, and quality.
• Work with cross-functional teams (architecture, BI, data science, DevOps) to translate business requirements into scalable data solutions.
• Partner with stakeholders to define data strategy, standards, and architectural roadmaps.
• Establish and enforce standards for data quality, testing, monitoring, operational observability, and governance.
• Implement secure, compliant data access and lineage frameworks across cloud data platforms.
• Implement CI/CD pipelines, infrastructure-as-code for data platform artifacts, and automated testing frameworks for data jobs and workflows.

Requirements

• 10+ years of hands-on experience in data engineering on Azure with deep expertise in ADF, Synapse, Databricks, and Microsoft Fabric.
• Proven experience with real-time data processing, streaming architectures, and Spark Structured Streaming.
• Strong proficiency in Azure Data Factory, Spark (PySpark), SQL, Azure Synapse Analytics, Databricks Runtime, and cloud storage.
• Solid knowledge of Unity Catalog for data governance, security, and access management.
• Experience designing and managing Databricks Dashboards, performance optimization, cost controls, and data platform resource tuning.
• Expertise in building scalable, fault-tolerant, and high-throughput batch & streaming data solutions.
• Excellent leadership, cross-team collaboration, and communication skills.