Hybrid Senior Azure Databricks Engineer

at Thinkahead Consultant Psychologist Pty Ltd

• Job title: Senior Azure Databricks Engineer
• Job type: Full Time
• Experience level: Senior
• Salary: Not specified
• Degree requirement: No Education Requirement
• Tech skills: Airflow, Apache, Azure, ETL, Kafka, PySpark, RDBMS, Scala, Spark, SQL, Unity, Vault
• Location requirements: Hybrid, Hyderabad, India

About the role

• Senior Azure Databricks Engineer at AHEAD building platforms for digital business. Architecting and optimizing Azure Databricks solutions and data pipelines for enterprise clients.

Responsibilities

• Architect, build, and optimize **Azure Databricks** solutions (batch + streaming) using **Apache Spark**.
• Design Lakehouse data models and curation layers using **Delta Lake** (Bronze/Silver/Gold), including SCD patterns where applicable.
• Lead end-to-end development of **ETL/ELT** pipelines integrating sources like APIs, RDBMS, SaaS apps, files, and event streams.
• Implement orchestration and scheduling using **Databricks Workflows** and/or **Azure Data Factory** (or Airflow where relevant).
• Drive performance and cost optimization: cluster policies, autoscaling, photon/runtime selection, partitioning, file sizing, job tuning.
• Establish best practices for code quality: modular PySpark/SQL, reusable libraries, unit/integration testing, and peer reviews.
• Implement CI/CD for Databricks (Repos/Git), automated deployments, environment promotion, and release governance.
• Partner with security/governance teams to implement access controls, secrets, encryption, and data governance (e.g., **Unity Catalog**).
• Create operational excellence: monitoring, alerting, runbooks, incident response, and root-cause analysis.
• Mentor engineers, set technical direction, and contribute to roadmap planning and estimation.

Requirements

• 6+ years of data engineering experience, including strong hands-on delivery with **Azure Databricks**.
• Expert-level **Spark** experience (PySpark and/or Scala) and strong **SQL**.
• Proven experience building **production-grade Delta Lake / Lakehouse** architectures and data pipelines.
• Strong experience with Azure services such as **ADLS Gen2**, **Azure Data Factory**, **Event Hubs/Kafka**, **Synapse**, **Key Vault**, and Azure networking/security concepts.
• Demonstrated ability to tune Spark workloads (joins/shuffles, partitioning, caching, skew handling, file compaction).
• Experience with **Git-based development** and CI/CD (Azure DevOps or GitHub Actions), plus automated testing approaches.
• Strong communication skills and ability to lead technical discussions, influence decisions, and translate requirements into designs.

Benefits

• Employment Benefits include: