Senior Databricks Engineer at Jade Global responsible for designing and optimizing large-scale data pipelines. Collaborating with teams to transform raw data into actionable insights.
Responsibilities
Design and implement scalable ETL/ELT pipelines using Apache Spark on Databricks.
Build and optimize data lakes and data warehouses using Delta Lake and related tools.
Develop, schedule, and monitor workflows using Databricks Workflows, Airflow, or other orchestration tools.
Collaborate with data scientists, analysts, and stakeholders to understand data needs and deliver reliable solutions.
Tune Spark jobs for performance, scalability, and cost-efficiency.
Implement CI/CD pipelines and DevOps best practices for data engineering projects.
Ensure data security, compliance, and governance using tools like Unity Catalog.
Troubleshoot and resolve issues related to data ingestion, transformation, and integration.
Participate in data platform migration projects (e.g., from on-premise to Databricks or from legacy platforms to modern cloud-native architectures).
Mentor junior team members and contribute to a culture of technical excellence and innovation.
Requirements
Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
6+ years of hands-on experience in data engineering or big data development
3+ years of experience working with Databricks, including Spark, Delta Lake, and MLflow
Strong programming skills in Python and SQL
Solid understanding of distributed systems and performance tuning in Spark
Experience with cloud platforms such as Azure, AWS, or GCP (Azure Databricks preferred)
Experience working with structured, semi-structured, and unstructured data
Familiarity with DevOps practices, including version control (Git), CI/CD, and infrastructure as code (Terraform, ARM templates)
Excellent problem-solving skills and ability to communicate complex technical ideas to non-technical stakeholders
Experience in Databricks migration projects (e.g., from Hadoop, on-premise, or legacy ETL tools to Databricks)
Ability to thrive in a fast-paced, dynamic, client-facing role where delivering solid work products to exceed high expectations is a measure of success
Excellent leadership and interpersonal skills
Eager to contribute to a team-oriented environment
Strong prioritization and multi-tasking skills with a track record of meeting deadlines
Ability to be creative and analytical in a problem-solving environment
Effective verbal and written communication skills
Adaptable to new environments, people, technologies, and processes
Ability to manage ambiguity and solve undefined problems.
Commercial Refrigeration Engineer troubleshooting and resolving refrigeration systems for clients. Providing service and maintenance while developing knowledge in industrial systems with a mobile work structure.
Wintel Engineer responsible for maintaining the reliability and performance of hybrid Windows Server platforms. Ensuring consistency and security for business - critical applications in Glasgow.
Broadcast Engineer providing first and second tier technical support for live shows and offline productions. Troubleshooting broadcast equipment and ensuring quality operation.
Complaint Handling Engineer managing quality issues in Digital Solutions for medical software. Timely handling of complaints and communication with country organizations for resolution.
Senior Middleware Engineer managing Kubernetes and Docker applications for Ameriprise India LLP. Leading incident response and collaborating with teams on middleware technologies in a hybrid work environment.
Project Engineer managing midstream crude oil and gas processing pipeline projects at MPC. Collaborating across departments to ensure project success from conception to commissioning.
Project Engineer managing midstream crude oil and gas processing pipeline projects. Overseeing engineering, procurement, construction, and commissioning while ensuring safety and project objectives.
CAD Engineer designing CAD drawings for water and wastewater treatment infrastructure in the UK water industry. Collaborating with project teams to ensure quality and timely deliverables.
Building Engineer responsible for operation, maintenance, and repair of Atlantic Union Bank facilities. Collaborating with the Facilities Coordinator and managing service requests and vendor activities.