Data Engineer with expertise in Databricks, SQL, and Python for scalable data solutions. Focused on ETL/ELT pipeline development, auditability, data quality, and automated testing.
Responsibilities
Lead the design, development, and optimization of ETL/ELT pipelines using Databricks, Python, Spark, and Delta Lake.
Architect scalable data solutions using Medallion architecture (Bronze, Silver, Gold layers).
Design and implement data models and transformations using SQL and Python.
Build and maintain audit frameworks to ensure traceability, compliance, and data lineage.
Develop data quality monitoring and automated testing frameworks for pipeline reliability.
Perform data analysis to support operational data requests and user queries.
Collaborate with clinical data teams to analyse IRT/RTSM datasets.
Create and maintain dashboards and reports using BI tools (e.g., Superset Power BI, Tableau, Qlik, or similar).
Help manage CICD and automated code branching/deployment.
Ensure compliance with GxP, CDISC, and other regulatory standards.
Mentor junior engineers and promote engineering best practices.
Requirements
8–10 years of experience in data engineering, with leadership or team lead responsibilities.
Strong hands-on experience with Databricks, Apache Spark, and Delta Lake.
Advanced proficiency in SQL and Python for data transformation and automation.
Experience with ETL/ELT orchestration, job optimization, and performance tuning.
Proven experience designing and implementing audit, data quality, and testing frameworks.
Hands-on experience with IRT/RTSM clinical trial data systems.
Strong data analysis skills and ability to interpret complex datasets.
Experience with BI/reporting tools such as Power BI, Tableau, or Qlik.
Knowledge of clinical data standards (e.g., CDISC, SDTM, ADaM).
Experience with cloud platforms (Azure, AWS, or GCP) and CI/CD pipelines.
IT Data Engineer passionate about data solutions supporting digital transformation at Sizewell C. Join a collaborative team working on building data pipelines and platforms for a major infrastructure project.
Cloud & Data Engineer working with large datasets in innovative projects for Marketing Technology team. Focus on cloud platforms and development of scalable systems for digital marketing support.
Data Engineer responsible for building and maintaining data solutions using Microsoft Fabric. Working within a consultancy environment to meet client expectations across various sectors.
Data Engineering Lead at Fetch owning end - to - end data platform for AI, pricing, and operations. Collaborate with teams to enable real - time data - driven decisions and trustworthiness.
Data Engineer responsible for building ELT/ETL pipelines and supporting data governance practices at Daniels Health. Joining a mission - driven company innovating in healthcare waste management across multiple countries.
Data Engineer designing and optimizing Azure - based data platforms for enterprise analytics. Developing scalable data pipelines and enabling insights through Power BI and Azure Synapse Analytics.
Senior Software Engineer focused on ingestion pipeline at Fullstory. Engineering distributed systems for processing data at scale while collaborating with technical leaders.
Junior Data Engineer contributing to data solutions in home24's Martech team. Focus on data pipelines, analytical workflows, and machine learning model scaling with cross - functional collaboration.
Data Engineer at Onepoint developing cloud - native architectures and scalable data solutions. Collaborating on data processing pipelines and guiding clients on best practices.