Senior Data Engineer supporting AI - enabled financial compliance initiative with data pipelines and ingestion processes. Collaborating with diverse teams in a mission - critical regulated environment.
About the role
- Senior Data Engineer focusing on ETL developments in Azure Databricks at OAG, a travel data platform. Collaborating with multi-skilled teams to modernize Flight Status data processing.
Responsibilities
- Design, build, and optimize scalable ETL and Structured Streaming pipelines in Azure Databricks for real-time and batch ingestion of Flight Status data
- Design and implement data ingestion and processing pipelines that consolidate heterogeneous data sources, including APIs, event streams, and file-based feeds, into the OAG lakehouse (Azure Databricks + Delta Lake), ensuring data consistency, reliability, and scalability
- Implement and monitor data quality using automated validation, alerting, and observability practices
- Develop and maintain orchestration workflows in Apache Airflow, coordinating ingestion and transformation processes across multiple data flows
- Build reusable frameworks for schema evolution, error handling, deduplication, and auditing
- Collaborate with data platform, analytics, and product teams to define SLAs, data contracts, and performance targets
- Optimize Spark and Delta Lake performance for scalability, latency, and cost efficiency
- Implement CI/CD pipelines and automation for data workflows using Azure DevOps or equivalent tools
- Mentor engineers, review code, contribute to platform design discussions and planning, and help grow data engineering competencies in the team and across OAG
Requirements
- Proven track record in data engineering, with a strong focus on ETL development and streaming data architectures
- Experience with Azure Databricks, Apache Spark (Structured Streaming), and Delta Lake
- Proficiency in Python (PySpark) and SQL, with experience transforming large-scale, complex datasets
- Hands-on experience in data orchestration and workflow automation (e.g., Apache Airflow or similar)
- Experience working in a cloud data environment (preferably Azure) across storage, compute, and pipeline services
- Familiarity with streaming or messaging technologies (e.g., Kafka, Event Hubs)
- Strong understanding of data quality, validation, and observability practices
- Ability to deliver production-grade solutions with a results-oriented and ownership-driven mindset
- Experience implementing CI/CD and version-control practices using Azure DevOps, GitHub Actions, or similar tools
- Excellent analytical, communication, and collaboration skills
- Strong understanding of modern data engineering patterns and ability to design scalable, modular, and reliable data systems
Benefits
- Company-provided free lunch every day
- Private health insurance
- Company bonus scheme
- Voluntary participation in a company-supported retirement scheme
- A generous annual leave policy, growing with each year of service, and a day off during your birthday month
- Participation in team-building activities, team workshops, and group learning sessions
Job title
Job type
Experience level
Salary
Degree requirement
Tech skills
Location requirements
Data Architect leading the definition and construction of cloud data architecture for Kyndryl. Participating in significant technological modernization initiatives, focusing on Google Cloud Platform.
Senior Data Engineer driving data intelligence requirements and scalable data solutions for a global consulting firm. Collaborating across functions to enhance Microsoft architecture and analytics capabilities.
Experienced AI Engineer designing and building production - grade agentic AI systems using generative AI and large language models. Collaborating with data engineers, data scientists in a tech - driven company.
Intermediate Data Engineer designing and building data pipelines for travel industry data management. Collaborating across teams to ensure reliable data for analytics and reporting.
Data Engineer managing and organizing datasets for AI models at Walaris, developing AI - driven autonomous systems for defense and security applications.
Data Engineer designing and maintaining data pipelines at Black Semiconductor. Collaborating with process, equipment, and IT teams to support manufacturing analytics and decision - making.
Junior Data Engineer role focusing on Business Intelligence and Big Data at Avanade. Collaborating on data analysis and SQL queries in a supportive learning environment.
GCP Data Engineer designing and developing data processing modules for Ki, an algorithmic insurance carrier. Working closely with multiple teams to optimize data pipelines and reporting.
Data Engineer at Securian Financial optimizing scalable data pipelines for AI and advanced analytics. Collaborating with teams to deliver secure and accessible data solutions.