Data Engineer managing and organizing datasets for AI models at Walaris, developing AI - driven autonomous systems for defense and security applications.
About the role
- Senior Data Engineer responsible for building and optimizing ETL pipelines for a leading travel data platform. Collaborating with software and data engineers to modernize data ingestion and processing.
Responsibilities
- Design, build, and optimize scalable ETL and Structured Streaming pipelines in Azure Databricks for real-time and batch ingestion of Flight Status data
- Design and implement data ingestion and processing pipelines that consolidate heterogeneous data sources, including APIs, event streams, and file-based feeds, into the OAG lakehouse (Azure Databricks + Delta Lake), ensuring data consistency, reliability, and scalability
- Implement and monitor data quality using automated validation, alerting, and observability practices
- Develop and maintain orchestration workflows in Apache Airflow, coordinating ingestion and transformation processes across multiple data flows
- Build reusable frameworks for schema evolution, error handling, deduplication, and auditing
- Collaborate with data platform, analytics, and product teams to define SLAs, data contracts, and performance targets
- Optimize Spark and Delta Lake performance for scalability, latency, and cost efficiency
- Implement CI/CD pipelines and automation for data workflows using Azure DevOps or equivalent tools
- Mentor engineers, review code, contribute to platform design discussions and planning, and help grow data engineering competencies in the team and across OAG
Requirements
- Proven track record in data engineering with a strong focus on ETL development and streaming data architectures
- Experience with Azure Databricks, Apache Spark (Structured Streaming), and Delta Lake
- Proficiency in Python (PySpark) and SQL, with experience transforming large-scale, complex datasets
- Hands-on experience in data orchestration and workflow automation (e.g., Apache Airflow or similar)
- Experience working in a cloud data environment (preferably Azure) across storage, compute, and pipeline services
- Familiarity with streaming or messaging technologies (e.g., Kafka, Event Hubs)
- Strong understanding of data quality, validation, and observability practices
- Ability to deliver production-grade solutions with a results-oriented and ownership-driven mindset
- Experience implementing CI/CD and version-control practices using Azure DevOps, GitHub Actions, or similar tools
- Excellent analytical, communication, and collaboration skills
- Strong understanding of modern data engineering patterns and ability to design scalable, modular, and reliable data systems
Benefits
- Company-provided free lunch every day
- Private health insurance
- Company bonus scheme
- Voluntary participation in a company-supported retirement scheme
- Generous annual leave policy that grows with each year of service
- Day off during your birthday month
- Participation in team-building activities
- Participation in team workshops
- Participation in group learning sessions
Job title
Job type
Experience level
Salary
Degree requirement
Tech skills
Location requirements
Data Engineer designing and maintaining data pipelines at Black Semiconductor. Collaborating with process, equipment, and IT teams to support manufacturing analytics and decision - making.
Junior Data Engineer role focusing on Business Intelligence and Big Data at Avanade. Collaborating on data analysis and SQL queries in a supportive learning environment.
GCP Data Engineer designing and developing data processing modules for Ki, an algorithmic insurance carrier. Working closely with multiple teams to optimize data pipelines and reporting.
Data Engineer at Securian Financial optimizing scalable data pipelines for AI and advanced analytics. Collaborating with teams to deliver secure and accessible data solutions.
IT Data Engineering Co‑Op at BlueRock Therapeutics supports development of scientific data systems. Collaboration on data workflows and foundational AWS data engineering tasks.
Data Engineer I building and operationalizing complex data solutions for Travelers' analytics using Databricks. Collaborating within teams to educate end users and support data governance.
Data Engineer shaping modern data architecture to drive golf’s digital transformation. Collaborating with teams to enhance data pipelines and insights for customer engagement and revenue growth.
Staff Data Engineer overseeing complex data systems for CITY Furniture. Responsible for architecting and optimizing data ecosystems in a hybrid work environment.
Data Engineer strengthening data platform team at Samba TV to improve data analytics and reporting capabilities. Building on AWS, Databricks, BigQuery, and Snowflake technology.