Data Engineering Lead at BlackStone eIT managing a team and driving data solutions aligned with strategic goals. Collaborating with departments to ensure effective data management.
About the role
- Data Engineer working on open-source Data Lakehouse and data pipeline development. Involves data integration and ensuring data quality in a hybrid work environment.
Responsibilities
- Development and maintenance of a fully open-source Data Lakehouse.
- Design and development of data pipelines for scalable and reliable data workflows to transform extensive quantities of both structured and unstructured data.
- Data integration from various sources, including databases, APIs, data streaming services and cloud data platforms.
- Optimisation of queries and workflows for increased performance and enhanced efficiency.
- Writing modular, testable and production-grade code.
- Ensuring data quality through monitoring, validation and data quality checks, maintaining accuracy and consistency across the data platform.
- Elaboration of test programs.
- Document processes comprehensively to ensure seamless data pipeline management and troubleshooting.
- Assistance with deployment and configuration of the system.
Requirements
- Master's degree in IT and minimum 9 years of relevant professional experience (or Bachelor's degree in IT and minimum 13 years of experience).
- Extensive hands-on experience as Data Engineer or Data Architect in modern cloud-based open-source data platform solutions and on data analytics tools.
- Excellent knowledge of data warehouse and/or data lakehouse design & architecture.
- Experience with open-source, code-based data transformation tools such as dbt, Spark and Trino.
- Previous experience with open-source orchestration tools such as Airflow, Dagster or Luigi.
- Experience with SQL and Python.
- Experience with AI-powered assistants like Amazon Q that can streamline data engineering processes.
- Good knowledge of relational database systems.
- Good knowledge of event streaming platforms and message brokers like Kafka and RabbitMQ.
- Extensive experience in creating end-to-end data pipelines and the ELT framework.
- Understanding of the principles behind storage protocols like Apache Iceberg or Delta Lake.
- Proficiency with Kubernetes and Docker/Podman.
- Good knowledge of data modelling tools.
- Good knowledge of online analytical data processing (OLAP) and data mining tools.
- Advanced English (C1) communication skills (written and spoken).
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Data engineer designing custom data architectures at Zenika. Developing and maintaining data transformation pipelines, optimizing performance, and collaborating with cross - functional teams.
Senior Data Engineer responsible for AspenTech IP.21 implementation, leading a team for industrial data solutions.
Senior Data Engineer responsible for implementing and optimizing industrial data infrastructure at AG Solution. Leading a team and collaborating with various stakeholders to ensure data quality and reporting.
Lead Data Engineer designing scalable data architecture on Google Cloud Platform for infas 360. Collaborating with cross - functional teams to develop tailored data products in Bonn.
Data Engineer developing data platforms for Octopus Electric Vehicles. Build and optimise data solutions to transform insights for sustainable transportation.
Senior Data Engineer developing and maintaining data pipelines and solutions for EMC's data analytics products. Collaborating with cross - functional teams to ensure high - quality data management and governance practices.
Principal Product Manager leading product strategy for health data platform at PointClickCare. Collaborating across teams to optimize health data for analytics and care delivery.
Senior Data Engineer designing and optimizing data platforms for clients using Microsoft Azure, Microsoft Fabric, Power BI, and Databricks. Working closely with clients to deliver scalable solutions.
Data Engineer providing technical expertise on mission - critical NAVSUP OIS program. Work involves data architecture and database management in AWS GovCloud environments.