Data Engineer responsible for building and maintaining data solutions on AWS and GCP. Focus on Lakehouse architecture to support analytics, reporting, and AI/ML use cases.
About the role
- Data Engineer at GFT managing data systems and workflows, focusing on data engineering and data science collaboration. Leveraging technologies like Python, Airflow, and AWS.
Responsibilities
- Architect, develop, and maintain scalable data infrastructure, including data lakes, pipelines, and metadata repositories
- Work closely with data scientists to build and support data models, integrate data sources, and support machine learning workflows and experimentation environments
- Develop and optimize large-scale, batch, and real-time data processing systems to enhance operational efficiency and meet business objectives
- Leverage Python, Apache Airflow, and AWS services to automate data workflows and processes
- Utilize AWS services such as S3, Glue, EC2, and Lambda to manage data storage and compute resources
- Implement robust testing and validation procedures to ensure the reliability, accuracy, and security of data processing workflows
- Stay informed of industry best practices and emerging technologies in both data engineering and data science
Requirements
- Proficiency in Python for data processing and scripting (pandas, pyspark)
- Workflow automation (Apache Airflow)
- Experience with AWS services (Glue, S3, EC2, Lambda)
- Experience with Kubernetes and Docker for managing containerized environments in the cloud
- Hands-on experience with columnar and big data databases (Athena, Redshift, Vertica, Hive/Hadoop)
- Strong familiarity with AWS services for cloud-based data processing and management
- Experience with CI/CD tools such as Jenkins, CircleCI, or AWS CodePipeline for continuous integration and deployment
- Expertise in building and managing robust data architectures and pipelines for large-scale data operations
- Ability to support data science workflows, including collaboration on data preparation, feature engineering, and enabling experimentation environments
Benefits
- Competitive salary
- 13th-month salary guarantee
- Performance bonus
- Professional English course for employees
- Premium health insurance
- Extensive annual leave
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Data Engineer for Verity, a digital transformation consultancy, designing data architectures and building scalable pipelines. Collaborates on data quality and analytical dataset structuring.
Data Engineer Pleno at Verity focusing on building data pipelines and analytics frameworks. Engage with teams to ensure data quality and enhance cloud - based solutions.
Senior Data Engineer at Shopmonkey building and maintaining data infrastructure. Driving strategic decisions for tools and processes while ensuring data quality across platforms.
Data Engineer at dsm - firmenich designing and maintaining robust data pipelines. Collaborating on impactful projects centered around data for health, nutrition, and beauty.
Data Scientist developing customer - focused data products to improve customer journey at Noibu. Collaborating with teams to analyze feedback data and shape analytics strategies.
Data Engineer at fundcraft ensuring seamless data movement and storage. Focus on data migrations, ETL maintenance, and collaboration with the Product team for insights.
Databricks Data Engineer developing scalable data pipelines within the insurance domain. Building ETL workflows on Azure and Databricks for analytics and business decision - making.
Senior Engineer with expertise in AWS, Microsoft Fabric, and Purview at EXL. Leading development of scalable cloud and data governance solutions while mentoring engineering teams.
Senior Engineer with expertise in AWS, Microsoft Fabric, and Purview at EXL. Driving architecture, development, and implementation of scalable cloud and data governance solutions.