Senior Data Engineer designing and improving software for business capabilities at Barclays. Collaborating with teams to build a data and intelligence platform for Equity Derivatives.
About the role
- Senior Data Engineer responsible for designing and implementing ETL pipelines. Collaborating with cross-functional teams to ensure data quality and optimizing data workflows with AWS tools.
Responsibilities
- Design and implement robust ETL pipelines using AWS Glue, Lambda, and S3.
- Monitor and optimize the performance of data workflows and batch processing jobs.
- Troubleshoot and resolve issues related to data pipeline failures, inconsistencies, and performance bottlenecks.
- Collaborate with cross-functional teams to define data requirements and ensure data quality and accuracy.
- Develop and maintain automated solutions for data transformation, migration, and integration tasks.
- Implement best practices for data security, data governance, and compliance within AWS environments.
- Continuously improve and optimize AWS Glue jobs, Lambda functions, and S3 storage management.
- Maintain comprehensive documentation for data pipeline architecture, job schedules, and issue resolution processes.
Requirements
- Strong experience with Data Engineering practices.
- Experience in AWS services, particularly AWS Glue, Lambda, S3, and other AWS data tools.
- Proficiency in SQL, python , Pyspark, numpy etc and experience in working with large-scale data sets.
- Experience in designing and implementing ETL pipelines in cloud environments.
- Expertise in troubleshooting and optimizing data processing workflows.
- Familiarity with data warehousing concepts and cloud-native data architecture.
- Knowledge of automation and orchestration tools in a cloud-based environment.
- Strong problem-solving skills and the ability to debug and improve the performance of data jobs.
- Excellent communication skills and the ability to work collaboratively with cross-functional teams.
- Good to have knowledge of DBT & Snowflake
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Senior AI & Data Engineer developing and implementing AI solutions in collaboration with clients and teams. Working on projects involving generative AI, predictive analytics, and data mastery.
Consultant driving IA business growth in Deloitte's Artificial Intelligence & Data team. Delivering innovative solutions using data analytics and automation technologies.
Data Engineer responsible for managing data architecture and pipelines at Snappi, a neobank. Collaborating with teams to enable data processing and analysis in innovative banking solutions.
Data Engineer at Destinus developing the data platform to support production and analytics needs. Involves migrating Excel sources to Lakehouse and integrating ERP systems in a hybrid role.
Senior Data Engineer developing solutions within the Global Specialty portfolio at an insurance company. Engaging with diverse business partners to ensure high quality data reporting.
Data Engineer at UBDS Group focusing on designing and optimizing modern data platforms. Collaborating in a multidisciplinary team to develop reliable data assets for analytics and operational use cases.
Data Engineer (dbt) at SDG Group involved in all phases of data projects. Collaborate on data ingestion, transformation, and visualization in a hybrid environment.
Data Consultant at SDG Group specializing in Data & Analytics projects. Collaborate on technical - functional definitions, ETL, data modeling, and visualization for cloud solutions.
Senior Data Engineer responsible for growing customer - defined targeting calculations and developing key/value databases for real - time data processing.