Senior Data Engineer working on GCP cloud data solutions and ETL processes in AI & Data Engineering team. Collaborating within a hybrid work setup in Bangalore, India.
About the role
- Lead Data Engineer designing and managing AWS data pipelines and platforms for AI & Data Engineering team. Involves collaborating with data scientists, analysts, and stakeholders for data-driven solutions.
Responsibilities
- Design and implement scalable ETL/ELT pipelines using AWS Glue, Spark (PySpark), and Step Functions
- Work with structured and semi-structured data using Athena, S3, and Lake Formation to enable efficient querying and access control
- Develop and deploy serverless data processing solutions using AWS Lambda and integrate them into pipeline orchestration
- Perform advanced SQL and PL/SQL development for data transformation, analysis, and performance tuning
- Build data lakes and data warehouses using S3, Aurora, and Athena
- Implement data governance, security, and access control strategies using AWS tools including Lake Formation, CloudFront, EBS/EFS, and IAM
- Develop and maintain metadata, lineage, and data cataloging capabilities
- Participate in data modeling exercises for both OLTP and OLAP environments
- Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver actionable insights
- Monitor, debug, and optimize data pipelines for reliability and performance.
Requirements
- Must have: Python, SQL/PLSQL, AWS, Postgresql, S3, Glue
- Good to have: CDK, GitHub
- Strong experience with AWS data services: Glue, Athena, Step Functions, Lambda, Lake Formation, S3, EC2, Aurora, EBS/EFS, CloudFront
- Proficient in PySpark, Python, SQL (basic and advanced), and PL/SQL
- Solid understanding of ETL/ELT processes and data warehousing concepts
- Familiarity with modern data platform fundamentals and distributed data processing
- Experience in data modeling (conceptual, logical, physical) for analytical and operational use cases
- Experience with orchestration and workflow management tools within AWS
- Strong debugging and performance tuning skills across the data stack.
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
No Education Requirement
Tech skills
Location requirements
Senior Data Engineer designing and developing scalable data pipelines using DBT and Python. Collaborating with internal stakeholders for analytics and reporting solutions.
Senior Data Engineer designing and optimizing the core data layer for Degreed's upskilling platform. Collaborating with internal teams and clients to ensure access to reliable and performant data.
Big Data Engineer handling both internal and external stakeholders for data processing related to fraud and compliance at ING. Managing and transforming high volume data and working closely with project teams.
Big Data Engineer role developing state - of - the - art solutions for financial crime prevention at ING. Collaborating with teams to manage high volume data and deliver effective technical solutions.
Senior Data Engineer focusing on Databricks in a market - leading company. Designing data architectures and optimizing data workflows in a collaborative environment.
Senior Data Architect Lead at Leidos developing enterprise data and analytics solutions for the Department of War. Collaborating with teams to implement data strategies and governance frameworks.
Senior Data Engineer Lead at Leidos, enhancing enterprise data solutions for DoD organizations. Collaborating with teams to deliver scalable data analytics and AI capabilities.
Data Engineer building advanced technology solutions for clients. Organizing and making disparate data meaningful to impact missions in fraud detection, cancer research, and national intelligence.
Staff Data Engineer on Real World Evidence team driving large - scale data initiatives. Collaborating with cross - functional teams to optimize data pipelines and improve healthcare outcomes.