AI Data Pipeline Engineer designing and operating high - throughput systems for petabyte - scale data delivery. Collaborating across teams to ensure data flows into AI workloads efficiently.
About the role
- AWS Data Engineer II responsible for designing and developing data pipelines in AWS cloud environment. Building scalable ETL jobs and collaborating with teams to enhance data integration.
Responsibilities
- Develop, maintain, and enhance data ingestion pipelines within the Enterprise Data Integration Framework (EDIF).
- Build and update AWS Glue ETL jobs in Python (PySpark) for validation, transformation, enrichment, and microbatch processing.
- Collaborate across engineering, QA, Cloud Operations, and vendor partners to implement new ingestion workflows.
- Contribute to architectural design, documentation, and best practices that improve EDIF scalability and resilience.
- Monitor, troubleshoot, and optimize ingestion workflow performance across AWS services (S3, Lambda, Glue, DynamoDB, PostgreSQL, Step Functions, CloudWatch, EventBridge, Athena).
- Assist with the onboarding of new vendor feeds, schemas, and operational schedules into EDIF.
- Participate in platform release management, change control, and nightly batch support activities.
- Maintain ingestion observability through CloudWatch dashboards and EDIF event-monitoring tools.
- Provide technical support to ensure successful nightly data ingestion and data quality compliance.
Requirements
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or related field.
- 3+ years of hands-on experience in data engineering or ETL development.
- Experience with core AWS data and compute services, including S3, Lambda, Glue, DynamoDB, PostgreSQL, Step Functions, CloudWatch, IAM, Athena.
- Proficiency with Python and/or PySpark for data transformation.
- Experience working with relational databases (PostgreSQL, SQL Server, Oracle, etc.).
- Understanding of data modeling, schema evolution, and data validation principles.
- Experience with Git-based version control and CI/CD workflows.
Benefits
- 401K matching
- health benefits
- employee stock options
- paid time off
- volunteer time off
Job title
Job type
Experience level
Salary
Degree requirement
Tech skills
Location requirements
AWS Data Engineer role focusing on AWS technologies in Gurugram, Haryana, India. Responsibilities include AWS data engineering tasks and collaboration with team members.
Data Engineer implementing innovative technology for various domains at Quantexa. Building data pipelines and providing insights to help clients solve complex business problems.
Principal Consultant Data Architecture leading complex Data and Analytics projects in a hybrid environment. Responsible for designing enterprise data architectures and mentoring technical teams.
Consultant / Senior Consultant in Data Engineering & Data Science contributing to data solutions. Collaborating with cross - functional teams in a hybrid environment in Germany.
Senior Data Engineer managing data platform strategy and analytics architecture at HALOS scaleup company. Owning design and implementation of analytical data platform.
Data Engineer building trusted data platforms for decision making at Lyrebird Health. Collaborating with teams to develop and maintain data pipelines and analytics - ready tables.
Data Engineer at leading online insurance platform for businesses. Delivering data pipelines and collaborating cross - functionally to enhance decision - making.
Senior Data Engineer developing and maintaining products for dentsu’s marketing using cloud technologies. Collaborating with teams to solve complex data problems and ensuring driving business value.
Senior Data Engineer responsible for designing and developing data products and solutions at Lloyds Banking Group. Collaborating with various teams to improve regulatory data processes and mentoring other engineers.