Senior Data Engineer supporting AI - enabled financial compliance initiative with data pipelines and ingestion processes. Collaborating with diverse teams in a mission - critical regulated environment.
MA
Hybrid AWS Data Engineer – Terraform
Posted 3 months ago
About the role
- AWS Data Engineer designing and deploying infrastructure using Terraform for a modern Data Lakehouse. Driving ETL processes, data ingestion, and analytics using AWS technologies.
Responsibilities
- Design, develop, and deploy AWS infrastructure using Terraform, including S3, Glue, IAM, Lake Formation, and Athena resources
- Develop and maintain AWS Glue ETL jobs (PySpark or Python shell) for data ingestion, transformation, and curation across raw → clean → curated layers
- Integrate Airflow (Amazon MWAA or self-managed) for orchestrating Glue jobs, data pipelines, and dependencies
- Build and maintain Glue Catalog, manage metadata, and align with Lake Formation security policies
- Write complex SQL queries for data validation, transformation, and reporting logic, ensuring efficient query performance
- Manage Terraform state files, backend setup (S3 + DynamoDB), and environment-based deployments
- Implement data ingestion frameworks for batch and near real-time pipelines
- Collaborate with Snowflake and BI teams for seamless data consumption
- Contribute to high availability (multi-AZ) and disaster recovery (multi-region) strategies for core data components
Requirements
- 6 years of experience as a Data Engineer or Cloud Engineer
- Strong expertise in AWS Services: S3, Glue, Glue Catalog, Lake Formation, IAM, Athena, CloudWatch, Lambda (preferred)
- Hands-on proficiency in Terraform (HCL) for infrastructure automation
- Experience with Airflow DAGs for orchestration of Glue, S3, and external data flows
- Solid understanding of PySpark / Python for ETL scripting
- Strong ability to write and optimize complex SQL (joins, window functions, CTEs, and analytical queries)
- Familiarity with data lake formats (Iceberg, Parquet, Delta, etc.)
- Experience with CI/CD pipelines (GitHub Actions, CodePipeline, or Jenkins)
Benefits
- Flexible work
- Healthcare including dental, vision, mental health, and well-being programs
- Financial well-being programs such as 401(k) and Employee Share Ownership Plan
- Paid time off and paid holidays
- Paid parental leave
- Family building benefits like adoption assistance, surrogacy, and cryopreservation
- Social well-being benefits like subsidized back-up child/elder care and tutoring
- Mentoring, coaching, and learning programs
- Employee Resource Groups
Job title
Job type
Experience level
Salary
Degree requirement
Tech skills
Location requirements
Data Architect leading the definition and construction of cloud data architecture for Kyndryl. Participating in significant technological modernization initiatives, focusing on Google Cloud Platform.
Senior Data Engineer driving data intelligence requirements and scalable data solutions for a global consulting firm. Collaborating across functions to enhance Microsoft architecture and analytics capabilities.
Experienced AI Engineer designing and building production - grade agentic AI systems using generative AI and large language models. Collaborating with data engineers, data scientists in a tech - driven company.
Intermediate Data Engineer designing and building data pipelines for travel industry data management. Collaborating across teams to ensure reliable data for analytics and reporting.
Data Engineer managing and organizing datasets for AI models at Walaris, developing AI - driven autonomous systems for defense and security applications.
Data Engineer designing and maintaining data pipelines at Black Semiconductor. Collaborating with process, equipment, and IT teams to support manufacturing analytics and decision - making.
Junior Data Engineer role focusing on Business Intelligence and Big Data at Avanade. Collaborating on data analysis and SQL queries in a supportive learning environment.
GCP Data Engineer designing and developing data processing modules for Ki, an algorithmic insurance carrier. Working closely with multiple teams to optimize data pipelines and reporting.
Data Engineer at Securian Financial optimizing scalable data pipelines for AI and advanced analytics. Collaborating with teams to deliver secure and accessible data solutions.