Lead Data Engineer overseeing engineers and advancing the data platform at American Family Insurance. Creating tools and infrastructure to empower teams across the company.
About the role
- Senior Data Engineer (AWS) with expertise in Python and data services. Working on enterprise-scale data processing and analytics initiatives in a hybrid model.
Responsibilities
- Design, develop, and maintain scalable data processing pipelines using Python, PySpark, and Spark SQL
- Build and optimize distributed data processing workflows on AWS platforms.
- Leverage AWS data services such as EMR, Glue, Lambda, and S3 for batch and real-time data processing.
- Design and manage data storage solutions using RDS/MySQL, Redshift , and other AWS-native databases.
- Implement effective data modeling, schema design, and schema evolution strategies.
- Perform performance tuning and optimization of Spark jobs and SQL queries.
- Monitor and troubleshoot data pipelines using AWS CloudWatch and logging frameworks.
- Manage secrets and credentials securely using AWS Secrets Manager.
- Collaborate with data architects, analysts, and stakeholders to translate business requirements into technical solutions.
- Debug complex data issues and provide root cause analysis with long-term fixes.
- Ensure data quality, reliability, and scalability across platforms
Requirements
- 10–13 years of overall experience in Data Engineering
- Strong proficiency in Python and SQL
- Extensive hands-on experience with PySpark and Spark SQL
- Strong experience with AWS data services , including: EMR Glue Lambda S3 RDS / MySQL Redshift CloudWatch Secrets Manager
- Solid understanding of distributed computing concepts
- Strong experience in data modeling, schema handling, and performance tuning
- Excellent debugging, analytical, and problem-solving skills.
- Ability to work effectively in a hybrid and collaborative environment
Benefits
- Immediate joiners only
- Hybrid work mode
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Working Student contributing to data engineering projects for AI/ML workflows while gaining hands - on experience with data pipelines and tools.
Data Architect designing end - to - end Snowflake data solutions and collaborating with technical stakeholders at Emerson. Supporting the realization of Data and Digitalization Strategy.
Manager of Data Engineering leading data assets and infrastructure initiatives at CLA. Collaborating with teams to enforce data quality standards and drive integration efforts.
Data Engineer building modern Data Lake architecture on AWS and implementing scalable ETL/ELT pipelines. Collaborating across teams for analytics and reporting on gaming platforms.
Chief Data Engineer leading Scania’s Commercial Data Engineering team for growing sustainable transport solutions. Focused on data products and pipelines for BI, analytics, and AI.
Entry - Level Data Engineer at GM, focusing on building large scale data platforms in cloud environments. Collaborating with data engineers and scientists while migrating systems to cloud solutions.
Data Engineer designing and building scalable ETL/ELT pipelines for enterprise - grade analytics solutions. Collaborating with product teams to deliver high - quality, secure, and discoverable data.
Data Engineer responsible for data integrations with AWS technology stack for Adobe's Digital Experience. Collaborating with multiple teams to conceptualize solutions and improve data ecosystem.
People Data Architect designing and managing people data analytics for Gen, delivering actionable insights for HR. Collaborating across teams to enhance data - driven decision - making.