Lead Data Engineer overseeing engineers and advancing the data platform at American Family Insurance. Creating tools and infrastructure to empower teams across the company.
About the role
- Data Engineer with expertise in Databricks, SQL, and Python for scalable data solutions. Focused on ETL/ELT pipeline development, auditability, data quality, and automated testing.
Responsibilities
- Lead the design, development, and optimization of ETL/ELT pipelines using Databricks, Python, Spark, and Delta Lake.
- Architect scalable data solutions using Medallion architecture (Bronze, Silver, Gold layers).
- Design and implement data models and transformations using SQL and Python.
- Build and maintain audit frameworks to ensure traceability, compliance, and data lineage.
- Develop data quality monitoring and automated testing frameworks for pipeline reliability.
- Perform data analysis to support operational data requests and user queries.
- Collaborate with clinical data teams to analyse IRT/RTSM datasets.
- Create and maintain dashboards and reports using BI tools (e.g., Superset Power BI, Tableau, Qlik, or similar).
- Help manage CICD and automated code branching/deployment.
- Ensure compliance with GxP, CDISC, and other regulatory standards.
- Mentor junior engineers and promote engineering best practices.
Requirements
- 8–10 years of experience in data engineering, with leadership or team lead responsibilities.
- Strong hands-on experience with Databricks, Apache Spark, and Delta Lake.
- Advanced proficiency in SQL and Python for data transformation and automation.
- Experience with ETL/ELT orchestration, job optimization, and performance tuning.
- Proven experience designing and implementing audit, data quality, and testing frameworks.
- Hands-on experience with IRT/RTSM clinical trial data systems.
- Strong data analysis skills and ability to interpret complex datasets.
- Experience with BI/reporting tools such as Power BI, Tableau, or Qlik.
- Knowledge of clinical data standards (e.g., CDISC, SDTM, ADaM).
- Experience with cloud platforms (Azure, AWS, or GCP) and CI/CD pipelines.
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Working Student contributing to data engineering projects for AI/ML workflows while gaining hands - on experience with data pipelines and tools.
Data Architect designing end - to - end Snowflake data solutions and collaborating with technical stakeholders at Emerson. Supporting the realization of Data and Digitalization Strategy.
Manager of Data Engineering leading data assets and infrastructure initiatives at CLA. Collaborating with teams to enforce data quality standards and drive integration efforts.
Data Engineer building modern Data Lake architecture on AWS and implementing scalable ETL/ELT pipelines. Collaborating across teams for analytics and reporting on gaming platforms.
Chief Data Engineer leading Scania’s Commercial Data Engineering team for growing sustainable transport solutions. Focused on data products and pipelines for BI, analytics, and AI.
Entry - Level Data Engineer at GM, focusing on building large scale data platforms in cloud environments. Collaborating with data engineers and scientists while migrating systems to cloud solutions.
Data Engineer designing and building scalable ETL/ELT pipelines for enterprise - grade analytics solutions. Collaborating with product teams to deliver high - quality, secure, and discoverable data.
Data Engineer responsible for data integrations with AWS technology stack for Adobe's Digital Experience. Collaborating with multiple teams to conceptualize solutions and improve data ecosystem.
People Data Architect designing and managing people data analytics for Gen, delivering actionable insights for HR. Collaborating across teams to enhance data - driven decision - making.