Hybrid Data Engineering Lead

at The Muse

• Job title: Data Engineering Lead
• Job type: Full Time
• Experience level: Senior
• Salary: $120,000 - $145,000 per year
• Degree requirement: Bachelor's Degree
• Tech skills: Apache, AWS, Azure, ETL, PySpark, Python, Scala, Spark, SQL
• Location requirements: Hybrid, Warren, United States

About the role

• Data Engineering Lead designing and optimizing scalable data pipelines for a property insurance company. Collaborating across teams, mentoring engineers, and driving data architecture initiatives.

Responsibilities

• Build and optimize distributed data processing jobs using Apache Spark on Databricks.
• Implement Delta Lake, DLT pipelines, dbt transformations and Medallion architecture for scalable and reliable data workflows.
• Design and automate ETL pipelines using Azure Data Factory, Databricks, and Synapse Analytics.
• Integrate data from diverse sources including Duck Creek, Intacct, Workday and external APIs.
• Develop dimensional models (Star/Snowflake schemas), stored procedures, and views for data warehouses.
• Ensure efficient querying and transformation using SQL, T-SQL, and PySpark.
• Leverage Azure DevOps, CI/CD pipelines, and GitHub for version control and deployment.
• Utilize Azure Logic Apps and ML Flow for workflow automation and model training.
• Implement role-based access control (RBAC), data encryption, and auditing mechanisms.
• Work closely with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.
• Mentor junior engineers and contribute to code reviews and architectural decisions.

Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering with at least 2 years on Databricks.
• Proficiency in Python, Scala, SQL, and Spark.
• Hands-on experience with Azure Data Services (ADF, ADLS, Synapse).
• Strong understanding of ETL, data warehousing, and data modeling concepts.
• Experience with Microstrategy, Power BI, including DAX and advanced visualizations.
• Familiarity with MLflow, LangChain, and LLM integration is a plus.
• Knowledge of Duck Creek a plus.
• Insurance Domain knowledge preferred.
• Preferred Certifications Databricks Data Engineering Professional Azure/AWS Data Engineering Certifications

Benefits

• Opportunities to stretch and grow: your professional and personal development matters to us.
• Clarity and kindness: you can rely on us to be open, honest and supportive, offering clarity on what success looks like.
• Support in good times and bad: we believe in showing up for each other consistently, not only when it’s easy.
• A community that cares: we are committed to sustaining a community in which each person feels cared for as an individual.