Data Engineer I building and operationalizing complex data solutions for Travelers' analytics using Databricks. Collaborating within teams to educate end users and support data governance.
About the role
- Data Engineer strengthening data platform team at Samba TV to improve data analytics and reporting capabilities. Building on AWS, Databricks, BigQuery, and Snowflake technology.
Responsibilities
- Build scalable data product architecture
- Responsible for modernizing data frameworks and integrations with Databricks and BigQuery
- Upgrade and reduce toil for developers on Apache Airflow
- Develop and optimize data transformations using Apache Spark (PySpark/Scala)
- Build procedures and guidelines to help teams operate with data
- Identify bottlenecks in our development lifecycle and find solutions to improve them
- Drive innovation throughout the tech org by evangelizing and educating teams on best practices and new technologies
- Work directly with our data teams and FinOps teams to drive efforts that span across teams
- Implement data governance, access control, and auditing using Databricks Unity Catalog
- Build and integrate automated, reusable data validation suites using data quality frameworks (Great Expectations or similar)
- Implement monitoring and anomaly detection systems for data quality, reliability and performance
- Develop and manage REST APIs to support secure data access, automation, and integration
- Collaborate with data scientists, analysts, and software engineers to deliver governed, reusable data assets
- Implement monitoring, logging, and alerting for data workflows
- Optimize cost and performance of cloud-based data infrastructure
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent experience)
- 5+ years of experience in data engineering or a related role
- Strong hands-on experience with Databricks, Apache Spark and BigQuery or Snowflake
- Proven experience with Modern table formats such as Delta Lake and Iceberg
- A deep understanding of the data lifecycle and how teams operate with data
- Hands-on experience implementing data governance and metadata management using Databricks Unity Catalog
- Experience managing and extending Apache Airflow (custom operators, plugins, infrastructure)
- Experience with Kubernetes
- Solid experience with AWS cloud services, especially S3 and data-related services
- Experience with data validation and data quality principles and working with SLA systems
- Proficiency in Python and SQL
Benefits
- Equal opportunity employer
- Celebration of diversity
- Commitment to an inclusive environment
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Data Engineer shaping modern data architecture to drive golf’s digital transformation. Collaborating with teams to enhance data pipelines and insights for customer engagement and revenue growth.
Staff Data Engineer overseeing complex data systems for CITY Furniture. Responsible for architecting and optimizing data ecosystems in a hybrid work environment.
Data Engineer focusing on secure ETL/ELT data pipelines and compliance in healthcare. Designing scalable ingestion frameworks and ensuring alignment with federal standards.
Data Migration Engineer at Capgemini delivering migration solutions for Guidewire Claim Center. Collaborating on cloud data migrations and validating processes in a sustainable tech environment.
Data Engineer responsible for collecting and analyzing data at Cruise Planners. Collaborate with teams for actionable insights using MySQL and Power BI.
Data Engineer for Leader Entertainment developing data solutions on Google Cloud Platform. Collaborating on data models, pipelines, and analytics in a hybrid role.
Senior Data Engineer designing and scaling data foundations for AI adoption across Ad Tech. Collaborating with cross - functional teams to deliver robust pipelines for high - profile AI applications.
Specialist in Data Engineering leading pipeline optimization at Inmetrics. Collaborating in innovative data - driven projects within a hybrid work environment.
Data Architect responsible for designing and implementing data architecture at Stefanini. Collaborate with technical teams and stakeholders in a hybrid work environment.