Associate Data Engineer building and maintaining data systems at Incedo. Transforming raw information into accessible datasets for decision - making and collaborating with analysts and data scientists.
About the role
- Data Engineer strengthening data platform team at Samba TV to improve data analytics and reporting capabilities. Building on AWS, Databricks, BigQuery, and Snowflake technology.
Responsibilities
- Build scalable data product architecture
- Responsible for modernizing data frameworks and integrations with Databricks and BigQuery
- Upgrade and reduce toil for developers on Apache Airflow
- Develop and optimize data transformations using Apache Spark (PySpark/Scala)
- Build procedures and guidelines to help teams operate with data
- Identify bottlenecks in our development lifecycle and find solutions to improve them
- Drive innovation throughout the tech org by evangelizing and educating teams on best practices and new technologies
- Work directly with our data teams and FinOps teams to drive efforts that span across teams
- Implement data governance, access control, and auditing using Databricks Unity Catalog
- Build and integrate automated, reusable data validation suites using data quality frameworks (Great Expectations or similar)
- Implement monitoring and anomaly detection systems for data quality, reliability and performance
- Develop and manage REST APIs to support secure data access, automation, and integration
- Collaborate with data scientists, analysts, and software engineers to deliver governed, reusable data assets
- Implement monitoring, logging, and alerting for data workflows
- Optimize cost and performance of cloud-based data infrastructure
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent experience)
- 5+ years of experience in data engineering or a related role
- Strong hands-on experience with Databricks, Apache Spark and BigQuery or Snowflake
- Proven experience with Modern table formats such as Delta Lake and Iceberg
- A deep understanding of the data lifecycle and how teams operate with data
- Hands-on experience implementing data governance and metadata management using Databricks Unity Catalog
- Experience managing and extending Apache Airflow (custom operators, plugins, infrastructure)
- Experience with Kubernetes
- Solid experience with AWS cloud services, especially S3 and data-related services
- Experience with data validation and data quality principles and working with SLA systems
- Proficiency in Python and SQL
Benefits
- Equal opportunity employer
- Celebration of diversity
- Commitment to an inclusive environment
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Intern Data Engineer at Flutter Studios learning to create applications in the gaming industry. Join a cross - functional team to implement a basic application in an Agile environment.
AI/Data Engineer at Comcast developing data pipelines and AI solutions for audit processes. Leading team efforts to ensure data quality and compliance with audit objectives across business units.
Senior Data Engineer involved in AWS Cloud and Big Data solutions for Financial Crime. Join CommBank's team to tackle complex data - centric problems with advanced technologies.
Senior Data Engineer leading and mentoring a team in building scalable data pipelines for digital transformation projects in an international software company.
Senior Data Operations Engineer at iKnowHow designing and implementing scalable data - driven applications. Focus on data pipelines, APIs, and collaboration across teams for project success.
Data/AI Engineer supporting innovations in corporate travel with AWS technologies at HRS Group. Collaborating with data teams to develop AI solutions and maintain data pipelines.
AS
Data Engineer developing data solutions for reporting and analytics at Assurant. Implementing and optimizing Data Warehouse solutions in cloud and on - premise environments with Agile methodology.
AI Engineer developing AI - driven analytics for trading at Deloitte. Focusing on scalable data pipelines and collaboration with traders for actionable insights.
Big Data Engineer developing and optimizing data systems at ESET. Collaborating with cross - functional teams to handle and transform vast datasets.