Senior Data Engineer optimizing ETL/ELT pipelines at Asahi Kasei. Evaluate programming concepts and support data science projects while ensuring solution stability in a hybrid work setup.
About the role
- Senior Data Engineer at Skillfield designing distributed data processing solutions using Apache Spark. Collaborates on cloud and on-prem solutions across enterprise levels in a hybrid work environment.
Responsibilities
- Design, develop, and maintain ETL and ELT pipelines using Apache Spark (batch and streaming)
- Build Spark applications in Scala, applying distributed processing best practices
- Optimise Spark workloads to improve performance, scalability, and reliability
- Work within the Hadoop ecosystem, including HDFS, Hive, and HBase
- Design and support high-performance analytical solutions using ClickHouse
- Build and manage data ingestion workflows using Apache NiFi
- Operate and troubleshoot data platforms in Linux-based environments
- Partner with data architects on solution design, schemas, and data models
- Investigate and resolve data pipeline failures and performance issues
- Maintain technical documentation and delivery artefacts in Confluence
- Track delivery progress and work items using Jira
Requirements
- Hands-on experience building solutions with Apache Spark
- Strong Scala development capability
- Experience working in Linux environments
- Practical knowledge of HDFS, Hive, and HBase
- Experience with ClickHouse or comparable analytical databases
- A solid understanding of distributed systems and performance optimisation
- Experience working across cloud, hybrid, or on-prem platforms
- Familiarity with Git-based workflows and CI/CD pipelines
- Nice to Have
- Experience with Kafka or other streaming platforms
- Exposure to Databricks, EMR, or managed Spark services
- Experience with orchestration tools such as Airflow
- Awareness of data security, identity management, and governance practices
Benefits
- Enjoy flexibility, support, and a focus on sustainable delivery
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Data Engineer responsible for building data intelligence system for the public sector. Ensuring data ingestion, quality, correlation, and helping with analytics for decision making.
Big Data Engineer focused on architecting and deploying scalable Apache Spark environments. Working on Nebul’s sovereign AI cloud to enhance data performance and security.
BI Specialist focusing on data architecture and insights for RPE. Collaborating with business and technology teams to enhance data governance and analytics.
Technical Lead responsible for delivering innovative software solutions for the healthcare sector. Leading technical teams and ensuring compliance with governance standards across Australia and New Zealand.
Senior Data Engineer dedicated to designing and optimizing cloud data solutions at Leega, a consulting firm focused on innovative client service.
Data Engineer optimizing data integrations for Retail industry. Combine multiple data sources and design ETL workflows while maintaining data quality and security.
Enterprise Data Architect shaping and executing data architecture to support Energy Trust of Oregon's objectives. Collaborating on standards and governance for effective data use.
Senior Data Engineer responsible for building high - performance data pipelines for satellite analytics. Collaborating with ML Engineers and product teams to enable actionable insights from satellite data.
IT Business Solutions Analyst - Data Engineer at JTI collaborating on centralized systems and data - driven solutions while ensuring governance and compliance.