Associate Data Engineer building and maintaining data systems at Incedo. Transforming raw information into accessible datasets for decision - making and collaborating with analysts and data scientists.
About the role
- Senior Data Engineer designing and maintaining cloud data solutions using Spark and Databricks. Collaborating with teams and clients to optimize data infrastructure and insights delivery.
Responsibilities
- Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and Databricks.
- Gather and analyze data requirements from business stakeholders and identify opportunities for data-driven insights.
- Build and optimize data pipelines for data ingestion, processing, and integration using Spark and Databricks.
- Ensure data quality, integrity, and security throughout all stages of the data lifecycle.
- Collaborate with cross-functional teams to design and implement data models, schemas, and storage solutions.
- Optimize data processing and analytics performance by tuning Spark jobs and leveraging Databricks features.
- Provide technical guidance and expertise to junior data engineers and developers.
- Stay up-to-date with emerging trends and technologies in cloud computing, big data, and data engineering.
- Contribute to the continuous improvement of data engineering processes, tools, and best practices.
Requirements
- Bachelor's or Master's degree in computer science, engineering, or a related field.
- 10+ years of experience as a Data Engineer, Software Engineer, or similar role, with a focus on building cloud-based data solutions.
- Strong experience with cloud platforms such as Azure or AWS.
- Proficiency in Apache Spark and Databricks for large-scale data processing and analytics.
- Experience in designing and implementing data processing pipelines using Spark and Databricks.
- Strong knowledge of SQL and experience with relational and NoSQL databases.
- Experience with data integration and ETL processes using tools like Apache Airflow or cloud-native orchestration services.
- Good understanding of data modeling and schema design principles.
- Experience with data governance and compliance frameworks.
- Excellent problem-solving and troubleshooting skills.
- Strong communication and collaboration skills to work effectively in a cross-functional team.
- Relevant certifications in cloud platforms, Spark, or Databricks are a plus.
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Intern Data Engineer at Flutter Studios learning to create applications in the gaming industry. Join a cross - functional team to implement a basic application in an Agile environment.
AI/Data Engineer at Comcast developing data pipelines and AI solutions for audit processes. Leading team efforts to ensure data quality and compliance with audit objectives across business units.
Senior Data Engineer involved in AWS Cloud and Big Data solutions for Financial Crime. Join CommBank's team to tackle complex data - centric problems with advanced technologies.
Senior Data Engineer leading and mentoring a team in building scalable data pipelines for digital transformation projects in an international software company.
Senior Data Operations Engineer at iKnowHow designing and implementing scalable data - driven applications. Focus on data pipelines, APIs, and collaboration across teams for project success.
Data/AI Engineer supporting innovations in corporate travel with AWS technologies at HRS Group. Collaborating with data teams to develop AI solutions and maintain data pipelines.
AS
Data Engineer developing data solutions for reporting and analytics at Assurant. Implementing and optimizing Data Warehouse solutions in cloud and on - premise environments with Agile methodology.
AI Engineer developing AI - driven analytics for trading at Deloitte. Focusing on scalable data pipelines and collaboration with traders for actionable insights.
Big Data Engineer developing and optimizing data systems at ESET. Collaborating with cross - functional teams to handle and transform vast datasets.