Data Engineer creating data pipelines in Databricks for a fast-growing digital banking platform. Responsible for ensuring data quality and optimising processes to support decision-making.
Responsibilities
Design, develop, test, deploy and monitor data pipelines in Databricks on AWS from a wide variety of data sources.
Design, develop, test, deploy and monitor scalable code with PySpark and SQL in Databricks.
Identify opportunities to improve internal process through code optimisation and automation.
Build data quality dashboards, lineage flows / and or monitoring tools to utilize the data pipeline, providing active monitoring and actionable insight into overall data quality and data governance.
Assist in migrating data from legacy systems onto newly developed solutions.
Follow and lead best practices on all data security, retention, and privacy policies.
Requirements
Bachelor’s degree.
** 3+ years’ experience of building ETL/ELT pipelines.**
Proven competency in solution design, development, implementation, reporting and analysis.
Proficiency in **Apache-Spark, Python and SQL languages**.
Proficiency in working with **Text, Delta, Parquet, JSON, CSV, and XML data formats.**
Working knowledge of Spark structured streaming.
**AWS infrastructure experience, specifically working with S3.**
**Solid understanding of git-based version control, DevOps, and CI/CD. Experience of working on Atlassian stack a plus.**
**Knowledge of common web API frameworks and web services.**
Strong teamwork, relationship, and client management skills, and the ability to influence peers and senior management to accomplish team goals.
Willingness to embrace modern technology, best practice, and ways of work.
Data Platform Expert developing and maintaining data solutions for analysis and reporting at Magna Electronics. Collaborating with various teams to enhance data - driven decision making and insights.
Data Engineer at Mobiz designing, building, and maintaining scalable data solutions for analytics. Collaborating with teams to leverage modern cloud technologies and improve data - driven decision - making.
Head of Data Engineering at Envitia overseeing data architecture services for public sector programs. Leading service mobilization with client stakeholders in a hybrid work environment.
Applied AI Health Data Architect - Senior Manager at PwC designing data architecture for healthcare operations. Contributing to innovative data solutions and mentoring teams for operational excellence.
CRM Functional Data Migration Engineer at Data - Core Systems, Inc. managing Dynamics 365 CE solutions and delivering technical expertise in a consulting role.
Data Engineer role for Alberta Securities Commission, focusing on data pipelines and infrastructure. Involves designing, building, and securing data assets for organizational decision - making.
Data Architect supporting UK MOD programme delivering data and integration capabilities for Defence. Join Amentum's growing IT team focused on critical national infrastructure and security projects.
Data Engineer joining Contour Software's development team to handle various big data projects. Focus on cloud data lake architecture and machine learning implementation.
Enterprise Architect Sr position at PNC collaborating with technology stakeholders to build enterprise data architecture. Focus on data models, governance, and guiding data decisions for business strategy.
Data Engineer architecting and maintaining data products using Databricks and Power BI for Nitro's digital transformation efforts. Collaborating to optimize processes and leverage cloud technologies.