Architect leading initiatives in data platform engineering at F5, solving challenges in unstructured data and enhancing governance frameworks.
EX
Hybrid Lead Assistant Manager – Data Engineering, Cloud Data Engineering
Posted 6 months ago
About the role
- Lead Assistant Manager overseeing Data Engineering projects with Databricks & Azure Cloud. Managing data pipelines, ensuring data integrity, and supporting insurance business operations.
Responsibilities
- Develop and maintain scalable and efficient data pipelines to ensure seamless data flow across various systems adddress any issues or bottlenecks in existing pipelines.
- Implement robust data checks to ensure the accuracy and integrity of data. Summarize and validate large datasets to ensure they meet quality standards.
- Monitor data jobs for successful completion. Troubleshoot and resolve any issues that arise to minimize downtime and ensure continuity of data processes.
- Regularly review and audit data processes and pipelines to ensure compliance with internal standards and regulatory requirements
- Familiar with working on Agile methodologies - scrum, sprint planning, backlog refinement etc.
Requirements
- 4-9 years experience on Data Engineering role working with Databricks & Azure Cloud technologies.
- Bachelor’s degree in computer science, Information Technology, or related field.
- Strong proficiency in PySpark, Python, SQL.
- Strong experience in data modeling, ETL/ELT pipeline development, and automation
- Proficient in working on Azure cloud components Azure Data Factory, Azure DataBricks, Azure Data Lake etc.
- Experience with data modeling, ETL processes, Delta Lake and data warehousing.
- Experience on Delta Live Tables, Autoloader & Unity Catalog.
- Preferred - Knowledge of the insurance industry and its data requirements.
- Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
- Excellent problem-solving skills and ability to work under tight deadlines.
- Hands-on experience with performance tuning of data pipelines and workflows
- Excellent communication and problem-solving skills to work effectively with diverse teams
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Data Architect designing and implementing data architectures supporting analytics and ML for federal clients. Collaborating with teams to translate mission needs into robust data solutions.
IT Data Engineer developing data pipelines and integrations for Scanfil Group's global IT organization. Collaborating across teams to enhance data solutions and reporting capabilities.
Data Engineer developing Azure data solutions at PwC New Zealand. Responsibilities include data quality monitoring, pipeline development, and collaboration with stakeholders in a supportive environment.
Senior Data Engineer designing and implementing the Enterprise Data Platform at Stellix. Focusing on analytics and insights with a growth path to Principal Data Engineer or Data Architect.
R&D Data Engineer at DXC, transforming complex data into digital assets for global analytics and Smart Lab solutions. Collaborating on ELN and LIMS tools for enhanced data management.
Data Engineer role focusing on data pipelines and processing at 42dot, a mobility AI company. Responsibilities include data collection, schema management, and pipeline monitoring.
Senior Data Engineer at mobility AI company designing large - scale data processing pipelines. Leading technical decisions and mentoring junior engineers in data architecture.
Senior Data Engineer at Booz Allen building advanced tech solutions for mission - driven projects. Utilizing data engineering activities, pipelines, and platforms for impactful data insights.
Senior Software Engineer contributing to Workday's AI/MLOps cloud ops platform. Involves data ingestion, computation, and generation of curated data sets with modern technologies.