Senior Data Engineer responsible for designing and implementing data solutions at Harambee. Collaborating with various stakeholders to enhance technology supporting work - seekers' journeys.
About the role
- AI Systems & Data Engineer at HyperFi designing Databricks pipelines and managing AI systems. Collaborating on data engineering tasks and optimizing workflows in a flexible tech environment.
Responsibilities
- Design and operate Databricks pipelines in Python to ingest and normalize large-scale unstructured data
- Build streaming and batch ingestion using Auto Loader, Delta Live Tables, and Workflows
- Model and maintain AI-ready lakehouse tables with Delta Lake and Unity Catalog
- Prepare retrieval and context datasets for RAG and agent systems
- Orchestrate Temporal-based workflows to coordinate data prep, validation, and AI handoff
- Enforce data quality, lineage, and access controls across pipelines
- Optimize PySpark jobs for performance, reliability, and cost
- Integrate pipeline outputs into production AI systems and APIs
- Monitor freshness, schema drift, and pipeline health
Requirements
- 5-7 years of experience building production-grade ML, data, or AI systems.
- Strong grasp of prompt engineering, context construction, and retrieval design.
- Comfortable working in LangChain and building agents.
- Experience with PySpark and Databricks to handle real-world data scale.
- Ability to write testable, maintainable Python with clear structure.
- Understanding of model evaluation, observability, and feedback loops.
- Excited to push from prototype → production → iteration.
- Familiarity with Databricks Data Intelligence Platform which unifies data warehousing and AI use cases on a single platform.
- Knowledge of Unity Catalog for open and unified governance of data, analytics, and AI on the lakehouse.
- Understanding of data security concerns related to AI and how to mitigate them using the Databricks AI Security Framework (DASF).
- Confident English skills to collaborate clearly and effectively with teammates
Benefits
- Flexible hours
- Async-friendly culture
- Engineering-led environment
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Senior Manager Data Engineer at Squarcle delivering technical leadership in data engineering and compliance with business objectives. Leading teams to optimize and develop data platforms for clients.
Senior Consultant Data Engineer in a consultancy firm focusing on data engineering and platform development. Collaborating with diverse teams to deliver high - quality data solutions.
Data Engineer at Mobileye building robust data pipelines for data infrastructure. Collaborating with teams to deliver high - quality data solutions for dynamic environments.
Cloud Data Engineer at Shift, focusing on building and operating data pipelines on Azure for Australian SMEs. Collaborating across teams to enhance data integration and quality.
Engenheiro de Dados especializado em PowerBI e Lakehouse no Campos Thomaz Advogados. Foco em preparar dados para dashboards e estruturação de ambientes com Microsoft.
Data Platform Expert developing and maintaining data solutions for analysis and reporting at Magna Electronics. Collaborating with various teams to enhance data - driven decision making and insights.
Data Engineer at Mobiz designing, building, and maintaining scalable data solutions for analytics. Collaborating with teams to leverage modern cloud technologies and improve data - driven decision - making.
Head of Data Engineering at Envitia overseeing data architecture services for public sector programs. Leading service mobilization with client stakeholders in a hybrid work environment.
Applied AI Health Data Architect - Senior Manager at PwC designing data architecture for healthcare operations. Contributing to innovative data solutions and mentoring teams for operational excellence.