Hybrid AI Engineer – AI System Calibration, Optimization

at Gugu Robotics

• Job title: AI Engineer – AI System Calibration, Optimization
• Job type: Full Time
• Experience level: Senior, Lead
• Salary: Not specified
• Degree requirement: Bachelor's Degree
• Tech skills: Azure, Cloud, Distributed Systems, Python, SDLC, Terraform
• Location requirements: Hybrid, Calgary, Canada

About the role

• AI Engineer specializing in AI system calibration and optimization for clients. Working directly with engineering teams to improve AI models and workflows in Azure environment.

Responsibilities

• Embed with strategic client as their technical partner for AI system calibration and prompt optimization.
• Build production-grade calibration systems using Python within the client's Azure environment.
• Implement DSPy framework and GEPA optimizer to systematically improve prompt quality and retrieval performance.
• Design and develop Golden Dataset curation workflows using Azure Data Labeling, establishing gold/silver data tier schemas.
• Create evaluation frameworks to measure model accuracy, precision/recall, latency, and hallucination rates.
• Architect prompt optimization pipelines for retrieval, context synthesis, and answer generation tailored to client needs.
• Own the path to production - evaluation pipelines, Azure ML workflows, KPI dashboards, and optimization automation.
• Iterate rapidly based on client feedback and KPI results, translate business goals into technical calibration improvements.
• Own end-to-end delivery of calibration systems from initial baseline to production-ready optimization workflows.
• Establish measurable KPIs and demonstrate accuracy improvements, latency reduction, and hallucination mitigation.
• Provide strategic guidance on RAG architecture improvements and retrieval parameter optimization.
• Accelerate client time-to-value through hands-on development and comprehensive knowledge transfer.
• Deliver operational playbooks and documentation enabling the client team to maintain calibration systems independently.
• Lead complex, multi-stakeholder calibration initiatives on-site and remotely; drive clarity, remove blockers, and keep execution on track.
• Set coding standards and architectural patterns for calibration components; write clear docs, runbooks, and technical specifications.
• Mentor client engineers through code reviews, pairing sessions, and technical workshops on DSPy, GEPA, and evaluation best practices.
• Make sound tradeoffs under real-world constraints - Azure cost optimization, data quality, performance requirements, and security.
• Align delivery with Robots & Pencils' responsible AI practices and client governance requirements.
• Work closely with client's AI SMEs and product engineering teams to understand product catalog structure and validation workflows.
• Collaborate with internal R&P product, engineering, and delivery teams on calibration methodology and best practices.
• Share insights from client engagement to improve R&P's prompt optimization frameworks and tooling.
• Contribute reusable patterns, evaluation frameworks, and documentation back to R&P's core platform.
• Collaborate across time zones with distributed teams.

Requirements

• Bachelor's degree in computer science, Engineering, or equivalent experience.
• 7+ years of professional software development with significant ownership of architecture and delivery.
• 3+ years of Python in ML/AI systems with a strong focus on data processing and evaluation pipelines.
• 2+ years building with Generative AI including hands-on prompt engineering and optimization work.
• Experience with prompt optimization frameworks - DSPy strongly preferred, or similar systematic approaches to prompt improvement.
• Deep understanding of RAG architectures - retrieval quality, latency/cost tuning, hallucination mitigation, and evaluation methods.
• Hands-on experience designing evaluation metrics and building assessment frameworks for LLM systems.
• Knowledge of systematic experimentation methods - A/B testing, parameter tuning, performance benchmarking.
• Experience with data curation, labeling workflows, and dataset quality management for AI systems.
• Strong Azure cloud experience with focus on AI/ML services - Azure Machine Learning, Azure AI Search, Azure OpenAI Service.
• Experience with Azure Data Labeling, Azure Blob Storage, and Azure infrastructure fundamentals.
• Understanding vector search platforms and retrieval optimization (Azure AI Search, Weaviate, Qdrant, Pinecone).
• Strong IaC background (Terraform or ARM templates) plus containerization and distributed systems knowledge.
• Solid SDLC practices - testing strategies, CI/CD, code reviews, observability, and operational excellence.
• Upper-intermediate English for client communication.
• Experience leading complex technical projects with multiple stakeholders.
• Strong communication skills for technical and executive audiences.
• Ability to context-switch and adapt to client environments.
• Willingness to travel to client sites.

Benefits

• Remote Friendly Office
• Professional Development Opportunities