AI Engineer Lead at Orica managing and optimizing AI models and datasets while mentoring engineers. Bridging R&D, Engineering, and Data Platforms in a collaborative environment.
About the role
- AI Engineer developing evaluation frameworks and observability for LLMs at Mistral AI. Collaborating with product teams to enhance AI models and ensure quality improvements.
Responsibilities
- Build and maintain an LLM evaluation framework (reference tests, heuristics, model-graded checks).
- Define and track metrics: task success, helpfulness, hallucination proxies, safety flags, latency/cost.
- Run A/B tests for prompts, models, and system prompts, analyze results, recommend rollout or rollback.
- Set up observability for LLM calls: structured logging, tracing, dashboards, alerts.
- Operate the model release: canary and shadow traffic, sign-offs, SLO-based rollback criteria, regression detection.
- Improve core behaviors: memory write/retrieve policies and evals, intent classification, follow-ups, routing, tool-call reliability.
- Create templates and docs so other teams can author evals and ship safely.
- Partner with Science, diagnose regressions and lead post-mortems.
Requirements
- Strong TypeScript or Python skills
- Production LLM experience: prompts, tool/function calling, and system prompts.
- Hands-on with evals and A/B testing, you can design metrics and make rollout decisions from data.
- Observability: logging, tracing, dashboards, alerting
- Product mindset: form hypotheses, run experiments, interpret results, iterate.
- Clear written and spoken communication, autonomous; and product-oriented.
- Now it would be ideal if you have experience with
- Safety systems: moderation, PII handling/redaction, guardrails.
- Release operations: canary/shadowing, automated rollbacks, experiment platforms.
Benefits
- 💰 Competitive salary and equity
- 🧑⚕️ Health insurance
- 🚴 Transportation allowance
- 🥎 Sport allowance
- 🥕 Meal vouchers
- 💰 Private pension plan
- 🍼 Parental : Generous parental leave policy
- 🌎 Visa sponsorship
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
AVP - Cloud & GenAI Lead & Architect driving enterprise cloud strategy and leading AI initiatives. Collaborating on AI programs and defining cloud architecture best practices in a leadership role.
Engineering intern contributing to technical projects in the music tech industry. Building production - level systems and gaining exposure to business and industry outcomes.
AI/ML Engineer shaping the future of legal AI at Ironclad, using cutting - edge tools for contract management. Collaborating with product teams to enhance AI capabilities and user experiences.
Lead the AI vision for AEC Construction as a Distinguished AI Architect at Autodesk. Shape how AI transforms construction towards intelligent and autonomous systems.
Senior Developer crafting AI - powered banking solutions with React and backend services for FNBO. Join the GenAI team to innovate in financial services while collaborating with cross - functional teams.
Director of AI Engineering developing AI systems for drug discovery at Pfizer. Collaborating with scientists to leverage AI in transforming R&D through innovative medicines and therapies.
AI Engineer designing and deploying AI/ML systems at Orion. Close mentorship and real - world production experience in a collaborative environment.
Senior AI Developer at a tech firm designing AI - enhanced automation solutions for business processes. Leading the development of tools integrating SAP and cloud data sources.
Director - Automation and AI Architect overseeing centralized data hub enabling collaboration for analytics and AI at Novartis.