Hybrid Founding Applied AI Engineer – Eval-Driven

at Airtree

• Job title: Founding Applied AI Engineer – Eval-Driven
• Job type: Full Time
• Experience level: Mid level, Senior
• Salary: Not specified
• Degree requirement: No Education Requirement
• Tech skills: Python
• Location requirements: Hybrid, Sydney, Australia

About the role

• Applied AI Engineer developing AI systems to prevent construction cost overruns. Blending ML engineering and data science to deliver reliable evaluation workflows in real customer settings.

Responsibilities

• Define evaluation problems: success criteria, failure modes, datasets, labelling guidelines, and score functions.
• Build and maintain an evaluation harness: regression tests, edge-case suites, and quality dashboards to prevent backsliding.
• Implement workflow systems end-to-end (data → model/LLM components → post-processing → acceptance testing) until they pass eval thresholds.
• Partner with product and domain stakeholders to translate messy real-world requirements into testable specs.

Requirements

• Strong Python skills and practical experience shipping ML/AI systems (not just experimentation).
• Demonstrated experience designing evals for ML/LLM systems (offline metrics, gold sets, error analysis, regression testing, monitoring).
• Comfort working across data science + engineering tasks: data wrangling, feature/label design, model/LLM iteration, and productionization.
• High ownership and intensity: persistence in closing the loop from “fails eval” to “passes consistently.”
• Nice to have
• Experience with document understanding (OCR, parsing, classification/extraction) and structured outputs (schemas, validators).
• Familiarity with AEC/construction workflows (design coordination, QA/compliance, BIM concepts like IFC/Revit).
• Experience building human-in-the-loop review systems and adjudication processes to improve training/eval data.