Hybrid Data Scientist – AI Agent

Posted last month

Apply now

About the role

  • Data Scientist working on enhancing AI agent performance metrics and experiments. Analyzing data and collaborating with cross-functional teams to drive product improvements.

Responsibilities

  • Design and analyze experiments to measure agent improvements—from model changes to UX variations—with statistical rigor and practical tradeoffs.
  • Define success metrics that connect agent trace data (prompts, responses, code changes, execution outcomes) to user outcomes like successful deploys, retention, and revenue.
  • Build the semantic layer for agent data in partnership with data engineering—defining the tables, metrics, and models that enable self-serve analysis across the AI team.
  • Surface insights from trace analysis that identify failure modes, successful patterns, and opportunities to improve agent effectiveness.
  • Partner with AI engineering, product, and leadership to translate data into roadmap decisions; you'll have a seat at the table for critical agent strategy discussions.
  • Create dashboards and reporting that surface agent performance metrics (task completion, latency, quality scores, user satisfaction) for the AI team and executives.

Requirements

  • 5+ years of experience in data science, analytics, or a quantitative role with a focus on product, growth, or experimentation.
  • Deep experimentation expertise: A/B testing, experiment design, power analysis, handling skewed data, interpreting results beyond p-values.
  • Strong SQL skills and experience designing data models for high-volume event data; experience with dbt or similar transformation tools.
  • Proficiency in Python and data science libraries (pandas, scipy, statsmodels, etc.).
  • Ability to translate ambiguous questions into structured analysis and communicate findings clearly to both technical and non-technical stakeholders.
  • Bias toward action: you ship insights that influence decisions, not just dashboards.

Benefits

  • 💰 Competitive Salary & Equity
  • 💹 401(k) Program
  • ⚕️ Health, Dental, Vision and Life Insurance
  • 🩼 Short Term and Long Term Disability
  • 🚼 Paid Parental, Medical, Caregiver Leave
  • 🚗 Commuter Benefits
  • 📱 Monthly Wellness Stipend
  • 🧑‍💻 Autonoumous Work Environement
  • 🖥 In Office Set-Up Reimbursement
  • 🏝 Flexible Time Off (FTO) + Holidays
  • 🚀 Quarterly Team Gatherings
  • ☕ In Office Amenities

Job title

Data Scientist – AI Agent

Job type

Experience level

Mid levelSenior

Salary

$140,000 - $280,000 per year

Degree requirement

Bachelor's Degree

Tech skills

Location requirements

Report this job

See something inaccurate? Let us know and we'll update the listing.

Report job