Data Scientist working on enhancing AI agent performance metrics and experiments. Analyzing data and collaborating with cross-functional teams to drive product improvements.
Responsibilities
Design and analyze experiments to measure agent improvements—from model changes to UX variations—with statistical rigor and practical tradeoffs.
Define success metrics that connect agent trace data (prompts, responses, code changes, execution outcomes) to user outcomes like successful deploys, retention, and revenue.
Build the semantic layer for agent data in partnership with data engineering—defining the tables, metrics, and models that enable self-serve analysis across the AI team.
Surface insights from trace analysis that identify failure modes, successful patterns, and opportunities to improve agent effectiveness.
Partner with AI engineering, product, and leadership to translate data into roadmap decisions; you'll have a seat at the table for critical agent strategy discussions.
Create dashboards and reporting that surface agent performance metrics (task completion, latency, quality scores, user satisfaction) for the AI team and executives.
Requirements
5+ years of experience in data science, analytics, or a quantitative role with a focus on product, growth, or experimentation.
Deep experimentation expertise: A/B testing, experiment design, power analysis, handling skewed data, interpreting results beyond p-values.
Strong SQL skills and experience designing data models for high-volume event data; experience with dbt or similar transformation tools.
Proficiency in Python and data science libraries (pandas, scipy, statsmodels, etc.).
Ability to translate ambiguous questions into structured analysis and communicate findings clearly to both technical and non-technical stakeholders.
Bias toward action: you ship insights that influence decisions, not just dashboards.
Data Scientist helping Qliro develop payment solutions through machine - learning in credit and fraud domains. Collaborating in a modern data platform environment to enhance decision - making capabilities.
Data Scientist at Capital One using ML to revolutionize customer engagements through personalization. Collaborate with elite teams to build cutting - edge recommendation systems.
AI Agent Developer responsible for designing and building AI agents to operate within digital environments. Collaborating with teams to deliver innovative AI solutions and ensure functionality within complex systems.
AI Agent Developer designing and implementing autonomous intelligent agents using AI frameworks. Collaborating across teams to optimize agent capabilities and performance.
Data Science/Analyst Intern working with risk analytics and data science teams at Credibly. Engaging in projects that leverage advanced analytics and statistical techniques for business growth.
Senior Data Scientist leading data science initiatives impacting global operations across various business units. Collaborating with cross - functional teams to architect scalable machine learning solutions in cloud environments.
Lead Data Scientist building GenAI - driven products and ML solutions for S&P Global. Mentoring team members and delivering high - impact projects in a global environment.
Data Scientist in Digital Transformation team at Lavazza implementing machine learning models and managing model lifecycle within Agile squads. Focusing on solving real business problems.
Data Scientist enhancing Sicredi's data pipeline and generating actionable insights across credit sectors. Collaborating closely with data engineers and analysts to improve decision - making.
Data Scientist role at Airbus developing AI/Data Science solutions for aircraft systems. Focused on value creation and collaboration with engineering teams for embedded computer vision applications.