Product Manager on Evaluators team managing dataset annotation and evaluator development pipeline for EdTech products. Collaborating with data scientists and educators, focusing on AI evaluation in educational tools.
Responsibilities
Own the annotation-to-evaluator pipeline: Manage the end-to-end process from identifying pedagogical constructs through dataset creation, model training, and evaluator validation; balance accuracy requirements with delivery timelines
Shape evaluator roadmap and prioritization: Conduct discovery with EdTech developers building literacy and content generation tools to understand which evaluators deliver the most value; provide input on prioritization based on impact, feasibility, and strategic alignment
Make strategic tradeoffs between accuracy and velocity: Partner with data scientists to set appropriate accuracy thresholds for different evaluator types; decide when to ship, iterate, or restart based on performance against baselines
Translate learning science into product specifications: Work with learning scientists to convert pedagogical frameworks for literacy instruction into clear, measurable evaluation criteria; write detailed specs that guide annotation and model development
Optimize and scale the annotation workflow: Identify opportunities to improve annotator efficiency, reduce ambiguity in guidelines, and increase inter-rater reliability; document learnings to enable expansion into new subject areas
Requirements
3-5 years of product management experience: A track record of shipping products, with direct experience working on AI/ML products, data annotation pipelines, or model evaluation workflows
Strong technical foundation in AI/ML - Comfort discussing annotation methodologies, model performance metrics, accuracy thresholds, inter-rater reliability, and evaluation frameworks with data scientists; ability to understand technical tradeoffs and make informed product decisions
Exceptional communication and storytelling skills - Ability to translate complex technical and pedagogical concepts into clear narratives for different audiences; skilled at building alignment across diverse stakeholders and making compelling cases for prioritization decisions
Self-directed execution mindset - Thrives in ambiguous, frontier environments where playbooks don't exist; takes initiative to identify problems, propose solutions, and drive outcomes; comfortable making decisions with incomplete information and learning through iteration
Customer-centric approach - Passion for understanding developer needs through direct engagement and feedback loops; commitment to building products that solve real problems; driven by mission-oriented work in education.
Benefits
Provides a generous employer match on employee 401(k) contributions to support planning for the future.
Paid time off to volunteer at an organization of your choice.
Funding for select family-forming benefits.
Relocation support for employees who need assistance moving
Salesforce Developer role requiring expertise in APEX, LWC, and Visualforce for multi - cloud transformation. Collaborate with Product Owners and Agile teams to deliver enhancements in Salesforce.
HR Systems Product Lead managing Workday HCM technologies to drive growth and improve HR processes at Gartner. Collaborating with stakeholders to enhance HR Systems functionalities and resolving issues.
Product Manager at Beach Bum driving mobile game feature development and player engagement. Overseeing the product strategy from ideation through launch in the gaming industry.
Senior Product Manager for Tutor Experience at Preply shaping tutor onboarding and retention strategies. Leading cross - functional initiatives to enhance tutor engagement and optimize product systems.
First Platform Product Manager at Preply, focusing on internal platform products to enhance developer experience and efficiency. Drive product vision, roadmap, and success metrics for core platform capabilities.
Product Owner maximizing product value and output for a global leader in real - time payments software. Collaborating with Agile teams and managing product backlog to deliver high - quality user stories in the financial sector.
CRM Product Owner managing Microsoft Dynamics CRM for a UK blood cancer charity, ensuring effective data support and leading ongoing improvements for accurate fundraising and engagement.
Product Manager supporting Entrust’s international expansion efforts within EMEA. Translating market insights into product requirements and collaborating with cross - functional teams.
Product Owner at Entrata enhancing delivery experience for property management software solutions. Collaborating with architects and development teams on product features and logistics optimization.
Head of Product Delivery managing end - to - end product delivery across Product, Engineering, Design, Data & AI at Automata. Enabling accelerative workflows and high - quality execution in lab automation.