Hybrid Senior Machine Learning Engineer

at Chalice AI

• Job title: Senior Machine Learning Engineer
• Job type: Full Time
• Experience level: Senior
• Salary: $180,000 - $200,000 per year
• Degree requirement: Postgraduate Degree
• Tech skills: AWS, Cloud, EC2, Grafana, Prometheus, PySpark, Python, PyTorch, Ray, Unity
• Location requirements: Hybrid, New York City, United States

About the role

• Senior Machine Learning Engineer developing scalable ML systems for AI applications at Chalice. Designing and deploying production ML models to impact advertising strategies and business outcomes.

Responsibilities

• Architect, train, and maintain scalable neural network systems for audience modeling and bid optimization using PyTorch and Ray distributed training (Ray Train, Ray Tune, DDP)
• Build and optimize multi-GPU training pipelines on Databricks, including hyperparameter search with ASHA scheduling and early stopping
• Develop feature engineering pipelines using PySpark, including embedding layers (EmbeddingBag, Embedding) for categorical and behavioral features
• Implement model comparison workflows with champion/challenger evaluation on holdout data
• Build resilient training and batch inference workflows with a focus on automation, reproducibility, and checkpoint recovery
• Implement robust model monitoring and observability solutions (MLflow, Prometheus, Grafana, Datadog) to track drift, performance metrics (AUC, AUPRC, F1), and system health
• Manage model versioning, experiment tracking, and artifact persistence using MLflow and Unity Catalog
• Work closely with engineering teams to integrate model outputs into production systems and optimize dataflows for fault-tolerance
• Partner with product stakeholders to align ML efforts with business impact, KPIs, and product strategy across AI Audiences, AI Allocator, CPA Algo, and Curate AI
• Lead technical design reviews, contribute to internal Python packages, and enforce engineering best practices (testing, CI/CD, modularity)
• Stay current on ML infrastructure advancements (distributed training, inference optimization, model serving patterns) and help guide adoption internally
• Document system architectures, create runbooks, and enable team members to adopt and extend the ML framework

Requirements

• Master's Degree or PhD in Computer Science, Statistics, Machine Learning, or related discipline with 5-10 years of industry experience
• Strong proficiency in PyTorch for neural network development, including custom architectures with embedding layers, MLP backbones, and binary classification heads
• Production experience with Databricks including Delta Lake, Unity Catalog, Asset Bundles, and cluster management
• Strong grasp of MLOps best practices: experiment tracking (MLflow), model versioning, model serving, monitoring, and reproducibility
• Expert-level Python and PySpark skills for data processing and feature engineering at scale
• Experience building and maintaining batch inference pipelines with schema versioning and artifact management
• Familiarity with cloud platforms (AWS: S3, EC2) and data warehousing (Snowflake)
• Experience with CI/CD workflows including build automation, testing, and packaging using GitHub Actions and Make
• Excellent collaboration and communication skills; ability to work effectively in a cross-functional environment with DS, Product, and Engineering teams.

Benefits

• Medical, Dental, and Vision coverage
• 401(k) options
• Unlimited PTO
• 11 Company Holidays
• Office-wide closure between Christmas Eve and New Year's