Hybrid Software Engineer – Model API's

at Baseten

• Job title: Software Engineer – Model API's
• Job type: Full Time
• Experience level: Mid level, Senior
• Salary: $150,000 - $230,000 per year
• Degree requirement: Bachelor's Degree
• Tech skills: Distributed Systems
• Location requirements: Hybrid, San Francisco, United States

About the role

• Software Engineer focused on Model APIs, ensuring performance for AI models at Baseten. Collaborating with product and infrastructure teams to streamline developer interaction with AI models.

Responsibilities

• Design, build, and operate the Model APIs surface with focus on advanced inference capabilities: structured outputs (JSON mode, grammar-constrained generation), tool/function calling and multi-modal serving
• Profile and optimize TensorRT-LLM kernels, analyze CUDA kernel performance, implement custom CUDA operators, tune memory allocation patterns for maximum throughput and optimize communication patterns across multi-GPU setups
• Productionize performance improvements across runtimes with deep understanding of their internals: speculative decoding implementations, guided generation for structured outputs, custom scheduling and routing algorithms for high-performance serving
• Build comprehensive benchmarking frameworks that measure real-world performance across different model architectures, batch sizes, sequence lengths, and hardware configurations
• Productionize performance improvements across runtimes (e.g.TensorRT, TensorRT‑LLM): speculative decoding, quantization, batching, and KV‑cache reuse.
• Instrument deep observability (metrics, traces, logs) and build repeatable benchmarks to measure speed, reliability, and quality.
• Implement platform fundamentals: API versioning, validation, usage metering, quotas, and authentication.
• Collaborate closely with other teams to deliver robust, developer‑friendly model serving experiences.

Requirements

• 3+ years experience building and operating distributed systems or large‑scale APIs.
• Proven track record of owning low‑latency, reliable backend services (rate‑limiting, auth, quotas, metering, migrations).
• Infra instincts with performance sensibilities: profiling, tracing, capacity planning, and SLO management.
• Comfortable debugging complex systems, from runtime internals to GPU execution traces.
• Strong written communication; able to produce clear design docs and collaborate across functions.

Benefits

• Competitive compensation package.
• This is a unique opportunity to be part of a rapidly growing startup in one of the most exciting engineering fields of our era.
• An inclusive and supportive work culture that fosters learning and growth.
• Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.