Hybrid Principal Engineer

at Palm Ventures

• Job title: Principal Engineer
• Job type: Full Time
• Experience level: Lead
• Salary: Not specified
• Degree requirement: Bachelor's Degree
• Tech skills: GraphQL, Kubernetes, Python, SDLC, Spark
• Location requirements: Hybrid, Redwood City, United States

About the role

• Principal Engineer at Gentoro defining standards for AI agent enterprise integration in a growth phase. Bridging high-level architecture and hands-on execution in a hybrid work environment.

Responsibilities

• Design and implement multi-agent systems and orchestration layers.
• Build and operate observability stacks (e.g., OpenTelemetry) to monitor agent reasoning paths, tool usage, and performance in real-time.
• Develop and enforce technical safety mechanisms—such as input/output filtering and behavioral boundaries—to mitigate risks like hallucinations, prompt injections, and bias.
• Analyze telemetry and execution traces to create feedback loops for continuous agent improvement and automated evaluation.
• Securely connect agents to external services, unstructured data, and enterprise APIs via robust tool-calling schemas.
• Implement fallback mechanisms, human-in-the-loop (HITL) checkpoints, and automated recovery for agentic failures.
• Implement best practices for MLOps, monitoring, and performance tuning of AI models in live environments
• Automate SDLC processes and CI/CD pipelines, elevate QA standards, and develop incident response protocols to enable high velocity, availability and reliability of our platform

Requirements

• 10+ years of senior engineering experience at a fast-paced, high-growth technology startup that has successfully scaled from early stage through Series A/B funding (or equivalent growth phase)
• 5+ years of ML, including 2+ years focused on LLMs or agentic workflows.
• Proficiency in agent orchestration and memory-augmented systems.
• Hands-on experience analyzing tracing and logging data.
• Experience using feedback loops to continuously improve ML systems
• Built agents that invoked tools or utilized Model Context Protocol (MCP) to access enterprise data sources
• Proficiency in modern technologies (e.g., Python, semantic search, vector DBs, GraphQL, queues, containers, Kubernetes, real-time data processing, Spark, Open Telemetry, Clickhouse)
• Thrives in startup ambiguity while maintaining the discipline of an enterprise-grade engineer
• Acts as a force multiplier who elevates the technical bar for the entire team
• Obsessed with practical application of AI systems and capable of building enterprise solutions that solve real-world customer problems