Tech Lead Manager, AI Evaluations

Job summary

Responsibilities

The bottleneck in building reliable and useful AI systems isn’t writing code anymore. It’s knowing what good looks like, applying that knowledge and experience from production usage into building rigorous evals, and using those evals to guide continuous product self-improvement loops.

• Own the AI evaluation eng roadmap and technical direction end to end: from defining what success means and building the evaluation platform to integrating results into release…
• Build scalable, reusable evaluation infrastructure for datasets, experiments, graders, trace capture and replay, result analysis, and dashboards.
• Create and maintain representative evaluation sets from expert-curated examples, appropriately governed production signals, synthetic cases, and historical failures.
• Define a balanced quality measurement system for AI products, including correctness, groundedness, retrieval and context relevance, completeness, instruction following, citation…
• Develop a fit-for-purpose mix of deterministic checks, model-based graders (including LLM-based autograders), and human evaluation.

What they're looking for

• 10+ years of experience in software engineering, ML infrastructure, data systems, or a related technical field. Experience must include building and operating production systems.
• 5+ years of engineering or applied AI team management experience, with a track record of technical leadership, developing engineers and scientists, and delivering…
• Hands-on experience designing, building, or operating evaluation systems for LLM/GenAI products, including model-based evaluators or autograders; practical understanding of how…
• Familiarity with evaluation methods and tradeoffs: curated and production-derived datasets, benchmark creation, deterministic metrics, rubric-based and pairwise judgments, human…
• Understanding of agent evaluation, including tool calling, multi-turn behavior, trajectory quality, task completion, and recovery from errors; experience with safety testing or…

Tech stack

• RAG
• Python
• Kubernetes