Member of Technical Staff, Enterprise Evals Platform

Job summary

Responsibilities

Enterprise agents are complex systems, and they only pay off when their work is reliable and economically viable. Evaluation is how you get both: checking correctness is the obvious case, and routing is the subtler one, since choosing a model against cost, latency, and quality requires quality to…

• Define golden sets: decompose real tasks and encode the expert quality bar.
• Build verifiers over agent trajectories and outputs, calibrated and hard to game.
• Build the eval platform that runs offline environments, task suites, and grading at scale.
• Run loss analysis over production trajectories and turn failure modes into regression tests.
• Run the optimization loop across models, prompts, skills, and harnesses.

What they're looking for

• Professional, academic, or research experience in agent engineering and evaluation, including how agent runtimes and harnesses produce a trajectory and where it fails.
• Bachelor’s Degree in Computer Science or a related technical degree such as Electrical Engineering, Physics, Mathematics, Computer Engineering, Information Systems or Statistics
• 1–5 years in industry level software engineering
• Experience building evaluation suites for LLM or agent systems, and familiarity with how benchmarks such as terminal-bench, tau-bench, and APEX are constructed and where they get…
• Judgment about task and rubric design: turning a fuzzy notion of quality into something measurable, with agent or model improvements to show for it.