Senior Product Operations Manager, Evaluation Quality

Job summary

Responsibilities

We’re looking for a senior operator to own the quality bar behind Harvey’s human evaluations. As we scale globally, the volume of eval work is growing 10x, but volume only matters if the output is trusted.

• Own the quality bar for Harvey’s human evaluations: define what “good” looks like for eval methodology and data analysis, and produce decision-grade outputs for EPD
• Author and maintain the evaluation guidelines, instructions, and databases that contract attorneys work from
• Standardize and streamline rubric and evaluation design into repeatable templates and one documented methodology, partnering with Applied Legal Research (ALR), who supplies…
• Own contract-attorney quality: onboarding, calibration training, inter-rater reliability, and the feedback loop (including benchmarks and gold references) that keeps judgment…
• Conducting quantitative and qualitative statistical analyses, diagnosing error states, investigating root causes, and turning raw eval results into a structured, prioritized…

What they're looking for

• 6+ years in product operations, research operations, evaluation/QA operations, or quality program management
• A track record of owning quality inputs (guidelines, instructions, benchmarks, QA procedures) for complex, expert-driven or human-in-the-loop work
• Experience onboarding, training, and calibrating a distributed pool of expert raters, annotators, or reviewers, and running the feedback loop that improves their quality over time
• Enough grounding in measurement concepts (calibration, inter-rater reliability, sampling, rubric design) to independently set up and own the quality of our evaluation loop yourself
• Experience with running quantitative and qualitative data analyses, interpreting and running statistical tests on evaluation data (natively or with AI tool support), and…