Research Scientist, APEX Benchmarks

Job summary

Responsibilities

We’re looking for a Research Scientist to lead the design of the next generation of APEX benchmarks and the expert-built datasets behind them.

• Benchmark design: Decide what the next APEX benchmark should measure, based on frontier model performance, saturation of existing evals, and where economically valuable work is…
• Dataset design: Design expert-built datasets and grading rubrics at scale — deciding what makes a task hard, what makes a grade defensible, and how to hold quality while…
• Measurement rigor: Set the standard for how we report results: confidence intervals, inter-rater agreement, human vs.
• Partnerships: Work with academic collaborators and industry partners (as we did with Cognition on APEX-SWE) to co-design benchmarks and get them adopted.
• External research voice: Publish — arXiv papers, open datasets, blog posts, conference talks, leaderboard releases — and represent Mercor’s research in conversations with…

What they're looking for

• Research background in evaluation: Strong applied or academic research background in LLM evaluation, benchmarking, NLP or a related field, with a track record of rigorous…
• Judgment about what to measure: You can look at a frontier model’s behavior and identify the measurement that would actually be informative, rather than the one that is easiest…
• Statistical rigor: You reason carefully about sampling, variance, contamination and grader reliability, and you don’t ship a number you can’t defend.
• Hands-on: Strong coding skills. You can build an eval harness, run the experiment and analyze the results yourself.
• Exceptional communication: You can present complex technical findings clearly to frontier lab researchers and to non-technical audiences, in writing and in person.