Evaluation Infrastructure

Job summary

Responsibilities

The RL infrastructure team is looking for an engineer to help develop our evaluation infrastructure.

• Build highly reliable, efficient, and easy to use infrastructure that runs all our evaluations at scale
• Collaborate closely with modeling & evaluation teams on developing new evals, maintaining evaluation signal quality, and building internal tooling to support training our…
• Identify and resolve performance bottlenecks in all layers of the eval infrastructure stack, including inference, capacity fleet management, asynchronous eval orchestration, and…

What they're looking for

• Experience in building, debugging, and optimizing efficiency of large-scale distributed systems
• Willingness to dive deep and solve hard problems at all levels of the stack
• Experience in LLM inference
• Experience in GPU compute management, workload scheduling, and dynamic resource optimization
• Experience in developing interfaces for comparing models and evals