Member of Technical Staff, ML Platform

Job summary

Responsibilities

We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals.

• Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale
• Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative
• Work directly with research teams on video, image, audio, agents, and robotics to understand what they need to measure and build it, then generalize the result into the platform
• Help set the standards for how we evaluate models at Runway: reproducibility, metric definitions, reporting formats, and when a result is trustworthy enough to act on
• Support broad adoption of the platform and its integration throughout Runway’s research efforts across training, production model serving and more

What they're looking for

• 5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or…
• Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes
• Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
• Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
• Comfort building internal tools end to end, from the command line to the browser

Tech stack

• Python
• PyTorch
• Kubernetes