Site Reliability Engineer, Post Training

Job summary

Responsibilities

We're hiring a Site Reliability Engineer (SRE) to keep our post-training and reinforcement learning (RL) systems fast, reliable, and easy for researchers to iterate on. Think of this as a production engineering or site reliability role built around model training: you'll own the health of the…

• Own the reliability, performance, and uptime of large-scale post-training and RL training jobs, from launch through completion
• Partner directly with research teams during active model runs, embedding with them to unblock training and speed up iteration
• Debug failures across the full stack — accelerators, networking, storage, schedulers, and training frameworks — and drive issues to root cause
• Build monitoring, alerting, and automated recovery so runs self-heal or fail fast instead of silently stalling
• Improve checkpointing, fault tolerance, and job scheduling so hardware failures cost minutes, not days of compute

What they're looking for

• 4+ years of experience as a production engineer, site reliability engineer, or infrastructure engineer operating large-scale distributed systems in production
• Track record debugging complex failures across distributed systems — networking, hardware, kernel, or scheduler issues
• Strong software engineering skills in Python and/or Go/C++, with the judgment to know when to script a fix versus build a system
• Solid grounding in Linux systems internals and networking fundamentals
• Comfortable owning production systems, including participating in on-call rotations

Tech stack

• Python
• Go
• C++
• PyTorch
• Ray
• Kubernetes