HPC Infrastructure Engineer - GPU Clusters

Job summary

Responsibilities

Every model we train runs on infrastructure this role owns. We operate NVIDIA GPU clusters across bare metal and rented capacity, and we're looking for an engineer to join our small research infrastructure team and make that compute fast, reliable, and boring - in the best sense.

• Operate and improve our GPU fleet end to end: provisioning, scheduling, monitoring, upgrades, capacity planning
• Build automation that keeps the fleet healthy without human intervention — node health checks, automated draining and remediation, burn-in pipelines for new capacity
• Own the stack beneath the training code: OS images, NVIDIA drivers, CUDA, container runtimes, NCCL, high-speed networking (InfiniBand/RoCE)
• Run and tune job scheduling (Slurm or similar) so researchers get compute fairly and fast
• Build and maintain high-performance storage for datasets and checkpoints

What they're looking for

• Have run large-scale Linux server or GPU environments in production and enjoy both building and operating
• Know the NVIDIA stack well — drivers, CUDA, NCCL, DCGM — or have deep systems experience and learn hardware stacks fast
• Are comfortable with bare-metal environments, server hardware, and high-speed networking
• Write solid automation in Python and/or Bash, with IaC tools like Ansible or Terraform
• Are happy digging into noisy data (metrics, logs, PromQL) to find what's actually wrong

Tech stack

• CUDA
• Python
• Terraform