ML Infra Engineer, Compute

Location

San Francisco

On-site

Contract

Permanent

Full-time

Salary

Not disclosed

Experience

Not specified

Job summary

Responsibilities

Physical Intelligence builds general-purpose AI for the physical world. Training our models requires orchestrating thousands of accelerators across a heterogeneous fleet of GPU and TPU clusters — spanning different hardware generations, cloud providers, and cluster topologies.

  • Own Intelligent Job Scheduling and Placement: Design and build multi-tenant scheduling systems that automatically place training jobs on the best available cluster based on…
  • Scale Multi-cluster Orchestration: Build the control plane that manages the job lifecycle across diverse clusters (mixed GPU/TPU, multi-generation hardware, on-prem/cloud) and…
  • Optimize Accelerator Utilization and Efficiency: Monitor and optimize GPU/TPU utilization across the entire fleet.
  • Ensure Scaling and Stability: Implement fault detection, automatic recovery, and resilience for long-running multi-node training jobs.
  • Support Inference and Robot Deployment: Extend scheduling and orchestration to inference workloads, including deploying models to edge devices on physical robots.

What they're looking for

  • Strong software engineering fundamentals
  • Experience building or operating job scheduling / resource management systems at scale
  • Experience with large-scale compute clusters (GPU and/or TPU)
  • Familiarity with schedulers and orchestration systems (SLURM, Kubernetes, GKE, K3S, or internal equivalents)
  • Comfort reasoning about resource allocation, bin-packing, priority scheduling, and multi-tenancy

Tech stack

Read the full job posting

This summary is written by Tokn from the original posting. Only the posting published by Physical Intelligence is authoritative. Request a removal

jobs.ashbyhq.com ↗