Training Performance Engineer

Job summary

Responsibilities

As a Training Performance Engineer, you’ll drive efficiency improvements across our distributed training stack. You’ll analyze large-scale training runs, identify utilization gaps, and design optimizations that push the boundaries of throughput and uptime.

• Profile end-to-end training runs to identify performance bottlenecks across compute, communication, and storage.
• Optimize GPU utilization and throughput for large-scale distributed model training.
• Collaborate with runtime and systems engineers to improve kernel efficiency, scheduling, and collective communication performance.
• Implement model graph transforms to improve end to end throughput.
• Build tooling to monitor and visualize MFU, throughput, and uptime across clusters.

Tech stack

• Python
• C++
• Rust
• CUDA
• PyTorch
• JAX
• TensorFlow