AI Performance Engineer

Job summary

Responsibilities

We are looking for a performance engineer who specializes in making large-scale machine learning workloads fast and cost-efficient in the datacenter. This role is focused on distributed training runs spanning many nodes, and high-throughput batch inference sweeping petabytes of real-world autonomy…

• Profile and optimize distributed training end to end - data loading and preprocessing, augmentation, kernel execution, gradient communication, and checkpointing
• Optimize large-scale offline and batch inference over petabyte-scale sensor logs: batching and scheduling strategies, quantization and low-precision execution, graph…
• Establish roofline and performance models for our workloads, quantify the gap between achieved and theoretical performance, and stack-rank optimization opportunities by impact…
• Improve multi-node scaling efficiency: sharding and parallelism strategies, collective communication, interconnect utilization, and memory-bandwidth and kernel-fusion bottlenecks
• Drive cluster goodput - reduce GPU idle time from input pipeline stalls, storage and network I/O, scheduling gaps, stragglers, and failure recovery on long-running jobs

What they're looking for

• Bachelor's degree or higher in computer science or adjacent fields.
• 1 or more years of hands-on experience with ML performance engineering: profiling, roofline analysis, throughput optimization, and root-cause investigation in production systems.
• 5 or more years of overall experience
• Experience with distributed multi-node training at scale (FSDP, DeepSpeed, Megatron, NCCL, or equivalent), including diagnosing scaling inefficiency as node count grows
• Deep familiarity with GPU or accelerator performance concepts - memory bandwidth, kernel launch overhead, occupancy, quantization, collective communication

Tech stack

• Go
• Triton
• Ray
• Python
• C++
• CUDA
• PyTorch
• Kubernetes