Member of Technical Staff, Training Performance Engineer

Job summary

Responsibilities

As a Performance Engineer in the Pre-Training team you will be responsible for optimizing the performance of our advanced language models and systems. Their primary focus is on improving key model training metrics, such as training throughput, ensuring high accelerator utilization.

• Design and write high-performant and scalable software for training.
• Understand architectural modifications and design choices and their effects on training throughput and quality.
• Write low-level CUDA, triton kernels to squeeze every last bit of performance from our accelerators.
• Research, implement, and experiment with ideas on our supercompute and data infrastructure.
• Learn from and work with the best researchers in the field.

What they're looking for

• Extremely strong software engineering skills.
• Proficiency in Python and related ML frameworks such as JAX, Pytorch and XLA/MLIR.
• Experience writing kernels for GPUs using CUDA, triton, etc
• Experience using large-scale distributed training strategies.
• Familiarity with autoregressive sequence models, such as Transformers.

Tech stack

• CUDA
• Triton
• Python
• JAX
• PyTorch
• Transformers