Research Engineer, Latency & Throughput

Job summary

Responsibilities

As a Research Engineer on the Latency & Throughput team, you will push the frontier of LLM inference by researching and prototyping techniques that reduce latency and increase throughput — from speculative decoding and quantization to KV compression and new attention variants.

• Research and prototype inference techniques — speculative decoding, quantization, KV compression, watermarking, and new model architectures — by reading recent literature and…
• Design and run experiments to validate or refute ideas, iterating quickly on small-scale setups before proposing production integration.
• Collaborate with the Inference Foundation team to integrate validated techniques into the vLLM-based serving stack.
• Partner with pretraining and RL teams when techniques require co-training, retraining, or training auxiliary models to improve inference.

What they're looking for

• A strong ML background with solid knowledge of LLM training techniques — architecture, tokenization, fine-tuning, and evaluation.
• Hands-on experience with Python and PyTorch; you write clean, experimental code and iterate fast. JAX is acceptable with strong Python proficiency.
• Research experience of some form: a thesis, publications, a dedicated research project, or efficiency-oriented work in a lab or industry research setting.
• A track record of conducting experiments: forming hypotheses, designing controlled comparisons, and drawing clear conclusions from results.
• Familiarity with the state of the art in LLM inference — you know what has been tried, what works, and where the open problems are.

Tech stack

• vLLM
• Python
• PyTorch
• JAX
• C++
• Rust
• CUDA