Senior Manager, Engineering - AI Inference

Job summary

Responsibilities

As a Senior Engineering Manager, you will lead an engineering team focused on making large language models run faster, cheaper, and more reliably in production. While managing and growing the team, you will stay deeply technical and hands-on with the inference stack end to end: profiling…

• Bring current inference techniques into production and refine them.
• Design and optimize serving architectures, including prefill and decode disaggregation, request routing, and related approaches.
• Work down into the serving stack, from frameworks like vLLM and SGLang to the CUDA kernels underneath, profiling and running in-depth analysis to find and fix performance problems.
• Adapt and scale optimization methods across many kinds of ML models, with an emphasis on large language models.
• Profile and tune deployments against clear targets for latency, throughput, and cost, and keep them dependable under real traffic.

What they're looking for

• 2+ years of experience directly managing and leading an engineering team in a high-performance or ML-focused environment.
• Strong technical, hands-on experience in software engineering, low-level optimization, or ML infrastructure, with a continued desire to stay close to code and architecture.
• A Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, Mathematics, or a related field.
• Hands-on experience shipping code in production with one or more general-purpose languages, such as Python or C++, with a strong preference for Python.
• Familiarity with methods for optimizing LLMs for high throughput / low latency inference.

Tech stack

• vLLM
• CUDA
• Python
• C++
• Docker
• Kubernetes