Software Engineer, Inference

Job summary

Responsibilities

Sierra’s AI agents depend on foundation models to reason and act in real time. The Inference team builds the systems that make those models fast, reliable, and efficient at scale.

• Partner with frontier labs and providers. At our scale, we rely on frontier labs, and inference providers to supply capacity, training and inference infrastructure.
• Shape Sierra’s inference architecture. Design how inference traffic flows across models, infrastructure, and providers, including new serving and proxy layers as Sierra scales.
• Build for low latency and high reliability. Develop systems for routing, failover, capacity management, and quota that keep inference performant and available across large-scale…
• Build and operate self-hosted inference. Run models on GPU infrastructure, from building containers and operating inference engines to managing the underlying compute capacity.
• Optimize inference performance. Work with the Applied Research team on techniques such as speculative decoding and serving-engine optimizations that improve latency, throughput…

What they're looking for

• Deep systems thinking and strong distributed systems fundamentals.
• Experience designing, building, and operating large-scale production systems.
• Strong judgment around tradeoffs involving latency, reliability, capacity, and cost.
• Experience taking ownership of complex infrastructure from architecture through production operation.
• Excitement about applying systems expertise to AI infrastructure and learning quickly as the underlying technology evolves.

Tech stack

• vLLM