Senior Backend Engineer, Inference Platform

Job summary

Responsibilities

Together AI is building the Inference Platform that brings the most advanced generative AI models to the world. Our platform powers multi-tenant serverless workloads and dedicated endpoints, enabling developers, enterprises, and researchers to harness the latest LLMs, multimodal models, image…

• Shape the core inference backbone that powers Together AI’s frontier models.
• Solve performance-critical challenges in global request routing, load balancing, and large-scale resource allocation.
• Work with state-of-the-art accelerators (H100s, H200s, GB200s) at global scale.
• Partner with world-class researchers to bring new model architectures into production.
• Collaborate with and contribute to the open source community, shaping the tools that advance the industry.

What they're looking for

• 5+ years of demonstrated experience building large-scale, fault-tolerant, distributed systems and API microservices.
• Strong background in designing, analyzing, and improving efficiency, scalability, and stability of complex systems.
• Excellent understanding of low-level OS concepts: multi-threading, memory management, networking, and storage performance.
• Expert-level programming in one or more of: Rust, Go, Python, or TypeScript.
• Knowledge of modern LLMs and generative models and how they are served in production is a plus.

Tech stack

• vLLM
• Rust
• Go
• Python
• TypeScript
• Kubernetes
• CUDA
• Triton