Staff Engineer, Distributed Storage and HPC & AI Infrastructure

Job summary

Responsibilities

In this role, you will operate, scale, and optimize multi-petabyte storage systems purpose-built for the world’s largest AI training and inference workloads. Storage now sits on the inference critical path as well, so you’ll also build the caching and data-movement layers that deliver KV cache and…

• Architect and implement the technical strategy and storage roadmap for Together AI, driving high-performance architectural decisions as we scale our GPU fleet.
• Engineer and scale multi-petabyte AI/ML storage systems by integrating Vast, Weka, and Ceph while executing deep cost optimization through automated tiering and lifecycle policies.
• Develop intelligent caching and tiered storage architectures to achieve extreme IOPS and cluster-wide throughput at GPU scale for training and inference workloads, including…
• Design and build KV-cache offload and reuse for inference: tiering KV blocks across GPU HBM, host memory, local NVMe, and distributed cache, enabling prefix reuse across requests…
• Tune storage isolation at the L2/L3 network layers to ensure secure, production-grade multi-tenancy for storage clients.

What they're looking for

• 8+ years in storage engineering, managing distributed storage at multi-petabyte scale
• Proven track record building, deploying and operating high-performance storage for GPU/HPC clusters, for training and/or inference
• Experience scaling storage for inference: KV-cache offload or disaggregated KV storage, prefix caching, model-weight distribution, or similar latency-critical data paths
• Deep Kubernetes and cloud-native storage experience in production environments
• Strong coding skills in Go and Python with demonstrated ability to build production-grade systems and tooling

Tech stack

• Kubernetes
• vLLM
• Go
• Python
• C++
• Rust
• Terraform