Software Engineer - Inference Platform

Job summary

Responsibilities

We're looking for distributed systems engineers and product-minded generalists to build the distributed runtime that powers large-scale LLM inference on Baseten. Our inference platform empowers customers to deploy and operate cutting-edge models with industry-leading performance, scalability, and…

• Build the infrastructure and orchestration systems that deploy and run large-scale distributed LLM inference, including routing, autoscaling, scheduling, and runtime management.
• Design, build, and operate Model APIs, with a focus on advanced inference capabilities: structured outputs (JSON mode, grammar-constrained generation), tool/function calling, and…
• Implement platform fundamentals such as API versioning, validation, usage metering, quotas, and authentication.
• Instrument deep observability (metrics, traces, logs) and build repeatable benchmarks for speed, reliability, and quality.
• Debug and harden complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads to improve reliability and scalability.

What they're looking for

• Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field, or equivalent practical experience.
• 3+ years building and operating distributed systems, backend infrastructure, or large-scale APIs where reliability, latency, and scale are first-class concerns.
• A proven track record of owning low-latency, reliable backend services, including rate limiting, auth, quotas, metering, and migrations.
• Infrastructure instincts with a feel for performance: profiling, tracing, capacity planning, and SLO management.
• Comfort debugging performance and reliability issues across multiple layers of the stack, from application behavior down to runtime and infrastructure internals.

Tech stack

• Spark
• Kubernetes
• vLLM