Site Reliability Engineer

Job summary

Responsibilities

As a Site Reliability Engineer at Baseten, you'll define and codify the gold standards of day 2 operations for our ML infrastructure platform. You'll envision and build robust systems, processes, automations, and observability tooling that keep our platform reliable at scale — and that empower the…

• Own the reliability of Baseten's multi-cloud Kubernetes infrastructure, including incident response, post-mortems, and remediation tracking.
• Build and maintain observability infrastructure — metrics, logging, dashboards, and alerting — as code.
• Author, validate, and improve runbooks for recurring failure patterns, ensuring they're structured for low-context, safe execution.
• Identify high-frequency failure patterns and convert them into automated mitigations or self-healing automations.
• Diagnose and resolve runtime issues related to latency, memory behavior, GPU utilization, concurrency, and model lifecycle management.

What they're looking for

• Extensive hands-on experience with Kubernetes (multi-cloud experience across EKS, GKE, or similar is a strong plus).
• Experience in building and maintaining scalable infrastructure.
• Strong foundation in observability tooling: metrics (VictoriaMetrics, Prometheus), logging (Loki, ELK), dashboards (Grafana), and alerting pipelines.
• Experience with infrastructure-as-code (Terraform, Helm) and GitOps workflows (Flux CD, ArgoCD).
• Experience writing and improving runbooks, leading incident response, and doing post-mortem analysis.

Tech stack

• Spark
• Kubernetes
• Terraform