Staff Software Engineer, Model Infrastructure

Job summary

Responsibilities

As a Staff Software Engineer on the Model Infrastructure team, you'll lead the design and development of the systems that power every AI request at Harvey. You'll partner closely with AI Research, Product Engineering, Infrastructure, and external model providers to build a platform that is highly…

• Lead the design and implementation of Harvey's Model Infrastructure platform.
• Build systems to ensure high availability, low latency, and operational excellence for AI inference.
• Design and improve Harvey's Unified Model Controller (UMC) and Model Selector platform to automatically detect model degradations and intelligently route traffic based on…
• Develop systems for model provisioning, capacity management, failover, and traffic engineering across multiple AI providers.
• Integrate new model providers and maintain provider APIs and SDKs, enabling Harvey to rapidly adopt emerging frontier models.

Tech stack

• Azure
• Go
• Java
• Python
• Rust
• C++
• Kubernetes
• Kafka