Backend Software Engineer, Cooperative AI (Evals)

Job summary

Responsibilities

We’re looking for a Backend Software Engineer with experience working in ML/LLM-heavy domains to help to design and build an evals infrastructure that measures the quality of OpenAI’s support automation.

• Design eval pipelines that are reliable, reproducible, and extendable
• Build the infrastructure for continuous eval monitoring frameworks (regression/drift monitoring, building robust golden datasets) along with feedback loops that ultimately…
• Design, build, and maintain backend services and APIs to support intelligent automation and knowledge systems
• Integrate and structure data across internal platforms, transforming it into formats optimized for use by downstream systems and AI workflows.
• Collaborate closely with data, research, and engineering teams to integrate OpenAI models into high-leverage workflows

What they're looking for

• 4+ years of backend engineering experience at product-driven companies (excluding internships)
• Proficiency in backend technologies. Our tech stack includes Python, FastAPI, and Postgres
• Experience designing and scaling distributed systems, APIs, or data processing pipelines
• Have experience building AI agents or applications, including designing evals and improving performance through prompting or scaffolding
• Are familiar with evaluation methods for LLMs and have worked with patterns like multi-agent workflows, tool use, or long context.

Tech stack

• Python
• FastAPI
• PostgreSQL