Staff Applied Scientist - Agentic Interfaces

Job summary

Responsibilities

This role is about that last piece. We're hiring a Staff Applied Scientist to define what "good" means for an Agentic interface at Datadog and to build the measurement systems that make it true.

• Own the evaluation strategy for Datadog's AI agent integrations. Define the metrics — offline and online, quality and cost, single-turn and trajectory-level — that the team and…
• Build the eval datasets, golden traces, and regression harnesses that catch quality changes before they hit customers, and make those assets reusable by every team contributing…
• Drive measurable improvements to retrieval relevance, tool-selection accuracy, and context efficiency, partnering closely with the AI engineers on the team who build the…
• Run applied research on the open problems in agent–data interaction: tool selection under large catalogs, multi-turn agent evaluation, grounding and hallucination control on live…
• Partner with the Bits SRE, Bits Assistant, and Bits Dev Agent teams so first-party agents benefit from the same measurement substrate as third-party integrations, and so…

What they're looking for

• You have a BS/MS/PhD in a scientific field, or equivalent experience.
• 10+ years of relevant engineering or applied science experience, including time as a technical lead.
• Proven track record of leading ML or GenAI initiatives in a product-driven environment, from research through production.
• Significant experience with evaluation, experimentation, or measurement of ML systems at scale.
• You bring a strong product mindset and are comfortable driving initiatives across cross-functional teams.