AI Research Engineer - Datadog AI Research (DAIR)

Job summary

Responsibilities

As a Research Engineer on our team, you will partner with Research Scientists to turn research ideas into working systems, building the data, tooling, and infrastructure that enable rapid iteration, trustworthy evaluation, and a smooth path from prototype to production.

• Build and operate multimodal data pipelines, training and evaluation infrastructure, benchmarks, and internal tooling
• Implement models, run experiments at scale, and profile for reliability, performance, and cost
• Build simulation environments and replay infrastructure for agent training and evaluation
• Orchestrate distributed training and distributed RL with Ray, including scheduling, scaling, and failure recovery
• Establish rigorous automated benchmarks and regression tests for world model predictions, agent performance, and simulation fidelity

What they're looking for

• You have depth in distributed computing, RL Infra, and ML systems for training and inference at scale; experience with Ray, Slurm, or similar frameworks is a plus
• You are proficient in Python, familiar with a systems language (e.g., Rust, C++, or Go), and comfortable with modern cloud and data infrastructure
• You have practical experience implementing and operating ML training and inference systems (e.g., PyTorch or JAX), including containerization, orchestration, and GPU acceleration
• You have practical experience with large-scale model training and fine-tuning, including frameworks like Megatron-LM, DeepSpeed, SkyRL, VeRL, or TorchTitan, and techniques such…
• You can explain design and performance trade-offs clearly to both technical and non-technical audiences

Tech stack

• Ray
• Python
• Rust
• C++
• Go
• PyTorch
• JAX
• CUDA