Data Scientist, Agent

Job summary

Responsibilities

TL;DR — You own how we measure and improve Lovable's AI agent. You build the eval systems and experiments that tell us whether a change makes the agent better or worse, and you turn agent telemetry into the fixes that raise success rates and cut errors.

• Define and own the metrics for agent quality: success, completion, error rates, and the behaviors that drive them.
• Build the eval systems and experiment framework that decide whether an agent change ships, like an A/B-tested rollout that catches a change increasing errors before it reaches…
• Turn agent traces and telemetry into concrete fixes, working directly with the agent engineering team.
• Build the tooling and agents that produce these evaluations continuously as the agent evolves.
• Set the bar for how we judge agent behavior where there is no answer key to check against.

What they're looking for

• A data scientist who wants to make an AI agent measurably better, not just report on it. You own agent quality metrics and drive them up.
• Experience or strong interest in LLM evaluation and observability: building evals, scoring outputs, tracing agent behavior, and catching regressions.
• Strong SQL and Python, applied statistics, and experimentation. Comfortable designing A/B tests for agent changes where outcomes are noisy.
• You build systems and agents that produce this insight continuously, rather than one-off analyses.
• Instinct for what "good" looks like in agent behavior (success, error rates, task completion) and how to measure it when there is no clean answer key.

Tech stack

• SQL
• Python
• BigQuery
• GCP