Mistral AI Posted 30+ days ago

AI Engineer, Product

Location

Paris

Contract

Permanent

Full-time

Salary

Not disclosed

Experience

3+ years

Mid-level level

Job summary

Responsibilities

Embedded directly in a product team as search, chat, documents, or audio, you'll improve AI-powered features through rigorous evaluation, prompt and orchestration design, and rapid experimentation.

Design and run evaluations for your product area: reference tests, heuristics, model-graded checks tailored to search relevance, chat quality, document understanding, or audio…
Define and track metrics that matter: task success, helpfulness, hallucination proxies, safety flags, latency, cost.
Own prompt and orchestration design: write, test, and iterate on prompts and system prompts as a core part of your work.
Run A/B tests on prompts, models, and configurations; analyze results; make rollout or rollback decisions from data.
Set up observability for LLM calls: structured logging, tracing, dashboards, alerts.

What they're looking for

3-4 years of experience; backgrounds that fit well include ML engineers moving closer to product, or software engineers with real AI/ML production experience.
Strong TypeScript or Python skills - we have both tracks depending on team fit.
Production LLM experience: prompts, tool/function calling, system prompts.
Hands-on with evals and A/B testing; you can design metrics, not just run them.
Comfortable implementing directly in product code, not only notebooks.