Evaluation Engineer, Applied AI

Job summary

Responsibilities

This is a chance to build the frontier LLM evals that shape the next generations of flagship models.

• Run discovery with customers, industry experts and teams across Mistral, then help decide which use cases are worth building first.
• Design new benchmarks for industry use cases and frontier capabilities, including how they're scored.
• Build evals where agents work inside real software (e.g., CAD and simulation tools), and run them on our models with checks that the results stay reliable over time.
• Stress-test existing evals to find flaws and bugs before we rely on them.
• Source data with expert vendors. Where customer data can't leave the customer, build realistic proxy or synthetic data instead.

What they're looking for

• 2+ years of industry experience (e.g., in software, AI, or consulting), or a PhD.
• A software engineering or research background, with experience writing code that others can build on.
• Experience working directly with customers, industry experts, or other teams (e.g., through consulting, solutions engineering, or research collaborations).
• Experience building something with LLMs.
• Comfort working with large datasets, including building and cleaning them.

Tech stack

• Python