Senior Staff Applied AI Engineer - Context Retrieval

Job summary

Responsibilities

Databricks agents are only as good as the context they can retrieve. Whether an agent is answering a question about last quarter's revenue, debugging a failing job, generating SQL against a 10,000-table lakehouse, or summarizing a Wiki page, its quality is bounded by what it can find — and how…

• Build the retrieval stack — query understanding, content understanding, ranking, retrieval, and evaluation — across the Enterprise SaaS data stored across multiple systems.
• Build the search subagents that sit on top of that stack and reason about what context is needed, how to retrieve it, and whether the right thing actually came back — closing the…
• Build the full retrieval stack from scratch. Own the end-to-end system: query understanding, content understanding and indexing, hybrid retrieval, ranking, and evaluation.
• Retrieve across heterogeneous data — structured and unstructured. Index and rank across structured assets (tables, columns, SQL queries, dashboards, code, notebooks, jobs) and…
• Connect to the SaaS surface area customers actually use. Build connectors and retrieval adapters for the systems where enterprise knowledge lives.

What they're looking for

• 10+ years of software engineering experience, with significant time spent building production retrieval, search, or RAG systems at scale.
• Deep Information Retrieval (IR) expertise: lexical retrieval (BM25, Lucene/Elasticsearch/OpenSearch), dense retrieval (embeddings, ANN indexes — FAISS, ScaNN, HNSW), hybrid…
• Hands-on experience with modern LLM-era retrieval: RAG architectures, query rewriting, re-ranking with cross-encoders, long-context strategies, and grounding techniques that…
• Experience designing agentic systems on top of retrieval — search planners, multi-hop / iterative retrieval, self-reflection and sufficiency checks, tool-using agents that decide…
• Strong grasp of relevance evaluation: nDCG, MRR, Precision@K, Recall@K; offline/online experimentation; LLM-as-judge frameworks; building human labeling pipelines.

Tech stack

• SQL
• RAG
• Elasticsearch
• Spark
• MLflow