When a RAG answer is wrong, which part failed — retrieval, chunking, or generation? Most teams guess, then swap the model. RAG Probe exists so you never have to guess: it benchmarks a RAG pipeline end to end and tells you where the failure actually lives.
What it does
- One swappable
answer()seam — point it at any pipeline. - A gold set mined from resolved support tickets: real question, verified answer, known source document.
- A ~20-config experiment matrix — query transform, chunking, embedder, retrieval strategy, k, prompt, model — one variable per run.
- Retrieval metrics (recall@k, precision@k, MRR, nDCG@k), LLM-as-judge scoring for faithfulness and correctness, refusal rate, latency, and estimated cost per run.
The decision rule that pays for the whole harness: if recall@k is low, fix retrieval — not the model. No amount of prompt engineering rescues a context window that never contained the answer.
Validated, not just written
Run end to end on WordPress 7.0 with live OpenAI calls: recall@5, MRR, and nDCG@5 at 1.0 on the gold set, judge faithfulness and correctness at 2/2, refuse-by-default behaving, roughly $0.0006 per run. Building it also surfaced two WordPress 7.0 AI-Client compatibility issues in the generation seam, which I fixed upstream.
Get it
Open source (WP-CLI plugin) on GitHub. Embeddings are pluggable — OpenAI, Voyage, Cohere, Jina, Google.