When a RAG answer is wrong, which part failed — retrieval, chunking, or generation? Most teams guess, then swap the model. **RAG Probe** exists so you never have to guess: it benchmarks a RAG pipeline end to end and tells you where the failure actually lives.

## What it does

- One swappable `answer()` seam — point it at any pipeline.
- A gold set mined from resolved support tickets: real question, verified answer, known source document.
- A ~20-config experiment matrix — query transform, chunking, embedder, retrieval strategy, k, prompt, model — one variable per run.
- Retrieval metrics (recall@k, precision@k, MRR, nDCG@k), LLM-as-judge scoring for faithfulness and correctness, refusal rate, latency, and estimated cost per run.

The decision rule that pays for the whole harness: **if recall@k is low, fix retrieval — not the model.** No amount of prompt engineering rescues a context window that never contained the answer.

## Validated, not just written

Run end to end on WordPress 7.0 with live OpenAI calls: recall@5, MRR, and nDCG@5 at 1.0 on the gold set, judge faithfulness and correctness at 2/2, refuse-by-default behaving, roughly $0.0006 per run. Building it also surfaced two WordPress 7.0 AI-Client compatibility issues in the generation seam, which I fixed upstream.

## Get it

Open source (WP-CLI plugin) on [GitHub](https://github.com/Ref34t/rag-probe). Embeddings are pluggable — OpenAI, Voyage, Cohere, Jina, Google.