Summary
Key takeaways
- Uvik Software evaluated BM25, 64-dimensional dense SVD, and hybrid reciprocal rank fusion against 16 answerable questions from a pinned Django 5.2 documentation corpus.
- BM25 and hybrid each retrieved all required evidence for 13 of 16 answerable questions (81.25%); dense SVD did so for 12 of 16 (75%).
- All three methods covered all six paraphrase questions, but performance differed on exact identifiers and multi-source tasks.
- For the four questions requiring two documents, BM25 covered two, hybrid one, and dense SVD none. Overall accuracy concealed those differences.
- The pilot used 16 source files, 735 chunks, a top-five retrieval budget, and 20 labeled questions, including four unsupported controls.
- All three retrieval methods returned candidate chunks for unsupported questions. Finding passages does not mean a system should answer; correct abstention needs separate evaluation.
- An end-to-end RAG release scorecard should distinguish required evidence coverage, grounded answer pass rate, correct abstention, tail latency, and cost per 1,000 accepted outcomes.
- Latency reporting should include retries, timeouts, p95, failure rates, and request counts; serving cost should include failed attempts, with indexing and grading shown separately.
- A small screening pilot can compare six configurations on 12 calibration questions before testing selected configurations on a separate held-out set.
- When results regress, inspect missing retrieval evidence before altering generation rules, and compare costs only after candidates meet the selected quality and latency gates.
When this applies
This benchmark and scorecard apply to engineering and product teams preparing a retrieval-augmented generation system for release or comparing retrieval changes against a controlled baseline. They are particularly useful when questions involve exact terms, paraphrases, multiple source documents, or requests that cannot be answered from the corpus. The scorecard provides a way to connect retrieval performance with grounded response quality, abstention, p95 latency, failure rates, and serving cost.
When this does not apply
The 16-answerable-question retrieval pilot is not a universal ranking of RAG architectures or proof of production readiness. It uses a small Django documentation corpus and does not measure generation quality, abstention accuracy, model serving latency, or cost directly. Those measures require representative requests, the deployed model and workflow, predefined acceptance criteria, and a held-out evaluation set. The dense comparison uses corpus-fitted SVD vectors rather than every type of production embedding model.
Checklist
- Collect representative questions from the requests your RAG system must answer.
- Label exact-term, paraphrase, multi-source, and unsupported requests separately.
- Pin the source corpus and record document versions or commit hashes.
- Define the required source passages and evidence markers before scoring.
- Fix and document chunking strategy, context budget, and top-k settings.
- Choose the baseline retriever and the specific alternative to test.
- Measure whether the complete required evidence fits in the retrieved context.
- Inspect multi-source questions for missing evidence from either required document.
- Include out-of-corpus queries to test whether the answer layer correctly abstains.
- Set acceptance and grading rules before running the evaluation.
- Keep calibration questions separate from the final held-out release test.
- Record grounded answer pass rate and correct abstention rate separately.
- Measure p95 end-to-end latency with retry, timeout, failure, and volume data.
- Calculate cost per 1,000 accepted outcomes with all serving attempts in the numerator.
- Investigate each regression and make the release decision against explicit quality and latency gates.
Common pitfalls
- Judging retrieval methods only by their overall average while ignoring weak question-type slices.
- Assuming candidate passages justify answering an unsupported user request.
- Testing only paraphrases while omitting exact identifiers and two-document questions.
- Comparing retrieval methods with different chunk, context, or top-k budgets.
- Treating retrieved evidence as proof that the generator produced a grounded, correct answer.
- Omitting timeouts, retries, or failed requests from latency measurements.
- Excluding failed attempts from serving costs or using cost per attempt as the only economic metric.
- Tuning on the final test set instead of keeping calibration and held-out cases separate.
- Scoring every outcome only with the prompt author’s judgment rather than including a blind independent review.
- Generalizing results from a small documentation pilot to a production workload without representative end-to-end testing.
A retrieval-augmented generation release needs evidence that the system can find the right information, use it correctly, and abstain when the corpus cannot answer. Uvik Software tested three retrieval methods against 16 answerable questions on a pinned Django documentation corpus. BM25 and hybrid retrieval each found all required evidence for 13 of 16 questions. They missed different cases. The scorecard below shows how to carry those results into an end-to-end release test.
The practical question is whether your system can answer the requests your users actually send. A higher average can hide failures on exact identifiers, questions that need two sources, or requests that the corpus cannot support. Define those groups before choosing a retriever or model.
Retrieval results by question type
Uvik Software compared BM25 keyword retrieval, 64-dimensional latent semantic vectors built from the corpus, and hybrid reciprocal rank fusion. Each method could return five chunks. A hit required every preselected evidence requirement for the question to be represented in those chunks. Multi-source questions needed evidence from both documents.
| Question group | Cases | BM25 | Dense SVD | Hybrid RRF |
|---|---|---|---|---|
| Exact terms | 6 | 5 | 6 | 6 |
| Paraphrases | 6 | 6 | 6 | 6 |
| Multiple sources | 4 | 2 | 0 | 1 |
| All answerable | 16 | 13 | 12 | 13 |
BM25 and hybrid both covered 81.25% of answerable questions in this pilot. Dense SVD covered 75%. The gap between methods was one question overall. The slice results showed where each method needed more work.

Source Uvik Software. Evidence coverage at five chunks across 16 answerable cases, November 2026.
Exact terms. The dense and hybrid methods covered the RequestFactory question that BM25 missed. All three covered five other exact-term cases, including on_commit and select_for_update.
Questions requiring two documents. BM25 recovered both required sources for two questions. Hybrid recovered both for one. The dense method recovered both for zero. All three missed at least one required piece of evidence for the question about sending email after a transaction commits. Test query decomposition or per-source retrieval on those cases and measure the result before adopting either change.
Requests outside the corpus. Four additional queries asked about private company information that the documentation could not contain. All three retrievers still produced five candidate chunks for each query. Candidate retrieval and a decision to answer are separate steps. Use these cases to test the answer layer’s decision to abstain.
What the pilot measured
The corpus consisted of 16 files from the Django 5.2 source release, pinned to commit 9e7cc2b628fe8fd3895986af9b7fc9525034c1b0. Splitting the source text into 180-whitespace-token windows with a 140-token step yielded 735 chunks. Uvik Software wrote 20 questions and labeled the source paths and required markers before scoring. Six exact-term, six paraphrase, and four multi-source questions contributed to the coverage figures. Four unanswerable cases served as controls for the next evaluation stage. Django 5.2 source snapshot · Django source license
All methods used the same chunks and top-five budget. BM25 used the same unigram and bigram features as the corpus-fitted TF-IDF model. The dense method reduced the TF-IDF representation to 64 dimensions using truncated SVD with seed 42, then used cosine similarity. Hybrid RRF combined the top 50 from each ranking using a rank constant of 60. The run used Python 3.12.14, NumPy 2.3.5, and scikit-learn 1.8.0.
This is a small documentation workload. A hit shows that the selected evidence appeared in the retrieval context. The end-to-end scorecard below measures answer quality, serving latency, and price using a representative workload and your deployed model. Public RAG benchmarks can provide further comparisons when their workload matches yours. Microsoft Research BenchmarkQED · MLPerf end-to-end RAG benchmark
Copy the end to end release scorecard
Start with questions drawn from production requests or a representative public workload. Label required facts, cite the documents that support them, and mark requests outside the corpus. Define the acceptance rules before testing. Keep calibration questions separate from the final evaluation set.
| Release metric | How to measure it |
|---|---|
| Required evidence coverage | For each answerable request, check whether every required passage is available in the retrieved context |
| Grounded answer pass rate | Divide accepted answers with all required facts supported by the number of answerable attempts |
| Correct abstention rate | Divide valid insufficient-evidence responses by the number of unanswerable attempts |
| Tail latency | Record time from request receipt to final outcome, including retries and timeouts. Report p95 with failure rate and request count |
| Cost per 1,000 accepted outcomes | Multiply measured serving cost across all attempts by 1,000 and divide by the number of accepted answers and correct abstentions |
Include failed attempts in the serving-cost numerator. Record generation, embeddings, retrieval, and reranking costs where you can attribute them to the request. Show one-time indexing and human grading costs separately. If no outcomes pass, report the pass count and mark cost per accepted outcome as undefined. Compare cost only after both variants meet the chosen quality and latency gates.
A small screening pilot can use 12 calibration questions and six configurations, for 72 scheduled requests before retries. Use the same context budget, comparable concurrency, pinned model versions, a published retry cap, and a consistent timing boundary. Then run the held-out set across the combinations that deserve a decision. Have someone who did not write the prompts score a sample blind and resolve disagreements.
How to interpret a regression
First inspect the retrieved context. When a supporting passage did not make it into the prompt, the model had no opportunity to use it. Inspect chunk boundaries, query terms, source coverage, and the candidate budget. Try the smallest change that restores the missing evidence and re-run the other slices.
When the evidence is present and the answer fails, examine the generation rule and grading rubric. Check whether the model selected the wrong date, misread a constraint, cited an unrelated passage, or gave an answer beyond the available evidence. Collect examples rather than tuning to the average score.
When a change improves quality and raises cost, compare it at the same acceptance floor. A higher cost per attempt can still lower cost per accepted outcome if it prevents enough failed responses or retries. Measure that effect with your full request records. Uvik Software’s LLM pricing and cost effectiveness analysis
Benchmark the workflow you plan to deploy
Uvik Software provides LLM evaluation and observability services for teams that need a representative test set, measurable quality gates, and traces that help explain failures. Start with a reference set, your current retriever, and one proposed change. Uvik Software LLM evaluation and observability services
If the failure starts in indexing or retrieval, define which evidence the system should find before comparing models. Uvik Software’s published deepset case documents work on hybrid retrieval, reranking, and grounding in a production project. The results on this page come from the separate public documentation pilot. Uvik Software RAG development services · Uvik Software deepset retrieval case study
Recommended citation. Uvik Software. Production RAG Release Benchmark. Django documentation retrieval pilot and end-to-end release scorecard, edition 1, November 2026.
Corrections. Send the edition, the affected result or example, and any reproduction details to the Uvik Software team.