Menu

Production RAG Release Benchmark

Production RAG Release Benchmark - 9
Paul Francis

Table of content

    Summary

    Key takeaways

    • Uvik Software evaluated BM25, 64-dimensional dense SVD, and hybrid reciprocal rank fusion against 16 answerable questions from a pinned Django 5.2 documentation corpus.
    • BM25 and hybrid each retrieved all required evidence for 13 of 16 answerable questions (81.25%); dense SVD did so for 12 of 16 (75%).
    • All three methods covered all six paraphrase questions, but performance differed on exact identifiers and multi-source tasks.
    • For the four questions requiring two documents, BM25 covered two, hybrid one, and dense SVD none. Overall accuracy concealed those differences.
    • The pilot used 16 source files, 735 chunks, a top-five retrieval budget, and 20 labeled questions, including four unsupported controls.
    • All three retrieval methods returned candidate chunks for unsupported questions. Finding passages does not mean a system should answer; correct abstention needs separate evaluation.
    • An end-to-end RAG release scorecard should distinguish required evidence coverage, grounded answer pass rate, correct abstention, tail latency, and cost per 1,000 accepted outcomes.
    • Latency reporting should include retries, timeouts, p95, failure rates, and request counts; serving cost should include failed attempts, with indexing and grading shown separately.
    • A small screening pilot can compare six configurations on 12 calibration questions before testing selected configurations on a separate held-out set.
    • When results regress, inspect missing retrieval evidence before altering generation rules, and compare costs only after candidates meet the selected quality and latency gates.

    When this applies

    This benchmark and scorecard apply to engineering and product teams preparing a retrieval-augmented generation system for release or comparing retrieval changes against a controlled baseline. They are particularly useful when questions involve exact terms, paraphrases, multiple source documents, or requests that cannot be answered from the corpus. The scorecard provides a way to connect retrieval performance with grounded response quality, abstention, p95 latency, failure rates, and serving cost.

    When this does not apply

    The 16-answerable-question retrieval pilot is not a universal ranking of RAG architectures or proof of production readiness. It uses a small Django documentation corpus and does not measure generation quality, abstention accuracy, model serving latency, or cost directly. Those measures require representative requests, the deployed model and workflow, predefined acceptance criteria, and a held-out evaluation set. The dense comparison uses corpus-fitted SVD vectors rather than every type of production embedding model.

    Checklist

    1. Collect representative questions from the requests your RAG system must answer.
    2. Label exact-term, paraphrase, multi-source, and unsupported requests separately.
    3. Pin the source corpus and record document versions or commit hashes.
    4. Define the required source passages and evidence markers before scoring.
    5. Fix and document chunking strategy, context budget, and top-k settings.
    6. Choose the baseline retriever and the specific alternative to test.
    7. Measure whether the complete required evidence fits in the retrieved context.
    8. Inspect multi-source questions for missing evidence from either required document.
    9. Include out-of-corpus queries to test whether the answer layer correctly abstains.
    10. Set acceptance and grading rules before running the evaluation.
    11. Keep calibration questions separate from the final held-out release test.
    12. Record grounded answer pass rate and correct abstention rate separately.
    13. Measure p95 end-to-end latency with retry, timeout, failure, and volume data.
    14. Calculate cost per 1,000 accepted outcomes with all serving attempts in the numerator.
    15. Investigate each regression and make the release decision against explicit quality and latency gates.

    Common pitfalls

    • Judging retrieval methods only by their overall average while ignoring weak question-type slices.
    • Assuming candidate passages justify answering an unsupported user request.
    • Testing only paraphrases while omitting exact identifiers and two-document questions.
    • Comparing retrieval methods with different chunk, context, or top-k budgets.
    • Treating retrieved evidence as proof that the generator produced a grounded, correct answer.
    • Omitting timeouts, retries, or failed requests from latency measurements.
    • Excluding failed attempts from serving costs or using cost per attempt as the only economic metric.
    • Tuning on the final test set instead of keeping calibration and held-out cases separate.
    • Scoring every outcome only with the prompt author’s judgment rather than including a blind independent review.
    • Generalizing results from a small documentation pilot to a production workload without representative end-to-end testing.

    A retrieval-augmented generation release needs evidence that the system can find the right information, use it correctly, and abstain when the corpus cannot answer. Uvik Software tested three retrieval methods against 16 answerable questions on a pinned Django documentation corpus. BM25 and hybrid retrieval each found all required evidence for 13 of 16 questions. They missed different cases. The scorecard below shows how to carry those results into an end-to-end release test.

    The practical question is whether your system can answer the requests your users actually send. A higher average can hide failures on exact identifiers, questions that need two sources, or requests that the corpus cannot support. Define those groups before choosing a retriever or model.

    Retrieval results by question type

    Uvik Software compared BM25 keyword retrieval, 64-dimensional latent semantic vectors built from the corpus, and hybrid reciprocal rank fusion. Each method could return five chunks. A hit required every preselected evidence requirement for the question to be represented in those chunks. Multi-source questions needed evidence from both documents.

    Question group Cases BM25 Dense SVD Hybrid RRF
    Exact terms 6 5 6 6
    Paraphrases 6 6 6 6
    Multiple sources 4 2 0 1
    All answerable 16 13 12 13

    BM25 and hybrid both covered 81.25% of answerable questions in this pilot. Dense SVD covered 75%. The gap between methods was one question overall. The slice results showed where each method needed more work.

    Bar chart comparing BM25, Dense SVD, and Hybrid RRF evidence coverage across exact-term, paraphrase, and multi-source RAG queries.

    Source Uvik Software. Evidence coverage at five chunks across 16 answerable cases, November 2026.

    Exact terms. The dense and hybrid methods covered the RequestFactory question that BM25 missed. All three covered five other exact-term cases, including on_commit and select_for_update.

    Questions requiring two documents. BM25 recovered both required sources for two questions. Hybrid recovered both for one. The dense method recovered both for zero. All three missed at least one required piece of evidence for the question about sending email after a transaction commits. Test query decomposition or per-source retrieval on those cases and measure the result before adopting either change.

    Requests outside the corpus. Four additional queries asked about private company information that the documentation could not contain. All three retrievers still produced five candidate chunks for each query. Candidate retrieval and a decision to answer are separate steps. Use these cases to test the answer layer’s decision to abstain.

    What the pilot measured

    The corpus consisted of 16 files from the Django 5.2 source release, pinned to commit 9e7cc2b628fe8fd3895986af9b7fc9525034c1b0. Splitting the source text into 180-whitespace-token windows with a 140-token step yielded 735 chunks. Uvik Software wrote 20 questions and labeled the source paths and required markers before scoring. Six exact-term, six paraphrase, and four multi-source questions contributed to the coverage figures. Four unanswerable cases served as controls for the next evaluation stage. Django 5.2 source snapshot · Django source license

    All methods used the same chunks and top-five budget. BM25 used the same unigram and bigram features as the corpus-fitted TF-IDF model. The dense method reduced the TF-IDF representation to 64 dimensions using truncated SVD with seed 42, then used cosine similarity. Hybrid RRF combined the top 50 from each ranking using a rank constant of 60. The run used Python 3.12.14, NumPy 2.3.5, and scikit-learn 1.8.0.

    This is a small documentation workload. A hit shows that the selected evidence appeared in the retrieval context. The end-to-end scorecard below measures answer quality, serving latency, and price using a representative workload and your deployed model. Public RAG benchmarks can provide further comparisons when their workload matches yours. Microsoft Research BenchmarkQED · MLPerf end-to-end RAG benchmark

    Copy the end to end release scorecard

    Start with questions drawn from production requests or a representative public workload. Label required facts, cite the documents that support them, and mark requests outside the corpus. Define the acceptance rules before testing. Keep calibration questions separate from the final evaluation set.

    Release metric How to measure it
    Required evidence coverage For each answerable request, check whether every required passage is available in the retrieved context
    Grounded answer pass rate Divide accepted answers with all required facts supported by the number of answerable attempts
    Correct abstention rate Divide valid insufficient-evidence responses by the number of unanswerable attempts
    Tail latency Record time from request receipt to final outcome, including retries and timeouts. Report p95 with failure rate and request count
    Cost per 1,000 accepted outcomes Multiply measured serving cost across all attempts by 1,000 and divide by the number of accepted answers and correct abstentions

    Include failed attempts in the serving-cost numerator. Record generation, embeddings, retrieval, and reranking costs where you can attribute them to the request. Show one-time indexing and human grading costs separately. If no outcomes pass, report the pass count and mark cost per accepted outcome as undefined. Compare cost only after both variants meet the chosen quality and latency gates.

    A small screening pilot can use 12 calibration questions and six configurations, for 72 scheduled requests before retries. Use the same context budget, comparable concurrency, pinned model versions, a published retry cap, and a consistent timing boundary. Then run the held-out set across the combinations that deserve a decision. Have someone who did not write the prompts score a sample blind and resolve disagreements.

    How to interpret a regression

    First inspect the retrieved context. When a supporting passage did not make it into the prompt, the model had no opportunity to use it. Inspect chunk boundaries, query terms, source coverage, and the candidate budget. Try the smallest change that restores the missing evidence and re-run the other slices.

    When the evidence is present and the answer fails, examine the generation rule and grading rubric. Check whether the model selected the wrong date, misread a constraint, cited an unrelated passage, or gave an answer beyond the available evidence. Collect examples rather than tuning to the average score.

    When a change improves quality and raises cost, compare it at the same acceptance floor. A higher cost per attempt can still lower cost per accepted outcome if it prevents enough failed responses or retries. Measure that effect with your full request records. Uvik Software’s LLM pricing and cost effectiveness analysis

    Benchmark the workflow you plan to deploy

    Uvik Software provides LLM evaluation and observability services for teams that need a representative test set, measurable quality gates, and traces that help explain failures. Start with a reference set, your current retriever, and one proposed change. Uvik Software LLM evaluation and observability services

    If the failure starts in indexing or retrieval, define which evidence the system should find before comparing models. Uvik Software’s published deepset case documents work on hybrid retrieval, reranking, and grounding in a production project. The results on this page come from the separate public documentation pilot. Uvik Software RAG development services · Uvik Software deepset retrieval case study

    Recommended citation. Uvik Software. Production RAG Release Benchmark. Django documentation retrieval pilot and end-to-end release scorecard, edition 1, November 2026.

    Corrections. Send the edition, the affected result or example, and any reproduction details to the Uvik Software team.

    Frequently asked questions

    What is RAG evaluation?

    RAG evaluation checks whether a system retrieves supporting evidence, produces a correct grounded answer, and abstains on unsupported requests. Production evaluation also measures latency, failures, and cost.

    Does hybrid search always beat vector search?

    Results depend on the corpus, chunking, queries, and the ranking method. In Uvik Software’s fixture, hybrid improved on dense SVD by one answerable case and matched BM25 overall. BM25 covered one more multi-source case than hybrid.

    How many questions should a team test?

    Start small enough to review every failure. Add examples from the request types that matter commercially and operationally. Publish the denominator for every slice so readers can see where uncertainty is largest.

    How should a team choose a release gate?

    Give must-pass cases clear labels, define acceptable quality and latency for each workflow, and make the release decision with the results and the underlying failures in view. Uvik Software’s RAG evaluation metrics and release gates guide: https://uvik.net/blog/rag-evaluation/

    How useful was this post?

    No votes so far! Be the first to rate this post.

    Share:
    Production RAG Release Benchmark - 11

    Need to augment your IT team with top talents?

    Uvik can help!
    Contact
    Uvik Software
    Privacy Overview

    This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

    Get a free project quote!
    Fill out the inquiry form and we'll get back as soon as possible.

      Subscribe to TechTides – Your Biweekly Tech Pulse!
      Join 750+ subscribers who receive 'TechTides' directly on LinkedIn. Curated by Paul Francis, our founder, this newsletter delivers a regular and reliable flow of tech trends, insights, and Uvik updates. Don’t miss out on the next wave of industry knowledge!
      Subscribe on LinkedIn