ChatGPT 
Summary
Key takeaways
- AI hallucination rates vary dramatically by task, so a single percentage cannot describe how reliable a model is in every context.
- In the Vectara document-summary benchmark, hallucination rates across 105 models ranged from 1.8% to 24.2%.
- Among large model providers on that benchmark, OpenAI gpt-5.4-nano scored 3.1% and Google gemini-2.5-flash-lite scored 3.3%.
- Higher reasoning effort does not automatically reduce hallucinations: in Uvik Software’s analysis of six same-model pairs, higher-effort settings were 0.98 percentage points worse on average.
- Retrieval does not eliminate hallucinations; legal AI research tools using retrieval still showed hallucination rates of 17% to 33%.
- Open-domain and news-related tasks are substantially harder than document-grounded summaries, with some AI search tools producing incorrect answers on more than 60% of tested news queries.
- Hallucination risk is especially important in high-stakes domains such as law and medicine, where fabricated citations or false clinical claims can cause real-world harm.
- Generated code has its own hallucination risk: one large study found that 19.7% of 2.23 million package references produced by code models did not exist.
- Answer rate must be considered alongside hallucination rate because models that refuse more often can appear artificially safer.
- The safest way to select and configure a model is to benchmark it on your own task, data, prompts, and acceptable error threshold rather than relying on a universal leaderboard.
When this applies
This applies when you are evaluating LLM reliability for document summarization, RAG, AI search, legal research, healthcare applications, coding assistants, customer-facing AI, or other systems where unsupported claims can create business or safety risk. It is especially useful when comparing models, reasoning settings, retrieval strategies, or mitigation techniques and when you need to define realistic hallucination thresholds for production. The data is also relevant for teams building evaluation suites and deciding where human review or stricter verification is required.
When this does not apply
This does not apply as a universal ranking of which AI model is always the most accurate. Different benchmarks measure different behaviors, use different definitions of hallucination, and test different tasks. A low hallucination rate on document summarization does not prove that the same model will perform equally well on news, legal research, coding, or open-domain questions. Published rates should therefore be compared only within the same benchmark and should not replace testing on your own production workload.
Checklist
- Define what counts as a hallucination for your specific application.
- Separate unsupported claims from directly contradictory claims when evaluating outputs.
- Choose a benchmark or evaluation set that closely matches the production task.
- Compare models only inside the same benchmark and under the same testing conditions.
- Measure answer rate or refusal rate alongside hallucination rate.
- Create a representative evaluation dataset from real user tasks and source material.
- Test multiple model families instead of assuming the largest or newest model will perform best.
- Benchmark different reasoning settings because more reasoning can sometimes increase unsupported claims.
- Evaluate retrieval quality separately from generation quality in RAG systems.
- Require citations or supporting evidence when factual verification matters.
- Add human review for legal, medical, financial, or other high-impact outputs.
- Verify every new software dependency suggested by an AI coding tool before installation.
- Test mitigation prompts, refusal rules, and input validation against known failure cases.
- Monitor hallucination rates after deployment because model, prompt, and data changes can alter behavior.
- Set an explicit acceptable error threshold and stop or redesign the workflow if the system cannot meet it.
Common pitfalls
- Quoting one hallucination percentage as if it represented a model’s accuracy across every task.
- Comparing rates from different benchmarks without accounting for differences in task, scoring method, and refusal behavior.
- Assuming higher reasoning effort always produces more factual answers.
- Believing that adding retrieval automatically removes hallucination risk.
- Choosing a model only by hallucination rate without checking how often it refuses to answer.
- Using document-summary benchmarks to predict performance on open-domain search, news, legal, or medical tasks.
- Trusting citations, package names, or references generated by a model without independent verification.
- Testing only ordinary examples instead of adversarial, ambiguous, or high-risk cases.
- Deploying high-stakes AI workflows without human review or a clear escalation path.
- Relying on public leaderboards without validating the model on your own data, prompts, and production conditions.
Short answer: how often does AI hallucinate? It depends on the task. On document summaries, the 105 models on the Vectara hallucination leaderboard add unsupported content to 1.8% to 24.2% of summaries. When AI search tools answer questions about the news, 37% to 94% of their answers are wrong (CJR Tow Center, 2025). This Uvik Software report puts the verified 2026 data in one place.
Key findings (Uvik Software analysis, October 2026)
- The lowest summary hallucination rate on the Vectara leaderboard is 1.8% (antgroup finix_s1_32b). The highest is 24.2% (Mistral ministral-3-3b).
- More reasoning does not cut hallucinations. In 6 same-model pairs, the higher-effort setting scored 0.98 percentage points worse on average. It never scored better (Uvik Software reasoning-tax analysis).
- OpenAI o3 hallucinated on 33% of PersonQA questions, twice the rate of o1 (16%). o4-mini reached 48%.
- Retrieval does not remove the risk. Legal research tools that use retrieval hallucinated 17% to 33% of the time.
- Code models invent dependencies. 19.7% of 2.23 million package references in generated code did not exist.
- AI hallucinations reached courts 2,149 times by 5 October 2026. Self-represented litigants filed 1,233 of these cases and lawyers 854.
- A short mitigation prompt cut adversarial clinical hallucinations from 66% to 44% on average.
Get the data. Download the numbers in this report, with their sources, as a CSV file. You can reuse it with credit to Uvik Software (CC BY 4.0).
What is an AI hallucination?
An AI hallucination is an output that a model presents as true, but that its sources or the facts do not support. Common examples are a fake court citation, a code package that does not exist and a summary that adds a detail the document does not contain.
Researchers use two main types:
- Unsupported content. The model adds information that is not in the source.
- Contradicting content. The model states something that conflicts with the source.
In this report, “hallucination rate” means the share of outputs that contain at least one hallucination, as each benchmark defines it. The definitions are different, so compare numbers only inside one benchmark. For the difference between generative and agentic systems, see agentic AI vs generative AI.
AI hallucination rates in 2026 at a glance
In the studies in this report, AI hallucination rates run from 1.8% to 94%. The task decides: summaries of a given document have the lowest rates, and AI search answers about the news have the highest.
Figure 1. Published AI hallucination and error rates by task, from 1.8% to 94%. Source: Uvik Software compilation of eight studies and leaderboards, October 2026.
| Task | What the study measures | Rate | Source and date |
|---|---|---|---|
| Document summaries | Summaries with unsupported content, 105 models | 1.8% to 24.2% | Vectara leaderboard, 11 May 2026 |
| Package names in generated code | Package references that do not exist, 16 models | 5.2% (commercial average) to 21.7% (open-source average) | Spracklen et al., USENIX Security 2025 |
| Questions about people | OpenAI PersonQA hallucination rate | 16% (o1) to 48% (o4-mini) | OpenAI o3 and o4-mini system card, 16 April 2025 |
| Legal research | Legal AI research tools that use retrieval | 17% to 33% | Magesh et al., Journal of Empirical Legal Studies, 2025 |
| False news claims | Share of chatbot answers that repeat false claims about the news | 35% (August 2025), up from 18% (August 2024) | NewsGuard, 4 September 2025 |
| News answers | AI assistant answers with at least one significant issue | 45% | EBU and BBC, 22 October 2025 |
| AI search about news | Wrong answers from 8 AI search tools | 37% (Perplexity) to 94% (Grok 3) | CJR Tow Center, 6 March 2025 |
| Clinical cases | Cases where the model repeated or expanded a planted false detail, average of 6 models | 66% (44% with a mitigation prompt) | Omar et al., Communications Medicine, 2025 |
Why do published hallucination rates differ so much?
Each benchmark measures a different thing. A summary benchmark checks if the output stays inside one document. A knowledge benchmark checks if the model guesses when it does not know. So one model can get very different scores on two tests.
Three questions explain most of the difference:
- What is the task? It is easier to stay accurate when you summarize a given document than when you answer open questions from memory.
- What counts as an error? Some benchmarks count any unsupported detail. Other benchmarks count only a wrong final answer.
- Can the model refuse? A model that refuses more often can show a lower hallucination rate. Always read the answer rate next to the hallucination rate.
| Benchmark | What it measures | Data | Direction | Coverage |
|---|---|---|---|---|
| Vectara Hallucination Leaderboard | Share of summaries that contain content the source does not support | HHEM-2.3 judge model, more than 7,700 articles of 50 to 24,000 words | Lower is better | 105 models (11 May 2026) |
| AA-Omniscience (Artificial Analysis) | Incorrect answers divided by all answers that are not correct (incorrect, partial or not attempted) | 6,000 questions in 6 domains and 42 topics | Lower is better | Live leaderboard |
| FACTS (Google DeepMind, Google Research and Kaggle) | Accuracy across grounding, multimodal, closed-book and search tasks | 4 sub-benchmarks | Higher is better. It is not a hallucination rate. | 15 models in the December 2025 paper |
| PersonQA and SimpleQA (OpenAI) | Hallucination rate on fact questions | OpenAI internal tests | Lower is better | OpenAI system cards |
Rule: compare models inside one benchmark. Do not compare a Vectara rate with an AA-Omniscience rate.
Which AI hallucinates the least? The 2026 leaderboard
On the Vectara Hallucination Leaderboard (11 May 2026, 105 models), antgroup finix_s1_32b has the lowest summary hallucination rate at 1.8%. Among models from the large labs, OpenAI gpt-5.4-nano (3.1%) and Google gemini-2.5-flash-lite (3.3%) rank highest.
Figure 2. Hallucination leaderboard: 20 of 105 models on the Vectara leaderboard (11 May 2026). Selection and chart: Uvik Software.
| Model | Developer | Hallucination rate | Answer rate |
|---|---|---|---|
| finix_s1_32b | Ant Group | 1.8% | 99.5% |
| gpt-5.4-nano | OpenAI | 3.1% | 100.0% |
| gemini-2.5-flash-lite | 3.3% | 99.5% | |
| Phi-4 | Microsoft | 3.7% | 80.7% |
| Llama-3.3-70B-Instruct-Turbo | Meta | 4.1% | 99.5% |
| DeepSeek-V3 | DeepSeek | 6.1% | 97.5% |
| gpt-5.4 | OpenAI | 7.0% | 99.9% |
| Llama-4-Scout-17B-16E-Instruct | Meta | 7.7% | 99.0% |
| DeepSeek-V4-Pro | DeepSeek | 8.6% | 97.2% |
| gpt-5.5 | OpenAI | 9.3% | 100.0% |
| claude-haiku-4-5 | Anthropic | 9.8% | 99.5% |
| gemini-3.1-pro-preview | 10.4% | 99.4% | |
| claude-sonnet-4-6 | Anthropic | 10.6% | 99.9% |
| qwen3.5-plus | Alibaba | 10.7% | 99.8% |
| DeepSeek-R1 | DeepSeek | 11.3% | 97.0% |
| claude-opus-4-7 | Anthropic | 12.0% | 98.0% |
| mistral-3-large | Mistral | 14.5% | 98.8% |
| grok-4-1-fast-reasoning | xAI | 19.2% | 99.7% |
| o3-pro | OpenAI | 23.3% | 100.0% |
| ministral-3-3b | Mistral | 24.2% | 74.3% |
Read the answer rate with care. Microsoft Phi-4 answered 80.7% of the tasks, so it skipped about 1 in 5 summaries. A low hallucination rate with a low answer rate can give a false picture.
Some small and fast models score well on this test. One reason is length: a shorter summary makes fewer claims that can be wrong. On the leaderboard, gpt-5.1 at high effort wrote 254.4 words per summary on average, against 165.5 words at low effort.
Do reasoning models hallucinate more?
Often, yes. In the Uvik Software analysis of six same-model pairs on the Vectara leaderboard, the higher reasoning setting had a higher or equal hallucination rate every time: 0.98 percentage points higher on average. OpenAI’s own tests show the same pattern: o3 hallucinated on 33% of PersonQA questions, against 16% for o1.
Figure 3. The reasoning tax: the same model at lower and higher reasoning effort on the Vectara leaderboard. Source: Uvik Software analysis, 11 May 2026 data.
| Model | Lower setting | Higher setting | Change |
|---|---|---|---|
| xAI grok-4-1-fast | non-reasoning: 17.8% | reasoning: 19.2% | +1.4 pts |
| xAI grok-4-fast | non-reasoning: 19.7% | reasoning: 20.2% | +0.5 pts |
| OpenAI gpt-5.2 | low: 8.4% | high: 10.8% | +2.4 pts |
| OpenAI gpt-5.1 | low: 10.9% | high: 12.1% | +1.2 pts |
| OpenAI gpt-5 | minimal: 14.7% | high: 15.1% | +0.4 pts |
| OpenAI o4-mini | low: 18.6% | high: 18.6% | 0.0 pts |
Why does this happen? OpenAI’s system card for o3 says the model “tends to make more claims overall”. More claims give more chances to be wrong. The same pattern shows in the DeepSeek family: the reasoning model DeepSeek-R1 scores 11.3% on the summary test, against 6.1% for DeepSeek-V3.
What to do: do not turn on maximum reasoning by default for summaries or data extraction. Test each setting on your own data, then pick the lowest setting that passes your accuracy target. Uvik Software engineers run this test as part of LLM evaluation and observability work.
ChatGPT, Claude, Gemini, Grok and Perplexity hallucination rates
On the Vectara summary test (May 2026), vendor flagships score between 7% and 19%: OpenAI gpt-5.4 7.0%, gpt-5.5 9.3%, Google gemini-3.1-pro-preview 10.4%, Anthropic claude-sonnet-4-6 10.6%, claude-opus-4-7 12.0% and xAI grok-4-1-fast-reasoning 19.2%.
| Vendor | Model (Vectara, 11 May 2026) | Summary hallucination rate | Other verified data |
|---|---|---|---|
| OpenAI (ChatGPT) | gpt-5.4 / gpt-5.5 / o3-pro | 7.0% / 9.3% / 23.3% | PersonQA: o1 16%, o3 33%, o4-mini 48% |
| Anthropic (Claude) | claude-haiku-4-5 / claude-sonnet-4-6 / claude-opus-4-7 | 9.8% / 10.6% / 12.0% | |
| Google (Gemini) | gemini-2.5-flash-lite / gemini-3.1-pro-preview | 3.3% / 10.4% | |
| xAI (Grok) | grok-4-1-fast non-reasoning / reasoning | 17.8% / 19.2% | CJR test: Grok 3 answered 94% of news queries incorrectly |
| Perplexity | Not on the leaderboard | CJR test: 37% of news queries answered incorrectly | |
| DeepSeek | DeepSeek-V3 / V4-Pro / R1 | 6.1% / 8.6% / 11.3% |
Assistants make more errors on news. In the EBU and BBC study, 45% of answers from ChatGPT, Copilot, Gemini and Perplexity had at least one significant issue, and 20% had major accuracy issues. In NewsGuard’s August 2025 audit of 10 chatbots, the chatbots repeated false claims 35% of the time, up from 18% one year earlier.
AI hallucination rates by industry
Legal: 2,149 court cases and counting
AI hallucinations reached courts 2,149 times by 5 October 2026, according to the AI Hallucination Cases database. The United States has 1,473 of these cases. Self-represented litigants filed 1,233 of them and lawyers filed 854.
Figure 4. Court decisions that involve AI hallucinations, by country and by party. Source: AI Hallucination Cases database (Damien Charlotin), 5 October 2026.
Uvik Software calculation from the same database: 57.4% of cases involve self-represented litigants, 39.7% involve lawyers and 68.5% come from the United States.
Professional tools are not immune. A Stanford study found that legal research tools from LexisNexis and Thomson Reuters, which use retrieval, hallucinated 17% to 33% of the time. Two cases show the cost:
- Mata v. Avianca (2023). A US federal judge fined two lawyers and their firm $5,000 after they filed a brief with fake cases from ChatGPT.
- Deloitte Australia (2025). Deloitte agreed to partly refund the Australian government for an A$440,000 report that contained a fabricated court quote and references to research that does not exist.
Medicine
In a 2025 Mount Sinai study, 6 models answered 300 clinical cases that each contained one planted false detail. On average, the models repeated or expanded the false detail 66% of the time. A short mitigation prompt cut the average to 44%.
The lesson for health products is clear: the model trusts what the user writes. Add input checks and human review for any clinical claim. More data is in our AI in healthcare statistics report.
News and search
- EBU and BBC (October 2025): 45% of AI assistant answers about the news had at least one significant issue. 22 public broadcasters in 18 countries and 14 languages took part.
- NewsGuard (August 2025): the 10 leading chatbots repeated false news claims 35% of the time, up from 18% in August 2024.
- CJR Tow Center (March 2025): 8 AI search tools answered more than 60% of queries incorrectly. Perplexity had the lowest error rate (37%) and Grok 3 the highest (94%).
Software development: package hallucination and slopsquatting
Code models invent packages. A USENIX Security 2025 study of 16 models found that 19.7% of 2.23 million package references did not exist, with 205,474 unique fake names. Commercial models averaged 5.2% and open-source models 21.7%.
Attackers can register these fake names and fill them with malware. This attack is called slopsquatting. A developer who installs a suggested package without a check can install the attacker’s code. Check every new dependency against an allow-list before install. Our Python AI Dependency Security Index and AI Coding Agent Security Benchmark show how to score this risk. For adoption data, see AI code generation statistics.
The workplace
56% of employees say they made mistakes in their work because of AI, in a KPMG and University of Melbourne survey of more than 48,000 people in 47 countries (2025). Errors from AI are now an everyday operational risk, not only a research topic.
How to reduce AI hallucinations: the control stack
No single fix removes hallucinations. Uvik Software engineers combine six layers: grounding, cautious instructions, abstention, automatic verification, human review and production monitoring. Each layer catches errors that the other layers miss.
Figure 5. The hallucination control stack: six layers that Uvik Software engineers use in production LLM applications.
- Ground answers in your own sources. Use retrieval-augmented generation (RAG) on approved data. RAG reduces the risk but does not remove it: legal RAG tools still hallucinated 17% to 33% of the time. See our RAG development services and the LlamaIndex vs LangChain comparison.
- Instruct the model to be cautious. Tell the model to use only the given context and to flag doubtful input. In the Mount Sinai study, one mitigation prompt cut the average hallucination rate from 66% to 44%.
- Reward “I don’t know”. Let the model refuse when the evidence is weak. OpenAI research from September 2025 explains that common training and evaluation methods reward guessing over admitting uncertainty.
- Verify automatically. Check each citation against your source index. Check each package name against an allow-list. Use a judge model to score summaries before a user sees them.
- Route risk to humans. Send legal, medical, financial and customer-facing claims to a human reviewer. Read our guide to human-in-the-loop AI.
- Monitor in production. Run a fixed evaluation set on every release and track the hallucination rate over time. Our AI production failure database shows what happens when teams skip this step.
Measure your own hallucination rate. Leaderboards test generic data. Your risk depends on your documents, prompts and users. Uvik Software engineers build an evaluation set from your data, add verification steps and track the hallucination rate on every release.
Talk to an LLM evaluation engineer | Fix an AI app that gives wrong answers
How to measure the hallucination rate of your own LLM app
- Collect 200 to 500 real questions or documents from your product.
- Write the correct answer, or list the allowed sources, for each item.
- Run your application and store each output with its retrieved context.
- Score each output with a judge model. Check a random sample by hand.
- Calculate two numbers: the hallucination rate (outputs with at least one unsupported claim, divided by all outputs) and the answer rate.
- Repeat the test on every model, prompt or retrieval change. Stop the release if the hallucination rate goes up.
AI agents add a new risk: a hallucinated fact can trigger a wrong action. See the verified data in our AI agent statistics 2026 report. If you serve your own models, the GPU Price Index shows what each extra evaluation run costs.
Methodology
- Sources. Uvik Software collected leaderboard snapshots, peer-reviewed studies, independent audits and official system cards published up to 7 October 2026.
- Verification rule. Every statistic in this report is either verified in at least three sources (the primary publisher and two independent outlets) or taken directly from a primary dataset (a leaderboard, a database or a system card) with its snapshot date. Uvik Software analyses are marked as such.
- Reasoning tax. Uvik Software matched six model pairs on the Vectara leaderboard (11 May 2026) that differ only in reasoning setting. We calculated the change in hallucination rate for each pair and the mean change.
- Update cycle. We refresh this report every month. Next update: 7 November 2026.
How to cite this report
Uvik Software (2026). AI hallucination rates in 2026: leaderboards, domain data and what reduces them. Data as of 7 October 2026. https://uvik.net/blog/ai-hallucination-rates/
You can reuse the charts and the dataset under CC BY 4.0. Please credit “Uvik Software” with a link to this page.
Sources
- Vectara. Hallucination Leaderboard, last updated 11 May 2026. https://github.com/vectara/hallucination-leaderboard
- Artificial Analysis. AA-Omniscience. https://artificialanalysis.ai/evaluations/omniscience
- Stanford HAI. AI Index Report 2026, chapter 3. https://hai.stanford.edu/assets/files/ai_index_report_2026_chapter_3_responsible_ai.pdf
- Google DeepMind, Google Research and Kaggle. The FACTS Leaderboard, December 2025. https://arxiv.org/abs/2512.10791
- OpenAI. OpenAI o3 and o4-mini System Card, 16 April 2025. https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf
- OpenAI. Why language models hallucinate, 5 September 2025. https://openai.com/index/why-language-models-hallucinate/
- Spracklen et al. We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. USENIX Security 2025. https://arxiv.org/abs/2406.10279
- Magesh et al. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies, 2025. https://nlp.stanford.edu/~manning/papers/Magesh-Hallucination%E2%80%90Free-2025.pdf
- Omar et al. Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support. Communications Medicine, 2025. https://cris.tau.ac.il/en/publications/multi-model-assurance-analysis-showing-large-language-models-are-/
- NewsGuard. AI False Claim Monitor, August 2025. https://www.newsguardtech.com/ai-monitor/august-2025-ai-false-claim-monitor/
- European Broadcasting Union. News Integrity in AI Assistants, 22 October 2025. https://www.ebu.ch/news/2025/10/ai-s-systemic-distortion-of-news-is-consistent-across-languages-and-territories-international-study-by-public-service-broadcaste
- Columbia Journalism Review, Tow Center. AI search has a citation problem, 6 March 2025. https://www.cjr.org/tow_center/we-compared-eight-ai-search-engines-theyre-all-bad-at-citing-news.php
- Damien Charlotin. AI Hallucination Cases database, last updated 5 October 2026. https://www.damiencharlotin.com/hallucinations/
- KPMG and University of Melbourne. Trust, attitudes and use of AI: a global study 2025. https://kpmg.com/xx/en/our-insights/ai-and-technology/trust-attitudes-and-use-of-ai.html
- Associated Press. Deloitte to partially refund Australian government for report with apparent AI-generated errors, 7 October 2025.
- ABA Journal. Lawyers who doubled down and defended ChatGPT’s fake cases must pay $5K, judge says, June 2023. https://abajournal.com/web/article/lawyers-who-doubled-down-and-defended-chatgpts-fake-cases-must-pay-5k-judge-says
Build AI features that stay accurate. Uvik Software provides senior Python and AI engineers: more than 50 engineers, no juniors, and at least 7 years of experience each. You get matched profiles 48 hours after the SOW and a 30-day no-cost replacement. Rates: $50 to $99 per hour. Uvik Software is a member of the Claude Partner Network.
