Why AI Makes Up Statistics (And How to Get Real Ones Instead)
An AI statistic fails in three ways, and only one of them is invention. A model can produce a number that does not exist; it can alter a real number while summarising; or — the one that gets published — it can state a real, correctly-quoted number and attach it to a claim the source never made. The third survives a search, survives a spot-check, and survives most editing, because the number is right. Only opening the source and reading the sentence it came from catches it.
Here is what causes each, what the research measures, and the workflow that produces numbers you can stand behind.
Key Takeaways
- The dangerous failure is misattribution, not invention. A fabricated number fails the first search. A real number attached to the wrong claim gets published.
- We did it ourselves. Our audit found a consultancy's finding that AI made a task 35–45% faster restated as AI automating 45% of coding tasks. Same number, different claim (ANALYSIS, audit 2026-09-02/03).
- Summarisation alters quantities, which is why a research system exists specifically to verify "quantity entities (dates, numbers, sums of money)" against the source document (Zhao et al., EMNLP Findings 2020, SOURCED).
- Rare facts are memorised badly. Model accuracy tracks how many training documents contained the fact — and a specific figure from a specific report is rare by construction.
- A usable statistic has six parts. Number, unit, population, date, method, source. A number with fewer than six is not yet a statistic.
The three failures, in order of how much trouble they cause
1. Invented
A number that does not exist anywhere. "73% of marketers report…" with no study behind it.
This one is loud. It fails the first search, it has no source to open, and any reviewer who looks finds nothing. It is embarrassing rather than dangerous.
2. Altered
A real number changed during summarising or rewriting — 3.2 becomes 3.5, a 2019 figure gets a 2024 date, millions become billions.
This is documented well enough that a system was built for it. Zhao, Cohen and Webber's Herman "learns to recognize and verify quantity entities (dates, numbers, sums of money, etc.)" in generated summaries and up-ranks the summaries whose quantities are supported by the original text (Zhao et al., arXiv:2009.13312, Findings of EMNLP 2020, retrieved 2026-09-06, SOURCED). The existence of the problem is the point: quantities specifically drift when text is regenerated.
3. Misattributed
The number is real. The quote is accurate. The claim it is attached to is not the claim the source made.
This is the one that reaches published pages, and we have a live example from our own corpus. A consultancy finding about developers completing tasks 35–45% faster with AI assistance appeared in one of our posts as AI "automating 45% of coding tasks". Same figure, entirely different assertion — speed of a human doing the work versus work no human does at all (ANALYSIS, audit of 80 posts, 2026-09-02/03).
Every check except one passes on that sentence. The number is real, the source exists, the organisation is correctly named. Only reading the sentence in the source catches it.
Why it happens
Rare facts are memorised badly. Kandpal and colleagues showed "a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training", and estimated that models "must be scaled by many orders of magnitude to reach competitive QA performance on questions with little support in the pre-training data" (Kandpal et al., arXiv:2211.08411, ICML 2023, retrieved 2026-09-06, SOURCED).
A statistic from a specific report is exactly that kind of fact. The shape of it — an organisation, a percentage, a year — appears constantly. The specific pairing appears rarely. So the model reliably produces the shape and unreliably produces the pairing, which is precisely the profile of a misattributed statistic.
Nothing in generation verifies. Text generation predicts likely continuations. There is no lookup step confirming a figure against a source unless retrieval or a tool is wired in and actually runs.
Plausible numbers are the easiest thing to generate. Round-ish percentages between 30 and 80 are dense in training text. A model asked for a statistic about almost anything will produce one in that band, and it will look exactly like the real ones.
What is not well studied. We looked for a benchmark measuring misattribution specifically — a real number sourced to the wrong study — as distinct from invented or altered numbers, and did not find one (ANALYSIS, searched 2026-09-06). The category most likely to be published is the category least measured. We would rather say that than cite something that does not measure it.
The six parts of a usable statistic
Before a number goes into a draft, it needs all six. Missing any one and it is not yet a statistic.
| part | example | why it matters |
|---|---|---|
| Number | 43% | — |
| Unit | percent of citations | 43% of what changes the meaning entirely |
| Population | non-fabricated GPT-3.5 citations in the study | a rate is meaningless without its denominator |
| Date | measured 2023 | model and market figures rot in months |
| Method | 636 citations checked against databases | tells you what the number can support |
| Source | Walters & Wilder, Scientific Reports | the thing a reader opens |
Applied to this post's own numbers: 43% of the real GPT-3.5 citations in a 636-citation sample contained substantive errors, measured in 2023 (Walters & Wilder, Scientific Reports, 2023, retrieved 2026-09-06, SOURCED). That sentence carries all six, which is why it can be checked in about thirty seconds.
How to get real numbers instead
Ask for the claim, then find the number yourself. The most reliable workflow inverts the usual one: have the model tell you what kind of evidence would support your point and who publishes it, then go and get the figure from the publisher. You are using the model for the search space, not the fact.
Supply the source and constrain the output to it. Paste the report and require every number to come from it, with the sentence it came from quoted alongside. This converts recall into reading, and it is the intervention with the strongest published support.
Demand the sentence, not the citation. "Quote the sentence containing this figure" is a better instruction than "cite your source", because a citation can be right while the claim is wrong. The sentence is where misattribution becomes visible.
Refuse round numbers with no denominator. "Most companies", "the majority of marketers", "up to 90%" — if you cannot state the population, you do not have a statistic. Cut it or state the uncertainty in the text.
Check the direction of the error. In our own audit, price errors were not randomly distributed: nearly all of them made products look cheaper or more generous than they are. Errors that flatter the argument deserve a second look, because they are the ones nobody questions.
Then verify anyway. Gate 7 of our 42-check fact-check checklist is six checks on statistics alone, and it exists because these mistakes reached our own published pages.
What this costs
Roughly two to four minutes per statistic, most of it spent reading the sentence around the number. For a post with six figures in it, that is half an hour.
We spent considerably more than that correcting the ones we published without it. The 14 error types we found in our own posts put fabricated and misattributed statistics near the top, and the remediation cost more than the original writing did.
How This Guide Was Sourced
Written and maintained by the LogicBalls editorial team (logicballs.com). Disclosure: LogicBalls builds AI writing tools. The misattribution example above is from our own published work, not a competitor's.
Sources. Quantity hallucination in summarisation: Zhao, Cohen and Webber, arXiv:2009.13312, Findings of EMNLP 2020. Long-tail memorisation: Kandpal et al., arXiv:2211.08411, ICML 2023. Citation error rates used as the worked example of a complete statistic: Walters and Wilder, Scientific Reports, 2023, read via PubMed Central. All abstracts read at source on 2026-09-06 and linked inline.
What is ours. The misattribution example and the price-error direction come from our own audit of 80 published posts on 2026-09-02/03, marked ANALYSIS. We publish no rate for how often AI invents statistics, because we have not measured one.
What we looked for and did not find. A published benchmark measuring misattribution — a real figure sourced to the wrong study — as a distinct category from invented or altered numbers. The section above says so rather than borrowing a benchmark that measures something else.
What could not be fetched. OpenAI's help documentation returned HTTP 403 to automated requests. No claim here is sourced to OpenAI material.
No LogicBalls telemetry is used in this guide.
Frequently Asked Questions
How can I tell an invented statistic from a real one?
Search the exact figure with the organisation's name. Invented ones return nothing or return only content repeating the same AI-generated claim. That check takes seconds and catches the loudest failure.
Is it safe if the AI cites a source for the number?
No. A citation tells you a source exists, not that the source says what the sentence claims. Open it and read the sentence containing the figure.
Do newer models invent fewer statistics?
Error rates have fallen measurably between model generations. The mechanism has not changed, and misattribution in particular is not something a better model reliably fixes, because the number it produces is real.
What about asking it to double-check?
Weak. You are asking the same process that produced the number to audit itself in a session where it has already committed to the answer. Check externally instead.
Can I use a statistic I found in another blog post?
Only after tracing it to the original. Second-hand figures are how misattribution spreads — and the chain usually ends at a report that says something slightly different.
