Why Does ChatGPT Make Up Sources? 6 Causes and Fixes
A specific citation — author, title, journal, year, volume, pages — is a rare string that appears in very few training documents, and language models memorise rare things badly. Nothing in ordinary text generation looks anything up, so the model produces the most plausible-looking reference rather than a real one, in exactly the same format and with exactly the same confidence as a real one. Six mechanisms drive it. Each has a fix, and none of the fixes is prompting alone.
The measured rates below come from studies of GPT-3.5, GPT-4, PaLM 2 and Llama 2 published in 2023 and 2024. Current models are better. They are not fixed, and the mechanisms have not changed.
Key Takeaways
- 55% of GPT-3.5's citations and 18% of GPT-4's were fabricated across 636 citations in 84 generated literature reviews (Walters & Wilder, Scientific Reports, 2023, SOURCED).
- Real citations were wrong too — 43% of GPT-3.5's non-fabricated citations and 24% of GPT-4's contained substantive errors. A real paper with the wrong year still fails a reference check.
- The model often does not know it is doing it. A study of over 800,000 legal queries found models "cannot always predict, or do not always know, when they are producing legal hallucinations" (Dahl et al., 2024, SOURCED).
- The root mechanism is long-tail memorisation — accuracy on a fact tracks how many training documents contained it, and one paper's exact citation string is about as long-tail as facts get.
- The only reliable fix is external: give it the source, make it search, or open every reference yourself.
The measured scale
Two studies are worth knowing by number.
Walters and Wilder had ChatGPT-3.5 and ChatGPT-4 produce short literature reviews on 42 multidisciplinary topics, then checked all 636 citations in the resulting 84 papers against databases. 55% of the GPT-3.5 citations and 18% of the GPT-4 citations were fabricated. Among the citations that pointed at real papers, 43% of GPT-3.5's and 24% of GPT-4's contained substantive errors — wrong author, wrong date, wrong journal, wrong volume or pages (Walters & Wilder, "Fabrication and errors in the bibliographic citations generated by ChatGPT", Scientific Reports, 2023, retrieved 2026-09-06, SOURCED).
Dahl and colleagues ran more than 800,000 queries about real federal court cases across four models. Hallucinations occurred "between 58% of the time with ChatGPT 4 and 88% with Llama 2, when these models are asked specific, verifiable questions about random federal court cases" (Dahl et al., "Large Legal Fictions", arXiv:2401.01301, 2024, retrieved 2026-09-06, SOURCED).
Both studies test models from 2023. Read them as evidence about the mechanism and its scale, not as this month's error rate.
Cause 1 — A citation is a long-tail fact
Model accuracy on a fact tracks how often that fact appeared in training. Kandpal and colleagues showed "a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training", across multiple datasets, corpora and model sizes up to 176B parameters. They also estimated that "today's models must be scaled by many orders of magnitude to reach competitive QA performance on questions with little support in the pre-training data" (Kandpal et al., arXiv:2211.08411, ICML 2023, retrieved 2026-09-06, SOURCED).
A specific reference string is close to the definition of a long-tail fact. The paper's title may appear thousands of times; its exact volume and page range appears in a handful of reference lists. So the model holds the shape of the citation and not the details.
The fix. Do not ask for citations from memory on anything niche. Supply the sources you want cited, or make the model retrieve them.
Cause 2 — The format is trivially imitable
A reference has a rigid, highly predictable structure. Author surname, initials, year in parentheses, title, journal in italics, volume, issue, pages. A model that has seen millions of them can generate a syntactically perfect one for a paper that does not exist, in a plausible journal, by plausible authors, in a plausible year.
Stated honestly, this one is our inference, not a finding. We looked for a study isolating format regularity — or the tokenisation of identifiers like DOIs and arXiv IDs — as a cause of citation fabrication specifically, and did not find one (ANALYSIS, searched 2026-09-06). The long-tail mechanism in cause 1 is documented. "The format is easy to fake" is a reasonable reading of why the output looks so convincing, and we are not going to dress it up as research.
The fix. Stop treating a well-formed citation as a signal of anything. Format is the one property a model can always get right.
Cause 3 — Nothing in generation checks existence
Text generation predicts likely continuations. There is no step in that process where the model queries a database to confirm a paper exists, unless a tool is explicitly wired in and actually called.
This is why the confident tone is not a bug. The model is not asserting that a reference exists; it is producing the text that would most likely follow your request, and confident text is what follows a request for citations.
The fix. Add the step the model does not have. Either retrieval — grounding the answer in documents supplied at answer time, the technique with the strongest published support — or a post-hoc pass. RARR is the clearest example of the second: a system that "automatically finds attribution for the output of any text generation model" and then "post-edits the output to fix unsupported content" (Gao et al., arXiv:2210.08726, ACL 2023, retrieved 2026-09-06, SOURCED). Note what that is — a retrofit after generation, not a prevention.
Cause 4 — Search is conditional, not automatic
When search is on, the model may look things up. It does not always. Google's own API documentation bills per search query "that the model decides to execute" (Gemini API grounding documentation, retrieved 2026-09-06, SOURCED), which confirms the decision sits with the model rather than with you.
So "search was enabled" is not the same as "this answer was retrieved". Some answers in a searching session still come from memory.
The fix. Check whether the answer actually cites a retrieved page, and open it. An answer with no citation in a search-enabled session is a recall answer wearing a retrieval badge.
There is a more selective version of this in the research: Self-RAG trains a model to decide when to retrieve and to critique its own output with special tokens, reporting gains in "factuality and citation accuracy for long-form generation" (Asai et al., arXiv:2310.11511, 2023, retrieved 2026-09-06, SOURCED). That is a research system, not a setting in a consumer product.
Cause 5 — The model cannot reliably tell when it is fabricating
If it knew, it could warn you. Dahl and colleagues' fourth finding is exactly this: models "cannot always predict, or do not always know, when they are producing legal hallucinations" (arXiv:2401.01301, retrieved 2026-09-06, SOURCED).
This is why "are you sure?" is such a weak check. You are asking a system to introspect on something it does not have reliable access to, in a session where it has already committed to the answer.
The fix. Detect from the outside instead. SelfCheckGPT works by sampling several responses and comparing them — facts the model actually holds tend to survive resampling, fabricated ones vary (Manakul et al., arXiv:2303.08896, EMNLP 2023, retrieved 2026-09-06, SOURCED). Note the limit clearly: that is detection, not correction. It flags where to look; it does not fix anything.
The manual version costs nothing. Ask for the same citation list three times in three fresh sessions. The references that change between runs are the ones to check first.
Cause 6 — The near-misses are worse than the inventions
An invented paper fails the first search you run. A real paper with the wrong year, wrong journal or wrong page range passes a careless check, gets copied into your reference list, and fails at peer review, in court, or in front of the one reader who looks it up.
The numbers make this the bigger exposure: 43% of GPT-3.5's real citations and 24% of GPT-4's carried substantive errors, against 55% and 18% fabrication (Walters & Wilder, above). Roughly as much of the risk sits in citations that are almost right.
The fix. Check the whole record, not existence. Author list, year, journal, volume, pages — against the publisher's own page or the DOI, not against a search-result snippet that may itself be repeating an error.
The one-minute version
For any AI-produced reference list:
- Open every citation. Every one. There is no sampling shortcut when the failure rate is this high.
- Check the details, not just that something exists with a similar title.
- Prefer supplying sources over asking for them.
- If search was on, confirm the answer actually cites a retrieved page.
- Regenerate the list twice and check whatever changed.
We put the full pass into a 42-check fact-check checklist; gate 5 and gate 7 cover this ground.
How This Guide Was Sourced
Written and maintained by the LogicBalls editorial team (logicballs.com). Disclosure: LogicBalls builds AI writing tools. We sell software that produces text like the text this post warns you to check, which is why cause 2 says plainly where our explanation is inference rather than research.
Sources. Citation fabrication rates: Walters & Wilder, Scientific Reports, 2023, read via the PubMed Central copy. Legal hallucination rates and the self-knowledge finding: Dahl et al., arXiv:2401.01301, Journal of Legal Analysis 2024. Long-tail memorisation: Kandpal et al., arXiv:2211.08411, ICML 2023. Post-hoc attribution: Gao et al., arXiv:2210.08726, ACL 2023. Adaptive retrieval: Asai et al., arXiv:2310.11511. Detection by resampling: Manakul et al., arXiv:2303.08896, EMNLP 2023. Conditional retrieval: Google's Gemini API grounding documentation. All retrieved 2026-09-06; every abstract quoted above was read at source.
What could not be fetched. OpenAI's help documentation returned HTTP 403 to automated requests, so no claim here is sourced to OpenAI material. Statements about ChatGPT's behaviour come from the studies named above, which tested it, and are described as such.
What we looked for and did not find. A study isolating citation format regularity, or the tokenisation of DOI and arXiv identifiers, as a cause of citation fabrication. Cause 2 is labelled ANALYSIS for that reason rather than being attached to a paper that does not say it.
Model versions matter here. Every rate above was measured on 2023-era models. They are the best public evidence on the mechanism and they are not a current error rate for any model you are using today.
No LogicBalls telemetry is used in this guide.
Frequently Asked Questions
Do newer models still make up sources?
Yes, at lower rates. The GPT-3.5 to GPT-4 comparison in the study above shows fabrication falling from 55% to 18% in one model generation — a large improvement and nowhere near zero.
Does turning on web search fix it?
It helps a great deal and it does not close the gap. Search is conditional, the model decides when to run it, and retrieval can surface a wrong page and repeat it confidently. Open what it cites.
Why does it apologise and then produce another fake citation?
Because the apology and the next citation come from the same process. Nothing about being corrected gives the model the reference it did not have.
Is asking for DOIs safer?
Slightly, because a DOI is easy to check, which makes your verification faster. It does not make the model less likely to produce one that resolves to nothing or to a different paper.
What is the single highest-value habit?
Paste the source in and ask for citations only from what you supplied. That converts recall into reading, which is the difference the research keeps finding.
