Is ChatGPT Accurate? What the Published Research Measures

There is no accuracy figure for ChatGPT, and the reason is the answer to the question. OpenAI's own published table scores one model at 93.3 on MMLU and 48.6 on SimpleQA — the same model, two tasks. On summary faithfulness, GPT-class models measured on one public leaderboard span 3.1% to 23.3% hallucination. On specific questions about federal court cases, one study found hallucinations 58% of the time. All of those are real, and none of them is "ChatGPT's accuracy rate".

This post collects the published measurements for GPT-class models, states what each one measured, and shows why they cannot be combined. It is written against OpenAI's developer documentation and its simple-evals repository, Vectara's hallucination leaderboard as it stood on 11 May 2026, and six peer-reviewed or arXiv papers, all retrieved 2026-09-18.

Key Takeaways

• One model, two numbers, both from OpenAI. o3-high scores 93.3 on MMLU and 48.6 on SimpleQA (OpenAI, simple-evals, retrieved 2026-09-18).
• OpenAI stopped updating that table. A deprecation notice dated July 2025 says simple-evals "will no longer be updated for new models or benchmark results" (same source).
• Within one vendor's line, on one task, the range is 3.1% to 23.3% hallucination when summarising documents (Vectara hallucination leaderboard, last updated 11 May 2026, retrieved 2026-09-18).
• Three separate studies report 58% — of atomic facts supported, of questions answered truthfully, and of legal answers hallucinating. Same number, three different meanings.
• Every headline benchmark score is measured with no tools and no search. OpenAI's own footnote says its o3 and o4-mini results are "with no tools enabled" (same source).

What OpenAI publishes, and what it stopped publishing

OpenAI open-sourced its benchmark library "so we can be transparent about the accuracy numbers we're publishing alongside our latest models" (OpenAI, simple-evals, retrieved 2026-09-18, SOURCED). That repository is the primary record of what the company itself claims, and three things in it matter more than any individual score.

It is frozen. The README opens with a deprecation notice: "July 2025: simple-evals will no longer be updated for new models or benchmark results. The repo will continue to host reference implementations for HealthBench, BrowseComp, and SimpleQA" (same source, SOURCED). So the newest models in the table are from 2025, and no comparable published table covers what shipped after it.

It says its own older benchmarks are exhausted. Footnote 5 reads: "We believe these evals are saturated for our newer models, but are reporting them for completeness" (same source, SOURCED). MMLU and MGSM are in that group.

The scores are measured without tools. Footnote 10 states the o3 and o4-mini results are "with no tools enabled" (same source, SOURCED). No web search, no file search. That is the opposite of how most people use the product, and it is the single most important caveat on this page (ANALYSIS).

For model facts as they stand today, the developer documentation lists gpt-6-astra with a knowledge cutoff of Apr 30, 2026, and the three gpt-5.6 models at Feb 16, 2026 (OpenAI, models documentation, retrieved 2026-09-18, SOURCED). Pin your reading: those rows change whenever a model ships.

The same model, two scores

Read across a row of OpenAI's table rather than down a column, and the question "is it accurate" stops having an answer.

Source: OpenAI, simple-evals, retrieved 2026-09-18, SOURCED.

MMLU is multiple-choice knowledge across many subjects. SimpleQA is short, fact-seeking questions with "a single, indisputable answer", graded correct, incorrect or not attempted, and deliberately "adversarially collected against GPT-4 responses" (Wei et al., arXiv:2411.04368, retrieved 2026-09-18, SOURCED).

The gap is not a contradiction. A model can recognise the right answer among four options far more often than it can produce a specific fact unprompted — and SimpleQA was built from questions that defeated an earlier model, so a low score is by design (ANALYSIS). What the gap does mean is that a vendor quoting "90%+" has chosen a column. The wider version of that argument is in what "99% accurate" actually means.

The published measurements, with what each one measured

All retrieved 2026-09-18, SOURCED. Two rows carry limits worth stating in the table's own terms: TruthfulQA tested "GPT-3, GPT-Neo/J, GPT-2 and a T5-based model", so it is not a measurement of any current model, and FreshQA reports behaviour by question type rather than one headline figure.

We have not averaged these and will not. The tasks differ, the datasets differ, the graders differ, and averaging them would produce a number that describes nothing.

Three studies, three 58%s

The coincidence is useful, because it shows what a bare percentage is worth.

• FActScore: ChatGPT achieved 58% — the share of atomic facts in generated biographies supported by a reliable source (Min et al., arXiv:2305.14251, EMNLP 2023, retrieved 2026-09-18, SOURCED).
• TruthfulQA: the best model tested was truthful on 58% of questions, against 94% for humans (Lin et al., arXiv:2109.07958, ACL 2022, retrieved 2026-09-18, SOURCED).
• Legal hallucinations: hallucinations occurred 58% of the time with ChatGPT 4 on "specific, verifiable questions about random federal court cases" (Dahl et al., arXiv:2401.01301, retrieved 2026-09-18, SOURCED).

One is a success rate on facts, one a success rate on resisting misconceptions, one a failure rate on a hard legal task. A percentage with no task attached is not evidence (ANALYSIS). That the three coincide is chance, and we are not aware of any relationship between them.

One vendor, one task, a sevenfold spread

Vectara's leaderboard holds the cleanest like-for-like comparison available, because every model does the identical job: summarise a document using only the facts in it. Method, stated by its maintainers: over 7,700 articles from news, technology, science, medicine, legal, sports, business and education, lengths from 50 to 24,000 words, scored by HHEM-2.3, at temperature 0, with the dataset kept "not publicly available to avoid overfitting by any LLM" (Vectara hallucination leaderboard, last updated 11 May 2026, retrieved 2026-09-18, SOURCED).

Across the 19 OpenAI models listed on 18 September 2026:

Selected rows; source as above, SOURCED.

Two things follow. The spread within one vendor's own line is sevenfold, so "which model" matters more than "which brand" for this task. And the ordering is not the marketing ordering — the smallest model on the list is the most faithful summariser, and the heavier reasoning variants sit at the bottom (ANALYSIS).

The leaderboard's maintainers are explicit about its scope: they are "not evaluating the quality of the summaries, only the factual consistency", and it "does not definitively measure all the ways a model can hallucinate" (same source, SOURCED). A good score here says the model stays faithful to a document in front of it. It says nothing about what it does with no document at all.

What none of these numbers tell you

Which model answered you. The product routes, and the interface does not always name the model behind a given reply. Every number above is attached to a specific model version.

Whether the answer used search. Retrieval is conditional, and benchmark scores here were measured with tools off. Why that matters, and what to do about it, is in why AI gives outdated answers.

Anything about your work. None of these datasets is your documents, your customers or your subject. A score on somebody else's test set does not transfer to your workflow.

Whether a wrong answer will be caught. That depends on your process, not the model. The reader-side version takes about two minutes: how to check if ChatGPT is right. The publication-side version is the 42-check fact-check checklist, and the causes behind fabricated references are in why ChatGPT makes up sources.

What to do instead of looking for the number

1. Ask what the number is a rate of. Task, dataset, grader, date, model version. Without all five it is decoration.
2. Match the benchmark to your job. Summarising supplied documents is a different risk from answering unsourced questions, and different published numbers govern each.
3. Run your own ten prompts. Any published figure is somebody else's task; twenty minutes on yours is worth more. The procedure is in the 10-prompt AI accuracy test.
4. Re-check when a model ships. The rows above carry dates because they expire.

For the same exercise done on Google's documentation rather than OpenAI's research, see Gemini accuracy: what Google's own documentation says. The general mechanics of reducing wrong answers are in how to stop ChatGPT from making things up.

How This Guide Was Sourced

Written and maintained by the LogicBalls editorial team (logicballs.com). Disclosure: LogicBalls builds AI writing tools. We route to GPT-class and other models, so we have an interest in this subject. Nothing here is a claim about our own accuracy: we publish no benchmark score for our own product, and a post arguing that unattached percentages are meaningless would be a poor place to add one.

AI involvement. AI assisted with research and drafting. Every figure, quotation and footnote above was fetched and read at its primary source on 2026-09-18 — OpenAI's repository and developer documentation, the leaderboard README, and each paper's abstract or record page. No figure is carried from a summary, a roundup or memory.

Sources. OpenAI simple-evals (deprecation notice, benchmark table, footnotes 5 and 10) and OpenAI's developer models documentation; Vectara's hallucination leaderboard, last updated 11 May 2026; Wei et al. (SimpleQA), Min et al. (FActScore), Lin et al. (TruthfulQA), Li et al. (HaluEval), Dahl et al. (legal hallucinations) and Vu et al. (FreshQA).

What could not be read. OpenAI's help centre returns HTTP 403 to automated requests, so nothing here is sourced to it — including how the product decides when to search, which is why that question is listed as unanswered rather than answered. The leaderboard's underlying dataset is deliberately not published, so its scores cannot be reproduced independently.

What is not claimed. No single accuracy figure for ChatGPT, no average across benchmarks, no ranking of vendors, and no measurement of our own tools. Where this post reasons beyond a source — that the MMLU-to-SimpleQA gap is a format effect, that the three 58% results are coincidence, that model size does not order summary faithfulness — the passage is marked ANALYSIS.

No LogicBalls telemetry is used in this guide. Every figure above is external and linked.

Frequently Asked Questions

So is ChatGPT accurate or not?

At what? On summarising a document it is given, published rates for GPT-class models run from about 3% to 23% hallucination depending on the model. On short fact-seeking questions with no tools, OpenAI's own table runs from 7.6 to 62.5 correct. On specific federal court questions, one study measured 58% hallucination. Pick the task that matches yours.

What is ChatGPT's hallucination rate?

There is no single one, and any source quoting one without naming a task and dataset has dropped the part that makes it meaningful. The nearest thing to a like-for-like figure is the summarisation leaderboard, which measures one task only.

Does a higher benchmark score mean fewer made-up facts?

Not reliably. On the summarisation leaderboard the heavier reasoning variants score worse than a nano model, and OpenAI's headline scores are measured with tools off.

Why does OpenAI not publish newer numbers?

Its simple-evals repository carries a July 2025 deprecation notice saying it will no longer be updated for new models or benchmark results. We found no equivalent published table covering later models.

Is ChatGPT getting more accurate over time?

On some tasks and by some measures, yes; the published record is too uneven to support a general claim, and the frozen repository makes a like-for-like trend across the newest models impossible to read from the vendor's own numbers.

Related reading

• AI Accuracy Rate: What "99% Accurate" Actually Means When You Buy AI
• Gemini Accuracy: What Google's Own Documentation Says
• Why Does ChatGPT Make Up Sources? 6 Causes and Fixes
• The 10-Prompt AI Accuracy Test (Run It on Any Tool)
• How to Stop ChatGPT From Making Things Up