AI Accuracy Rate: What '99% Accurate' Actually Means When You Buy AI

An AI accuracy rate is a score on one task, measured against one test set, using one metric. None of those three things is standardised, and vendors are not required to disclose any of them. So "99% accurate" is not a property of the product — it is the result of an experiment, and without the experiment's details the number cannot be checked, compared to a rival's number, or used to predict what the tool will do on your work.

We checked six vendors' published accuracy claims at source on 2026-09-06, across AI detection, speech-to-text, document OCR and customer-support automation. This post covers what those numbers measure, why two "99%" figures are not the same measurement, and what to ask instead.

Key Takeaways

• Of six vendor accuracy claims checked at source, one named public test sets and a stated normalisation method. The rest named internal benchmarks, access-restricted datasets, or nothing at all (ANALYSIS, retrieved 2026-09-06).
• Two vendors' "99%" figures cannot be compared. They measure different tasks, on different data, with different metrics.
• A 99% accuracy score is trivial to get on an imbalanced test set — Google's own machine-learning documentation uses exactly that example to warn against the metric.
• No framework requires a standard disclosure. NIST's AI Risk Management Framework recommends documenting test sets and limits; it mandates no metric and no public format.
• The useful question is not "how accurate" but "on what task, measured how, against what data, and can I reproduce it."

An accuracy rate is a measurement, not a property

Weight is a property of an object. Accuracy is not a property of a model. It is the output of a procedure: take a set of examples, run the model on them, count how many outputs match the expected answer, divide.

Change any input to that procedure and the number changes. Use easier examples and it goes up. Use a different definition of "correct" and it moves. Use a metric that weights errors differently and it moves again. All of these produce a defensible, honestly-derived, completely incomparable number.

That is why a percentage with no procedure attached tells you almost nothing. It is a claim about an experiment you cannot see.

Six accuracy claims, checked at source

Every row below was read on the vendor's own page on 2026-09-06. We are quoting these claims as evidence of what vendors publish, not citing any vendor as an authority.

On the Otter.ai row, stated plainly. The vendor help-centre page carrying that figure returned HTTP 403 to our automated request, so we could not read it at source. The claim above comes from secondary summaries and is not verified. We are including it because the inaccessibility is part of the point: a number you cannot open is a number you cannot check. Treat that row as unconfirmed.

On the Google Document AI row. The widely-repeated OCR accuracy figures for this product — 98% character accuracy and similar — do not appear on the vendor's own overview documentation, which publishes no accuracy percentage at all (retrieved 2026-09-06, SOURCED). Those numbers circulate through resellers and third-party blogs. The absence of a vendor figure is a more honest position than most of this table, and it is worth noting that it gets misreported as a specific number anyway.

One row out of six gives you enough to reproduce the measurement.

Why two "99%" figures are not the same number

The two AI-detection vendors above both publish figures above 99%. They are not measuring the same thing, and neither is measuring what a speech-to-text word error rate measures.

• The task differs. Binary classification (is this text machine-written?) and word error rate (what fraction of transcribed words are wrong?) are different questions with different failure modes.
• The test set differs. One vendor's internal Benchmark V6, another's 4,000 domain-split samples, a third's named public audio corpora. A model tuned on the kind of text in its own benchmark will score well on it.
• The metric differs. Raw accuracy, balanced accuracy, precision, recall, F1 and word error rate all compress a confusion matrix into one number, and they disagree with each other by design.

To compare two vendors, you need all three to match. Across the six claims above, no two matched on all three.

What the well-known benchmarks actually measure

Benchmark scores get quoted as general intelligence. They are narrower than that, and the maintainers usually say so.

MMLU is a multiple-choice knowledge test spanning humanities, social sciences, STEM and other subjects (Hendrycks et al., test repository, retrieved 2026-09-06, SOURCED). It measures recall across many domains in a fixed question format. It does not measure reasoning under ambiguity, performance on your workflow, or whether the model will invent a citation.

The Vectara hallucination leaderboard measures one task: summarising roughly 7,700 documents, scored for factual consistency against the source text. Its maintainers state directly that they are "not evaluating the quality of the summaries, only the factual consistency," and that the leaderboard "does not definitively measure all the ways a model can hallucinate" (leaderboard repository, retrieved 2026-09-06, SOURCED).

Read that second disclaimer carefully. A good score means the model stays faithful to a document placed in front of it. It says nothing about what the model does when no document is provided — which is the situation most people are actually in when it makes something up. We wrote about that gap separately in how to stop ChatGPT from making things up.

Three traps that make a high number easy

1. Class imbalance

If the thing you are detecting is rare, a model that never detects it scores well. Google's machine-learning documentation uses exactly this example: "For heavily imbalanced datasets, where one class appears very rarely, say 1% of the time, a model that predicts negative 100% of the time would score 99% on accuracy, despite being useless" (Google ML Crash Course, classification metrics, retrieved 2026-09-06, SOURCED).

This is why balanced accuracy exists. The scikit-learn documentation describes it as designed to "avoid inflated performance estimates on imbalanced datasets" (model evaluation reference, retrieved 2026-09-06, SOURCED). If a vendor reports raw accuracy without the class distribution of the test set, you cannot tell which situation you are in.

2. Test-set contamination

If the test questions were in the training data, the score measures memorisation. This is not a hypothetical: OpenAI's GPT-4 technical report discloses that BIG-bench data had "inadvertently mixed into the training set" and was excluded from evaluation for that reason, and separately discloses deliberately mixing MATH and GSM-8K training data into pretraining (GPT-4 Technical Report, retrieved 2026-09-06, SOURCED).

That disclosure is good practice — the lab checked and said so. Most vendors quoting a benchmark number have not told you whether they checked.

3. Task non-transfer

A score on one task does not carry to another. The Vectara disclaimer above is the cleanest statement of it from an evaluation's own maintainers: a factual-consistency score on summarisation is not a general hallucination rate. A vendor that scores well on document-grounded summarisation has told you nothing about open-ended drafting, and a detector that scores well on essays has told you nothing about how it handles technical documentation.

There is no reporting standard, and that is the finding

NIST's AI Risk Management Framework asks organisations to document the test sets, metrics and tools they used (MEASURE 2.1), to measure performance "for conditions similar to deployment" (MEASURE 2.3), and to document "limitations of generalizability beyond the conditions under which the technology was developed" (MEASURE 2.5) (NIST AI RMF Playbook, Measure, retrieved 2026-09-06, SOURCED).

What it does not do is mandate a metric, a test set, or a public disclosure format. It is voluntary and context-dependent by design.

ISO/IEC 42001 sits behind a paywall and we did not read it, so we make no claim about what it requires.

So the position today is: no framework tells a vendor what it must disclose alongside an accuracy number, and nothing stops a vendor publishing a percentage with no procedure attached. Six out of six claims we checked were legal, and one was reproducible.

Six questions that make an accuracy number mean something

Each of these targets a specific way the number can be true and useless. The expanded version of this list, with the follow-ups for when a vendor deflects, is coming as a separate post in this series.

1. What exact task was measured? Classification, word error rate and summarisation faithfulness are not interchangeable.
2. What was the test set — size, source, public or proprietary — and can we reproduce the run? If key data is access-restricted, the figure is an assertion.
3. Was contamination between training and test data checked, and what did you find? A lab that has done this can answer in one sentence.
4. Is that accuracy or balanced accuracy, and what is the class distribution? Without the distribution, raw accuracy is unreadable.
5. What are the stated limits of generalisability beyond the tested conditions? NIST MEASURE 2.5 asks for exactly this; a vendor that has done the work has it written down.
6. For anything defined by the vendor — "resolution rate", "most accurate" — what counts as success, and who verified it? A self-defined, self-scored metric is marketing.

If a vendor cannot answer questions 1, 2 and 4, you do not have a measurement. You have a number.

What we do about it here

We hold ourselves to the same standard, which is why this post publishes no accuracy figure of our own. We have not run a benchmark that would survive question 2, so we do not quote one.

What we do publish is the audit of our own blog, including the 14 error types we found in our own AI-assisted posts and the checks that catch them. That is a smaller claim than 99%, and it is one you can open and check.

How This Guide Was Sourced

Written and maintained by the LogicBalls editorial team (logicballs.com). Disclosure: LogicBalls builds AI writing tools. A company in this market has an obvious interest in how AI accuracy claims are read, which is why this post publishes no accuracy figure of its own and applies its six questions to itself in the section above.

Sources. Vendor claims were read on each vendor's own page on 2026-09-06 and are quoted as evidence of what vendors publish, not as authorities. Metric definitions come from Google's ML Crash Course and the scikit-learn model-evaluation reference. Benchmark descriptions come from the MMLU and Vectara leaderboard repositories, including the maintainers' own stated limits. The contamination disclosures come from the GPT-4 technical report on arXiv. Governance findings come from the NIST AI RMF Playbook.

What could not be verified. The Otter.ai help-centre page returned HTTP 403 to automated requests, so its figure is unconfirmed and labelled as such in the table rather than quoted as fact. ISO/IEC 42001 is paywalled; we did not read it and make no claim about its contents.

One category was deliberately left out. Machine translation belongs in this analysis and we excluded it, because the category is adjacent to tools LogicBalls sells and a head-to-head against a vendor we compete with is not something this blog does. The exclusion is a conflict-of-interest call, not an absence of examples.

What is not claimed. We ran no benchmark. Every figure above is someone else's, linked and dated.

No LogicBalls telemetry is used in this guide.

Frequently Asked Questions

Is a 99% accuracy claim a lie?

Usually not. It is usually a true result from an experiment you cannot see, on a task that may not be yours. The problem is not dishonesty, it is incomparability.

What is a good accuracy rate for an AI tool?

There is no threshold that transfers across tasks. A 95% word error rate would be catastrophic; 95% on a rare-event classifier might be worse than useless. The task and the class distribution decide what "good" is.

Why do vendors not publish their test sets?

Some cite competitive concerns, some cite licensing on the underlying data, and some do not say. Whatever the reason, an unpublished test set means the figure cannot be independently reproduced, and you should read it as an assertion rather than a measurement.

Does a higher benchmark score mean fewer hallucinations?

Not reliably. Benchmarks measure defined tasks under fixed conditions. The evaluation maintainers quoted above say so themselves.

Can I just test it myself?

Yes, and it is the strongest option available. Assemble 50 to 100 examples from your own real work with known correct answers, run them, and count. That number is smaller than a vendor's, and it is about you.

Related reading

• Verified AI Writing: How to Publish AI-Assisted Content You Can Stand Behind
• How to Detect AI Hallucinations
• What Is AI Hallucination? 7 Things Everyone Gets Wrong
• 8 AI Trustworthiness Frameworks Every User Should Know About
• Do AI Detectors Actually Work? What the Research Shows