In 2022, Vanderbilt University’s instructors submitted about 75,000 papers to Turnitin. When the university later weighed the company’s AI detector, it did a simple calculation. At a 1% false positive rate, around 750 student papers could have been wrongly labeled as AI-written (Vanderbilt, August 2023). Vanderbilt switched the feature off.
Seven hundred and fifty students. From a tool that, on paper, was 99% right.
That’s the problem with accuracy claims. The number sounds like a guarantee. In practice, it hides at least three different questions, and the answer to each one depends on who’s being tested.
Accuracy Is Several Numbers Wearing One Label
The four outcomes every detector produces
Every time a detector scores a document, one of four things happens:
- True positive: AI text, correctly flagged
- True negative: human text, correctly cleared
- False positive: human text, wrongly flagged
- False negative: AI text, missed
“Accuracy” lumps all four into one share of right answers. That’s where the trouble starts, because those errors don’t cost the same. Missing some AI text is annoying. Accusing a student who wrote every word is serious.
The metrics that actually matter
Recall, or true positive rate
Out of all the AI text, how much did the detector catch? A tool with 95% recall misses one AI document in twenty.
False positive rate
Out of all the human text, how much was wrongly flagged? This is the number that decides how many innocent people get hurt.
Precision
When the detector does raise a flag, how often is it right? This is what a teacher or editor really wants to know, and it’s rarely published.
AUROC
Short for “area under the receiver operating characteristic curve,” it summarizes how well a detector separates human from AI text across every possible threshold. It’s useful for researchers comparing models. It says little about the setting a school actually uses.
Why the Same Detector Looks Better or Worse Depending on the Room
The base rate problem
Here’s what many accuracy claims leave out. A detector’s precision depends heavily on how much AI text is in the pile to begin with.
Picture 10,000 essays and a detector with 95% recall and a 1% false positive rate. If 10% of the essays are AI-written, it catches 950 of them and wrongly flags 90 human ones. About one flag in eleven is a mistake.
Now say only 2% are AI-written. It catches 190, but still wrongly flags 98 human essays. More than a third of all flags now land on innocent writers. Same tool, same settings, very different reality.
Expected human papers wrongly flagged among Vanderbilt’s roughly 75,000 papers (2022), at different false positive rates. Calculation based on Vanderbilt’s figures, August 2023.
Thresholds change everything
Detectors don’t just say yes or no. They produce a score, and someone picks the cut-off. Lower it and you catch more AI, while flagging more humans. Raise it and you do the opposite.
That’s why serious benchmarks fix the false positive rate first. The RAID benchmark, which used more than 6 million generated texts, set every detector to a 5% false positive rate before comparing them (Dugan et al., ACL 2024). At that setting, the best detector caught 85.0% of unattacked AI text. Under a simple trick that swaps letters for look-alike characters, it fell to 9.3%. The authors also noted that few detectors can operate below a 1% false positive rate at all.
The fine print vendors do publish
Some companies do share the details, if you look. Turnitin said in 2023 that its document-level false positive rate was under 1% for documents with more than 20% AI writing, and that its sentence-level rate was around 4%. The company now shows scores between 1% and 19% as an asterisk rather than a number, explicitly to reduce false positives (Turnitin Help Center). Turnitin had reviewed more than 200 million papers by April 2024. At that scale, even a fraction of a percent adds up to a lot of people.
What Independent Tests Keep Finding
Lab claims and outside results tend to drift apart. A 2023 study by Debora Weber-Wulff and colleagues tested 14 tools and found that all scored below 80% accuracy, with only five above 70%. More recently, a tracker maintained by programs.com found that six of eight major detectors claim 99% accuracy or higher, while 19 of 39 independent results put detection below 80% (programs.com, updated November 2026).
Universities have noticed. The University of Waterloo stopped using Turnitin’s AI detection in September 2025, saying the costs outweighed the benefits. Curtin University switched it off from January 1, 2026. Washington State University cancelled its AI detection contract in February 2026.
Length, editing and language all move the number
Accuracy also shifts with the text itself. Short passages give a detector less to work with, so scores on a 150-word paragraph are shakier than on a 1,500-word essay. Text that a person has edited heavily, or that mixes human and AI sentences, sits in a gray zone most tools handle poorly. And most published results come from English prose, so a claim tested there may not hold for other languages or for technical writing full of formulas and jargon.
That’s why a single accuracy figure, with no description of the texts behind it, is close to meaningless. A detector can be excellent on long English essays and unreliable on short product descriptions, and both statements can come from the same tool in the same month.
How to Read a Detector or Rewriter Claim
The same skepticism applies to tools on the other side, the ones that promise to rewrite text so it passes detection. Their claims use the same slippery language. A “99% pass rate” could mean almost anything.
A good test publishes its full scoring rules. Phrasly’s September 2026 Benchmarks, for instance, reports four separate measures for each rewriting tool, defines them precisely, and shows the sample count behind every figure. “100% human outputs,” for example, means outputs scoring exactly 1.0 on Pangram’s human fraction, not “mostly human.” Phrasly sells one of the tested tools, so weigh the results accordingly. The format is still worth copying.
When you see an accuracy claim, ask:
- Accuracy on what kind of text, and how long?
- What’s the false positive rate, specifically?
- At what threshold?
- On how many samples, and who chose them?
- Has anyone without a stake in the result repeated it?
If a vendor can only answer the first question with a single percentage, you don’t have an accuracy figure. You have a slogan.
To see how individual tools compare on these measures, Phrasly has a useful explainer on how different AI checkers handle false positives, including a plain breakdown of sensitivity and specificity.
Read the Errors, Not the Headline
“99% accurate” is true in the narrow way a weather forecast of “no snow in July” is accurate. It can be technically right and still useless for the decision in front of you.
What matters is which mistakes a tool makes, how often, and to whom. A detector that rarely misses AI text but regularly flags careful human writers is a liability in a classroom. A rewriter that claims perfect scores without a published method is a gamble.
So when someone hands you a single impressive number, ask for the other ones. The false positive rate, the threshold, the sample size and the people who checked. Those four answers will tell you far more than any headline percentage ever could.
