accesspatched

Research

Your accessibility scanner is lying to you: why "zero errors" proves almost nothing

Published 19 July 2026

Short answer: no, an automated accessibility scan is not enough, and a green "zero errors" result does not mean your site is accessible. The best automated tools detect roughly 30 to 40% of WCAG issues (Level Access), and, more damning, a peer-reviewed study of real blind users found that fully conforming to WCAG covered only about half of the problems they actually ran into. So even a perfect scan against a perfect standard leaves a lot of your customers stuck.

We run automated scans on every audit, and we give one away for free, so this is not a case against tooling. It is a case against the story the industry tells about tooling. A clean scan is a smoke test. It is not a certificate, and anyone selling it as one is quietly dropping the hard part.

What the research actually found about WCAG conformance

The single most useful study here is Power, Freire, Petrie and Swallow, published at CHI in 2012. The team sat 32 blind users in front of 16 websites and logged what tripped them up. They recorded 1,383 distinct problem instances. Then they asked a pointed question: if these sites had fully conformed to WCAG 2.0, how many of those problems would have been prevented? The answer was about half. Roughly 50% of the barriers real blind users hit were not addressed by conforming to the guidelines at all (Power et al., 2012).

Sit with that for a second, because it reframes everything. WCAG is the standard your scanner checks against. This study is not saying tools are worse than the standard. It is saying the standard itself, even when met perfectly, only reaches about half the lived problems. Your scanner checks a subset of a standard that covers half the reality. Two lossy steps, stacked.

This is not a lone result, either. Petrie and Kheir showed back in 2007 that accessibility problems for disabled users and usability problems for everyone overlap but are genuinely different sets, and that guideline priority ratings did not line up with what users actually struggled with (Petrie and Kheir, 2007). A high-priority guideline is not automatically the thing that breaks a real journey, and a low-priority one can be exactly what stops someone from paying you.

The tools do not even agree with each other

If automated tools all caught the same reliable third, you could at least treat that third as solved ground. They do not. Vigo, Brown and Conway benchmarked six automated accessibility evaluation tools and measured, bluntly, "the harm of sole reliance on automated tests." Their finding: coverage of the WCAG 2.0 success criteria was only partial, the tools missed many criteria outright, and, crucially, they disagreed with one another about what they did check (Vigo et al., 2013). Run two scanners on the same page and you can get two different verdicts.

So "zero errors" is not even a stable claim. It means zero errors that this particular tool, on this particular day, was built to look for. Swap the tool and the number moves. The earlier work by Vigo and Brajnik on automatic accessibility metrics reached a compatible conclusion: automated metrics are useful signals but a long way from a complete verdict on whether a page works (Vigo and Brajnik, 2011).

What a scanner literally cannot see

The 30 to 40% ceiling is not laziness in the engines. It is a hard limit on what rules can decide. The gap falls exactly where you need to understand meaning, not just structure.

These map directly onto what frustrates screen reader users in practice. The classic Lazar study of 100 blind users found their top frustrations were confusing page layouts, conflicts between the screen reader and the application, poorly labelled forms, missing alt text, and misleading links. Some of those a scanner catches. Several it cannot, because "confusing" and "misleading" are human verdicts.

So why is "zero errors" sold as compliance?

Because it is cheap to produce and it looks decisive. This is the whole business model of accessibility overlays: a script you paste in that claims to detect and fix problems automatically. The people who actually do this work took the trouble to put their names to a rebuttal. The Overlay Fact Sheet, signed by Karl Groves and around 800 accessibility professionals, states plainly that overlays cannot achieve WCAG conformance, and its signatories pledge never to recommend overlays that deceptively claim automated compliance (Overlay Fact Sheet). Groves' own expert-witness analysis of 50 sites running accessiBe found they were no more and no less accessible than sites without it.

Regulators have started to agree in writing. In January 2025 the US Federal Trade Commission ordered accessiBe to pay $1,000,000 over deceptive claims that its AI overlay could make websites WCAG-compliant (FTC, 2025). To be clear about our own house rule: no monetary fines have been issued under the European Accessibility Act anywhere yet. The accessiBe order is a US consumer-protection matter about false advertising, not an EAA penalty. But it is a formal, on-the-record finding that "an automated tool made us compliant" is a claim you can get punished for making.

How to use automated testing correctly

None of this means throw the scanner away. Automated engines are genuinely excellent at the high-frequency, rule-checkable problems, and those happen to be the ones that appear most often. The six errors behind 96% of detected failures, missing alt attributes, empty buttons and links, missing form labels, low contrast, and a missing language declaration, are all things code can verify every time without getting bored. Run a scanner early, run it often, and clear that layer cheaply.

Then treat the result honestly. A clean scan means you have cleared the machine-detectable third. It says nothing about the other two thirds, and, per Power, nothing about the roughly half of real problems that live outside the guidelines entirely. The only way to close that gap is a human driving the actual journey with the actual assistive technology your customers use. If you are wondering which tools those are, we wrote about which screen readers your customers actually run.

Common questions

Does a passing automated scan mean my site is WCAG compliant?

No. A pass means the specific tool found no failures among the checks it can automate, which is roughly a third of WCAG criteria (Level Access). Conformance requires evaluating criteria that need human judgment, and different tools disagree on even the automatable ones (Vigo et al., 2013).

If I fully conform to WCAG, is my site accessible to everyone?

Conformance gets you a long way and you should aim for it, but it is not the finish line. In a study of 32 blind users, WCAG conformance addressed only about half of the problems they actually encountered (Power et al., 2012). Real testing with disabled users finds barriers the guidelines miss.

Are accessibility overlays a shortcut around all this?

No. Around 800 accessibility professionals have signed a statement that overlays cannot deliver conformance (Overlay Fact Sheet), and the FTC fined one overlay vendor $1,000,000 for claiming otherwise (FTC, 2025).

The way we handle the gap is straightforward. We start with the automated scan to clear the cheap third, then a human audits the real journeys to WCAG-EM methodology, tests the checkout with real screen readers, ships the fixes as pull requests your developers can review and merge, and issues an EN 301 549 conformance statement at the end. The scan finds the smoke. The human finds whether the building actually works. You can see how we scope that under our services.

See where your own shop stands. Our free scan runs the same automated checks a regulator's tester starts with, then a human goes through your site with a screen reader and emails you the findings, in plain language. If your shop passes, we say so.

Get a free scan

Sources

Back to all research