{"id":1569,"date":"2026-01-19T10:38:19","date_gmt":"2026-01-19T18:38:19","guid":{"rendered":"https:\/\/galm.sg-host.com\/?p=1569"},"modified":"2026-03-30T08:16:12","modified_gmt":"2026-03-30T15:16:12","slug":"what-an-automation-company-learned-from-manual-audits","status":"publish","type":"post","link":"https:\/\/evinced.com\/blog\/what-an-automation-company-learned-from-manual-audits\/","title":{"rendered":"What an automation company learned from manual audits"},"content":{"rendered":"\n

The question every organization asks when evaluating accessibility tools seems simple: if we choose this particular solution, how much manual testing will we still have to do?<\/em><\/p>\n\n\n\n

This has been difficult to answer industry-wide for a few reasons, not least of which because many players in the industry (including us) maintain anti-benchmarking clauses in their terms of service. But it\u2019s also true that there have been few third-party data sets for analysis.<\/p>\n\n\n\n

Looking at audits<\/strong><\/h2>\n\n\n\n

To address this gap, we decided to ask friends in the industry to send us a copy of an audit that was recently done for one of their websites. We have assumed that these audits were performed entirely manually, but even if some tools were used, the point is that they were performed on a production website, at the highest standards in the business for audits.<\/p>\n\n\n\n

How do we know? Because to date, we\u2019ve collected 35 of these audits, and virtually all were done by the two most reputable leaders in the auditing business. Here are some basics of the data set as of December 2025:<\/p>\n\n\n\n

Descriptive statistics for the audit set<\/strong><\/p>\n\n\n\n

n<\/td>35<\/td><\/tr>
Avg. number of issues reported<\/td>178.6<\/td><\/tr>
Avg. number of critical issues<\/td>43.5<\/td><\/tr>
Avg. number of serious issues<\/td>105.4<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Now that they\u2019re in hand, we\u2019re able to run some helpful analysis on them.<\/p>\n\n\n\n

Because each issue reported in an audit is listed and described, we can assess whether that issue would have been automatically detectable by our tools. All of our ability to detect problems comes from a code base of checkpoints that we\u2019ve developed called our validation set<\/a>. And because our very large validation set includes<\/em> the much smaller set of axe-core validations, we can also assess which of these issues would also<\/em> have been detectable by a tool running axe-core. (Because we include axe-core validations, it isn\u2019t theoretically possible for a tool solely running axe-core to detect issues that we don\u2019t.)<\/p>\n\n\n\n

So we can \u201cbackcast,\u201d if you will, the performance of Evinced vs. axe-core over the range of the issues reported in the manual audits we have gathered to date.1<\/a><\/sup><\/p>\n\n\n\n

What we found<\/strong><\/h2>\n\n\n\n

Across the audits gathered, we found that Evinced can detect nearly 63%<\/strong> of the issues reported, whereas axe-core detected roughly 23%<\/strong>.<\/p>\n\n\n\n

Percentage of accessibility issues<\/strong> detectable in our sample of manual audits<\/strong><\/p>\n\n\n\n

Tool<\/strong><\/td>% Detected<\/strong><\/td><\/tr>
Axe-core only<\/td>22.6%<\/td><\/tr>
Evinced<\/td>62.8%<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

That\u2019s nearly three times more coverage than axe-core alone, on actual production websites. This is in line with our prior work comparing our coverage, but the data set here is particularly simple and easy to understand.<\/p>\n\n\n\n

But the raw percentages, while clear, aren\u2019t the most important learning. What truly matters is what those numbers mean for your program workload, and for its costs.<\/p>\n\n\n\n

What this means for accessibility programs<\/strong><\/h2>\n\n\n\n

Audits, even manually performed audits, have their place. They are an important part of understanding the accessibility of a production website. <\/p>\n\n\n\n

But our analysis suggests something important. <\/p>\n\n\n\n

The opportunity for optimizing a manual audit process is substantial. This data raises the question: if you can detect a problem automatically, why wouldn\u2019t you do it? <\/p>\n\n\n\n

To us, if an accessibility issue can be detected earlier in the development process, it should be. Automated tooling can run at a developer\u2019s or designer\u2019s desk, in context, without slowing delivery or requiring specialized accessibility expertise. When those issues are caught early, they never need to appear in an audit report at all. <\/p>\n\n\n\n

That shift matters because every issue found late represents work that had to wait: triage, reproduction, scheduling, and remediation long after the original developer\u2019s context has faded. By contrast, audits are for spot checking the automation, and most importantly for deploying human talent for coping with the new: new designs, new approaches, new components, new products. <\/p>\n\n\n\n

In addition, it also frees human accessibility experts to focus on the critical and frankly much needed work on driving adoption. <\/p>\n\n\n\n

This opportunity, while substantial, can also be stated in the negative: if you are spending tons of resources on using humans to do things automation could do, by the laws of economics, you\u2019re robbing your organization of resources spent on things like driving adoption. <\/p>\n\n\n\n

There is no free lunch in accessibility. <\/p>\n\n\n\n

Our next blog post will tackle just how expensive this lunch really is. Stay tuned!<\/p>\n\n\n\n

\n\n\n
  1. Axe-core licensing agreement (held by the Mozilla Public License Version 2.0)<\/a> \u21a9\ufe0e<\/a><\/li><\/ol>\n\n\n

    <\/p>\n","protected":false},"excerpt":{"rendered":"

    The question every organization asks when evaluating accessibility tools seems simple: if we choose this particular solution, how much manual testing will we still have to do? This has been difficult to answer industry-wide for a few reasons, not least of which because many players in the industry (including us) maintain anti-benchmarking clauses in their […]<\/p>\n","protected":false},"author":15,"featured_media":1573,"comment_status":"open","ping_status":"open","sticky":true,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":"[{\"content\":\"Axe-core licensing agreement (held by the Mozilla Public License Version 2.0)<\/a>\",\"id\":\"e6594f6d-2e68-4d89-9755-d14b9b7e7b85\"}]"},"categories":[2],"tags":[],"class_list":["post-1569","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-accessibility"],"acf":{"authors_to_show":[922]},"_links":{"self":[{"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/posts\/1569","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/comments?post=1569"}],"version-history":[{"count":5,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/posts\/1569\/revisions"}],"predecessor-version":[{"id":1579,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/posts\/1569\/revisions\/1579"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/media\/1573"}],"wp:attachment":[{"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/media?parent=1569"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/categories?post=1569"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/tags?post=1569"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}