{"id":1940,"date":"2026-09-18T12:39:34","date_gmt":"2026-09-18T19:39:34","guid":{"rendered":"https:\/\/evinced.com\/blog\/?p=1940"},"modified":"2026-09-18T13:20:09","modified_gmt":"2026-09-18T20:20:09","slug":"when-skills-are-not-enough","status":"publish","type":"post","link":"https:\/\/evinced.com\/blog\/when-skills-are-not-enough\/","title":{"rendered":"When skills are not enough"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\"><em>Post 3 of 5 in our series on AI-assisted code and accessibility.<\/em><br><\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">So far in our series on AI and accessibility, we\u2019ve covered two key points:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li style=\"font-size:18px\"><a href=\"https:\/\/evinced.com\/blog\/the-heart-of-the-problem-with-llms\/\" data-type=\"link\" data-id=\"https:\/\/evinced.com\/blog\/the-heart-of-the-problem-with-llms\/\">LLMs don\u2019t produce accessible code<\/a> by default.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li style=\"font-size:18px\"><a href=\"https:\/\/evinced.com\/blog\/the-promise-and-peril-of-prompting\/\" data-type=\"link\" data-id=\"https:\/\/evinced.com\/blog\/the-promise-and-peril-of-prompting\/\">Prompts can improve results<\/a>, but not enough, and not reliably so.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">In this post, we\u2019re going to cover the most well-known example of a third approach to using LLMs in combination with agentic coding, and that\u2019s to use a skill.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">In the world of agentic coding, a skill is a reusable, task-specific set of instructions, knowledge, and procedures that an operator can give an LLM. They are versatile, in that they can include not just instructions but also copies of correct and incorrect code examples, scripts for running third-party tools, and rules. Once the skill is built, it can be invoked easily when a relevant task presents itself, and all of the information in the skill doesn\u2019t have to squeeze, over and over again, into the prompt window of an LLM.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">Researcher Michael Fairchild, who\u2019s a key figure behind the <a href=\"https:\/\/microsoft.github.io\/a11y-llm-eval-report\/index.html\">LLM A11y Eval<\/a> project that we have cited before, published a skill in May 2026 called <a href=\"https:\/\/github.com\/microsoft\/a11y-llm-eval\/tree\/main\/config\/skills\/building-accessible-ui\">Building Accessible UI<\/a>. Fairchild published some promising results, so we decided to add to the state of knowledge here by running and documenting our own tests using it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">Our results were also good. But they were also, as it turns out, not nearly good enough.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Setting up the experiment<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">A skill like Building Accessible UI works best at code-generation time: it rides along while the model writes the code, steering it toward accessible patterns and checking the output as it goes. So that is what we tested. We returned to the nine-app protocol from our first two posts: three models, three frameworks, one identical single-page app spec, this time with the skill installed and invoked during the build. For comparison, we ran the same protocol with Evinced Harness, our MCP-based detection and repair tooling, in the loop instead. (Skill runs used GPT 5.2, which had replaced GPT 5.1.)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">First, a reality check on model progress: benchmarked for a separate study, every newer model still produced frankly terrible accessibility results out of the box, several worse than their predecessors.<\/p>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\"><strong>Exhibit&nbsp;1. Baseline Accessibility Performance of Models, with Release Date<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\" style=\"font-size:18px\"><table><thead><tr><th class=\"has-text-align-left\" data-align=\"left\"><strong><em>Model, Release Date<\/em><\/strong><\/th><th class=\"has-text-align-left\" data-align=\"left\"><strong><em>Errors*<\/em><\/strong><\/th><th class=\"has-text-align-left\" data-align=\"left\"><strong><em>Model, Release Date<\/em><\/strong><\/th><th class=\"has-text-align-left\" data-align=\"left\"><strong><em>Errors*<\/em><\/strong><\/th><\/tr><\/thead><tbody><tr><td class=\"has-text-align-left\" data-align=\"left\"><em>Claude 4.5 Sonnet, Sept 2025<\/em><\/td><td class=\"has-text-align-left\" data-align=\"left\"><strong>36<\/strong><\/td><td class=\"has-text-align-left\" data-align=\"left\"><em>Claude Opus 4.8, May 2026<\/em><\/td><td class=\"has-text-align-left\" data-align=\"left\"><strong>94<\/strong><\/td><\/tr><tr><td class=\"has-text-align-left\" data-align=\"left\"><em>GPT 5.1, Nov 2025<\/em><\/td><td class=\"has-text-align-left\" data-align=\"left\"><strong>46<\/strong><\/td><td class=\"has-text-align-left\" data-align=\"left\"><em>GPT 5.5, Apr 2026<\/em><\/td><td class=\"has-text-align-left\" data-align=\"left\"><strong>64<\/strong><\/td><\/tr><tr><td class=\"has-text-align-left\" data-align=\"left\"><em>Gemini 2.5 Pro, Mar 2025<\/em><\/td><td class=\"has-text-align-left\" data-align=\"left\"><strong>53<\/strong><\/td><td class=\"has-text-align-left\" data-align=\"left\"><em>Gemini 3.1 Pro, Feb 2026<\/em><\/td><td class=\"has-text-align-left\" data-align=\"left\"><strong>51<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"has-text-align-center has-small-font-size wp-block-paragraph\"><em>Average critical + serious accessibility errors made when building our reference app, as measured by Evinced using default settings.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">The newer models did not just fail to improve: for two of the three vendors, the newest model shipped substantially more critical and serious defects than its predecessor. Claude\u2019s count nearly tripled, and GPT\u2019s rose about 40%, with only Gemini holding roughly flat. The trend is not slowly improving. It points the wrong way.<br><br>&#8220;Wait for better models&#8221; is not a plan we would recommend.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Counting defects, not echoes<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">A methodology note. Scanners count instances: one broken checkbox row, repeated 46 times in a list, counts as 46 issues when it is really one mistake. So in this post we count components: distinct root causes, deduplicated by issue type and element pattern. It is the more conservative number, and the one a developer actually has to fix. We also limit the counts to critical and serious defects, the two grades that block or badly impede assistive-technology users.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The work to be done<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">So what did the unguided models build? They built 9 baseline apps, with: 18.1 defective components each, on average. Exhibit 2 shows the breakdown by type:<\/p>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\"><strong>Exhibit&nbsp;2. Baseline Component Accessibility Defects by Type<\/strong><br><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"506\" src=\"https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image1-1-1024x506.png\" alt=\"Bar chart showing baseline critical and serious defects by type. \nRanking most common to least:\nAccessible name (46)\nTarget size (24)\nColor contrast (24)\nKeyboard accessible (19)\nInteractable role (18)\nARIA required parent (9)\nNested interactive (9)\nEverything else (14)\" class=\"wp-image-1941\" srcset=\"https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image1-1-1024x506.png 1024w, https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image1-1-300x148.png 300w, https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image1-1-768x379.png 768w, https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image1-1-1536x758.png 1536w, https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image1-1.png 1659w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">Controls with no accessible name, keyboard access, or proper role make up most of the critical defects.&nbsp;Target size and color contrast dominate the serious defects, with a tail of ARIA structure and focus-order problems.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">This is the to-do list any tool would need to clear. How did our models and skills perform?<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>The results<\/strong><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">Here is how each approach handled it.<\/p>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\"><strong>Exhibit&nbsp;3.&nbsp;Defective Components Per App Built, by Approach<\/strong><br><\/p>\n\n\n\n<figure class=\"wp-block-table is-style-stripes\" style=\"font-size:18px\"><table class=\"has-fixed-layout\"><thead><tr><th>Approach<\/th><th class=\"has-text-align-center\" data-align=\"center\">Critical<\/th><th class=\"has-text-align-center\" data-align=\"center\">Serious<\/th><th class=\"has-text-align-center\" data-align=\"center\">Critical<br>+ Serious<\/th><th class=\"has-text-align-center\" data-align=\"center\"><strong>Apps with Zero Critical Defects Remaining<\/strong><\/th><\/tr><\/thead><tbody><tr><td>No guidance<\/td><td class=\"has-text-align-center\" data-align=\"center\">11.4<\/td><td class=\"has-text-align-center\" data-align=\"center\">6.7<\/td><td class=\"has-text-align-center\" data-align=\"center\">18.1<\/td><td class=\"has-text-align-center\" data-align=\"center\">0 of 9<\/td><\/tr><tr><td>Accessibility prompt (<a href=\"https:\/\/evinced.com\/blog\/the-promise-and-peril-of-prompting\/\" data-type=\"link\" data-id=\"https:\/\/evinced.com\/blog\/the-promise-and-peril-of-prompting\/\">see post #2<\/a>)<\/td><td class=\"has-text-align-center\" data-align=\"center\">1.2<\/td><td class=\"has-text-align-center\" data-align=\"center\">11.9<\/td><td class=\"has-text-align-center\" data-align=\"center\">13.1<\/td><td class=\"has-text-align-center\" data-align=\"center\">6 of 9<\/td><\/tr><tr><td><em>Building Accessible UI<\/em> skill<\/td><td class=\"has-text-align-center\" data-align=\"center\">2.1<\/td><td class=\"has-text-align-center\" data-align=\"center\">5.8<\/td><td class=\"has-text-align-center\" data-align=\"center\">7.9<\/td><td class=\"has-text-align-center\" data-align=\"center\">4 of 9<\/td><\/tr><tr><td><strong>Evinced Harness<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\"><strong>0.0<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\"><strong>0.7<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\"><strong>0.7<\/strong><\/td><td class=\"has-text-align-center\" data-align=\"center\"><strong>9 of 9<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"has-text-align-center has-small-font-size wp-block-paragraph\"><em>Averages are across nine apps (three models, three frameworks), measured by Evinced Web Flow Analyzer.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">As a topline note, the skill is the best free option we have tested. It cut critical components by 82%, a large improvement over baseline. And while this is marginally less effective than the accessibility prompt alone, its improvement came without the accessibility prompt\u2019s surge in serious component defects. If your budget is absolute zero, the skill is a better choice than the prompt.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">Of course, the \u201cfree\u201d option could still turn out to be <a href=\"https:\/\/evinced.com\/blog\/how-fixing-an-accessibility-bug-can-cost-15000\/\">quite expensive<\/a>, and even putting aside token costs. Note above that none of the free options get to true accessibility.&nbsp;What you save up front will be spent on the manual effort to find, reproduce, assign, fix, and verify everything that was missed. And while that plays out, your site or app will be inaccessible in some way to some people.<\/p>\n\n\n\n<div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\"><strong>Exhibit 4.&nbsp;Percent Reduction (and Increase) in Critical and Serious Defective Components Per App, By Model and Trial<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"635\" src=\"https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image2-1024x635.png\" alt=\"Horizontal bar chart of accessibility improvement across nine builds, three each from Claude 4.5 Sonnet, ChatGPT 5.1\/5.2, and Gemini 2.5 Pro. Evinced Harness reaches at or near fully accessible on every build, while the Skill alone averages between 31 and 71 percent and regresses on one build. Analysis below.\" class=\"wp-image-1942\" srcset=\"https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image2-1024x635.png 1024w, https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image2-300x186.png 300w, https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image2-768x476.png 768w, https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image2-1536x952.png 1536w, https:\/\/evinced.com\/blog\/wp-content\/uploads\/2026\/09\/image2.png 1999w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center has-small-font-size wp-block-paragraph\"><em>Accessible = a 100% reduction in autodetected critical and serious defects.\u00a0<\/em><\/p>\n\n\n\n<p class=\"has-text-align-center has-small-font-size wp-block-paragraph\"><em>Frameworks: 1 = React + Bootstrap, 2 = Svelte + Tailwind, 3 = Plain JS + CSS.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">Looking at the skill\u2019s performance by trial, note that in at least one trial \u2013 Claude 4.5 paired with Svelte and Tailwind \u2013 the skill actually made things <em>worse<\/em>. But by and large it had a positive impact.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">On balance, the net improvement from the skill was slightly less than half that of <a href=\"https:\/\/www.evinced.com\/blog\/the-end-of-guessing\">Evinced Harness<\/a>, our own tool for helping developers build, test, and fix code for accessibility. Across everything we have measured in this series, Harness is the only approach that gets to zero components with critical defects.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The takeaway<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">Given that the skill did show some improvements, we asked ourselves whether it could perform better. Our best answer, at the moment, is that this current performance for a free skill is something of a plateau.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">Why? Because a skill guides the model with patterns and verifies the result with checks, and a free skill must build those checks on free tools, which, in practice, means axe-core.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">At this point, we are in familiar territory.&nbsp;In every analysis we have done, Evinced identifies dramatically more defects than axe-core, and depending on the study and methodology it ranges from 2.5X to 3.0X. (This comparison is for all types of issues, but for critical issues only the gap is often much, much wider. See also the comparative validations count in our article \u201c<a href=\"https:\/\/www.evinced.com\/blog\/whats-in-a-checkpoint-charlie\">What\u2019s In a Checkpoint, Charlie?<\/a>\u201d)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">So in effect, it\u2019s not so much a limit of the skill itself, as it is of the tool it\u2019s using to detect accessibility bugs. Anything built on a tool that has lots of false negatives (i.e., bugs it didn\u2019t detect but should have)&nbsp;will itself have lots of false negatives. And strictly speaking, it\u2019s a little worse than that. As some have pointed out, skills can help a base model pass most \u2013 <a href=\"https:\/\/www.aaron-gustafson.com\/notebook\/can-your-ai-pass-the-accessibility-test\/\">but not even all<\/a> \u2013 of axe-core\u2019s checks.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">This performance ceiling holds for any tool you evaluate, from any vendor.&nbsp;Before you ask anything else, ask what engine it is built on. Because a defect your detection tool cannot see does not go away. It ships, it waits, and one day it will grow into an expensive problem.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\" style=\"font-size:18px\">In our next post, we\u2019ll examine how we can automatically and continuously monitor and repair these problems, before they have a chance to grow. See you soon.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Post 3 of 5 in our series on AI-assisted code and accessibility. So far in our series on AI and accessibility, we\u2019ve covered two key points: In this post, we\u2019re going to cover the most well-known example of a third approach to using LLMs in combination with agentic coding, and that\u2019s to use a skill. [&hellip;]<\/p>\n","protected":false},"author":15,"featured_media":1885,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[7],"tags":[],"class_list":["post-1940","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"acf":{"authors_to_show":[1650]},"_links":{"self":[{"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/posts\/1940","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/comments?post=1940"}],"version-history":[{"count":17,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/posts\/1940\/revisions"}],"predecessor-version":[{"id":1972,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/posts\/1940\/revisions\/1972"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/media\/1885"}],"wp:attachment":[{"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/media?parent=1940"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/categories?post=1940"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/evinced.com\/blog\/wp-json\/wp\/v2\/tags?post=1940"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}