The Accessibility Testing Gap Between a Clean Scan and a Legal Defense

A mid-sized e-commerce team runs its checkout flow through an automated accessibility scanner before every release. The score has read 100 out of 100 for six consecutive quarters. Then a demand letter arrives, alleging that a blind customer using JAWS could not complete a purchase because the promo code modal trapped keyboard focus and never returned it to the page. The scanner never flagged it, because nothing in the modal's markup was technically invalid. The <div> had a role. The button had a label. Every rule the tool checks passed. The one thing that mattered — whether a real person navigating without a mouse could get in and out of that dialog — was never tested, because no automated tool can press Tab, Shift+Tab, and Escape and observe what actually happens.

This is not a rare misconfiguration. It is the default state of accessibility testing at most software companies, and it is becoming a more expensive default every year. Accessibility testing, as most engineering organizations practice it, means running axe, WAVE, or Lighthouse against a set of pages, fixing whatever the tool flags, and treating a clean report as evidence of compliance. That practice made sense when the primary goal was catching obvious markup errors cheaply. It stops making sense the moment the question shifts from "does our markup follow patterns a machine can check" to "can a person using a screen reader, a screen magnifier, voice control, or keyboard-only navigation actually complete the transaction we built." Regulators, courts, and — most immediately — actual users are asking the second question. Most testing programs are still answering the first.

This article maps that gap precisely: what automated accessibility testing structurally cannot see, what the current legal and regulatory landscape in the United States and the European Union actually requires, and what a testing program has to include to hold up when someone with a legitimate grievance — or a regulator, or a plaintiff's attorney — looks past the scan report and tries to use the product themselves.

The Measurement Illusion Behind a High Accessibility Score

Every popular automated tool — axe DevTools, WAVE, Lighthouse's accessibility audit, Pa11y, IBM Equal Access Checker — works the same way at its core. It parses the rendered DOM, applies a rules engine built from patterns known to violate WCAG success criteria, and reports matches. This is genuinely useful. It catches missing alt attributes, unlabeled form fields, insufficient color contrast, duplicate IDs, missing document language attributes, and a long list of other structural defects, fast and at scale, on every build.

What it cannot do is evaluate meaning, sequence, or behavior. A scanner can confirm that an aria-label exists. It cannot confirm that the label is accurate, or that it still makes sense after the element's state changes. It can confirm that a tabindex attribute is syntactically valid. It cannot confirm that the order in which focus lands on twelve interactive elements matches the order a sighted user would visually expect. It can confirm an aria-live region exists in the markup. It cannot confirm that a screen reader actually announces its content at the right moment, in the right way, without cutting off mid-sentence because the DOM node was replaced too quickly.

There have been two meaningfully different attempts to quantify this gap, and it is worth understanding both, because they measure different things and get cited interchangeably in ways that mislead people who only read the headline number.

Deque Systems, the company behind the widely used axe-core engine, published a 2021 analysis of more than 2,000 accessibility audits covering roughly 13,000 pages and nearly 300,000 identified issues. Framed by total volume of issues found, Deque's automated tooling identified 57% of all logged accessibility issues — a higher figure than the industry's older, informally cited benchmark of 20–30% coverage. Dylan Barrell, Deque's CTO at the time, explained the shift explicitly: the 57% figure comes from changing the denominator from "percentage of WCAG success criteria that can be tested" to "percentage of total issue volume automated tools catch." That is a legitimate way to measure tool value, and it says something real: a large share of the individual defects that accumulate on a typical page are the kind of low-contrast text, missing labels, and empty links that pattern-matching catches efficiently.

It says something different from — and is often confused with — the question a compliance program actually needs answered: what fraction of the distinct success criteria in WCAG can a scanner test at all, regardless of how many individual instances of a given problem exist on a page. That second framing produces a much lower number, and it is the one that matters for legal defensibility, because a regulator or a plaintiff's expert is not counting flagged <img> tags. They are asking whether the product satisfies each of the specific, named success criteria in WCAG — and a page can have zero automated findings while still failing several of them outright.

An independent breakdown by accessibility consultancy Accessible.org, which walked through all 55 success criteria in WCAG 2.2 at Level AA and classified each one by how reliably a scanner can evaluate it, found:

Neither number is "wrong." Deque's 57% describes how much of the raw defect volume a well-configured automated pipeline eliminates before a human ever looks at the page, which is a meaningful efficiency gain worth capturing. Accessible.org's 13% describes how much of the legal and functional compliance surface — the actual list of criteria a product must satisfy — a scanner alone can certify. A testing program that quotes the 57% figure to justify skipping manual and assistive-technology testing is applying a defect-volume statistic to a compliance question it was never designed to answer.

The independent WebAIM Million project, which runs automated scans against the home pages of the top one million websites every year, offers a third, complementary data point: it measures how much fails even when the target is only what a scanner can see. WebAIM's most recent analysis found detectable WCAG 2 failures on 95.9% of the home pages tested (up from 94.8% the year before), with an average of 56.1 detectable errors per page — a 10.1% increase in average error count year over year. Six failure types accounted for 96% of all flagged errors: low-contrast text (83.9% of pages), missing alternative text (53.1%), missing form input labels (51%), empty links (46.3%), empty buttons (30.6%), and missing document-language declaration (13.5%). WebAIM also flagged a less obvious trend: pages using ARIA attributes averaged more errors (59.1 per page) than pages without ARIA (42 per page) — a strong signal that ARIA is frequently added incorrectly, often by teams trying to fix accessibility issues without understanding the semantics they are introducing. ARIA attribute usage rose 27% year over year even as failure rates worsened, which is precisely the pattern you would expect if teams are adding ARIA to satisfy scanners rather than to communicate accurate state to assistive technology.

Put together, these three sources tell a consistent story from three different angles: even the narrow slice of accessibility that automated tools are built to catch is failing on the overwhelming majority of live websites, and that narrow slice covers a minority of what the actual standard requires.

What a Scanner Structurally Cannot See

It helps to be specific about why these gaps exist, because the reasons are architectural, not a matter of tools needing another year of development. A scanner evaluates a snapshot of the DOM against static rules. It has no concept of user intent, no ability to operate a device the way a person does, and no access to what an assistive technology's synthesized output actually sounds or behaves like. Several categories of WCAG success criteria depend on exactly those things.

Focus order and keyboard operability (WCAG 2.4.3, 2.1.1, 2.1.2). A scanner can confirm that every interactive element is theoretically reachable by keyboard — it can see tabindex values and detect elements with no keyboard handler at all. It cannot walk through the interface pressing Tab in sequence and judge whether the order in which focus lands makes logical sense, whether a custom widget correctly traps and releases focus, or whether a user can ever get out of a component once they are in it. Consider a date picker built as a set of <div> elements with click handlers. A scanner may find nothing wrong: every element that needs a label has one, contrast passes, and no ARIA attribute is malformed. But if the developer never wired keyboard event handlers to the calendar grid, a keyboard-only user cannot select a date at all. The failure is total, and invisible to pattern matching, because pattern matching only looks at what exists in the DOM, not at what a device driven purely by a keyboard is actually able to do with it.

Meaningful sequence and visual-to-programmatic correspondence (WCAG 1.3.2). CSS can visually reposition content in ways that make the page look correct to a sighted user while the underlying DOM order — the order a screen reader reads content in — is scrambled. A three-column layout built with flex-direction: row-reverse or absolute positioning can look perfectly normal visually while a screen reader announces the content in an order no sighted person would recognize. No automated tool renders the page audibly and compares that experience to the visual layout; it only inspects markup, which by definition looks unremarkable when the problem is purely about sequence and presentation diverging.

Whether dynamic content is actually announced (WCAG 4.1.3, Status Messages). This is one of the most consequential gaps, and one of the least understood outside accessibility specialist circles. WCAG 2.1 added a success criterion specifically requiring that status messages — form validation errors, "item added to cart" confirmations, search result counts, save confirmations — be programmatically determinable so assistive technology can announce them without requiring focus to move to them. The typical implementation pattern is an aria-live region. A scanner can confirm that an element with aria-live="polite" or role="alert" exists somewhere in the markup. It cannot confirm any of the following, each of which determines whether the message is actually heard:

• Whether the live region existed in the DOM before the content changed (a region that is created and populated in the same operation is frequently not announced by several screen reader and browser combinations, because the assistive technology never had a chance to register the node).
• Whether the content is replaced so quickly by a subsequent update that the first message is cut off mid-announcement.
• Whether the live region is visually hidden in a way (display: none, visibility: hidden) that also removes it from the accessibility tree, silently defeating the entire mechanism.
• Whether competing live regions on the same page cause announcements to collide or interrupt each other.
• Whether the specific screen reader and browser combination in actual use by the target audience handles the pattern consistently — behavior here still varies meaningfully across JAWS, NVDA, and VoiceOver, and across browser engines, in ways no static rule captures.

A worked example makes this concrete. Here is a login form's error-handling markup that passes every common automated accessibility check:

<form id="login-form">
  <label for="email">Email</label>
  <input type="email" id="email" name="email" aria-describedby="email-error" />
  <span id="email-error" class="error-text" role="alert"></span>

  <button type="submit">Sign in</button>
</form>

<script>
  form.addEventListener('submit', (e) => {
    e.preventDefault();
    const errorSpan = document.getElementById('email-error');
    errorSpan.textContent = ''; // clear
    errorSpan.textContent = 'Please enter a valid email address.'; // set immediately after
  });
</script>

Every element has a label, an aria-describedby relationship, and a role="alert" container. An automated scan reports zero issues. But because the script clears and repopulates the same node in the same synchronous execution block, several screen reader and browser pairings never register the intermediate empty state as a change worth announcing, and in some configurations the announcement is dropped entirely on the first submission attempt — the user hears nothing and has no idea why the form did not proceed. Only a person testing with an actual screen reader, submitting the form and listening to what happens, will catch this. It requires no code change flagged by any linter; it requires interaction testing.

Custom widget conformance to expected interaction patterns (WCAG 4.1.2, 2.1.1). Any custom-built combobox, tab panel, accordion, tree view, or menu needs to replicate the exact keyboard and ARIA behavior that the W3C's ARIA Authoring Practices Guide (APG) documents for that widget type — specific arrow-key behavior, specific aria-expanded and aria-activedescendant state management, specific focus behavior on open and close. A scanner can confirm that role="combobox" is present. It has no way to confirm that pressing the Down arrow moves through the option list the way a screen reader user expects, that typing a letter jumps to a matching option, or that aria-expanded correctly flips to true when the list opens. Every one of these behaviors requires a person operating the widget with a keyboard and, ideally, a screen reader, and comparing the experience against the documented pattern.

Color and meaning beyond contrast ratio (WCAG 1.4.1). Automated tools check contrast ratios well. They cannot judge whether color is the only mechanism conveying meaning — a red border on an invalid field with no accompanying icon, text, or ARIA attribute passes every contrast check while remaining functionally invisible to anyone with certain forms of color vision deficiency or anyone using a screen reader, who receives no information that the field is invalid at all beyond the (possibly broken, per the previous section) live region announcement.

Timing, motion, and input alternatives (WCAG 2.2.1, 2.2.2, 2.5.7, 2.5.4). Whether a session timeout gives adequate warning and an extension mechanism, whether a carousel can be paused, whether a drag-and-drop interaction has a non-drag alternative, and whether functionality triggered by device motion has an equivalent that does not require it — all of these require a human to interact with the live behavior of the interface over time. None of them are visible in a static DOM snapshot.

The practical conclusion is not that automated tools are worthless. It is that they are a necessary, cheap, high-frequency filter for a specific and limited category of defects — the roughly 13% of criteria they can evaluate reliably, plus a meaningful share of the raw issue volume within that category — and they are structurally incapable of evaluating everything that depends on sequence, behavior, timing, or how a specific assistive technology actually renders an interface to a specific user. Treating a clean scan as proof of the rest is not a testing gap. It is a category error about what the tool was ever built to measure.

The Legal and Regulatory Landscape Now Tests the Part Scanners Miss

The reason this gap has moved from a quality concern to an executive risk conversation is that every enforcement mechanism gaining traction right now — litigation volume in the United States, a newly binding federal rule for government entities, and a new EU-wide law with real penalties — evaluates the product the way a person using assistive technology actually experiences it, not the way a scanner scores it.

United States: ADA Title III litigation remains a steady, material cost of doing business online

Title III of the Americans with Disabilities Act prohibits disability discrimination in "places of public accommodation," and federal courts have increasingly applied it to commercial websites and mobile apps, even though the ADA itself predates the web and contains no explicit statutory language about digital properties. There is no federal regulation specifying exactly what "compliant" means for a private company's website under Title III — courts commonly look to WCAG 2.0 or 2.1 Level AA as a reference point in settlements and rulings, but it is a de facto benchmark rather than a codified legal requirement for private businesses (this is a meaningful distinction from the DOJ's Title II rule for government entities, covered below, which does specify WCAG 2.1 AA explicitly).

Filing volume has stayed in a narrow, persistently high band for several years. According to Seyfarth Shaw's ADA Title III tracking, which compiles federal court filing data:

The 2025 figure represents a roughly 2% decline from 2024's total, not a meaningful retreat — it sits within the same range the industry has occupied for years, and it does not include the state-court filings (notably under New York and California state disability statutes) that plaintiffs' firms have increasingly shifted toward as some federal courts apply more rigorous standing requirements. California, Florida, and New York remain the top three filing jurisdictions by a wide margin, with Illinois emerging as a fast-growing fourth as filing patterns shift among plaintiffs' firms. Three states recorded zero federal filings in 2025 (Montana, North Dakota, South Dakota), underscoring how concentrated this litigation activity is in specific jurisdictions with plaintiff-friendly precedent and active plaintiffs' bars — a detail worth knowing if your risk model assumes uniform national exposure.

What actually happens in these cases is the part that matters for a testing program. Complaints typically allege that a named plaintiff, using a screen reader (JAWS and NVDA are named most often, consistent with their dominance in the WebAIM Screen Reader User Survey data below), was unable to complete a specific task: browsing a specific product category, applying a discount code, completing checkout, filling out a contact form, or navigating a specific interactive component. Plaintiffs' experts in these matters typically produce accessibility reports built from actual assistive-technology testing, not automated scan output — because a scan report showing "0 critical issues" is not what caused the plaintiff's real, lived difficulty using the site, and a defendant's scan report showing the same thing does not rebut testimony about an actual failed user experience. A clean Lighthouse score is not admissible evidence that a blind user could complete checkout; it is evidence that the DOM contains no pattern the tool was built to flag, which is a narrower and different claim.

The DOJ's Title II rule: the first US regulation to name WCAG 2.1 AA explicitly, with a real compliance calendar

In April 2024, the US Department of Justice finalized a rule under Title II of the ADA (which covers state and local government entities, not private businesses) requiring that web content and mobile applications conform to WCAG 2.1 Level AA — the first time a US federal regulation has named a specific WCAG version and level as a binding legal standard rather than a persuasive reference point. The rule set two original compliance deadlines based on jurisdiction population, both measured from the April 24, 2024 publication date:

In April 2026, DOJ issued an interim final rule extending both deadlines by one year, in response to implementation concerns raised by public entities. This extension is directly relevant to any company selling software to state or local government agencies, school districts, public universities, transit authorities, or public utilities: your government customers now carry a specific, named legal obligation to conform to WCAG 2.1 AA on the timelines above, and they are increasingly pushing that obligation contractually onto software vendors through procurement requirements, VPATs (Voluntary Product Accessibility Templates), and accessibility conformance reports. A SaaS vendor with no credible accessibility testing program is going to find this rule showing up in RFPs and security/compliance questionnaires well before any deadline arrives, because government purchasers need to document their own compliance chain and will not accept a vendor's self-reported "we ran a scanner" attestation as sufficient evidence, precisely because it is not.

Two things about this rule matter beyond its direct scope. First, it establishes WCAG 2.1 AA as an explicit, named US federal standard for the first time — a fact litigants and commentators can and do point to when arguing what "reasonable" accessibility looks like more broadly, even outside Title II's direct coverage. Second, it did not name automated testing as a compliance methodology anywhere in its text; conformance is defined by the actual WCAG success criteria, which — per the analysis above — are only 13% reliably verifiable by a scanner.

The European Accessibility Act: enforcement is now live, not upcoming

The European Accessibility Act (EAA), an EU directive requiring member states to harmonize accessibility requirements for a defined set of products and services, reached its enforcement date on June 28, 2025. Member states had until June 28, 2022 to transpose the directive into national law, with the substantive application date arriving three years later. This is not a future deadline to plan around; for any company selling covered products or services into the EU market, it is a present legal obligation.

The EAA's scope is broader than websites alone. It covers, among other categories: computers and operating systems, smartphones and tablets, self-service terminals such as ATMs and ticketing machines, e-readers, consumer banking services, e-commerce, transportation ticketing and information services, and audiovisual media services. Coverage applies to manufacturers, service providers, importers, and distributors placing covered products or services on the EU market, regardless of where the company is headquartered — a US-based SaaS company selling e-commerce or fintech software used by EU consumers falls within scope if its product or service category is covered, not just EU-domiciled companies.

The technical presumption-of-conformity standard is EN 301 549, the European harmonized standard for ICT accessibility, whose current version (V3.2.1, published March 2021) incorporates WCAG 2.1 Level AA success criteria directly into its web-content clause, while also extending further into non-web software, documents, and hardware requirements that WCAG itself does not address. A product conforming to EN 301 549 is presumed to satisfy the EAA's accessibility requirements, per the standard's official technical summary. Penalties for non-compliance are set by each EU member state individually rather than by the EAA directly, and range from financial penalties (with at least one member state permitting criminal penalties for serious or repeated violations) to withdrawal of the non-compliant product or service from the market — a materially different and more severe enforcement mechanism than the monetary settlements typical of US ADA Title III litigation, because market withdrawal is an operational, not just financial, consequence.

The throughline across all three enforcement mechanisms — US litigation practice, the DOJ's Title II rule, and the EAA — is the same: none of them define compliance as "an automated scan reports no critical issues." All three define it, explicitly or through consistent practice, as the product's actual behavior against a named, human-experience-based standard (WCAG 2.1 AA in the US federal and EU cases; a similar de facto benchmark in Title III practice), evaluated by real interaction, not by static pattern matching.

Single-Page Applications Multiply the Failure Modes a Scanner Cannot See

Most of the examples above generalize across any web product, but modern single-page applications built with React, Vue, Angular, or similar frameworks introduce an additional category of accessibility defects tied specifically to how these frameworks manage rendering, and this category is almost entirely invisible to automated tools because it depends on the sequence of DOM mutations over time, not the state of the DOM at any single instant a scanner inspects.

Route changes without focus or announcement management. In a traditional multi-page website, navigating to a new page triggers a full document reload, and screen readers automatically announce the new page's title and reset focus to the top of the document. A single-page application intercepts navigation and swaps content via JavaScript, without a full page load, and the browser has no built-in mechanism to tell assistive technology that a "new page" has effectively appeared. Unless the application explicitly moves focus to a logical location (commonly the new view's heading) and updates the document title on every route change, a screen reader user who clicks a navigation link hears nothing change at all — the interface has moved on, visually, while their assistive technology continues to behave as though they are still on the previous view. This defect is completely invisible to a scanner evaluating a single rendered route in isolation, because the problem exists only in the transition between two states, not in either state alone.

Live region timing broken by virtual DOM diffing. Frameworks that use a virtual DOM (React is the most common example) reconcile changes efficiently by comparing a new render tree to the previous one and applying only the minimal set of real DOM mutations needed. This is excellent for performance and can be quietly hostile to aria-live regions, because the framework's diffing algorithm may choose to update a text node's content in place rather than removing and re-inserting the element, and the precise sequence and timing of those mutations is what determines whether a given screen reader registers a change worth announcing. A component that renders correctly, has valid ARIA attributes, and passes every automated check can still fail to announce reliably purely because of how the framework's rendering engine chose to apply the update — a detail with zero visibility in the markup a scanner inspects, since the DOM looks identical whether the update path was one mutation or three.

State restoration after conditional rendering removes and re-adds elements. Conditional rendering patterns ({condition && <Component />}) are idiomatic in component frameworks and routinely used to show and hide error messages, modals, and expandable sections. Each time the condition flips, the component's underlying DOM node is destroyed and a new one created, rather than an existing node being shown or hidden. For a live region intended to announce a validation error, this pattern reproduces exactly the same registration problem described in the login-form example earlier, but now embedded as a default pattern across an entire codebase built by teams who never separately audited how their framework's conditional rendering interacts with assistive technology announcement timing. Because the resulting markup is syntactically identical to a working implementation, no automated rule distinguishes the two.

Client-side routing breaking skip links and landmark navigation. Skip-to-content links and ARIA landmark regions (main, nav, banner) are foundational screen reader navigation aids, and they depend on stable, predictable page structure. Some single-page application architectures re-render the entire application shell on certain state changes, briefly removing and re-creating landmark elements, which can disrupt a screen reader user's established mental map of the page structure in ways a static, single-render snapshot never reveals.

The practical implication is that framework choice and rendering architecture are accessibility decisions, not purely performance or developer-experience decisions, and teams building on component frameworks need manual, assistive-technology-driven testing specifically around navigation and state transitions, not just around static component appearance.

Mobile Applications Carry the Same Gap, With Different Instrumentation

Everything argued above about web accessibility testing applies with equal force to native and hybrid mobile applications, and the same enforcement mechanisms cover them: the DOJ's Title II rule explicitly includes mobile applications alongside web content, and the EU Accessibility Act's coverage of smartphones, e-commerce, and banking services applies regardless of whether the interface is delivered through a browser or a native app. The specific tooling differs, but the underlying limitation is identical.

iOS testing relies on VoiceOver, Apple's built-in screen reader, and Android testing relies on TalkBack, Google's equivalent. Both platforms ship automated auditing tools of their own — Xcode's Accessibility Inspector on iOS, and Android's Accessibility Scanner app — and both operate on the same fundamental principle as their web counterparts: they check whether interface elements expose the platform-required accessibility properties (accessible name, role, state, and value), not whether the actual experience of navigating the app with VoiceOver or TalkBack running makes sense. A custom-drawn interactive control in a mobile app — a swipeable card, a custom slider, a gesture-based interaction — can expose a technically valid accessibility label while remaining functionally unusable by swipe-based screen reader navigation if the developer never tested the actual swipe-through order or implemented the custom actions VoiceOver and TalkBack both support for exposing non-standard gestures through an accessible alternative.

Mobile introduces failure categories with no direct web equivalent worth naming specifically: gesture-only interactions (a swipe-to-delete pattern with no accessible alternative button), dynamic type and text scaling support (whether the app's layout survives a user's system-wide font size increase without clipping or overlapping content), and orientation-lock decisions that can block screen magnification workflows some low-vision users depend on. None of these are evaluated by an automated accessibility scan of the app's view hierarchy; each requires operating the actual app with VoiceOver or TalkBack enabled, exactly as a real user would, including testing common gestures (swipe left/right to move between elements, double-tap to activate, two-finger swipe for scrolling) rather than only confirming that accessibility labels exist in the underlying code.

Three Scenarios That Show What the Gap Actually Costs

The following three scenarios are hypothetical, built to be realistic and specific rather than generic, and are not based on any real QAtronic client, project, or reported outcome. Any figures used are illustrative estimates for the purpose of explaining the mechanism, not real benchmarks or reported results.

Scenario one: the e-commerce keyboard trap that a 100% Lighthouse score never caught

Initial situation. A direct-to-consumer apparel retailer runs Lighthouse's accessibility audit as a required CI gate before every deploy; the team treats a score below 95 as a blocking failure. The score has been at or near 100 for over a year. The checkout flow includes a "apply promo code" interaction implemented as a modal dialog built with a third-party UI library, styled to match brand guidelines with custom CSS.

Hidden assumption. The engineering team assumed that because the modal component came from a well-regarded open-source library with documented accessibility support, and because the automated scanner found nothing wrong with the rendered markup, keyboard and screen reader behavior was handled correctly by construction.

Technical cause. The library's default focus-trap behavior had been partially overridden by a custom onOpen handler added to auto-focus the promo code input field, which inadvertently removed the library's Escape-to-close keyboard binding and its return-focus behavior (returning focus to the triggering button on close). A keyboard user could tab into the modal and focus the input field, but pressing Escape did nothing, and Tab cycled only within a subset of the modal's own interactive elements without ever reaching the close button, because the close button had been visually repositioned via CSS in a way that also altered its position in the DOM's focus order.

Consequence. A keyboard-only or screen-reader user who opened the promo code modal had no way to close it and return to the checkout flow without reloading the entire page, losing their cart's applied state in the process. This is exactly the kind of failure that generates an ADA Title III complaint: a specific, describable, task-level breakdown a real user encountered, not an abstract markup deficiency.

Decision that needed to be made. Whether to continue treating a CI-gated automated score as sufficient release criteria for checkout-critical interactive components, or to require manual keyboard testing specifically for any custom or overridden interactive widget before it ships, regardless of automated score.

Better approach. Component-level manual keyboard testing (tab in, tab out, Escape, Enter, Space, arrow keys as applicable) added as a required step for any interactive component with custom focus-management code, performed once per component and re-verified whenever the focus-handling logic changes — not run against every page on every deploy, but specifically targeted at the components where custom behavior overrides a library's tested defaults.

Scenario two: the fintech dashboard where "success" and "failure" sounded identical

Initial situation. A B2B fintech SaaS platform allows finance teams to schedule outbound vendor payments. When a payment submission succeeds, a green toast notification appears in the corner of the screen for four seconds and then disappears. When a payment submission fails (insufficient balance, invalid routing number, duplicate transaction detected), a red toast notification appears in the same location, styled differently, also for four seconds.

Hidden assumption. The design and engineering teams assumed that because both toast types used the same aria-live="polite" container and both contained descriptive text, a screen reader user would receive equivalent information to a sighted user, just through a different sensory channel.

Technical cause. The toast container was a single, shared, persistent DOM node reused for every notification, with its text content and CSS class replaced on each new toast. Because the replacement happened via a single synchronous DOM mutation, and because the previous toast's content was cleared and replaced within the same render cycle rather than being removed and re-inserted as a new node, several screen reader and browser combinations either did not announce the change at all on rapid successive submissions, or announced only a fragment of the message before the four-second auto-dismiss removed it from the DOM. A user attempting to retry a failed payment in the same session sometimes never heard why the first attempt failed.

Consequence, as a realistic illustrative outcome: a blind finance operations user, unaware that their first submission had failed, resubmitted the same payment, creating a duplicate transaction that required manual reconciliation and vendor contact to reverse — a direct operational and financial consequence stemming from a testing gap, not a code defect a linter would ever catch.

Decision that needed to be made. Whether transactional confirmation and error messaging for financial actions should rely on a transient, auto-dismissing toast pattern at all, versus a persistent, dismissible status message that remains in the DOM and accessibility tree until the user acknowledges it — a product and interaction-design decision, not only an engineering fix.

Better approach. For any financially consequential confirmation or error state, use a persistent status region that does not auto-dismiss, insert it as a genuinely new DOM node per message (rather than mutating a shared node) to ensure reliable announcement, and verify the actual announcement behavior with real screen reader and browser combinations reflecting the organization's actual user base — not just confirm the presence of aria-live in the markup.

Scenario three: the settings page a blind engineering manager could not use

Initial situation. A project-management SaaS product built a custom multi-select dropdown component as part of its internal design system, used across the product for filtering, tagging, and — critically — for a permissions and entitlement settings screen where account administrators assign feature access to team members.

Hidden assumption. The design system team assumed that because the component visually resembled a native <select> element and had been reviewed for color contrast and labeling, it inherited the keyboard operability that a native HTML select element provides automatically.

Technical cause. The component was built entirely from styled <div> and <span> elements with click handlers, with no keyboard event listeners attached at all — no response to Enter or Space to open the list, no arrow-key navigation between options, and no role="listbox" or related ARIA pattern implemented. It was visually indistinguishable from an accessible native control while being completely inoperable without a mouse.

Consequence. An account administrator using keyboard-only navigation (a mobility-related reason unrelated to vision, illustrating that accessibility failures affect more than screen reader users) could not open the dropdown at all, meaning they could not assign or revoke feature entitlements for their team — a core administrative function of the product, not a peripheral feature. Because this component was part of a shared design system, the same defect existed across every screen that reused it, multiplying a single implementation gap into dozens of broken workflows across the product.

Decision that needed to be made. Whether accessibility conformance testing for shared design-system components should be a one-time review gate before a component is published to the library, given that any defect at that layer propagates to every consuming team automatically — versus relying on each consuming team to independently discover and report the problem.

Better approach. Treat the design system's component library as the highest-leverage point for manual keyboard and screen reader testing in the entire organization: a defect fixed once at the component level is fixed everywhere it is used, while a defect caught only at the page level has to be rediscovered and reported separately for every page that happens to use the broken component.

A Three-Layer Testing Model Built for Actual Defensibility

The organizations that get sued or fail procurement accessibility reviews repeatedly are rarely the ones with no automated testing. They are the ones whose testing program stops at automated testing. A model that would hold up under real scrutiny — a plaintiff's expert, a government procurement review, an EU market surveillance authority — needs three distinct layers, each catching what the others structurally cannot.

Layer 1: automated scanning, run continuously. This layer belongs in CI, gating builds on newly introduced violations, and should cover every page and every build, because it is cheap and fast enough to run at that frequency. Its job is to prevent regressions in the roughly 13% of criteria it can reliably check and to reduce the raw volume of low-level defects (contrast, missing labels, empty links) that would otherwise consume manual testing time on issues a machine catches perfectly well. This layer should never be described internally as "accessibility testing" without qualification; it is automated accessibility scanning, one input into testing, not the whole activity.

Layer 2: structured manual and assistive-technology testing, targeted by risk. This layer cannot run on every page on every build economically, and does not need to. It should be targeted at: any new or substantially modified custom interactive component (modals, dropdowns, date pickers, drag interactions, multi-step forms); any workflow that is core to the product's value proposition (checkout, sign-up, primary task completion, administrative and settings functions); and any component published into a shared design system, per the leverage point illustrated in scenario three above. Testing at this layer means, at minimum: completing the full task using keyboard only, with no mouse; completing the full task using a screen reader with the display off or ignored, listening only to what is announced; and verifying behavior against the specific screen reader and browser combinations most representative of real usage. The WebAIM Screen Reader User Survey's most recent results are a useful reference point for prioritizing combinations: JAWS with Chrome (24.7% of respondents), NVDA with Chrome (21.3%), and JAWS with Edge (11.4%) were the three most common pairings, with JAWS (40.5% primary, 60.5% used at all) and NVDA (37.7% primary, 65.6% used at all) dominating desktop screen reader usage and VoiceOver (9.7% primary, 43.9% used at all) trailing but still significant, particularly for macOS and iOS-heavy user bases. A testing program that only ever tests with VoiceOver on macOS because that is what is installed on the team's laptops is testing against the least representative combination for a typical desktop-oriented B2B SaaS user base.

Layer 3: documentation and governance that produces a defensible record. This is the layer most testing programs skip entirely, and it is the layer that determines whether the first two layers matter to anyone outside the engineering team. It includes maintaining a current Accessibility Conformance Report (commonly built on the VPAT template), documenting specifically which success criteria were tested manually, with which assistive technology, on which date, by whom, and with what outcome; tracking identified issues to closure rather than to a backlog that never gets prioritized; defining and following a retest cadence tied to product changes, not just a point-in-time annual audit; and, for organizations in EU market scope, publishing the accessibility statement the EAA and related EU accessibility rules require. This layer is what turns "we tested this" into "here is what we tested, how, and when," which is the actual question a procurement reviewer, a regulator, or opposing counsel will ask.

The Audit Trail Question Most Programs Cannot Answer

Ask an engineering or QA leader whether their product has been accessibility tested, and most say yes. Ask the follow-up questions, and the confidence usually collapses:

• Which specific WCAG 2.1 or 2.2 success criteria were evaluated, and which were explicitly out of scope for this round?
• Was each criterion evaluated automatically, manually, or both — and if manually, by testing with an actual screen reader and keyboard, or by visual inspection of code?
• Which screen reader and browser combinations were used, and do they reflect the organization's actual customer base rather than whatever happened to be installed on the tester's machine?
• When was the most recent retest, and what changed in the product since then that might have invalidated previous findings?
• For every issue found, what is its current status — fixed, in progress, accepted risk, or simply logged and forgotten?
• Is there a current, accurate VPAT or Accessibility Conformance Report that a customer's procurement team, a government agency, or opposing counsel could be handed today without embarrassment?

None of these questions are answered by a scan report showing zero critical issues. A scan report answers exactly one question — "did our automated ruleset find any of the pattern matches it is built to detect, as of this run" — and every one of the six questions above requires a separate, deliberate answer.

This is not bureaucratic overhead for its own sake. In litigation, in regulatory review, and in enterprise or government procurement, the absence of a credible record is treated as equivalent to the absence of testing, regardless of what engineers privately believe they did. A team that manually tested a checkout flow with NVDA six months ago but kept no record of what was tested or found has, from an evidentiary standpoint, no meaningfully stronger position than a team that never tested it at all — because there is nothing to show anyone asking.

Metrics That Mislead, and Metrics That Do Not

Accessibility programs get measured, and the choice of metric quietly determines what the organization actually optimizes for. Several commonly reported metrics look reassuring while measuring almost nothing about real usability, and a smaller set of less convenient metrics tracks something closer to the truth.

Automated scan score, tracked over time, is the most common and most misleading metric in isolation. A rising score can reflect genuine defect reduction, or it can reflect a shrinking set of pages included in the scan's crawl scope, or components being rebuilt in ways that happen to avoid the specific patterns the tool checks for while introducing new problems the tool cannot see, such as the keyboard trap in the first scenario above. A score by itself, without a record of what pages and components it covers and what specifically changed between measurements, tells a leadership team almost nothing about trend direction in real usability.

Number of open accessibility issues in a backlog is similarly unreliable without qualification. A team can drive this number toward zero by closing easy, low-impact automated findings (a missing alt attribute here, a contrast tweak there) while a small number of severe, task-blocking manual findings — the kind found in the three scenarios above — sit untouched because they require more engineering effort to fix and get continually deprioritized against feature work. A shrinking issue count with no severity or impact weighting can represent declining real risk or a false sense of progress built entirely on the easiest fixes.

More useful, if less flattering, metrics include: the percentage of core workflows (not pages — workflows, meaning an entire task from start to finish) that have been manually verified with keyboard and screen reader testing within a defined period; the age of the most recent manual test for each core workflow, surfaced explicitly rather than left implicit, so that a workflow untested in the last two release cycles is visible as a gap rather than assumed fine by default; the number of severe, task-blocking findings (not total findings) discovered per manual testing session, tracked over time, since a rising or flat rate here after months of remediation work indicates the testing process itself may be missing things, not that the product has genuinely improved; and time-to-remediation specifically for findings classified as blocking a core task, as distinct from cosmetic findings, since blending both into a single average remediation time obscures whether the issues that actually generate legal and usability risk are being fixed with appropriate urgency.

The distinction worth internalizing is that a metric built from what is easy to count (total findings, aggregate score) tends to reward activity that is easy to perform, while a metric built around task completion and workflow coverage tends to reward activity that actually reduces the specific risk this article describes. Leadership reporting that only shows the first category should be treated as a yellow flag regardless of how favorable the numbers look.

An Accessibility Testing Maturity Model

The following is an original framework developed for this article, not a published industry standard, intended to help engineering and QA leaders locate their organization honestly and identify the next concrete step rather than trying to leap to full maturity at once.

Most organizations that believe they are at Level 3 or 4, when asked the specific audit-trail questions above, are actually at Level 1 or 2. The gap between self-perception and actual maturity is itself a risk factor, because it produces false confidence at exactly the moment a real test — a lawsuit, a procurement review, a regulatory inquiry — would expose the difference.

A Diagnostic Checklist: What a Clean Scan Report Will Never Verify

Use this as a starting checklist for manual and assistive-technology testing on any release involving new or modified interactive functionality. None of these items can be fully verified by an automated scanner; each requires a human performing the action described.

1. Complete the primary task using only a keyboard. No mouse, no trackpad. Can every interactive element be reached, and does Tab order match a sensible reading order?
2. Verify visible focus indication at every step. Is it always clear, visually, which element currently has keyboard focus, including inside custom components?
3. Test every modal, dialog, and overlay for focus trapping and release. Does focus move into the dialog on open, stay contained within it while open, and correctly return to the triggering element on close, via both a close button and the Escape key?
4. Submit every form with invalid data using a screen reader, with the display off or ignored. Is the error actually announced, immediately, in a way that identifies which field is invalid and why?
5. Trigger every transient status message (toasts, save confirmations, loading states) with a screen reader running. Is the message announced in full before it disappears from the DOM?
6. Operate every custom widget (combobox, listbox, tree, tab panel, accordion, slider) using only the keyboard interactions documented in the W3C ARIA Authoring Practices Guide for that pattern. Does it behave as documented, not just visually resemble the pattern?
7. Zoom the page to 400% and reflow the viewport to a narrow width simultaneously (WCAG 1.4.10). Does content remain usable without requiring horizontal scrolling for primary reading content?
8. Check whether any content relies on color alone to convey meaning. Cover or desaturate the page; is all necessary information still available?
9. Test any timed interaction (session timeout, auto-advancing carousel, timed form) for a working pause, extend, or disable mechanism.
10. Test any drag-and-drop interaction for a non-drag alternative (WCAG 2.5.7).
11. Verify reading order with a screen reader against visual layout, particularly on any page using CSS Grid, Flexbox reordering, or absolute positioning.
12. Confirm page titles, headings, and landmark regions allow a screen reader user to understand and navigate page structure without reading linearly from the top.
13. Retest all of the above after any design system component update, not just after page-level changes, since a shared component defect propagates silently to every page that consumes it.

Findings from this checklist should be triaged by task impact, not by how they were discovered. A defect that fully blocks completion of a core workflow — the keyboard trap in the first scenario, the silent payment error in the second — warrants the same release-blocking severity regardless of whether it came from manual testing or an automated scan, and regardless of how many or few individual instances of it exist on the page. Treating every manually-found issue as automatically lower priority than automated findings simply because it did not come from a CI-gated tool inverts the actual risk ordering: the manually-found issues are disproportionately the ones tied to real task failures, while automated findings are disproportionately the ones with the least direct relationship to whether a task can be completed at all.

Where Automated Testing Still Earns Its Place

None of this argues for abandoning automated scanning; it argues against treating it as sufficient. Automated tools remain the right choice for exactly what they do well: catching regressions continuously, at zero marginal human cost per run, across an entire site or application on every single build. A contrast ratio regression introduced by a design system color token change, a missing alt attribute on a newly added image, a form field that lost its label during a refactor — these are precisely the defects a CI-gated scanner catches before a human ever needs to look, and running one on every build is unambiguously worth the cost.

The discipline required is keeping the tool's role scoped honestly. A scanner passing should be read as "no regressions in the roughly one in eight WCAG criteria this tool can evaluate," not as "accessible." Teams that rename the automated gate internally from "accessibility check" to "accessibility regression check" — a small language change — tend to stop conflating the two, because the more accurate name makes the limitation explicit every time someone sees it in a CI log.

There is a second, less obvious value automated tooling provides: it makes manual testing more efficient by clearing away the volume of low-severity, easily-detected defects before a human tester's time gets spent on them. A manual tester who opens a page riddled with missing labels and contrast failures the scanner should have already caught is spending scarce, expensive attention on defects a machine handles for free, instead of on the sequence, timing, and behavioral questions only a human can evaluate. Keeping the automated gate strict and current is what makes the manual layer affordable to run consistently, rather than an either/or trade-off between the two.

Who Should Own This, at Different Stages of Growth

Early-stage startups typically cannot justify a dedicated accessibility engineer, and should not try to. The highest-leverage investment at this stage is building the design system's core interactive components (buttons, form fields, modals, dropdowns) correctly from the start, since a small number of well-built shared components propagate correctness across the whole product far more efficiently than page-by-page remediation later. A single engineer with genuine screen reader and keyboard testing competence, embedded in the team building the design system, delivers more risk reduction per hour than a broad but shallow scanning initiative across every page.

Scale-ups with a growing product surface and multiple engineering teams typically need to formalize Layer 2 and Layer 3 practices before they become unmanageable: a defined set of triggers for when manual and AT testing is required (new custom components, core workflow changes, anything touching forms or checkout), a shared, documented set of AT/browser combinations the whole organization tests against consistently, and a lightweight but real issue-tracking and retest process. This is also the stage where enterprise and government sales cycles start surfacing VPAT and accessibility questionnaire requirements, making Layer 3 documentation a sales-enablement need as much as a risk-reduction one.

Enterprises operating at a scale where hundreds of engineers touch the product independently need dedicated ownership — an accessibility engineering function or center of excellence responsible for the design system's conformance, training and enabling other teams, running or commissioning periodic third-party audits for independent verification, and owning the governance layer (VPAT maintenance, accessibility statement, regulatory tracking across every jurisdiction the product operates in). At this scale, relying on individual engineer initiative to catch accessibility defects is not a strategy; it is an admission that no one owns the outcome.

A pattern worth naming explicitly because it recurs across all three stages: accessibility ownership frequently gets assigned to whichever function is most visible at the moment a complaint or sales blocker surfaces, rather than to the function actually positioned to prevent the next one. A legal team fielding a demand letter is not positioned to fix focus management in a shared modal component; a sales team blocked by a procurement questionnaire is not positioned to retest a checkout flow with NVDA. Durable ownership belongs with whoever controls the design system and release process, because that is where a fix, once made, actually stays fixed across the whole product rather than being patched page by page in response to whichever complaint arrived most recently.

Internal Capability Versus Third-Party Audit: A Decision Most Teams Make by Default Instead of by Analysis

A recurring question once an organization accepts that manual and assistive-technology testing is necessary is whether to build that capability internally or commission it from an outside specialist. Most companies never actually decide this question; they default into whichever option is easiest to procure at the moment a customer or legal question forces the issue, which is rarely the option best suited to their actual risk profile. The two approaches are not mutually exclusive, and the right mix depends on release frequency, product surface area, and what triggered the need in the first place.

The pattern that best matches how litigation, procurement, and regulatory review actually work is layered, not either/or: internal manual and assistive-technology testing embedded into the regular release process for continuous coverage of changes as they ship, combined with a periodic independent audit — commissioned often enough to validate that the internal process is actually working, and to produce the more credible, third-party-verified documentation that a VPAT, a procurement reviewer, or opposing counsel will weigh more heavily than a self-reported internal claim. Neither one substitutes for the other; internal testing without periodic independent validation drifts toward self-serving blind spots over time, and periodic audits without continuous internal testing between them leave every feature shipped after the audit date completely unverified until the next engagement, which for a fast-moving product can be the majority of the product's surface area at any given time.

Frequently Asked Questions

Does conforming to WCAG 2.1 AA guarantee legal compliance in the United States? No, for private companies under ADA Title III there is no codified federal regulation naming WCAG as the exact legal standard, and courts and settlements vary in exactly what they require, though WCAG 2.1 or 2.2 AA is consistently used as the practical reference point by courts, plaintiffs, and defendants alike. For US state and local government entities, the DOJ's Title II rule does name WCAG 2.1 AA as a specific, binding legal standard with defined compliance dates. Conformance substantially reduces risk and gives you a credible, evidence-based position in either context, but "conformance" has to mean actually satisfying the criteria through real testing, not passing an automated scan against a subset of them.

Can AI-based accessibility tools replace manual and screen-reader testing? Newer AI-assisted tools can help triage findings, suggest fixes, and in some cases simulate certain interaction patterns faster than manual testing alone, but they operate on the same fundamental limitation as rule-based scanners when it comes to verifying subjective, behavioral, or context-dependent criteria: they infer rather than directly observe what a real assistive technology does with real markup in a real browser. Treat AI-assisted tools as a way to make Layer 1 faster and Layer 2 more efficiently targeted, not as a substitute for either.

Is a VPAT the same thing as an accessibility audit? No. A VPAT (Voluntary Product Accessibility Template) is a standardized document format for reporting conformance status against a given standard (typically WCAG and Section 508 in the US, or EN 301 549 in the EU context). It is the output document, not the testing methodology. A VPAT filled out based on developer self-assessment without underlying manual and AT testing is a documentation exercise, not evidence of actual conformance, and sophisticated procurement reviewers increasingly ask how the VPAT's claims were verified.

Does a US-based SaaS company need to worry about the European Accessibility Act if it has no EU offices? The EAA's obligations attach to placing covered products or services on the EU market, not to where the company is headquartered. A US company selling e-commerce, consumer banking, or certain other covered service categories to consumers in the EU falls within scope based on where the product or service is offered and used, not the seller's domicile. Companies without EU-based legal counsel should get a specific determination for their product category rather than assuming exemption based on headquarters location.

What is the fastest way to reduce risk without commissioning a full third-party audit immediately? Apply the three-layer model's Layer 2 checklist above to the single highest-traffic, highest-value workflow in the product first — typically checkout, sign-up, or the core task the product exists to perform — using real keyboard-only and screen reader testing, not visual review of code. This targets the workflow most likely to generate a complaint or litigation, and most likely to be tested directly by any plaintiff's expert, before spreading effort thinly across lower-risk areas.

How often should manual accessibility testing be repeated? Tie it to change, not to a calendar. Any new or substantially modified interactive component, any change to focus-management or dynamic-content logic, and any addition to a shared design system should trigger a retest of the affected pattern. Independent of specific changes, core workflows (checkout, sign-up, primary task completion) warrant periodic full manual verification because small, individually low-risk changes across many releases can accumulate into a workflow-level failure that no single code review catches.

If our product passed a third-party accessibility audit last year, are we covered? Covered against what was tested, as of when it was tested, and nothing shipped since. An audit is a snapshot, not an ongoing guarantee, and its evidentiary value degrades with every subsequent release that touched the audited surface without a corresponding retest. A year-old clean audit sitting alongside a dozen unreviewed releases is a weaker position than most leadership teams assume it is, precisely because the gap between the audit date and today is exactly where new defects of the kind described throughout this article are most likely to have been introduced.

What This Means for the Release Process You Already Have

Reducing this risk does not require building an accessibility program from nothing. It requires being honest about what your existing automated gate actually proves, adding a targeted layer of real keyboard and screen-reader testing at the points of highest leverage — shared components and core workflows — and keeping a record specific enough to answer the six audit-trail questions above without scrambling. QAtronic works with engineering teams to build exactly this kind of layered verification into an existing release process: automated checks that catch regressions continuously, manual and assistive-technology testing targeted at the components and workflows where real usability risk concentrates, and a documentation practice that turns testing activity into a defensible record rather than an assumption. The goal is not a perfect audit; it is a testing discipline that reflects what your product actually does when a real person, using real assistive technology, tries to use it.

The Distinction That Actually Matters

A scanner tells you whether your markup matches a pattern it was built to recognize. It has never told you, and structurally cannot tell you, whether a person using a screen reader can complete the task your product exists to perform. Conflating those two facts is not a minor technical imprecision; it is the specific gap that litigation, government procurement review, and EU market enforcement are now built to expose, because every one of those mechanisms evaluates the second question and ignores the first as irrelevant to it.

The organizations most exposed right now are not the ones with the most markup defects. They are the ones whose engineering and QA leadership genuinely believes a 100% automated score means the product is accessible, and has never had that belief tested by someone who actually navigated the product without a mouse or without sight. That belief is comfortable, cheap to maintain, and wrong in a specific, demonstrable way for the majority of what the applicable standards actually require.

The question worth taking back to your engineering team is not whether the accessibility scanner passed on the last build. It is whether anyone on the team has personally completed your product's core workflow using only a keyboard, or with a screen reader and the monitor off, in the last release cycle. If the honest answer is no, the scanner's score is not evidence of anything beyond its own narrow definition of a pass.