What Automated Accessibility Simply Cannot Fix

Updated: April 4, 2022

It’s not just Accessibe

I had a conversation with a prospective client last week who was considering installing the UserWay automated accessibility widget on his site. When I shared some of the reasons why automated accessibility overlays are ineffective, he was surprised that I thought UserWay likely suffered from these same problems. Accessibe’s shortcomings at bringing its customers into ADA and WCAG compliance has featured in multiple federal lawsuits and as a result has gotten most of the negative media attention, but in reality most of the issues are inherent to any attempt to automate accessibility via an “AI” overlay (at least as the technology stands today).

As I dug a little more, I found that this prospective client’s confusion was not as surprising. UserWay has used the lawsuits and negative press around Accessibe in its marketing (see UserWay vs acessiBe) just as Accessibe has published claims that UserWay is likely to put its customers at risk of litigation.

For reference:

• Using a free plugin like UserWay? You are at risk of litigation
• UserWay “AI” Widget can risk your accessibility compliance

It's also worth noting that Userway has itself been tested in court. In the case of Paguada v. Yieldstreet, the defendant added Userway to its site after being sued and then attempted to file for dismissal on the basis that the issues had been corrected and the suit was therefore "moot". The court slapped this down and has ruled that the lawsuit should move forward.

The Overlay Fact sheet, written and signed by some of the leading accessibility experts, calls out a number of other automated accessibility service providers in addition to UserWay and Accessibe, but ultimately there is no need to single out any specific company. I’m nowhere near an expert in AI and I’ll never say never, but for now there are things that are simply impossible to address automatically. The honest overlay and widget vendors will acknowledge this, but no matter what they say, there are some items that need to be addressed manually.

These items can be divided between those that automated accessibility vendors can’t find, can’t fix, or won’t fix. This post gets a little bit into the weeds on the current technical limitations of automated accessibility. If you don't have a technical background, you should still be able to follow along.

Undetectable accessibility violations

There are some great automated accessibility scan tools including WAVE and Google’s Lighthouse that can review a webpage URL and share a list of accessibility issues that need to be fixed. Those familiar with the full set of Web Content Accessibility Guidelines (WCAG) know that most potential accessibility violations can’t be detected at all by an automated scanner. And there are some areas where the scanners can tell you that you might have a problem but can’t confirm for sure.

Depending on your tolerance for false positives, these automated scanners can flag somewhere between 25% to 40% of possible accessibility violations at most.

The $10,000 question is: if it’s not possible to automatically find an issue, how can an overlay automatically fix it?

Items that overlays can’t detect

This is a non-exhaustive list with examples of accessibility issues that cannot be detected automatically (and require manual testing):

• Audio-only content without a text transcript
• Video content without audio descriptions of essential information that is conveyed visually
• Video content without closed captions
• Use of color as the only means of sharing information (i.e. color coding)
• Audio that plays for longer than 3 seconds without a mechanism to stop, pause or mute it
• Users navigating only using the keyboard getting trapped on a page element

Items that overlays can’t reliably detect

There are some accessibility issues that can be detected under some circumstances, but there are enough exceptions that you can’t rely on an automated scanner to find them:

• Data tables need to have the headers encoded, but it’s not possible to programmatically determine whether the headers are in the top row, first column or both.
• Images that convey meaningful content require an alt text but overlays can’t determine the quality of the alt text from context.
• For that matter, they can’t even tell if the image is decorative (and therefore does not require an alt text in the first place.)
• When text is overlaid on an image or video background, it’s not possible to tell whether it has sufficient color contrast to be legible to users with vision impairments.
• If a list is not coded using the proper HTML markup, the automated scanner can’t know for sure whether it is a list (and therefore can’t automatically fix it).

Overlays can’t reliably fix all accessibility violations

Even for the subset of accessibility violations that can be automatically detected with some certainty, automated overlays can’t necessarily bring your site into compliance. Here are some examples of issues that an overlay may struggle to fix:

• For sites that don’t have a responsive mobile view built in, even a site with an overlay will be unable to zoom the page without content going offscreen or requiring horizontal scroll.
• For a page that is missing content headings, an overlay won’t add them.
• When images lack alt text, the AI-generated alt text might describe the image contents but the description may not be contextually correct for your page.
• On forms that are missing field labels, it isn’t possible to add the correct labels reliably.
• It is very common for authors in content management systems to copy and paste highly formatted text from external programs in Microsoft Word and accidentally introduce formatting or semantic errors into a webpage. While an automated overlay can theoretically strip out the extra styling, there is no guarantee that it would not break the design and layout (as well as strip out semantic formatting like headings or list bullets).

Where overlays don’t even try

Setting aside the inherent limitations of automated accessibility overlays, there are areas where the vendors don’t even try. If you look at the terms of service for any of the relevant service providers, you’ll notice that there are some significant exclusions that are simply not covered. The specifics might vary a bit depending on the service, but some common exclusions include:

• Any non-HTML content (including XML or Flash)
• HTML validation errors
• Audio content
• Video (even though ironically video captioning via artificial intelligence is reasonably accurate these days)
• SVG vector images
• PDFs

To add insult to injury, these vendors will sometimes turn around and offer to fix these excluded issues manually for an additional fee. 

Don’t put a bullseye on your website

There may very well be differences between different automated accessibility overlay vendors, but it’s important not to be taken in by the marketing. None of them can deliver on a claim of 100% compliance with the ADA and WCAG. Repeat plaintiffs that make up the vast majority of surf-by lawsuits know this and it is only a matter of time before they start using the presence of an overlay to determine whether to sue.