Automated accessibility testing: capabilities, limitations & responsible use

What is digital accessibility?

Digital accessibility ensures that everyone can use digital content, regardless of disability.

At UW–Madison, we measure accessibility against the Web Content Accessibility Guidelines (WCAG 2.1). WCAG’s major criteria are that digital content must be Perceivable, Operable, Understandable and Robust (POUR).  

Accessibility testing helps us meet these criteria by identifying barriers that prevent people from accessing information or completing tasks. While some barriers can be detected automatically, others require human judgment.

Automated accessibility tools

Automated accessibility testing tools include website scanners, browser extensions, Content Management System (CMS) plugins, Integrated Development Environment (IDE) extensions, linters, color contrast analyzers, and other tools that use software to identify accessibility barriers.

While these tools vary in scope and complexity, they all automate accessibility checks that can be performed reliably without human judgment.

What automated tools do well

Testing at scale: 

• Auditing large websites
• Monitoring accessibility across many pages
• Integrating accessibility checks into development workflows

Objectively measurable conditions, such as: 

• Numeric thresholds, e.g., color contrast ratios
• Presence of required attributes, e.g., alttext, form labels, etc.
• ARIA roles and structural patterns
• Syntactic correctness,e.g., heading levels are in the correct order

Automated testing performs best when the accessibility requirement being evaluated:

• Is based on clear, objective rules
• Has a clear pass/fail outcome
• Does not require understanding the purpose or meaning of the content

Where automated testing falls short

Automated testing cannot reliably evaluate meaning, intent, usability, or the user experience when accessing digital content using assistive technologies such as screen readers, voice recognition software and switch devices. It may also miss barriers involving custom components and dynamic content. It struggles with questions such as:

• Is alternative text useful?
• Does link text make sense out of context?
• Is the reading order logical?
• Are error messages clear and actionable?

These require:

• Human judgment
• An understanding of context and user goals
• Evaluation of the actual user experience

What about artificial intelligence? 

Emerging AI-assisted tools show promise in evaluating some accessibility barriers that were previously difficult to assess automatically. However, these tools still rely on inference and cannot replace human judgment or an understanding of user needs.

The role of human evaluation

Many aspects of accessibility still require human evaluation. Human evaluations help confirm that:

• Keyboard users can clearly see which element has focus.
• Screen readers announce content in a logical and meaningful order.
• Dynamic content and multimedia remain accessible.
• Content remains usable at high levels of magnification.

Common manual testing techniques include:

• Keyboard-only testing to verify navigation and focus behavior.
• Screen reader testing to evaluate how content is announced and whether users can navigate efficiently.
• Magnification testing to confirm that content remains usable when zoomed to 400%.

For more information about manual testing, see the relevant section of the Testing Web Accessibility guide. 

Bringing it all together

Accessibility testing is most effective when automated and manual testing techniques are used together throughout the content lifecycle, with each method applied where it provides the greatest benefit.

When	Why	How
Design	Use manual review to identify potential accessibility barriers before development begins. Evaluate usability, readability, page structure, color choices, and interaction patterns.	Design reviews, accessibility design checklists, color contrast analyzers, keyboard walkthroughs, reviews by accessibility specialists or people with disabilities
Development	Use automated testing to catch common barriers early and manual testing to evaluate areas that require human judgment.	Browser accessibility tools, IDE extensions, linting tools, automated accessibility tests, keyboard testing, code reviews
Content creation and publishing	Use automated testing to identify common errors and manual review to ensure content is clear, understandable, and appropriately structured.	CMS accessibility checkers, browser extensions, heading structure reviews, alternative text reviews, plain language reviews
Quality assurance (QA)	Use both automated and manual testing to validate the overall accessibility of the user experience before release.	Website scanners, browser developer tools, screen reader testing, keyboard-only testing, user journey testing, accessibility conformance testing

Conclusion

Automated tools are necessary but are not sufficient by themselves. Think of the shoes on your feet; you can travel farther with both than with just one. The same is true of accessibility testing: using both automated and manual testing techniques helps you find more barriers with less effort than using either approach by itself.

Automated testing excels at finding barriers that can be checked reliably and consistently. Manual testing provides the human judgment and contextual understanding of the real-world experience of people with disabilities.

The most effective accessibility programs do not choose between automated and manual testing. They incorporate both throughout the content lifecycle and use each where it provides the greatest value.

A note on AI use: I used Microsoft Copilot to help outline and edit this article, including tightening the language in several sections. The reporting, analysis and conclusions are my own.