Criterion508 Accessibility Solutions Skip to Main Content


Dangers of Website Accessibility Testing Tools

April 2017 – Criterion

Many Section 508 and WCAG accessibility vendors, who make most of their revenues licensing automated website accessibility testing tools, have recently begun posting articles that claim their HTML validation tools are the answer to Section 508 and WCAG 2.0 compliance testing. Although the sales pitch may sound promising, the results are not.

Any reputable website accessibility vendor understands the only way to validate compliance with Section 508 and WCAG 2.0 guidelines is to conduct extensive manual accessibility testing utilizing experienced accessibility professionals and expert end-users with disabilities using a variety of assistive technologies.

Quality HTML validators have their place in every web developer’s toolbox, but they are certainly not the answer to website accessibility testing given that they capture about 20% of Section 508 / WCAG 2.0 guideline failures

Criterion is retained every week by companies who had previously engaged other accessibility vendors that promised them Section 508 and WCAG 2.0 compliance simply by using their automated testing tools. Because these tools failed to ensure Section 508 or WCAG 2.0 compliance as promised, these companies now face issues that either threaten their federal contracts or have resulted in threats of litigation.

Let’s take a look at some of the more common sales pitches we hear from our new clients by other vendors regarding automated website accessibility testing tools and HTML validators:

Sales Pitch Reality
Automated testing is faster

Automated testing is not faster when the results of the scan don’t ensure Section 508 and WCAG 2.0 compliance.

Typically, it takes less than 10 business days for a team of Criterion’s accessibility testers to document all Section 508 and WCAG failures within a website.

Additionally, due to the extensive documentation created through our manual testing, the average duration from the start of a website accessibility initiative to Section 508 and WCAG 2.0 certification is only four months.

Automated testing is cheaper

Automated testing tools are not cheaper than manual testing. Many vendors license their HTML validation tools for 2 to 3 times the cost of manually testing ALL pages within even the largest and most complicated dynamic websites.

The cost for Criterion to manually test ALL pages on a website utilizing several accessibility experts and end-users with disabilities generally costs 25% to 50% LESS than the cost of licensing the most robust automated testing tools that will merely skim accessibility failures and result in a website that is not compliant to WCAG and Section 508.

Automated testing is 100% accurate

Automated testing is only as good as the algorithms behind the scan. Unfortunately, many HTML validators report false positives and have major compatibility issues with certain browsers.

All “Free” HTML validators should be avoided. These are ineffective in gaining true accessibility.

Automated testing exposes all Section 508 and WCAG 2.0 violations on a page Automated testing exposes less than 17% of relevant Section 508 and WCAG 2.0 compliance issues.
Automated testing is all you need to make your site accessible to screen reader users

Automated testing is just a small piece of any comprehensive website accessibility testing protocol. In fact automated testing typically misses key components of allowing screen reader users to fluently move through a website.

Successful website accessibility initiatives are dependent on:

  • Web developer training
  • Dedicated subject matter expertise
  • Expert end-users with disabilities testing
  • Extensive independent third-party documentation and more

Remember, if you need to know that your website is Section 508 and WCAG 2.0 compliant, its common sense to have the website tested by people who actually have disabilities.

Automated testing tools combined with project management features allow for easier tracking and remediation of Section 508 and WCAG 2.0 failures

Website accessibility is not something above and beyond good website development. It is good website development!

The majority of Criterion’s clients are major corporations who have already invested considerable money, resources and training in their existing IT project management systems.

There is no reason for corporations to purchase a separate project management system that only tracks website accessibility issues and repairs. To do so reinforces the inaccurate perception that accessibility is an extra step in day to day website project management, design, development and maintenance.

Automated testing simply dissects code to identify inconsistencies with W3C standards and this is not the same as accessibility testing. Most objects and content on a page require human evaluation and the application of reasoning skills as content is compared to the more complicated guidelines set forth under Section 508 and the WCAG 2.0. Remember, context is critically important when testing website content and there is simply no algorithm capable of performing this type of evaluation.

Simply put, automated testing tools and HTML validators do not replace the need for manual testing by accessibility professionals and expert end-users with disabilities.

Below is a partial list of WCAG 2.0 and Section 508 accessibility guidelines that automated testing is incapable of resolving:


Call (888) 508-3973 to discuss your WCAG 2.0 and Section 508 compliance requirements today!