Is accessibility conformance an elusive property? A study of validity and reliability of WCAG 2.0

The Web Content Accessibility Guidelines (WCAG) 2.0 separate testing into both \u201cMachine\u201d and \u201cHuman\u201d audits; and further classify \u201cHuman Testability\u201d into \u201cReliably Human Testable\u201d and \u201cNot Reliably Testable\u201d; it is human testability that is the focus of this paper. We wanted to investigate the likelihood that \u201cat least 80% of knowledgeable human evaluators would agree on the conclusion\u201d of an accessibility audit, and therefore understand the percentage of success criteria that could be described as reliably human testable, and those that could not. In this case, we recruited twenty-\ufb01ve experienced evaluators to audit four pages for WCAG 2.0 conformance. These pages were chosen to differ in layout, complexity, and accessibility support, thereby creating a small but variable sample. We found that an 80% agreement between experienced evaluators almost never occurred and that the average agreement was at the 70\u201375% mark, while the error rate was around 29%. Further, trained\u2014but novice\u2014evaluators performing the same audits exhibited the same agreement to that of our more experienced ones, but a reduction on validity of 6\u201313%; the validity that an untrained user would attain can only be a conjecture. Expertise appears to improve (by 19%) the ability to avoid false positives. Finally, pooling the results of two independent experienced evaluators would be the best option, capturing at most 76% of the true problems and producing only 24% of false positives. Any other independent combination of audits would achieve worse results. This means that an 80% target for agreement, when audits are conducted without communication between evaluators, is not attainable, even with experienced evaluators, when working on pages similar to the ones used in this experiment; that the error rate even for experienced evaluators is relatively high and further, that untrained accessibility auditors be they developers or quality testers from other domains, would do much worse than this