Testability Costs Too Much

by Gian Sampson-Wild

49 Reader Comments

Back to the Article
  1. This is a great explanation of some of the issues surrounding a “testable” in any standard, particularly in reference to usability. Interesting this exact issue is becoming an issue for the new U.S. federal voting machine standards.

    I recommend anyone interested in usability standards to review the latest draft of the “standards”:http://vote.nist.gov/meeting-05212007/VVSG-Draft-05242007.pdf , known as the Voluntary Voting System Guidelines (VVSG). Relevant comments should be sent to voting@nist.gov

    Copy & paste the code below to embed this comment.
  2. I’ve voiced my frustration with WCAG in other venues. It’s academic and arcane and hasn’t fostered a movement towards accessible sites, leaving it still as nothing more than a hammer for lawyers or a Shining Torch of Truth for standardistas.

    But I’m not sure if removing testability is the right idea. The problem is that common users need testability. Most sites for businesses and academic units aren’t created by design firms or standardistas; they’re created by people with basic knowledge and basic tools. The best way to get them to accessibility is to give them tools to build and validate, and the creators of those tools need guidelines reflecting what they should be testing for.

    Is the problem here the mission? Maybe WCAG 2.0 is trying to do too much here. Maybe the problem here is that the working group is trying to write Leviticus when all they need is to deliver the Ten Commandments and let others meet to come up with the 637 laws that stem from them.

    In non-religious terms, maybe WCAG 2.0 needs to be just the most basic guidelines, written to be as clear and unwordy as possible, with an eye to other working groups interpreting these guidelines into rules germane to specific uses. In other words, blow out 90% of the document, write some basic and general statements, and then punt it to others to try and figure out what that means for specific instances.

    I’m not arguing for a market-based creation of standards. What I’m saying is that the WCAG works on not coveting their neighbor’s ass, not about whether that ass is kosher for Passover.

    Unfortunately, I think it’s too late for that. We’re stuck with the muddle we’re in now, and it’s just going to continue what I consistently see at my level—the common web person blowing off accessibility because it’s not “easy” and lacks a simple validation methodology.

    Copy & paste the code below to embed this comment.
  3. In my view, technology-specific guidelines are better because of the relative ease of use. Yes, they expire with technology, but what is the alternative? So far, it seems to be either 1) inordinately long development time for technology-agnostic guidelines which require technology-specific examples for clarity, or 2) updated and/or new guidelines for the natural evolution in technologies.

    If there’s a third way, I haven’t seen it yet, and I’m starting to think option 2 above is the more natural choice.

    Copy & paste the code below to embed this comment.
  4. The WCAG as an organisation is becoming increasingly unaccessible to the wider web community which, in itself, is an amusing paradox.

    The core consideration in regards to WCAG, that prevents its relevance, is accessibility of the web to the authoring community, not just website users. Guidelines that prevent the average punter from publishing web content easily are counter-productive to the development of the internet as a whole. Which, is probably why the WCAG2.0 is never going to find endorsement by any legal authority.

    The internet’s core area of accessibility is the disemmenation of content by individuals who have access to a device capable of FTP/HTTP. When guidelines are created which prevent this, then they will largely be ignored. Which, is what we have seen to date and we will continue to see in the future.

    I definitely agree that the main issue at stake here is the lack of consultation with community groups who work directly with disabled people . It is from these groups that guidelines should be formed, on which, those of us with the technical skills to translate those needs into real coding approaches should then translate those guidelines into a formal working document. At the moment, I see far too many websites with a WCAG validation logo that don’t even come close to being accessible to large sections of the disabled community. Now, do you think this helps the WCAG’s credibity, or, hinder it?

    Copy & paste the code below to embed this comment.
  5. maybe WCAG 2.0 needs to be just the most basic guidelines, written to be as clear and unwordy as possible

    Hmm. Like… this?

    • 1.1 Provide text alternatives for any non-text content so that it can be changed into other forms people need such as large print, braille, speech, symbols or simpler language
    • 1.2 Provide synchronized alternatives for multimedia
    • 1.3 Create content that can be presented in different ways (for example spoken aloud, simpler layout, etc.) without losing information or structure
    • 1.4 Make it easier for people with disabilities to see and hear content including separating foreground from background
    • 2.1 Make all functionality available from a keyboard
    • 2.2 Provide users with disabilities enough time to read and use content
    • 2.3 Do not create content that is known to cause seizures
    • 2.4 Provide ways to help users with disabilities navigate, find content and determine where they are
    • 3.1 Make text content readable and understandable
    • 3.2 Make Web pages appear and operate in predictable ways
    • 3.3 Help users avoid and correct mistakes
    • 4.1 Maximize compatibility with current and future user agents, including assistive technologies

    If you don’t understand these in enough detail, there are checkpoints to spell it out. If you don’t understand those, there are techniques to define whether you’ve met them. And if you still don’t comprehend, the Working Group has even documented why each one is necessary.

    It seems most of the criticism here is that WCAG 2 isn’t at the same time universally applicable, comprehensive, precise beyond a shadow of a doubt, and capable of being taught to developers over the course of an elevator ride. I know! Maybe a unicorn could deliver it with a free iPhone, too!

    Copy & paste the code below to embed this comment.
  6. I agree with Gian that interested designers and developers should submit comments about the current draft.  One additional request should be made however, and that is to read the draft document first.  I’ve put my “geez this document has taken forever” and “I don’t want to learn new standards” thoughts aside and given the draft as objective a read as possible and am pretty happy with the result.  And if you read the document and have negative comments at least it won’t sound like you’re just writing because Gian suggested it.

    I’d also like to add that efforts are being made to make the new Section 508 standards and WCAG 2.0 and the ISO accessibility standards harmonize with each other.  Wouldn’t it be nice to have a single set of standards that you need to follow?  Participation in the Section 508 process is open to the public, so if you have thoughts and want to participate join the mailing lists and calls.  Find out more at “http://www.teitac.org”:http://www.teitac.org

    Copy & paste the code below to embed this comment.
  7. I must second Matt and Andrew’s comments. Much of the criticism of WCAG 2.0 is due to a few, relatively minor rough edges. Forget the W3C process and politics for a minute and read the document from an objective point of view and you’ll find it to be quite good. It’s far from perfect, but good. Suggestions that we toss out the baby with the bath water will result in no progress in accessibility.

    If we want WCAG 2.0 to be adopted or adapted in broader realms, particularly in legal arenas where it will have the broadest impact, then some level of testability is vital. Does this make things more difficult? Absolutely. Is the 8 out of 10 thing absurd? Yep. Should WCAG 2.0 include some provisions for guidelines that are not absolutely testable? Yes, or they should drop the guidelines that are pseudo-testable (what is “lower secondary reading level” anyways?).

    I think many in the community simply give WCAG 2.0 way too much credence. Our goal should be accessibility, not simply compliance. WCAG 2.0 should be one of many tools we use. If the tool isn’t perfect, use it for what it’s good for and ignore the parts you think are broken. Better yet, if you think you can make it better, get submitting those comments before Friday.

    Copy & paste the code below to embed this comment.
  8. I agree that testability has interfered with the ability to include some cognitive Success Criteria and that as a result those cognitive techiques were included as advisory. But I have trouble with the logic of wanting to remove testability. WCAG 2.0 has most of the common techniques for cognitive disabilities that we would find among recommendations from leading cognitive experts. The objection is that WCAG has made them advisory, not that they have not included cognitive techniques.

    But if we remove testability to get the Cognitive Techniques out of advisoy then the entire WCAG becomes advisory. We would not have not accomplished anything that I can see.

    The spectum of cognitive disabiliies is wider than any other disability group. I think there is much benefit in looking at all the guidelines for places where they impact some people with cognitive issues.

    I agree that there are some Level 3 issues such as Acronyms that I would have liked to see at level 2 but in general there are many Guideline whose primary target may be blind people or people other disabilities which also give substantial improvement of accessibility to some people with cognitive issues also.

    In looking at Level one I would say there are 9 Success Criteria that improve access for some people with cognitive disabilities. For instance, I know a lady who has a form of dyslexia that prevents her from using a mouse. For her, every Success Criteria that makes the web site keyboard accessible is a benefit to her, in fact these Success Criteria are crucial to her employment. Some people with cognitive issues may benefit from have headings which programmatically determined because they may use a User Agent which takes advantage of Heading levels. Guidelines that prevent the web site from changing focus unexpectedly help some people with cognitive issues. Guidelines that extend time outs help some people with cognitive issues who are slower to respond. Contrast and flashing related guidelines help some people with cognitive disabilities. The 4.1 guideline that makes sure all content of the site meets level one swings us back around to apply these issues to other technologies or at least provide the content in another technology which does conform.

    When we look at the guidelines through the eyes of different kinds of cognitive issues we find many Success Criteria that help many different kinds of cognitive disabilities.

    Copy & paste the code below to embed this comment.
  9. It’s only when WCAG tries to become law that issues such as conformance, conformance level, testability, normative vs. non-normative, advisory,  etc become issues. 

    Let’s look at WCAG for what is it, a great collection of guidance created by a body of experts from around the globe.  Let’s stop trying to make it a standard.  Please let’s stop trying to harmonize the world’s law to WCAG.  After all it’s the Web Content Accessibility GUIDELINES not the Web Content Accessibility Law.  (or is it?)

    I could never agree that we need to end testability.  As a member of Industry, I need a validator that will let me know if the thousands of the pages here will conform.  I can’t check each by hand.  I realize that at best only part of the guidelines can be machine testable, but even having part of them is better then none when faced with such a large web site.  I would rather see more testing over none!

    Copy & paste the code below to embed this comment.
  10. “Patrick Lauke”:http://www.alistapart.com/comments/testability?page=2#14
    bq. if they’re subjective, and not testable, how can you ever hope to enforce them, say within an institution? and if the guidelines are taken as a basis for legislation, would you still want to wait for case-law to clarify each point (maybe having a jury of 10 people, or just a — non tech savvy — judge)?

    There are so many things that can’t be quantified, but can still be verified. Many points of law do require subjective interpretation, and there’s no problem with that.

    There are plenty of things that are mechanistically testable, and that’s great. We can still have automated reports that analyse most aspects of accessibility, which will be very useful to developers, particularly those new to accessibility.

    But we shouldn’t sacrifice good practice for the sake of having testable guidelines. There are some aspects of accessibility that are not mechanistically testable, and I think it’s OK to say that, and leave it there. The farcical attempts to define “reliably human testable” makes it messy.

    The guidelines are here to serve accessibility, not the other way round!

    Copy & paste the code below to embed this comment.
  11. Interesting article.  I must admit I sometimes read W3C specifications with some frustration – sometimes because the language feels too academic (by trying to be too precise, and ending up being obscure), often because the specifications AREN’T testable, e.g. the CSS specifications which leave too much interpretation (personally I’d have preferred a set of tests produced as part of the specification, rather than as an incomplete after thought.

    However, the WCAG guidelines are guidelines and not a specification, so I agree, aiming to make them completely testable doesn’t work.  Personally, I’d like them to be expressed as a series points, each illustrated by a set of patterns and anti-patterns, each explaining why one is good from an accessibility point of view, and the other bad…

    Copy & paste the code below to embed this comment.
  12. In three subsequent paragraphs under the heading ‘Cognitive disabilities neglected’, Gian Sampson-Wild implies that she was removed from the Working Group because of the way she voted.

    To me, this is bullying. Vote the way we want you to, or you are out. I find this repugnant and morally reprehensible.

    This is a very serious allegation and it needs to be taken seriously. If there is basis to these claims, and I’m sure Gian Sampson-Wild believes there is, then action needs to be taken. Bullying in any environment is unacceptable.

    Copy & paste the code below to embed this comment.
  13. Joe,
    you argue that “The final version of the WCAG Samurai Errata aren’t out yet, and it’s disingenuous to blow up your favourite failing of the Errata into a claim that the whole thing is naff”.

    Yet it seems to be perfectly acceptable to attack what is not the final version of WCAG 2.0 on the basis of certain bits of it that people don’t like. That’s not just you by the way; I did it as did many others.

    But to avoid having double standards it has to be reasonable to critique early versions of the WCAG errata thingummy in exactly the same way… you’ve proven you can dish it out; now you’ve got to show you’re big enough to take it.

    FWIW, I think you are, and I think you’ll get it right. But that doesn’t mean you should be immune from criticism, and nor do the self-appointed and secretive Samurai deserve any more protection than do the WCAG WG…

    Copy & paste the code below to embed this comment.
  14. In three subsequent paragraphs under the heading “˜Cognitive disabilities neglected’, Gian Sampson-Wild implies that she was removed from the Working Group because of the way she voted.

    I was a participant in the WCAG WG at the time, and I can say that nobody “removed” her from the group, period. It is possible that she was notified that she had not met the requirements for good standing (i.e., she missed 2 of the last 3 teleconferences and/or 2 of the last 3 face-to-face meetings), which would have been restored once she resumed attending the calls. What she neglects to mention is that she missed six consecutive calls (and six techniques calls) between February 26 and April 22, 2004, according to the group’s minutes.


    And yet, even she admits that those in attendance were opposed — unanimously — to her proposal, and reiterated that opposition the following week.

    Occam’s razor, people. Her standing was changed because she was not actively participating. No conspiracy necessary.

    Copy & paste the code below to embed this comment.
  15. Now, Jack, our esteemed colleague Patrick H. Lauke did what I said he did. My previous article on WCAG 2 was a somewhat different animal. Care to compare and contrast?

    Copy & paste the code below to embed this comment.
    • Your one-liner list of WCAG 2 headlines is great as far as it goes, but in fact people do need accessibility guidelines to be, at once, instantly gleanable, fully detailed, implementable without indecision, and actually likely to improve accessibility for people with disabilities. That may be an impossible dream, which would be a fair point of criticism. But maybe — just maybe — WCAG 2 as it stands is too far away from these requirements.
    • Gian has stated publicly that she was very sick during the time she missed the phone calls. At least one WCAG WG member knows this firsthand. Illness is a disability, and the WCAG Working Group did not accommodate that disability short of undue hardship. The issue passed a vote “unanimously”? only after W3C technicalities were deliberately used to remove Gian from the group.

    Matt, you don’t work for the W3C anymore. It is no longer necessary to engage in Stockholm-syndrome-like defences of the organization. WCAG WG busts people who disagree or who threaten to withhold “consensus.”? That’s how they operate; that’s why they’re bullies; that’s why so very few of us trust them anymore.

    Copy & paste the code below to embed this comment.
  16. my right honourable friend joe may have misinterpreted my reason for the apparent statements of “naffness” of WCAG + Samurai.

    for the record, i do like the Samurai document a lot, barring a few minor nitpicks.

    what i was referring to in my comments ( “such as number 8”:http://www.alistapart.com/comments/testability?page=1#8 ) was not meant to take away from it, but rather to ask Gian for clarification on why she felt it was a better solution to her fundamental problem with WCAG 2, leading to one of her final statements:

    With the publication of the WCAG Samurai Errata, the web community finally has a choice …

    which seemed like a non-sequitur to me, since Samurai has the same fundamental problem for cognitive disabilities in particular.

    Copy & paste the code below to embed this comment.
  17. First off, consensus is not unanimity. If one participant stands apart from the rest on a point like this, they can raise a formal objection, which Gian has, and which will result in the group having to defend their decision. (Again.) Still, good standing would never have afforded her the privilege of lying down in the road to get her way.

    In this case, the argument seems to be that the proposal she made, which didn’t receive any support from a group made up of a pretty even mix of advocates, academics and technical folks, should be overturned because… why? Because some bureaucratic status that is about as relevant to the situation as her eye color was changed around the same time? Because it somehow got into ALA? And now, because she was sick? (The messages she sent to the list to excuse herself at the time usually indicated she was “travelling.”) No matter how many times you try to explain away how it didn’t go the way you wanted, it’s still a bad idea.

    With regard to the perceived trustworthiness of W3C, and WAI in particular, well, take a bow. You’re the one who’s responsible for that, thanks in large part to your vividly creative storytelling. But that’s worth an article unto itself.

    Copy & paste the code below to embed this comment.
  18. …nobody “removed”? her from the group, period…

    I said: “my status as a member (in good standing) was revoked due to ‘non-participation.’”

    There were other members in the Working Group that did not fulfil the requirements of attendance. And, as Matt points out, I had not been meeting my attendance requirements for some time. However it was only when I stopped consensus on testability that my status was revoked. It was not “some bureaucratic status that is about as relevant to the situation as [my] eye color” – it was the reason why I could not vote against testability. And this “bureacratic status” wasn’t “changed around the same time” as my vote against testability – it was changed within 24 hours of my vote against testability.

    To clarify I was actually removed from the Working Group at a latter date (which Joe Clark refers to). In the above instance I was living in rural Queensland and travelling to various capital cities for work. Being on the Working Group may not seem too onerous – but it means reviewing up to 100 emails a day, and, in Australia it means weekly 6am teleconferences on a Friday – as a volunteer. Because I was running my own business for most of that time, I was the one who had to foot the bill for the 90 minute international telephone calls (long before Skype became available). I did miss some teleconferences. However it was only when I voted against testability was my status (and voting rights) revoked. I was so disgusted by this that I remained inactive within the Working Group.

    Eight months later all Working Group Members were required to re-apply due to the Patent Policy. I used that opportunity to become active within the Working Group. It took me six months of emails to the W3C Staff Member before I was allowed back on the Working Group – and only after I had sent an email to the entire Working Group and CCed the WAI Steering Committee.

    I became very ill in October 2005. I did not work for ten months. During that time I did not participate in Working Group teleconferences or via the mailing list. However it was only after I lodged a record number of comments on the Last Call Working Draft and after I had mentioned to an IBM Working Group member that I would be joining the next teleconference did I get told that I was no longer a Member. On the 20 July 2006 I received an email from Judy Brewer saying that my Invited Expert status was about to expire “since you haven’t been an active participant in the WCAG WG recently”. I emailed back that I had been ill. The reason for my removal changed to my apparent lapsed association with my affiliate company. When I argued that my association had not lapsed, the emails ceased.

    Copy & paste the code below to embed this comment.
  19. I have posted the stream of emails between Judy Brewer and “myself”:http://www.tkh.com.au/?p=39
    I have also included comments about my removal from the Working Group by various non-corporate members of the “Working Group”:http://www.tkh.com.au/?p=39#wg

    Copy & paste the code below to embed this comment.
  20. Matt May knows that I back up my assertions about WCAG WG with original documents or links to them. This, in the industry, is called documentation, not “vividly creative storytelling,”? a cutesy phrase I assume means “lying.”? If anyone can prove I have lied about any topic in the last 30 years, please publish your documentation.

    Copy & paste the code below to embed this comment.
  21. The original article (remember that?) was a very good piece about testability.

    The last bunch of comments aren’t, and should be taken to personal blogs. Why not comment on Gian’s blog entry that includes the discussions with the working group. They’d be on-topic there.

    Copy & paste the code below to embed this comment.
  22. The notion that the best set of guidelines would be entirely testable seems ludicrous to me but so does the idea that any testability whatsoever is unwarranted.  I mean, we are doing this on computers, after all – you’d be leaving entire orchards of fruit on the vine by refraining from having any testable guidelines whatsoever.

    For example, “Make all functionality available from a keyboard” is eminently testable, as are many specific cases of the other guidelines, for example, in compliance with “Help users avoid and correct mistakes” no form control should start off in a state that it cannot be returned to – e.g. the infamous set of radio buttons that start with nothing selected, but then once you click one you can’t back out even if it wasn’t a required field… and although the quality of particular alt text can’t be judged by a machine its presence – whether the alt tag is missing or empty, or even whether it contains an entire sentence – can be machine-tested.  You could even go so far as to test whether all photographs have alt tags, whether images containing text have alt tags, whether images containing faces have alt tags, et cetera.

    And in the spirit of guiding rather than strict lawgiving, tests need not be for the strict purpose of passing or failing on compliance, they can be informative.  I must concur with the previous poster that a technological guideline, an actual test tool or suite of tools, may be appropriate for these parts of the standards.

    I do think that the testable and non-testable guidelines should be clearly delineated.  I also think that the cost of testability should be considered and ameliorated, particularly when publicly-available tools could be developed, or the existing ones better-used.  (But I would note that despite its title, neither the article nor the subsequent discussion has addressed cost.)

    Copy & paste the code below to embed this comment.
  23. I’m sorry, Bruce, but I disagree with you. Discussion of Gian Sampson-Wild’s claims in regards to her removal from the Working Group are valid.

    How many on that Working Group are voting in a specific way because that is the way that they have been told/shown is the ‘correct’ way? It could be that others agreed with Gian on testability but were too afraid to vote in the way that they truly felt.

    Is there bullying in other Working Groups in the W3C? Is the W3C creating documents to the best of all combined abilities?

    These questions need to be asked considering that at least one other person has made similar claims of this Working Group and there seems to be no response from the W3C about it; tacitly condoning the alleged type of behaviour.

    Copy & paste the code below to embed this comment.
  24. I’ve written up “my thoughts in more detail”:http://www.thepickards.co.uk/index.php/200707/wcag-20-testability-testing-times-tetchiness/ but taking into account Bruce Lawson’s quite sensible pointing out that we’re discussing testability rather than Working Group status or otherwise, I’ll not go into detail here.

    You can either nip over to my site to read my long and rambling opinionated rant, or not. I honestly don’t mind.

    Copy & paste the code below to embed this comment.
  25. For ALT text, the test is surely a self-critical question to the designer: “Does the alt-text I’ve used give the reader an idea of what the image is trying to convey, or what my purpose in including it in the web page was, if for some reason the reader can’t see the it?”

    For any image, there will be a range of acceptable alt-text, so the answer to the question will only be a matter of opinion, informed or not.  That’s not something that’s testable by machine, is barely testable by humans, and certainly not with any degree of consistency (at the required 80% level for WCAG2).

    If its not testable, you can’t measure compliance and hence the perceived reduction in usefulness of WCAG2 to purchasers of websites as a standard to aim for.  Sounds like sticking strictly to the letter of WCAG2 could result in less usable/accessible (sensu lato) websites. I’d rather stick to the spirit and use techniques shown to enhance accesibility/usability. Actual demonstration of that to a user/client would be better than certification.

    Copy & paste the code below to embed this comment.
  26. We do test our product though accessibility testing is often like many companies seen as important. If there are problems found by customers the issues are fixed pretty swiftly.

    As a designer I have never got into WCAG2 as it seemed to less used and to vague. Maybe I just haven’t got the time to learn another set of standards as WCAG1 was heavy enough.

    Copy & paste the code below to embed this comment.
  27. Testability is unfortunately a must for the many many people, including the W3C and its members, who make money through accessibility consultancy – it allows them to say “This is exactly right, and this is exactly wrong”, which helps a lot and leaves no uncertainties or arguments or incorrect online testing systems.

    However, Gian is exactly right – many of the most important elements of the current set of the WCAG are untestable – these elements also go hand in hand with search engine optimisation and usability/general user experience for even able-bodied users.

    The document defines testable as “machine-testable” or “reliably human testable”? — which means that eight out of ten human testers must agree on whether the site passes or fails each success criterion.”

    The truth is that disabled users are not machines and 8 out of ten human beings aren’t disabled. In reality, the W3C should get these guidelines approved not by a group of able-bodied front-end developers who are going to have to use and adhere to them, but by the very people they are intended to help the most: disabled users. And with the multitude of disabilities out there, it’s about time we started asking all of them what they think and what they would want from websites.

    Copy & paste the code below to embed this comment.
  28. I cannot agree strongly enough with Aidan Williams – we need to start testing with, and talking to, people with disabilities.

    This group of people is unfortunately under-represented in the WCAG Working Group. Often when we worked in Working Group sub-groups to revise specific success criterion we were split by geographic location, not by interest or knowledge of a particular area of disability. But this is not a problem relevant only to the Working Group: I believe the entire accessibility community needs to engage with their users more.

    Copy & paste the code below to embed this comment.