Comments on Testability Costs Too Much

48 Reader Comments

Back to the Article
  1. Complicated stuff, but it sounds like you know what you’re talking about and have my backing!

    Copy & paste the code below to embed this comment.
  2. I’ve said it for a while now, over and over. The various working groups at the W3C just don’t get it at all.  Are people forgetting that in some cases people are volunteering their time and effort on these working groups. But then (as we have seen) people on the groups that don’t like the outcome of the discussion, and like a political roundtable, just remove the troublemakers. It is these troublemakers that are often the only thing keeping the working group in check.

    I did wonder about the testability.  It did seem a little inconsistent.  But Gian you have explained in detail and I now understand why.  It sounds strange to remove this.  But testing that doesn’t have a yes or no result is basically a waste of time.

    Copy & paste the code below to embed this comment.
  3. Let me clarify this. (before Gian flames me) What we should be doing is removing all testability from the equation. Maybe this needs to be documented elsewhere with examples. So the enforcement and demonstration of the guideline are not in the guideline.

    Also why remove “cognitive disabilities” because its too hard.  This I’m still trying to understand.  Working Group please explain!

    Copy & paste the code below to embed this comment.
  4. Gian,
    I think you’ve made a good case for this. In general, I’ve been very happy with this draft of WCAG 2.0, and I think it’s helpful for developers if checkpoints are testable, but not at the expense of not being able to help those with cognitive disabilities.

    What about something that says checkpoints (oh, all right, “success crtieria”) must be testable wherever possible? That way success criteria relating to cognitive disabilities etc can be included but testability is still brought into the equation wherever it can be?

    Copy & paste the code below to embed this comment.
  5. Excellent article, it’s rare to find people talk intelligently about WCAG/Accessibility.  However, I’m afraid I’d go significantly further.  The story you tell of your dealings with the group dovetails with other experts’ experience: that the group has long since stopped being about accessibility, and is now essentially meaningless.

    Having worked for a firm that took accessibility _standards_ extremely seriously, I’ve seen the damage that can be done by incomplete comprehension of accesibility guidelines and slavish following of the same.  I think it’s highly unlikely WCAG 2 will improve actual usability for anyone, but the damagae it is likely to cause externally is huge.

    WCAG 2 is incomprehensible in the extreme, prescriptive in the the wrong ways and permissive in the wrong ways.  I understand that testability is the cause of a lot of this, but so is the dynamic of the working group.  The absence of any actual research into the behaviours of impaired users renders the whole endeavour bizarre.  (a criticism that could also be levelled at WCAG 1)

    In short, it’s time to stop taking WCAG 2 even vaguely seriously and work on a competing standard.  It seems the number of accessibility experts with serious reservations about WCAG 2 exceeds the number in the working group.  I know it’s a significant endeavour, but it’s the only way a number of significant standards advances have occurred.  Look at SOAP…

    Copy & paste the code below to embed this comment.
  6. A well stated case, hopefully one that will be listened to!
    Standards, guidelines, definitions; they’re all there to help us do our jobs, when they stop helping and start hindering they loose all relevance.
    Accessibility, like other user-focused disciplines, relies on a thorough understanding of the needs of the end user, and guidelines can’t replace that, no matter how testable they are…

    Copy & paste the code below to embed this comment.
  7. Maybe I’m missing the obvious, but I’m a tad confused about how the WCAG + Samurai are any better.

    How are things like “do not set one confusable colour on top of another”, followed by just one example of “red type on green or black backround”, any clearer than what WCAG, for instance, is doing by saying “assistive technology” without specifying which one?

    And the almost anti-testability statement in the Samurai’s bit about HTML semantics “Other authors’ disagreement with your choices is not relevant to these errata” ... how does that help in checking whether a site actually complies with the errata or not?
    Lastly, if even the Samurai clearly states (just like WCAG 2.0) that it “cannot be a claim of full accessibility to people with cognitive disabilities”, how is that “finally a choice”, if one of the primary concerns was indeed how users with cognitive disabilities are being disenfranchised by WCAG 2.0?

    Anyway, I’ve gathered these thoughts in a quick post on Accessify - “Gian Sampson-Wild on WCAG 2.0’s concept of testability”:

    Copy & paste the code below to embed this comment.
  8. Good article, Gian. Yet more proof that the brains behind WCAG2 aren’t the best that money can buy.

    I mean, c’mon! Every accessibility guide since the year dot has said “don’t rely on automated tests”, because we’ve known all along that there are some things that you just can’t automatically test for.

    The bit about asking ten people and seeing if eight of them agree - that’s like saying “it’s raining today, it’s proof that the climate is becoming wetter”. How do you select your ten people? How do you know they represent a fair and unbiased sample group? How do you know that the next ten people won’t result in eight of those ten giving the opposite viewpoint? Do we need to sample ten groups of ten, to make sure that 8 out of 10 give the same answer eight times out of ten? But then we might have as little as 64% agreement - so that would no longer be sufficient.

    You don’t need to be a degree-level statistician to see just how fatuous the “human-testable” guideline is.

    A lot of accessibility guidelines _can_ be machine-tested. But there are also a lot that can’t. Colour combinations may not always be testable, if you’re looking at the effects of juxtaposition of colours. Kincaid reading scores will tell you if you’re using too many words like “juxtaposition”, but there’s a *lot* more to “use the simplest language that is appropriate” than that. Relevant alt text, consistent layout, logical grouping of elements (esp form elements), even semantic code - these are all, to a greater or lesser extent, subjective. They can’t reliably be tested, but it is still vital that all authors do _as much as they can_ to ensure that these guidelines are met.

    Copy & paste the code below to embed this comment.
  9. bq. for instance the WCAG Samurai Errata are ten pages, whereas WCAG2 is close to the 500 page mark

    that’s not really comparing like for like, though. the normative part of WCAG 2, minus the appendices, is just over 10 pages as well. you could say that, to understand WCAG 2, you need all the other material and techniques relating to each SC, but that is just as true for the Samurai - only that there is no techniques document for Samurai yet, so you have to track down the related best practices etc yourself.

    also, Samurai only applies to HTML/CSS, with added bits specific to PDF and some things that could reasonably be applied, at least in principle, to other formats like Flash…while WCAG 2, as you noted of course, tries to be tech agnostic, thus making it far more relevant for areas such as e-learning, for instance…

    Copy & paste the code below to embed this comment.
  10. ...and sorry for the current commenting issues on Accessify - there seem to be some gremlins in the works there.

    Copy & paste the code below to embed this comment.
  11. bq. Relevant alt text, consistent layout, logical grouping of elements (esp form elements), even semantic code — these are all, to a greater or lesser extent, subjective. They can’t reliably be tested, but it is still vital that all authors do as much as they can to ensure that these guidelines are met.

    if they’re subjective, and not testable, how can you ever hope to enforce them, say within an institution? and if the guidelines are taken as a basis for legislation, would you still want to wait for case-law to clarify each point (maybe having a jury of 10 people, or just a - non tech savvy - judge)?

    i don’t think there was any need to rail on about the brains behind WCAG 2 and automated testing, as that’s not what the guidelines emphasise, quite clearly.

    Copy & paste the code below to embed this comment.
  12. I am just beginning to scratch the surface of web standards and accessibility, and it takes real-world experience and personal discovery before I am capable of understanding why any of it is important at all. Sure, in school they wax poetic about the importance of the W3C and the WCAG in particular, but none have bothered to participate in the WCAG personally.

    It sounds to me like the people pushing testability through are the ones who enjoy the political game or are encased in the impenetrable walls of academia, where time is in abundance and implementing their guidelines/lofty goals are within reach of no one else. Testability at the expense of accessibility? Perhaps this is why I get so frustrated and confused when attempting anything beyond the basics.

    Copy & paste the code below to embed this comment.
  13. If WCAG 2.0 is too hard to understand, or not clear enough to be sure we can (and how we can) comply with it, I doubt that very many people on the job will be looking to the Samurai Errata for clarification, and fewer still managers (people who decide what the standards and the testing criteria will be) will use Samurai. How many web workers even KNOW about that?

    Instead, I see WCAG 1.0 hanging around for a long, long while. We’re used to it. We can comply with it (when we really, really have to).

    When 2.0 is doable, we’ll do 2.0.

    Copy & paste the code below to embed this comment.
  14. that’s been the problem with WCAG 1, and it could become a problem if WCAG 2 was rewritten to be technology-specific rather than agnostic + good techniques documents. also, new best practices emerge all the time (even just for HTML). enshrining the current practices into guidelines (which seem to take forever to be amended and updated) may result in the same issues that actually led to WCAG + Samurai in the first place.

    as for SOCOG, weren’t the things that were wrong with the site technical (missing ALT text on images and image maps, tables that didn’t linearise properly) rather than subjective?

    Copy & paste the code below to embed this comment.
  15. there shouldn’t have been an underline in my above post.

    Copy & paste the code below to embed this comment.
  16. The final version of the WCAG Samurai Errata aren’t out yet, and it’s disingenuous to blow up your favourite failing of the Errata into a claim that the whole thing is naff. Give us time, please.

    Copy & paste the code below to embed this comment.
  17. This is a great explanation of some of the issues surrounding a “testable” in any standard, particularly in reference to usability. Interesting this exact issue is becoming an issue for the new U.S. federal voting machine standards.

    I recommend anyone interested in usability standards to review the latest draft of the “standards”: , known as the _Voluntary Voting System Guidelines_ (VVSG). Relevant comments should be sent to

    Copy & paste the code below to embed this comment.
  18. I’ve voiced my frustration with WCAG in other venues. It’s academic and arcane and hasn’t fostered a movement towards accessible sites, leaving it still as nothing more than a hammer for lawyers or a Shining Torch of Truth for standardistas.

    But I’m not sure if removing testability is the right idea. The problem is that common users _need_ testability. Most sites for businesses and academic units aren’t created by design firms or standardistas; they’re created by people with basic knowledge and basic tools. The best way to get them to accessibility is to give them tools to build and validate, and the creators of those tools need guidelines reflecting what they should be testing for.

    Is the problem here the mission? Maybe WCAG 2.0 is trying to do too much here. Maybe the problem here is that the working group is trying to write Leviticus when all they need is to deliver the Ten Commandments and let others meet to come up with the 637 laws that stem from them.

    In non-religious terms, maybe WCAG 2.0 needs to be just the most basic guidelines, written to be as clear and unwordy as possible, with an eye to _other_ working groups interpreting these guidelines into rules germane to specific uses. In other words, blow out 90% of the document, write some basic and general statements, and then punt it to others to try and figure out what that means for specific instances.

    I’m not arguing for a market-based creation of standards. What I’m saying is that the WCAG works on not coveting their neighbor’s ass, not about whether that ass is kosher for Passover.

    Unfortunately, I think it’s too late for that. We’re stuck with the muddle we’re in now, and it’s just going to continue what I consistently see at my level—the common web person blowing off accessibility because it’s not “easy” and lacks a simple validation methodology.

    Copy & paste the code below to embed this comment.
  19. In my view, technology-specific guidelines are better because of the relative ease of use. Yes, they expire with technology, but what is the alternative? So far, it seems to be either 1) inordinately long development time for technology-agnostic guidelines which require technology-specific examples for clarity, or 2) updated and/or new guidelines for the natural evolution in technologies.

    If there’s a third way, I haven’t seen it yet, and I’m starting to think option 2 above is the more natural choice.

    Copy & paste the code below to embed this comment.
  20. The WCAG as an organisation is becoming increasingly unaccessible to the wider web community which, in itself, is an amusing paradox.

    The core consideration in regards to WCAG, that prevents its relevance, is accessibility of the web to the authoring community, not just website users. Guidelines that prevent the average punter from publishing web content easily are counter-productive to the development of the internet as a whole. Which, is probably why the WCAG2.0 is never going to find endorsement by any legal authority.

    The internet’s core area of accessibility is the disemmenation of content by individuals who have access to a device capable of FTP/HTTP. When guidelines are created which prevent this, then they will largely be ignored. Which, is what we have seen to date and we will continue to see in the future.

    I definitely agree that the main issue at stake here is the lack of consultation with community groups who work directly with disabled people . It is from these groups that guidelines should be formed, on which, those of us with the technical skills to translate those needs into real coding approaches should then translate those guidelines into a formal working document. At the moment, I see far too many websites with a WCAG validation logo that don’t even come close to being accessible to large sections of the disabled community. Now, do you think this helps the WCAG’s credibity, or, hinder it?

    Copy & paste the code below to embed this comment.
  21. bq. maybe WCAG 2.0 needs to be just the most basic guidelines, written to be as clear and unwordy as possible

    Hmm. Like… this?

    * 1.1 Provide text alternatives for any non-text content so that it can be changed into other forms people need such as large print, braille, speech, symbols or simpler language
    * 1.2 Provide synchronized alternatives for multimedia
    * 1.3 Create content that can be presented in different ways (for example spoken aloud, simpler layout, etc.) without losing information or structure
    * 1.4 Make it easier for people with disabilities to see and hear content including separating foreground from background
    * 2.1 Make all functionality available from a keyboard
    * 2.2 Provide users with disabilities enough time to read and use content
    * 2.3 Do not create content that is known to cause seizures
    * 2.4 Provide ways to help users with disabilities navigate, find content and determine where they are
    * 3.1 Make text content readable and understandable
    * 3.2 Make Web pages appear and operate in predictable ways
    * 3.3 Help users avoid and correct mistakes
    * 4.1 Maximize compatibility with current and future user agents, including assistive technologies

    If you don’t understand these in enough detail, there are checkpoints to spell it out. If you don’t understand _those_, there are techniques to define whether you’ve met them. And if you still don’t comprehend, the Working Group has even documented _why_ each one is necessary.

    It seems most of the criticism here is that WCAG 2 isn’t at the same time universally applicable, comprehensive, precise beyond a shadow of a doubt, and capable of being taught to developers over the course of an elevator ride. I know! Maybe a unicorn could deliver it with a free iPhone, too!

    Copy & paste the code below to embed this comment.
  22. I must second Matt and Andrew’s comments. Much of the criticism of WCAG 2.0 is due to a few, relatively minor rough edges. Forget the W3C process and politics for a minute and read the document from an objective point of view and you’ll find it to be quite good. It’s far from perfect, but good. Suggestions that we toss out the baby with the bath water will result in no progress in accessibility.

    If we want WCAG 2.0 to be adopted or adapted in broader realms, particularly in legal arenas where it will have the broadest impact, then some level of testability is vital. Does this make things more difficult? Absolutely. Is the 8 out of 10 thing absurd? Yep. Should WCAG 2.0 include some provisions for guidelines that are not absolutely testable? Yes, or they should drop the guidelines that are pseudo-testable (what is “lower secondary reading level” anyways?).

    I think many in the community simply give WCAG 2.0 way too much credence. Our goal should be accessibility, not simply compliance. WCAG 2.0 should be one of many tools we use. If the tool isn’t perfect, use it for what it’s good for and ignore the parts you think are broken. Better yet, if you think you can make it better, get submitting those comments before Friday.

    Copy & paste the code below to embed this comment.
  23. I agree that testability has interfered with the ability to include some cognitive Success Criteria and that as a result those cognitive techiques were included as advisory. But I have trouble with the logic of wanting to remove testability. WCAG 2.0 has most of the common techniques for cognitive disabilities that we would find among recommendations from leading cognitive experts. The objection is that WCAG has made them advisory, not that they have not included cognitive techniques.

    But if we remove testability to get the Cognitive Techniques out of advisoy then the *entire* WCAG becomes advisory. We would not have not accomplished anything that I can see.

    The spectum of cognitive disabiliies is wider than any other disability group. I think there is much benefit in looking at all the guidelines for places where they impact *some* people with cognitive issues.

    I agree that there are some Level 3 issues such as Acronyms that I would have liked to see at level 2 but in general there are many Guideline whose primary target may be blind people or people other disabilities which also give substantial improvement of accessibility to some people with cognitive issues also.

    In looking at Level one I would say there are 9 Success Criteria that improve access for some people with cognitive disabilities. For instance, I know a lady who has a form of dyslexia that prevents her from using a mouse. For her, every Success Criteria that makes the web site keyboard accessible is a benefit to her, in fact these Success Criteria are crucial to her employment. Some people with cognitive issues may benefit from have headings which programmatically determined because they may use a User Agent which takes advantage of Heading levels. Guidelines that prevent the web site from changing focus unexpectedly help some people with cognitive issues. Guidelines that extend time outs help some people with cognitive issues who are slower to respond. Contrast and flashing related guidelines help some people with cognitive disabilities. The 4.1 guideline that makes sure all content of the site meets level one swings us back around to apply these issues to other technologies or at least provide the content in another technology which does conform.

    When we look at the guidelines through the eyes of different kinds of cognitive issues we find many Success Criteria that help many different kinds of cognitive disabilities.

    Copy & paste the code below to embed this comment.
  24. It’s only when WCAG tries to become law that issues such as conformance, conformance level, testability, normative vs. non-normative, advisory,  etc become issues. 

    Let’s look at WCAG for what is it, a great collection of guidance created by a body of experts from around the globe.  Let’s stop trying to make it a standard.  Please let’s stop trying to harmonize the world’s law to WCAG.  After all it’s the Web Content Accessibility GUIDELINES not the Web Content Accessibility Law.  (or is it?)

    I could never agree that we need to end testability.  As a member of Industry, I need a validator that will let me know if the thousands of the pages here will conform.  I can’t check each by hand.  I realize that at best only part of the guidelines can be machine testable, but even having part of them is better then none when faced with such a large web site.  I would rather see more testing over none!

    Copy & paste the code below to embed this comment.
  25. “Patrick Lauke”:
    bq. if they’re subjective, and not testable, how can you ever hope to enforce them, say within an institution? and if the guidelines are taken as a basis for legislation, would you still want to wait for case-law to clarify each point (maybe having a jury of 10 people, or just a — non tech savvy — judge)?

    There are so many things that can’t be quantified, but can still be verified. Many points of law *do* require subjective interpretation, and there’s no problem with that.

    There are plenty of things that are mechanistically testable, and that’s great. We can still have automated reports that analyse *most* aspects of accessibility, which will be very useful to developers, particularly those new to accessibility.

    But we shouldn’t sacrifice good practice for the sake of having testable guidelines. There are some aspects of accessibility that are not mechanistically testable, and I think it’s OK to say that, and leave it there. The farcical attempts to define “reliably human testable” makes it messy.

    The guidelines are here to serve accessibility, not the other way round!

    Copy & paste the code below to embed this comment.
  26. Interesting article.  I must admit I sometimes read W3C specifications with some frustration - sometimes because the language feels too academic (by trying to be too precise, and ending up being obscure), often because the specifications AREN’T testable, e.g. the CSS specifications which leave too much interpretation (personally I’d have preferred a set of tests produced as part of the specification, rather than as an incomplete after thought.

    However, the WCAG guidelines are guidelines and not a specification, so I agree, aiming to make them completely testable doesn’t work.  Personally, I’d like them to be expressed as a series points, each illustrated by a set of patterns and anti-patterns, each explaining why one is good from an accessibility point of view, and the other bad…

    Copy & paste the code below to embed this comment.
  27. In three subsequent paragraphs under the heading ‘Cognitive disabilities neglected’, Gian Sampson-Wild implies that she was removed from the Working Group because of the way she voted.

    To me, this is bullying. Vote the way we want you to, or you are out. I find this repugnant and morally reprehensible.

    This is a very serious allegation and it needs to be taken seriously. If there is basis to these claims, and I’m sure Gian Sampson-Wild believes there is, then action needs to be taken. Bullying in any environment is unacceptable.

    Copy & paste the code below to embed this comment.
  28. Joe,
    you argue that “The final version of the WCAG Samurai Errata aren’t out yet, and it’s disingenuous to blow up your favourite failing of the Errata into a claim that the whole thing is naff”.

    Yet it seems to be perfectly acceptable to attack what is not the final version of WCAG 2.0 on the basis of certain bits of it that people don’t like. That’s not just you by the way; I did it as did many others.

    But to avoid having double standards it has to be reasonable to critique early versions of the WCAG errata thingummy in exactly the same way… you’ve proven you can dish it out; now you’ve got to show you’re big enough to take it.

    FWIW, I think you are, and I think you’ll get it right. But that doesn’t mean you should be immune from criticism, and nor do the self-appointed and secretive Samurai deserve any more protection than do the WCAG WG…

    Copy & paste the code below to embed this comment.
  29. bq. In three subsequent paragraphs under the heading “˜Cognitive disabilities neglected’, Gian Sampson-Wild implies that she was removed from the Working Group because of the way she voted.

    I was a participant in the WCAG WG at the time, and I can say that nobody “removed” her from the group, period. It is possible that she was notified that she had not met the requirements for good standing (i.e., she missed 2 of the last 3 teleconferences and/or 2 of the last 3 face-to-face meetings), which would have been restored once she resumed attending the calls. What she neglects to mention is that she missed six consecutive calls (_and_ six techniques calls) between February 26 and April 22, 2004, according to the group’s minutes.

    And yet, even she admits that those in attendance were opposed — _unanimously_ — to her proposal, and reiterated that opposition the following week.

    Occam’s razor, people. Her standing was changed because she was not actively participating. No conspiracy necessary.

    Copy & paste the code below to embed this comment.
  30. Now, Jack, our esteemed colleague Patrick H. Lauke did what I said he did. My previous article on WCAG 2 was a somewhat different animal. Care to compare and contrast?

    Copy & paste the code below to embed this comment.
  31. * Your one-liner list of WCAG 2 headlines is great as far as it goes, but in fact people _do_ need accessibility guidelines to be, at once, instantly gleanable, fully detailed, implementable without indecision, and actually likely to improve accessibility for people with disabilities. That may be an impossible dream, which would be a fair point of criticism. But maybe — just maybe — WCAG 2 as it stands is _too far away_ from these requirements.

    * Gian has stated publicly that she was very sick during the time she missed the phone calls. At least one WCAG WG member knows this firsthand. Illness is a disability, and the WCAG Working Group did not accommodate that disability short of undue hardship. The issue passed a vote “unanimously”? only after W3C technicalities were deliberately used to remove Gian from the group.

    Matt, you don’t work for the W3C anymore. It is no longer necessary to engage in Stockholm-syndrome-like defences of the organization. WCAG WG busts people who disagree or who threaten to withhold “consensus.”? That’s how they operate; that’s why they’re bullies; that’s why so very few of us trust them anymore.

    Copy & paste the code below to embed this comment.
  32. my right honourable friend joe may have misinterpreted my reason for the apparent statements of “naffness” of WCAG + Samurai.

    for the record, i do like the Samurai document a lot, barring a few minor nitpicks.

    what i was referring to in my comments ( “such as number 8”: ) was not meant to take away from it, but rather to ask Gian for clarification on why she felt it was a better solution to her fundamental problem with WCAG 2, leading to one of her final statements:

    bq. With the publication of the WCAG Samurai Errata, the web community finally has a choice ...

    which seemed like a non-sequitur to me, since Samurai has the same fundamental problem for cognitive disabilities in particular.

    Copy & paste the code below to embed this comment.
  33. First off, consensus _is not_ unanimity. If one participant stands apart from the rest on a point like this, they can raise a formal objection, which Gian has, and which will result in the group having to defend their decision. (Again.) Still, good standing would never have afforded her the privilege of lying down in the road to get her way.

    In this case, the argument seems to be that the proposal she made, which didn’t receive _any_ support from a group made up of a pretty even mix of advocates, academics and technical folks, should be overturned because… why? Because some bureaucratic status that is about as relevant to the situation as her eye color was changed around the same time? Because it somehow got into ALA? And now, because she was sick? (The messages she sent to the list to excuse herself at the time usually indicated she was “travelling.”) No matter how many times you try to explain away how it didn’t go the way you wanted, it’s still a bad idea.

    With regard to the perceived trustworthiness of W3C, and WAI in particular, well, take a bow. You’re the one who’s responsible for that, thanks in large part to your vividly creative storytelling. But that’s worth an article unto itself.

    Copy & paste the code below to embed this comment.
  34. Matt May knows that I back up my assertions about WCAG WG with original documents or links to them. This, in the industry, is called documentation, not “vividly creative storytelling,”? a cutesy phrase I assume means “lying.”? If anyone can prove I have lied about any topic in the last 30 years, please publish your documentation.

    Copy & paste the code below to embed this comment.
  35. The original article (remember that?) was a very good piece about testability.

    The last bunch of comments aren’t, and should be taken to personal blogs. Why not comment on Gian’s blog entry that includes the discussions with the working group. They’d be on-topic there.

    Copy & paste the code below to embed this comment.
  36. The notion that the best set of guidelines would be entirely testable seems ludicrous to me but so does the idea that any testability whatsoever is unwarranted.  I mean, we _are_ doing this on computers, after all - you’d be leaving entire orchards of fruit on the vine by refraining from having any testable guidelines whatsoever.

    For example, “Make all functionality available from a keyboard” is eminently testable, as are many specific cases of the other guidelines, for example, in compliance with “Help users avoid and correct mistakes” no form control should start off in a state that it cannot be returned to - e.g. the infamous set of radio buttons that start with nothing selected, but then once you click one you can’t back out even if it wasn’t a required field… and although the _quality_ of particular alt text can’t be judged by a machine its _presence_ - whether the alt tag is missing or empty, or even whether it contains an entire sentence - _can_ be machine-tested.  You could even go so far as to test whether all photographs have alt tags, whether images containing text have alt tags, whether images containing faces have alt tags, et cetera.

    And in the spirit of guiding rather than strict lawgiving, tests need not be for the strict purpose of passing or failing on compliance, they can be informative.  I must concur with the previous poster that a technological guideline, an actual test tool or suite of tools, may be appropriate for these parts of the standards.

    I do think that the testable and non-testable guidelines should be clearly delineated.  I also think that the cost of testability should be considered and ameliorated, particularly when publicly-available tools could be developed, or the existing ones better-used.  (But I would note that despite its title, neither the article nor the subsequent discussion has addressed cost.)

    Copy & paste the code below to embed this comment.
  37. I’m sorry, Bruce, but I disagree with you. Discussion of Gian Sampson-Wild’s claims in regards to her removal from the Working Group are valid.

    How many on that Working Group are voting in a specific way because that is the way that they have been told/shown is the ‘correct’ way? It could be that others agreed with Gian on testability but were too afraid to vote in the way that they truly felt.

    Is there bullying in other Working Groups in the W3C? Is the W3C creating documents to the best of all combined abilities?

    These questions need to be asked considering that at least one other person has made similar claims of this Working Group and there seems to be no response from the W3C about it; tacitly condoning the alleged type of behaviour.

    Copy & paste the code below to embed this comment.
  38. I’ve written up “my thoughts in more detail”: but taking into account Bruce Lawson’s quite sensible pointing out that we’re discussing testability rather than Working Group status or otherwise, I’ll not go into detail here.

    You can either nip over to my site to read my long and rambling opinionated rant, or not. I honestly don’t mind.

    Copy & paste the code below to embed this comment.
  39. For ALT text, the test is surely a self-critical question to the designer: “Does the alt-text I’ve used give the reader an idea of what the image is trying to convey, or what my purpose in including it in the web page was, if for some reason the reader can’t see the it?”

    For any image, there will be a range of acceptable alt-text, so the answer to the question will only be a matter of opinion, informed or not.  That’s not something that’s testable by machine, is barely testable by humans, and certainly not with any degree of consistency (at the required 80% level for WCAG2).

    If its not testable, you can’t measure compliance and hence the perceived reduction in usefulness of WCAG2 to purchasers of websites as a standard to aim for.  Sounds like sticking strictly to the letter of WCAG2 could result in less usable/accessible (sensu lato) websites. I’d rather stick to the spirit and use techniques shown to enhance accesibility/usability. Actual demonstration of that to a user/client would be better than certification.

    Copy & paste the code below to embed this comment.
  40. We do test our product though accessibility testing is often like many companies seen as important. If there are problems found by customers the issues are fixed pretty swiftly.

    As a designer I have never got into WCAG2 as it seemed to less used and to vague. Maybe I just haven’t got the time to learn another set of standards as WCAG1 was heavy enough.

    Copy & paste the code below to embed this comment.
  41. Testability is unfortunately a must for the many many people, including the W3C and its members, who make money through accessibility consultancy - it allows them to say “This is exactly right, and this is exactly wrong”, which helps a lot and leaves no uncertainties or arguments or incorrect online testing systems.

    However, Gian is exactly right - many of the most important elements of the current set of the WCAG are untestable - these elements also go hand in hand with search engine optimisation and usability/general user experience for even able-bodied users.

    The document defines testable as “machine-testable” or “reliably human testable”? — which means that eight out of ten human testers must agree on whether the site passes or fails each success criterion.”

    The truth is that disabled users are not machines and 8 out of ten human beings aren’t disabled. In reality, the W3C should get these guidelines approved not by a group of able-bodied front-end developers who are going to have to use and adhere to them, but by the very people they are intended to help the most: disabled users. And with the multitude of disabilities out there, it’s about time we started asking all of them what they think and what they would want from websites.

    Copy & paste the code below to embed this comment.
  42. Sorry, commenting is closed on this article.