Testability Costs Too Much

by Gian Sampson-Wild

49 Reader Comments

Back to the Article
  1. Matt May knows that I back up my assertions about WCAG WG with original documents or links to them. This, in the industry, is called documentation, not “vividly creative storytelling,”? a cutesy phrase I assume means “lying.”? If anyone can prove I have lied about any topic in the last 30 years, please publish your documentation.

    Copy & paste the code below to embed this comment.
  2. The original article (remember that?) was a very good piece about testability.

    The last bunch of comments aren’t, and should be taken to personal blogs. Why not comment on Gian’s blog entry that includes the discussions with the working group. They’d be on-topic there.

    Copy & paste the code below to embed this comment.
  3. The notion that the best set of guidelines would be entirely testable seems ludicrous to me but so does the idea that any testability whatsoever is unwarranted.  I mean, we are doing this on computers, after all – you’d be leaving entire orchards of fruit on the vine by refraining from having any testable guidelines whatsoever.

    For example, “Make all functionality available from a keyboard” is eminently testable, as are many specific cases of the other guidelines, for example, in compliance with “Help users avoid and correct mistakes” no form control should start off in a state that it cannot be returned to – e.g. the infamous set of radio buttons that start with nothing selected, but then once you click one you can’t back out even if it wasn’t a required field… and although the quality of particular alt text can’t be judged by a machine its presence – whether the alt tag is missing or empty, or even whether it contains an entire sentence – can be machine-tested.  You could even go so far as to test whether all photographs have alt tags, whether images containing text have alt tags, whether images containing faces have alt tags, et cetera.

    And in the spirit of guiding rather than strict lawgiving, tests need not be for the strict purpose of passing or failing on compliance, they can be informative.  I must concur with the previous poster that a technological guideline, an actual test tool or suite of tools, may be appropriate for these parts of the standards.

    I do think that the testable and non-testable guidelines should be clearly delineated.  I also think that the cost of testability should be considered and ameliorated, particularly when publicly-available tools could be developed, or the existing ones better-used.  (But I would note that despite its title, neither the article nor the subsequent discussion has addressed cost.)

    Copy & paste the code below to embed this comment.
  4. I’m sorry, Bruce, but I disagree with you. Discussion of Gian Sampson-Wild’s claims in regards to her removal from the Working Group are valid.

    How many on that Working Group are voting in a specific way because that is the way that they have been told/shown is the ‘correct’ way? It could be that others agreed with Gian on testability but were too afraid to vote in the way that they truly felt.

    Is there bullying in other Working Groups in the W3C? Is the W3C creating documents to the best of all combined abilities?

    These questions need to be asked considering that at least one other person has made similar claims of this Working Group and there seems to be no response from the W3C about it; tacitly condoning the alleged type of behaviour.

    Copy & paste the code below to embed this comment.
  5. I’ve written up “my thoughts in more detail”:http://www.thepickards.co.uk/index.php/200707/wcag-20-testability-testing-times-tetchiness/ but taking into account Bruce Lawson’s quite sensible pointing out that we’re discussing testability rather than Working Group status or otherwise, I’ll not go into detail here.

    You can either nip over to my site to read my long and rambling opinionated rant, or not. I honestly don’t mind.

    Copy & paste the code below to embed this comment.
  6. For ALT text, the test is surely a self-critical question to the designer: “Does the alt-text I’ve used give the reader an idea of what the image is trying to convey, or what my purpose in including it in the web page was, if for some reason the reader can’t see the it?”

    For any image, there will be a range of acceptable alt-text, so the answer to the question will only be a matter of opinion, informed or not.  That’s not something that’s testable by machine, is barely testable by humans, and certainly not with any degree of consistency (at the required 80% level for WCAG2).

    If its not testable, you can’t measure compliance and hence the perceived reduction in usefulness of WCAG2 to purchasers of websites as a standard to aim for.  Sounds like sticking strictly to the letter of WCAG2 could result in less usable/accessible (sensu lato) websites. I’d rather stick to the spirit and use techniques shown to enhance accesibility/usability. Actual demonstration of that to a user/client would be better than certification.

    Copy & paste the code below to embed this comment.
  7. We do test our product though accessibility testing is often like many companies seen as important. If there are problems found by customers the issues are fixed pretty swiftly.

    As a designer I have never got into WCAG2 as it seemed to less used and to vague. Maybe I just haven’t got the time to learn another set of standards as WCAG1 was heavy enough.

    Copy & paste the code below to embed this comment.
  8. Testability is unfortunately a must for the many many people, including the W3C and its members, who make money through accessibility consultancy – it allows them to say “This is exactly right, and this is exactly wrong”, which helps a lot and leaves no uncertainties or arguments or incorrect online testing systems.

    However, Gian is exactly right – many of the most important elements of the current set of the WCAG are untestable – these elements also go hand in hand with search engine optimisation and usability/general user experience for even able-bodied users.

    The document defines testable as “machine-testable” or “reliably human testable”? — which means that eight out of ten human testers must agree on whether the site passes or fails each success criterion.”

    The truth is that disabled users are not machines and 8 out of ten human beings aren’t disabled. In reality, the W3C should get these guidelines approved not by a group of able-bodied front-end developers who are going to have to use and adhere to them, but by the very people they are intended to help the most: disabled users. And with the multitude of disabilities out there, it’s about time we started asking all of them what they think and what they would want from websites.

    Copy & paste the code below to embed this comment.
  9. I cannot agree strongly enough with Aidan Williams – we need to start testing with, and talking to, people with disabilities.

    This group of people is unfortunately under-represented in the WCAG Working Group. Often when we worked in Working Group sub-groups to revise specific success criterion we were split by geographic location, not by interest or knowledge of a particular area of disability. But this is not a problem relevant only to the Working Group: I believe the entire accessibility community needs to engage with their users more.

    Copy & paste the code below to embed this comment.