Comments on Usability Testing Demystified

23 Reader Comments

Back to the Article
  1. Maybe Jakob Nielsen should allow a designer to style, since this is the site I most often hear referenced by people who associate good usability with poor, outdated design

    Copy & paste the code below to embed this comment.
  2. Great article, Dana. Thanks for clarifying the value, purpose, and flavors of usability testing. From what I’ve seen, the bad rap seems to stem from teams working with less experienced practitioners who ask the wrong questions, leading questions, or believe they must use the formal lab approach vs. doing quick or in-context testing. Hope this helps squelch the naysayers.

    Copy & paste the code below to embed this comment.
  3. Thanks, Dana.  Very good information.  I’m always hearing that some testing is better than none at all. That being said, what do you think about  I’m considering it as an option for quick and inexpensive testing for some of our firm’s clients.

    Copy & paste the code below to embed this comment.
  4. @ronansprake - there was a project to redesign one of Nielsen’s pages, you can see the results at “”: . Nielsen does not pretend that his website is an example of good aesthetic design, he knows that it is not, but says that he does not have the skills to make it aesthetically pleasing and so he would rather leave it plain and simple than make it a mess.

    Despite its looks, it is one of the most popular websites out there for web authoring and usability research, which just proves his point - if you build it well, they will come ... even if it looks cruddy.

    Copy & paste the code below to embed this comment.
  5. I’ve only done one session, but I found people didn’t like being asked to “test” something - they didn’t want to take a test. I learned a lot that time, mainly about what I should and shouldn’t say.

    It’s difficult to work behind a screen everyday and then go into such a public and more than likely humiliating situation. Through humiliation comes humility though. I guess.

    Next time I will try the “gathering feedback” approach.

    Copy & paste the code below to embed this comment.
  6. Recently while talking to a “Usability Expert” at one of our local utility companies, she asked if I had any experience with different usability software programs that spit out usability data. I must have had a blank look on my face as I thought to myself, “how many software programs are going to be using your website?”

    Copy & paste the code below to embed this comment.
  7. bq. I’ve only done one session, but I found people didn’t like being asked to “test” something — they didn’t want to take a test. I learned a lot that time, mainly about what I should and shouldn’t say.

    What you need people to understand is that they aren’t being tested. They are the examiners, your website is under the microscope. If they are having difficulty, that means that the website probably isn’t well-enough designed to meet their needs. Because even if they are clearly not demonstrating a lot of intelligence, logic or resourcefulness - and there are a lot of people out there who don’t! - they are still the people that your website is there to serve, and you’ve got a better chance of changing your website to fit what people can work with than you have of changing the way they work with your website.

    Yes, sometimes it’s hard. When you’ve created a design that you think is really easy to use, and some knucklehead comes along and can’t follow simple instructions, ignores the big highlighted links and buttons that you’ve put in prime spots and clicks around randomly for a while before giving up, it’s hard not to scream and cry and ask them why they can’t see the blindingly obvious when it’s right in front of them ... but you need to be able to understand their thought processes and what they are looking for/at, so that you can come up with a design that they can follow.

    Copy & paste the code below to embed this comment.
  8. To @romansprake and Stephen Down

    Stephen is right - Jakob has said this about Visual design is not his forte.

    Okay, so looks very 1997, but so what? It’s about content. Look at craigslist. Not beautiful but tons of great content that is pretty easy to get to.

    What you can learn through getting feedback from users is whether which part of design matters and how much, so you can make an informed decision about where to spend resources. Jakob has decided that, based on his audience, it’s about content, not slick visual design.

    Copy & paste the code below to embed this comment.
  9. @kwalser Thanks much!

    It’s easy to do a bad job of moderating usability test sessions or user research, but I think most people are well-intentioned about it.

    I actually find it much easier to ask the right kinds of questions in quick, in the wild tests because they tend to be very focused. And the context offers so much to observe. In formal lab tests there’s a lot of pressure to do it the “right” way, and less opportunity to recover if you screw up.

    But mostly it just takes practice. So I take your point about people having bad experiences with usability testing because their facilitator is not experienced.

    Copy & paste the code below to embed this comment.
  10. @usersrule,

      Glad you liked the piece. I am all about observing people using designs to reach their own goals. It’s about being present to witness the crazy things they say and do that you could never have predicted—it’s that “ah ha!” moment that you get when a participant creates a workaround to some hindrance in the design that gives you what you need to make good design decisions. There’s no substitute for that.

      And more data isn’t necessarily better. It depends on the source.

      Two things offend me about

    1. What I do is get people to help me evaluate a design. It is the design that is being tested, not the user. So I *hate* the term “user testing.” It’s usability testing.

    2. reduces “user testing” to a QA process. It isn’t QA. It’s a tool for the team to understand users. So, to do remote, unmoderated usability testing, you have to make sure that your test is *very* well designed, so there are no questions about the tasks. The other thing with QA is that it’s just running stuff through a test kit. Usability testing is about learning *why* users do what they do in addition to the *what*. So although with you get video of people, there’s no guarantee they’re going to think aloud, and you have no opportunity to ask follow up questions.

    Short answer: I’m not a fan.


    Copy & paste the code below to embed this comment.
  11. Many of us would like to do user testing, but how can resources which are typically thin be allocated? Resources meaning time and/or money. 

    Specifically, how can you gain meaningful input on something that s quite frankly not even to the “usable” stage in production yet, and getting it there would eat up some of the budget… only to learn from the testing that things have to be re-done? How is this reconciled?

    Copy & paste the code below to embed this comment.
  12. @Bill Leonard

    I come from a background in game design and have implemented some of the techniques that I learned from game development into my web development projects.

    I always create a design document upfront that not only describes in text exactly what the goal of the project is, but also provides well thought out visual diagrams of how the application will work. The diagrams do not have to be an exact replication of what the user-interface will look like. Think of it as the sketch that a painter lays on a canvas before actually painting.  The main goal is to be able to quickly create and modify a tangible representation of the process that the user will have to go through in order to achieve the desired goal.

    This design procedure provides the ability to immediately identify unnecessary actions that the user will have to go through to navigate your application.  You can also easily create several diagrams of different ways that the navigation could work.  You can then have a simple pre-test by presenting the diagrams to multiple people from your user demographic and getting a consensus on which UI seems to be the most user friendly.

    This will allow you to focus your development from the start before spending too much of your budget.  You should always do a usability test once you’ve got a working prototype, but a good design document goes a long way in making sure that the first prototype is not a waste of time, money or resources.

    Copy & paste the code below to embed this comment.
  13. The problem about testing the usability of early prototypes is that you have to find many more people to test it on. I realized that most of the people who tested early versions of a design do not want to spend time over again. Probably enterprises with a considerable budget do not have difficulties solving this problem, but for small projects good usability testing is challenging anyway.

    Copy & paste the code below to embed this comment.
  14. @Bill Leonard

    Great discussion. Costs are always a concern; budget and time for gathering feedback from users must be built in to a project plan from the beginning.

    For help with costs, you might want to check out a book by Mayhew and Bias called *Cost Justifying Usability Second Edition.*

    The idea of testing early on prototypes is that the team must be testing something that it is willing to experiment and if the design being experimented on doesn’t work, then be willing to throw it away (or part of it), and start over (or revise heavily). Prototypes are trials - they’re meant to be disposable but also to be cumulative. As you learn more, each successive one gets more usable and more detailed. For excellent help with prototyping software, web app, and web site UIs, see Carolyn Snyder’s *Paper Prototyping.* It’s brilliant and makes the whole process easy.

    To say that you need a lot of users to test a prototype isn’t quite right. You can test the first prototype with 1 or 2 or 3 users, and you’ll probably get data right away. That is, just within those few people, you will see where there are frustrations and hindrances. Then you can change the prototype and test with a few more people. Though there’s a lot of discussion about how many people to have test a design, my rule is use as many as you need to feel confident in the design decisions you make. If you observe one person having a particular problem and feel like you know just what to do about it, go ahead and change the prototype UI and test again. With luck, you’ll find different problems, or be able to refine what you did in the last round.

    In any case, usability testing is never meant to be a one-shot thing. It’s part of an iterative process of improving a design. The idea is that you’re learning what you need to know from real people to help you make well-informed design decisions.

    Copy & paste the code below to embed this comment.
  15. Several commenters have expressed some concern about recruiting participants for user research and usability studies.

    This is the most important step to get right in user research. If you don’t have the right users, you don’t have the right data.

    So, that sounds hard and expensive, but it doesn’t have to be.

    When you’re just starting out with a design, you can use people close to you. People in your company, people in your family, or friends—but they still must be like your real users. And they shouldn’t know anything about what you’re trying to do. That is, they should not be people on your project. Make sense?

    Usually, you can just say, “Can you give me a few minutes to try something out?” And they almost always will. Buy a cup of coffee or soda for them and thank them profusely. Not costly at all.

    As the prototypes become more real and you want to do more formalized testing, you can keep costs lower by going to the participants rather than setting up a formal test in a lab (unless your company has a lab and doesn’t charge you back for using it). Visit them in their offices or wherever they would normally use the design.

    This is a less expensive option than asking participants to come to you because you’re using less of their time. When you ask them to come to you, though it isn’t obvious, you’re also compensating them for travel time and the hassle of getting to you. If you visit them, you also get the benefit of bonus data because you’ll learn things by being in their environment that you won’t pick up from their being in your space.

    I’ve got a lot of articles on my blog about recruiting. You might want to check those out. This link will get you to all the posts tagged with “recruiting”:

    You might also want to read my article on Boxes and Arrows about how to treat study participants:

    Copy & paste the code below to embed this comment.
  16. @ Dana Chisnell

    Thanks for providing the great resources.  The article on Boxes and Arrows is spot on.  Many of the projects that I have been working on recently have extremely tight budgets.  That being said, we are usually pinching pennies during development.  Treating my testers well has really paid off for me. 

    I’ve been able to establish a good relationship with 5-6 people who will participate in tests as often as I need them.  These users are excited to see how their feedback effects our applications and always willing to test the prototypes for no cost.  Usually, we will provide lunch or other amenities before or after the test. That is enough to keep them happy.

    These guys are great at finding bugs in our systems and declaring what changes they would like to see.  I always test on a larger group once my core group of testers seem comfortable with the app. This process saves us a good amount of time and money.

    Copy & paste the code below to embed this comment.
  17. Thanks for that article, very informative.
    Whenever I’m designing and considering usability, I always think of the classic example of the newly built student campus. Put the buildings in place, but not the footpaths. After a couple of months, the students will naturally choose the easiest path between buildings and wear a groove in the grass. Build the footpaths there!

    Copy & paste the code below to embed this comment.
  18. Interesting article.

    I’m a big believer in testing but sometimes I get lazy. I think the feedback idea rather than testing is a great idea.

    I think content is key but visuals are important

    Copy & paste the code below to embed this comment.
  19. Usability testing may be an arduous process and it could very well be a luxury for some websites, but no matter what, it will always provide some insight into your website and your users.

    It’s amazing to realize that you can spend weeks hammering out a quality sitemap, navigation structure, and labels for important links, only to find that the actual visitors you are targeting don’t use the same terminology you do, and therefore can’t find what they’re looking for (even if its right in front of them).

    Usability testing doesn’t have to be burdensome — hold a simply focus group and ask participants to complete a “homework assignment” you’ve come up with. In the end, you can see how many were successful and give them a survey to add in their thoughts and frustrations with the project.

    Copy & paste the code below to embed this comment.
  20. Thanks Dana, great article. Thought you might be interested in our new fast and inexpensive service.

    It can be used for any type of online property (websites, website prototypes, online adverts, search processes, etc).

    It has a very attractive delivery speed, and price point (24-48 hours from order, 299$ for a 5 person user experience test). The features are as follows:

    Clients define a target url (their own or that of their competition or best practice)
    Clients define a goal for the testers to perform (e.g.; “find product x and take it through the checkout process..”)
    Clients define the demographics of the kind of testers they would like
    Clients define survey questions

    Within 24-48 hours clients receive a report that includes, for a minimum of 5 testers:

    Web cam recordings of the testers conducting their assigned goal/task
    A synchronized recording of the entire screen session during the test
    ClickFlow Analysis
    Contextual written “bubble” commentary on screenshots
    Survey results
    Other quantitative data

    Visitors to the site ( can request a free 1 person sample test.

    Kind regards,


    Copy & paste the code below to embed this comment.
  21. We use this and it is as Dana says, about evaluation,testing and great design. I think that the method of Alpha to Beta to release is key to success in any web application. Usertesting benchmarks the experience and I think you are right not to be a fan of this. I am for the before launch and after launch approach which emplys actually user information, market research and post launch click heat mapping.

    Copy & paste the code below to embed this comment.
  22. If you are a small company and dont really have the resources for such a large scale usability test, would you just carry out in house usability testing for a new piece of software or invite a select number of your regular customers to test the software online and give you feedback. As an online company its kind of difficult to set up a testing environment where you can visually see how people react, you can only get written feedback via a form. Where I think asking the right questions will determine how good and useful that feedback will be.

    Copy & paste the code below to embed this comment.
  23. For those that are looking to get feedback on your websites, I would recommend trying  I do not own this site, however, I am one of the testers that review websites.  The site shows website owners how visitors (such as myself)view/would use their website through the power of video.  Site owners write out task that they want the testers to complete.  The testers complete the tasks and send their videos to the site owner.

    Copy & paste the code below to embed this comment.
  24. Sorry, commenting is closed on this article.