Comments on Collaborative User Testing: Less Bias, Better Research

9 Reader Comments

Back to the Article
  1. Alla, great article! I have to admit, when I started reading it I was quite cynical, prepared for yet another piece about usability testing that regurgitates the same things. However, I found your focus on the various types of bias very refreshing! I’m also impressed by the efficiency with which you seem to be running these test sessions. Thanks for all the little tricks! It gives me some renewed energy to push for more usability testing - as you say, it’s easy as a product designer to believe that you know your users well enough to no longer test.

    Copy & paste the code below to embed this comment.
  2. Such a great article, especially for teams that struggle for practical suggestions on how to work testing into short sprint cadences.

    Copy & paste the code below to embed this comment.
  3. Alla, Really accurate observations. When I volunteered as a facilitator in the Usability Lab at the Academy of Art University, I constantly evaluated my vocabulary to ensure that I was not prompting the user. I noticed that it also helped having the test facilitated by someone who was not so attached to the outcome.

    Copy & paste the code below to embed this comment.
  4. Great article, I am always concerned about my own biases when testing concepts with users. I am wondering with the two teams testing at the same time, do they test the same thing with the same user twice? Or two users at the same time? If it is two users at the same time, then am I right in assuming that it doesn’t have to be at the same time?

    Copy & paste the code below to embed this comment.
  5. Great article, I really enjoyed reading it!

    I work in industry now where we do most of our testing at our offices, and the critical thing for us is making sure that we get UX, Content, Design and Product to watch the sessions (using Morae or WebEx), even if it means we have to buy them food. That increases both diversity of points of view watching the sessions and buy-in on the results. If there’s a point where the team disagrees about what they saw, the ensuing discussion is usually very productive. Generally by the time the study is over, the team has a pretty clear idea of the next steps, even before the report is written.

    I’ve always felt that half the benefit of usability testing was taking the product team and locking them in a room and really making them look at their own product for several hours, and really THINK about how users will use it. A lot of the insights are from the team just from doing those mental walk-throughs, in addition to what the actual participants do.

    I like the idea of giving observers homework to do while observing, to get them more engaged. I think you could also take advantage of Google Doc’s ability to allow multiple people to edit the same document simultaneously to let folks see each other’s notes during the session and make it something of a discussion. Perhaps give them each their own column in a spreadsheet so they’re not editing on top of each other.

    I have a few other articles you may be interested in: http://thecapras.org/mcapra/UXArticles.html

    Thanks for taking the time to read my dissertation!

    Copy & paste the code below to embed this comment.
  6. Great article. I echo Matty’s thoughts and really appreciate the perspective of this piece. My big question is how do we get stakeholders and managers to have less bias (both confirmation and hindsight)? The Paul Goodwin article talk a little about this, although the line “Simply warning people of its dangers has no effect” is a frustrating start. Are there more articles expanding on this subject?

    Furthermore, how do we get stakeholders to begin investing in testing in the first place? Especially on small details (rather, details that appear to be small). This is something I am struggling with at the moment.

    Copy & paste the code below to embed this comment.
  7. Your experiences in testing are not unlike those found in boardgame design. Ask people their opinion, and they will share it in ways that may be soul crushing/exactly what you needed to hear.

    Cf. Designing Modern Strategy Games (Studies in Game Design Book 1)Aug 14, 2012
    by George Phillies and Tom Vasel.

    Copy & paste the code below to embed this comment.
  8. Sorry, commenting is closed on this article.