Explore our Digital Experience Management Toolkit

10 mistakes to avoid in usability testing

Author's avatar By Expert commentator 03 Jun, 2014
Essential Essential topic

Better usability testing for websites and apps: the top 10 mistakes to avoid

Done well, usability testing enables organisations to validate ideas and designs for websites and applications with real-world users, helping increase engagement, satisfaction and conversions. But executing a successful user testing strategy brings with it numerous challenges and potential hazards. Think about your redesign and conversion optimisation projects. Are you at risk of committing any common testing mistakes?

Introduction to the value of usability testing

Research from the IEEE has found that fixing problems with software once development’s already begun is twice as expensive as resolving them before development commences.

Conducting lab-based usability testing that provides valuable insight into how users perceive and interact with your site or application before your project goes live can, therefore, deliver significant returns; in fact, it’s been found that for every dollar a company invests to increase usability it receives $10-$100 in benefits, and wins customer satisfaction and loyalty.

Ten most common mistakes in usability testing

However, successfully crafting, conducting and analysing these tests requires careful planning, and there are a number of challenges that you may stumble across. Here are some of the most common mistakes people fall victim to, along with tips to ensure you get the most out of any testing activity you undertake.

  • 1. Recruiting unsuitable participants

Even if your testing set-up is perfect, if you’re not testing with the right people your results won’t be as useful as they might be. It’s therefore important to research participants in advance, and build a test base that reflects your audience (for example, if visitors to your website are largely female, splitting test subjects 50/50 according to gender may not be representative).

  • 2. Not testing early and often during the project lifecycle

The overheads associated with a lab setup mean that often people wait until the very end of a project to conduct testing sessions, which can lead to costly issues being uncovered when they could have been much more easily solved at an early stage.

Testing throughout the project lifecycle is therefore recommended and there are, in fact, a wide variety of methods available, meaning that whatever your requirements there’s an option to suit you. Here’s a brief comparison of some of the most popular approaches:

Method Cost Description Results
Guerrilla testing * Asking members of the public to complete tasks. Low cost. High number of tests Quantitative & qualitative results. Unpredictable
Unmoderated remote testing ** Participants have no contact with a facilitator and complete tasks in their home environment. Low cost. High number of tests Quantitative results. Slightly unpredictable
Moderated remote testing *** Participants test remotely. Sessions are run by a facilitator. Similar costs to office-based testing. Sessions can be watched live Qualitative results. Heavily reliant on internet connection
Laboratory testing **** Participants test in the office and sessions are run by a facilitator. They can be watched live Qualitative results. Most likely to have more statistical validity

A combination of qualitative and quantitative testing throughout the project lifecycle has been found to produce the best results, so think about how you will build this into your own project. For example, a recent project I was involved with tested:

HTML Prototype Sessions Setup
Early draft 5 facilitated sessions Laptop, web cam, meeting room on client’s premises
Version 2 10 unmoderated sessions 3rd party testing solution
Version 3 5 facilitated sessions Box UK Usability Lab, observer at Box UK’s London office
  • 3. Following too rigid a test plan

Preparing a test plan is vital to keep track of sessions and, perhaps more importantly, guarantee that all important areas have been covered during the testing.

However, you shouldn’t be afraid to go off-course when necessary, as you may find that the answers you get when users are acting freely provide interesting insights you hadn’t considered previously.

Ranking your questions in order of importance can help you manage this unplanned activity while ensuring that critical topics are addressed.

BOxUK usability

  • 4. Not rehearsing your setup

It may sounds obvious but it’s crucial to check your equipment well in advance of your sessions to confirm that everything works as it should, from the devices participants will be using to the recording software you have in place, and even basics such as power points and internet connections.

  • 5. Using a one-way mirror

Many usability labs feature one-way mirrors that enable observers to monitor sessions without being seen by participants. However if the user knows they are being watched this can lead to unnatural behaviours and, subsequently, have a negative effect on your results.

For this reason, transmitting audio and visual feeds to a different room is often a better option, and one that also caters to remote teams anywhere in the world.

box uk ux

  • 6. Not meeting participants in reception

It’s important to remember that participants may be nervous about the session ahead, particularly if they’ve never been involved in laboratory testing before and are unsure about what’s required. Greeting them as soon as they arrive and making them feel at home helps create a relaxed and comfortable environment to support an open and insightful session.

  • 7. Asking leading questions

When creating your test plan, make sure you review all questions for any possible bias, as otherwise you may find that you’re actually leading the user to certain answers or actions based on what you expect (or want) to happen. Also be mindful of body language, as this can provide you with an even greater understanding of user responses.

  • 8. Interrupting the participant

While test facilitators should support the participant and, as covered above, explore potentially valuable new areas of enquiry, the main purpose of usability testing is to gather feedback from representative end-users. As such, make sure you give them enough time to express their thoughts, and avoid interrupting until you’re sure they are finished.

  • 9. Undertaking two roles in a testing session

Lab-based usability testing sessions involve both a facilitator who guides the user through the tasks, and an observer responsible for making notes and conducting the initial analysis.

If a single consultant is performing both these roles then critical responses could be overlooked, so be sure to allocate appropriate time and budget to testing.

  • 10. Not considering external influences

Ideally laboratory usability testing should be performed in facilities purpose-built for the task, as this provides a controlled environment that’s tailored for best results. No matter where your sessions are conducted, though, external factors such as noise, interruptions and building work may impact your activity, but preparing well in advance can help you manage these issues.

Find out more
So, that’s an introduction to some of the issues you should aim to avoid when embarking on a usability testing project, in order to unlock the full value from every one of your sessions and so improve the quality of your finished products and services. If you want find out more about any of the pitfalls discussed in this post, download our white paper 'Ten Common Usability Testing Mistakes', and if you think I’ve missed anything important please leave your comments below.


Author's avatar

By Expert commentator

This is a post we've invited from a digital marketing specialist who has agreed to share their expertise, opinions and case studies. Their details are given at the end of the article.

Recommended Blog Posts