Better usability testing for websites and apps: the top 10 mistakes to avoid
Done well, usability testing enables organisations to validate ideas and designs for websites and applications with real-world users, helping increase engagement, satisfaction and conversions. But executing a successful user testing strategy brings with it numerous challenges and potential hazards. Think about your redesign and conversion optimisation projects. Are you at risk of committing any common testing mistakes?
Introduction to the value of usability testing
Research from the IEEE has found that fixing problems with software once development’s already begun is twice as expensive as resolving them before development commences.
Conducting lab-based usability testing that provides valuable insight into how users perceive and interact with your site or application before your project goes live can, therefore, deliver significant returns; in fact, it’s been found that for every dollar a company invests to increase usability it receives $10-$100 in benefits, and wins customer satisfaction and loyalty.
Ten most common mistakes in usability testing
However, successfully crafting, conducting and analysing these tests requires careful planning, and there are a number of challenges that you may stumble across. Here are some of the most common mistakes people fall victim to, along with tips to ensure you get the most out of any testing activity you undertake.
- 1. Recruiting unsuitable participants
Even if your testing set-up is perfect, if you’re not testing with the right people your results won’t be as useful as they might be. It’s therefore important to research participants in advance, and build a test base that reflects your audience (for example, if visitors to your website are largely female, splitting test subjects 50/50 according to gender may not be representative).
- 2. Not testing early and often during the project lifecycle
The overheads associated with a lab setup mean that often people wait until the very end of a project to conduct testing sessions, which can lead to costly issues being uncovered when they could have been much more easily solved at an early stage.
Testing throughout the project lifecycle is therefore recommended and there are, in fact, a wide variety of methods available, meaning that whatever your requirements there’s an option to suit you. Here’s a brief comparison of some of the most popular approaches:
||Asking members of the public to complete tasks. Low cost. High number of tests
||Quantitative & qualitative results. Unpredictable
|Unmoderated remote testing
||Participants have no contact with a facilitator and complete tasks in their home environment. Low cost. High number of tests
||Quantitative results. Slightly unpredictable
|Moderated remote testing
||Participants test remotely. Sessions are run by a facilitator. Similar costs to office-based testing. Sessions can be watched live
||Qualitative results. Heavily reliant on internet connection
||Participants test in the office and sessions are run by a facilitator. They can be watched live
||Qualitative results. Most likely to have more statistical validity
A combination of qualitative and quantitative testing throughout the project lifecycle has been found to produce the best results, so think about how you will build this into your own project. For example, a recent project I was involved with tested:
||5 facilitated sessions
||Laptop, web cam, meeting room on client’s premises
||10 unmoderated sessions
||3rd party testing solution
||5 facilitated sessions
||Box UK Usability Lab, observer at Box UK’s London office
- 3. Following too rigid a test plan
Preparing a test plan is vital to keep track of sessions and, perhaps more importantly, guarantee that all important areas have been covered during the testing.
However, you shouldn’t be afraid to go off-course when necessary, as you may find that the answers you get when users are acting freely provide interesting insights you hadn’t considered previously.
Ranking your questions in order of importance can help you manage this unplanned activity while ensuring that critical topics are addressed.
- 4. Not rehearsing your setup
It may sounds obvious but it’s crucial to check your equipment well in advance of your sessions to confirm that everything works as it should, from the devices participants will be using to the recording software you have in place, and even basics such as power points and internet connections.
- 5. Using a one-way mirror
Many usability labs feature one-way mirrors that enable observers to monitor sessions without being seen by participants. However if the user knows they are being watched this can lead to unnatural behaviours and, subsequently, have a negative effect on your results.
For this reason, transmitting audio and visual feeds to a different room is often a better option, and one that also caters to remote teams anywhere in the world.
- 6. Not meeting participants in reception
It’s important to remember that participants may be nervous about the session ahead, particularly if they’ve never been involved in laboratory testing before and are unsure about what’s required. Greeting them as soon as they arrive and making them feel at home helps create a relaxed and comfortable environment to support an open and insightful session.
- 7. Asking leading questions
When creating your test plan, make sure you review all questions for any possible bias, as otherwise you may find that you’re actually leading the user to certain answers or actions based on what you expect (or want) to happen. Also be mindful of body language, as this can provide you with an even greater understanding of user responses.
- 8. Interrupting the participant
While test facilitators should support the participant and, as covered above, explore potentially valuable new areas of enquiry, the main purpose of usability testing is to gather feedback from representative end-users. As such, make sure you give them enough time to express their thoughts, and avoid interrupting until you’re sure they are finished.
- 9. Undertaking two roles in a testing session
Lab-based usability testing sessions involve both a facilitator who guides the user through the tasks, and an observer responsible for making notes and conducting the initial analysis.
If a single consultant is performing both these roles then critical responses could be overlooked, so be sure to allocate appropriate time and budget to testing.
- 10. Not considering external influences
Ideally laboratory usability testing should be performed in facilities purpose-built for the task, as this provides a controlled environment that’s tailored for best results. No matter where your sessions are conducted, though, external factors such as noise, interruptions and building work may impact your activity, but preparing well in advance can help you manage these issues.
Find out more
So, that’s an introduction to some of the issues you should aim to avoid when embarking on a usability testing project, in order to unlock the full value from every one of your sessions and so improve the quality of your finished products and services. If you want find out more about any of the pitfalls discussed in this post, download our white paper 'Ten Common Usability Testing Mistakes', and if you think I’ve missed anything important please leave your comments below.