(Almost) Everything You Wanted to Know About User Testing but Were Too Afraid to Ask! (Almost) Everything You Wanted to Know About User Testing but Were Too Afraid to Ask!

(Almost) Everything You Wanted to Know About User Testing but Were Too Afraid to Ask!

  • date-ic 27 Apr 2016
  • time-ic 4 minutes read

If you are building a website, app, or other digital product, you may have already considered testing your prototypes with real users. User testing is an integral part of any design process, and when done at the right time, can give you useful insights and ideas to improve the experience of using your website, app or digital product.

There are so many ways to recruit users to test your product. UXPin’s Guide to Usability Testing references over thirty different ways to test products on users. They range from grabbing the nearest person next to you and asking them to use your prototype, to focus groups, tree testing, remote user testing, and many other methods.

Whichever method you use, the main point of testing is the same: To get real users in front of your product so that you can make improvements or identify issues that you may not have been aware of.

When it comes to sample size, sometimes less is more

The type of testing you decide to do can also affect your sample size, or how many people you are able to test your product on. You have access to many more users if you use a testing platform than if you are conducting hallway user tests where you are finding a person or group of people closest to you at the moment.

However, hallway testing allows you to observe the participant in real time, not only recording their actions but their behaviors and comments. This can actually lead to a better understanding overall, and means that you do not need as many test subjects the first place.

In fact, Jakob Nielsen of the Nielsen Norman Group determined that conducting such tests on a bare minimum of five people should give you data that is accurate enough. He even created a formula:


Where N is the number of users and L is the proportions of usability problems found by a user, typically around 30 percent. The formula is graphed in the chart below:

neilsen usability

In other words, it is still extremely valuable to test your product on a limited number of people and in fact, testing with more than five people gives diminishing returns: You would be putting more work into those tests than you are getting new information out of them.

This is why TestMate recommends that each testing project runs with a minimum of four or five testers and up to a maximum of eight or nine. Our usability tests are effectively ‘hallway tests’ conducted remotely, with users sourced to reflect the target demographic of the website, app or digital product being reviewed.

Moderated vs. Unmoderated testing

Testing methods typically fall under two categories: moderated and unmoderated. Both can be used at different phases of the design process, and your preference of the two depends on the specific goals you are trying to achieve.

Moderated tests usually involve guidance and direction from a moderator who is already familiar with the design and purpose of the app. Moderators can help guide users and prod them to delve deeper in order to get a more complete picture of the user’s flow. Moderators can also record body language and verbal reactions.

Unmoderated tests let the users “run free” in a way, and they are usually administered by a usability testing tool much like TestMate. Unmoderated tests can happen remotely, and allow designers to gather feedback from hundreds of participants (or whatever their budgets allow).

The Advantage of Unmoderated Tests

Unmoderated, remote tests are typically faster and easier to perform, cost less, and are an easy way to build user testing into a company that previously has not thought about testing much in the past.

TestMate can provide designers with all of these benefits, including helping to create a test plan and analyzing results obtained. But what can you really learn from unmoderated tests? The answer might surprise you!

It is possible to mine a wealth of quantitative and qualitative information, including:

  • Task completion rate
  • Time spent on a task or a page
  • Clickstream paths
  • Satisfaction ratings
  • Design feedback
  • Points of frustration or confusion
  • Whether users can find what they are looking for
  • If users find your product credible or trustworthy

You can do a lot with this data, so long as you set up your test correctly and ask your users the right questions.

If you’re testing your product frequently, it is important to know how to get the most out of your user testing sessions. Make sure that you and other team members identify clear goals by answering the question: “What do we hope to learn from this?” The answer to this question could involve both broad, open-ended tasks like first impressions of a website, or it could be specific tasks like learning how to place an item in the shopping cart or navigating to the “Settings” page.

“…Unmoderated tests

It is also important to consider running tests for different demographics and groups of users, since they might respond differently to the same questions.

It is key to ask your users the right questions and to not fall into the trap of asking leading questions that could induce bias. Don’t ask users to find specific content on your Web site or app – instead, ask them to perform a task. For instance:

Do Ask: “Show me how you would buy a gift on this website”

Don’t Ask: “Was it hard to find the preferences page?” or even worse, “What do you think about this icon?”

Open-ended questions such as the former can be incredibly valuable.

It is well-known that the results of any research study can be affected by an observer or moderator, and unmoderated tests provide a method of obtaining unbiased feedback that can greatly help your design in the long run.

Why TestMate?

Poor user experience is a common problem among digital products, and TestMate’s services can help you differentiate your offering in the market, increase conversion rates, and ensure customer engagement. TestMate combines the efficiency of testing platforms with the value generated by testing with real users from your chosen demographic.

We can tailor our services to meet your company’s individual needs, and will help to create a strong test plan and recruit users. When testing is complete, you will be able to view video feedback of their interaction with your site, app, or digital product. Our team can also extract key insights from your user tests in order to help you identify pain points and can give you actionable recommendations to fix problems. Please contact us if you want to learn more about any aspect of testing, or just to chat!

Identify breaking points or issues in your product before they become a real problem with TestMate.

Sarah Khan,
UX Writer @ TestMate


Keen to know more
about our Full Service?

Our world class qualitative user testing can help explain the why behind your data and give you the answers to your burning questions.