Remote User Testing

User testing is a fundamental step in the journey towards launching a website. Businesses don’t want to invest time and money into site development if they aren’t certain the changes will satisfy what real users actually need.

There are a few options for conducting user testing. The most commonly employed are moderated user testing, where users come to your business premises to complete a test under observation, and remote user testing, where users complete tests at home, recording their journey with screen capture software.

This article focuses on remote user testing, why you might choose it and how to go about conducting it.

Remote user testing

Remote user testing can be done at various stages of a new site build, from wireframe layout to advanced stages of design, and for existing websites that are being upgraded to include new information and functionality.

A site doesn’t need to be near completion to benefit from user testing. You can get useful information testing a wireframe by asking users what information they expect to see on the page, and what functionality they would expect from the site. Even if certain windows and icons aren’t yet active, you can observe if users click on them, expecting to be led somewhere.

Advantages of remote user testing

• A moderator is not required while tests take place, which leaves them free to work on other things.
• You don’t need to supply a venue or organise appointments and meeting rooms.
• You save money, as you are not including travel expenses in the amount you pay users.
• The user can complete the test at their convenience, and because they aren’t being observed, they are less likely to give responses they think moderators want to hear.
• You can access a broader range of demographics because there are no geographic limitations. Remote user testing is inclusive of people of all ages, locations, and physical capabilities. This results in more meaningful data.

Challenges of remote user testing

• Users may be wary of downloading screen capture software to their computer, or may struggle using it. (For in-house moderated tests, you can set the device up in advance and assist with technical hiccups as they arise.)
• Since testers use their own devices, you may encounter personal information in the screen capture. It is therefore important to instruct users to only commence recording when they are ready for the task.
• You may save time by not moderating the test, but you don’t escape the need to view, record and analyse the test results when they come in.

User Testers

TestMate has a pool of perceptive users from all demographics to navigate and give feedback on your site. We can assist you in connecting to user testers, however, if you prefer to handle it yourself, you can source your own users. Consider testing internal employees, customers, family and friends in the target demographic, or advertising for testers.

Who can be a user tester?

Anyone can be a tester, as long as they have an internet connection and access to the site, app or digital product you are testing on. At TestMate, users apply to be testers with us. We ask them to complete a sample test, and from there, we select based on their ability to communicate feedback effectively, and their demographic. At times, we further whittle down our target list to identify users who fall into particular consumer categories. For instance, if our client is a health insurance company that wants to engage users who have recently purchased health insurance, we would ask our users if they fall into this category.

On the topic of screening, it’s worth noting that users generally want to score the testing gig. It’s therefore possible they may say they are eligible to test when they aren’t. It’s no big deal, but to avoid it happening you can ask yes/no screener questions when you do your call out. As users won’t know which answer guarantees participation, you get an accurate response. (For the written test, you can include screener questions in your survey design.)

What users need

• If you are designing changes to both a website and an app, you need to nominate which device your users should test on. It goes without saying, they need access to that device and an internet connection. They also need to download the screen recording software appropriate to the test.
• It’s worthwhile getting users to complete a pre-test to ensure they are comfortable using the testing software, and feel confident with delivering their feedback clearly.
• Users need to agree to maintaining confidentiality, as they may be testing sites for products or services that haven’t been publicly launched yet.
• Each user should be given a user number/code to quote at the start of the test, rather than using their real name.

Setting up remote user testing

The following is a checklist for conducting remote user testing:

• Create a test plan for both a screen recorded test, and a written test.
• Run a pilot test.
• Confirm your demographic. Are you testing only your existing customer demographic, or throwing the net wider?
• Contact users and confirm participation.
• Provide testers with clear instructions. This includes what device to test on, how long the test should take, and advise them to keep their test plan handy during the running of the test (either printed or on a separate device) as they should not leave the website during screen recording.
• Users should read the test plan questions aloud for each task, and provide a commentary of their search so that the analyst knows why they halted at a particular part of the site. For example, they may say, ‘I’m clicking on the Products menu, however I can’t find Mortgages here. I’ll try another menu.’

The Test Plan

The test plan for the screen capture test is created around usability goals for your company or client. What do you want to learn from the test? When you clarify that, you create your plan.

The plan provides the user with clear instructions for the test and includes:

• A brief about the company.
• A scenario/context for the test. For example, ‘In this test you are looking to buy a house for under $900,000. You have come to the company’s website to see what they offer.’
• A persona for the user. For example, ‘This will be your first home loan. You have a $60,000 deposit. You are single and work full-time.’
• An outline of the main task. ‘You need to find out where on the site you will find information about buying your first home, and how you can apply.’
• The test tasks. Generally, these ask the user to start on a particular webpage, and navigate the site to search for pertinent information.

Written test

After the screen capture test, it is beneficial to include a written test. The written test allows you to more easily quantify the data you receive. At TestMate, our written tests incorporate NPS (Net Promoter Score) and SUS (System Usability Scale) questions, such as:

• I found the tasks unnecessarily complex
• I thought the tasks were easy
• I needed to learn a lot of things before I could get going with the tasks
• I felt very confident doing the tasks

For the written test, you can use one of the many survey tools available online, such as Google Forms and Survey Monkey. Some tools charge to export survey data, however Google Forms does not, so it is an economical option. (Click here to see a comparison of different survey tools.)

Pilot testing

Pilot testing your test plan gives you the opportunity to eliminate confusing terminology and remove questions that are too long or are causing people to get lost. (The test, ideally, shouldn’t exceed 15 minutes, as testers become fatigued and the value of the findings is diminished.)

After trialling the test plan yourself, ask at least one other person to complete it, and for larger projects, two to three people. At TestMate, for complex projects, we run the pilot with a selection of users from our testing base, before we roll out a test to other users. Our clients also review the test plan, and we incorporate their feedback into the final version.

Analysis

After your users complete their tests, it’s time to watch the footage and note the results in a spreadsheet. You need to note the tester ID, their response for each question, and the time it took them to find the information outlined in each task. (Use the screen capture time code for this.)

For reporting purposes, it’s useful to categorise feedback. At TestMate, we use the following categories:

• First impression (general thoughts on the site)
• Design (feedback related to the site design)
• Usability (observations of how the customer used the site)
• Functionality (observations of how the site functions)
• Suggestion (suggestions for how the site could be enhanced)

Once you have completed the annotation and placed feedback into relevant categories, you can report on the test results.

Reporting on results

Reporting falls into Key Findings and Recommendations, and should be delivered to the client (or your managerial team if it’s in-house) both verbally and in written form. After this delivery, decisions are made regarding what improvements go ahead. Budgetary, technological and time constraints mean not all improvements identified in user testing can be made. Regardless of this, the test results indicate adjustments that are most vital and businesses can proceed more confidently with this insight.