3 Usability Testing Methods That Work For UX Insights
What comes to mind when you hear the phrase ‘Usability Testing’? Like many frequently used terms in Conversion Rate Optimisation this phrase is used to refer both to specific activities and broad categories of analysis. In this article, we’ll look at three different usability testing methods and answer the question “which type of usability testing should you use?”
These three types of usability testing offer different levels of insight into the issues and sources of confusion users may face when using your website. Understand the difference between these types of usability testing, and the benefits of each one, to learn which type is best for your specific requirements.
Table of Contents
What Is Usability Testing?
Before talking about the different types of usability testing here’s a quick recap of what’s usability testing.
Usability testing is a technique used to evaluate a product by testing it on real users. The goal is to identify usability issues, collect qualitative and quantitative data, and determine how well the product meets the needs of its intended users.
Usability testing is important because it provides direct input from real users on how they use the product, which in turn helps identify areas for improvement. This leads to products that are more intuitive, easier to use, and more user-friendly.
When choosing a usability testing method, businesses commonly face challenges like finding the right number and demographic mix of test participants, choosing between remote or in-person testing, balancing budget and resources with test scope, and picking the most suitable usability metrics to measure. Proper planning and being clear about the goals of testing can help businesses select the right usability testing approach for their needs.
What Usability Testing Is Not
1. It’s Not Just About Aesthetics
Usability testing is often mistaken as only evaluating the visual design and aesthetics of a product. While visual appeal is related to usability, usability testing looks at the entire user experience.
The goal is to uncover problems and gather insights into how real users interact with the product, not just critique the look and feel. Things like navigation, workflows, and how intuitive features are to use are key focuses.
2. It Doesn’t Happen At The End
Effective usability testing should be conducted early and often during product development, not only at the end.
When testing happens iteratively, issues can be identified early and fixed, saving time and resources.
End-of-process testing picks up critical flaws late when it’s costly to make changes. Ongoing testing provides valuable user feedback throughout the product’s evolution.
3. It Isn’t Just About Tasks
While usability testing often involves users completing predefined tasks, it shouldn’t strictly follow a rigid script. Open-ended observation can reveal insights like where users struggle without guidance and hearing their honest impressions. Mixing scripted tasks with free exploration better balances getting feedback on planned features while still catching unexpected issues.
Compared to A/B testing, usability testing provides more detailed qualitative data on why certain designs work better. Surveys collect opinions at scale but lack the direct observational insights of usability testing. Usability testing directly involves and engages with real users for a deeper understanding of how products work in the real world.
4. It’s Not Just Recruiting Users
Recruiting representative users is important, but effective usability testing is much more than that.
You need to carefully plan the tasks, draft the test script, determine the environment, choose the appropriate moderator, and consider the metrics to measure. Without paying attention to these factors, you won’t get quality, actionable data.
5. It Doesn’t Replace Market Research
While valuable, usability testing focuses on feedback on an existing product, not upfront market exploration.
To understand target user pains, motivations, and needs, employ market research techniques like interviews, surveys, and focus groups first.
This provides critical context for developing product requirements and functionality. Later usability testing assesses real user interactions with the product, complementing early-stage market research.
6. It Doesn’t Eliminate Other Testing
While critical, usability testing doesn’t replace functionality, security, performance, and other forms of testing.
Usability testing provides user experience insights, while other methods verify back-end quality attributes. The testing types are distinct but complement each other in a comprehensive test strategy.
7. It Isn’t a Replacement for Design Standards
Even if some usability issues didn’t surface during testing, widely accepted design conventions should still be followed when possible.
Relying solely on usability testing is risky, as no single test round can catch every issue. What users expect in terms of common navigation, input methods, and flows evolves gradually over time.
Types Of Usability Testing Methods
1. Laboratory usability testing
In a laboratory usability test, participants are studied in person while interacting with your website to monitor how they are using it. In addition, many laboratory studies also have the capability to gather data on eye movement, which provides additional insight into how they view your website, as well as recording the participant’s face, which often show some expressions that show the participant’s real feelings whilst they’re using your site. Laboratory testing sessions are usually filmed for in-depth analysis and insight generation at a later date.
Advantages of laboratory usability testing
- Allows for in-depth qualitative and quantitative analysis
- Can easily track eye movement, user stumbling blocks, satisfaction
- Provides control over test environment and equipment
- Ability to closely observe and measure user behavior
Disadvantages of laboratory usability testing:
- More expensive due to the need for special equipment and the environment.
- Participants must come to a specific location for testing.
- Artificial test environments may influence participant behavior.
- Does not provide insight into the natural context of use.
- Difficult to recruit representative users to a physical location.
- Logistically complex to set up and manage in-person testing
- Less scalable than remote testing methods
2. Remote Usability Testing
While laboratory usability testing is strictly controlled and expensive to track and monitor, remote unmoderated usability testing is inexpensive and easy for even the smallest of businesses to perform thanks to software like UserTesting or WhatUsersDo.
Remote unmoderated usability testing involves test participants completing specific, pre-defined tasks on your website, without additional guidance from a facilitator. They are completed at home on the tester’s own device and their screen and audio are recorded and sent to you to watch and analyse.
Some software providers offer the option of using their experts to analyse the videos for you but we recommend that, if you have time, you should watch the videos yourself to make sure you don’t miss out on any valuable insight.
While laboratory testing is often broad, with test participants completing big tasks and testers monitoring their interactions with your website, we find the best results from remote unmoderated testing tend to be when delving deeper into a single, highly specific area of your website. Providing testers with small, clearly defined tasks, such as ordering a certain product on your website or finding out more about a particular product or service, keeps them on topic and delivers usable insight time and again.
Advantages of remote unmoderated usability testing
- Fast turnaround, get results in as little as 48 hours.
- Cost-effective compared to moderated testing.
- Gain targeted insights on specific site areas.
- Participants test in their natural environment.
- Scalable – can run many tests simultaneously.
- No need to bring participants to a physical location.
Disadvantages of remote unmoderated usability testing
- Participants may not represent real users.
- Participants may lack the motivation to complete the tasks seriously.
- Lack of observation means missed contextual insights.
- Cannot ask follow-up questions or probe issues.
- Reliant on participant’s technology and internet.
- Difficult to identify vague or subtle issues.
- Data quality depends on test design and instructions.
Note: WhatUsersDo have since clarified that, unlike the majority of remote unmoderated usability testing sites, they limit testers to 3 desktop tests per month to address the disadvantages of people becoming ‘professional’ usability testers.
3. Remote moderated usability testing
Remote moderated usability testing is a testing method that involves participants completing their usual customer journey on your website whilst sharing screens with a facilitator using tools such as join.me, GoToMeeting or Skype.
In a remote moderated test, participants and test facilitators work remotely alongside each other while the test occurs. The test facilitator monitors the actions of the participant, communicating with them as and when required via chat or phone as they complete the test.
Since the test facilitator can communicate with the participant during the test, a moderated remote test allows the facilitator to ask follow-up questions for the test participant to gain additional insight into why they made the decisions they made while interacting with your website.
It also allows the test participant to ask for clarification on their instructions – an ability that prevents misunderstandings from occurring. This can lead to test data that’s more reliable and actionable than data from an unmoderated test.
Advantages of remote moderated usability testing
- Low cost compared to in-person moderated testing.
- Participants test in their natural environment.
- Can recruit real users that match the target audience
- The moderator can ask follow-up questions and probe issues.
- Gain insights into the overall customer experience.
- The large scale allows for more comprehensive feedback.
- Observation provides contextual insights
- The moderator can clarify vague issues in real time.
Disadvantages of remote moderated usability testing:
- It’s a time-intensive recruitment process
- It may need substantial traffic to find the right users
- Scheduling tests can be logistically challenging i.e finding the right time
- Harder for the moderator to observe subtle reactions
- Can be difficult to incentivize participation
- Requires trained moderators to conduct sessions
When To Use Laboratory Or Remote User Testing
Laboratory and remote user testing each have advantages that make them more suitable for certain scenarios:
Use in-person laboratory testing when:
- Subtle non-verbal behaviors, facial expressions, and body language are key to understand. Being physically present allows moderators to notice nuanced reactions that can be missed remotely. This helps gain insights into users’ emotions, hesitations, and confusion during tasks.
- The test environment is impossible to accurately recreate outside the lab, like testing a complex prototype or product still in development. The control of a lab environment ensures consistent conditions for all test users.
- You want to completely avoid distractions and focus user attention. A controlled lab with no outside interruptions or noise ensures users concentrate fully on test tasks.
- Stakeholder observation and direct experience with user interactions is important. Viewing in-person tests can elicit more empathy and engagement from stakeholders than watching remote video recordings.
Consider remote moderated testing when:
- Sourcing a representative sample requires geographic diversity impractical for in-person. Remote testing allows you to conveniently include users from anywhere.
- Testing the product in users’ natural environments yields more valid insights for certain products. Websites and apps can be best evaluated in real usage contexts.
- The comfort of home environments removes pressure users may feel being watched in a lab, yielding more honest feedback. The anonymity of remote testing encourages open sharing.
- Tight budget and travel constraints make in-person testing cost-prohibitive. Remote testing reduces costs by eliminating travel and facility expenses.
- Frequent quick testing iterations are needed, which online services enable by simplifying participant recruiting and test moderation.
Guidelines For Effective Usability Testing
1. Define Goals and Research Questions
Clearly define 2-3 high-priority goals and research questions to focus the test. Common goals include improving task success rates, reducing errors, easing points of struggle, and boosting subjective satisfaction. Well-defined goals guide effective task design and keep efforts focused on outcomes that truly matter to the product experience.
2. Recruit 5-8 Representative Users Per Test Round
Carefully recruit 5-8 participants per round that represent your core target users. Screen potential participants to match user demographics, backgrounds, expertise levels, and behaviors. For consumer products, avoid teammates to reduce bias. Segment screening criteria by user type or expertise level if appropriate. Offer fair incentives valued at $100-200 for participation time.
3. Plan Realistic Tasks Aligned to Goals
Design 5-7 key tasks aligned to goals, from onboarding to critical workflows. Prioritize testing tasks predicted to struggle most based on past data or team assumptions to confirm issues. Balance realistic end-to-end scenarios with targeted variable manipulations to isolate pain points.
4. Script a Logical Test Flow with Flexibility
Script a logical workflow guiding users through tasks and scenarios. But retain the flexibility to go off script and probe reactions, explore insights from open exploration, and deviate as needed based on moderator observations. Allow 20-30 minutes of free exploration time.
5. Set Up an Ideal Test Environment
Prepare a quiet, distraction-free space optimized for audiovisual recording, with all necessary prototypes, products, and tools available. For remote moderated testing, pilot test equipment and connections extensively beforehand.
6. Moderate Neutrally with Minimal Guidance
Greet users warmly and explain the test goals neutrally. Avoid leading behavior or excessive guidance during tasks, only answering direct questions. Silently observe and take timestamped notes, allowing natural reactions and struggles to emerge.
7. Record Quantitative and Qualitative Data
Record task success, completion times, errors, clicks, NPS, and sentiments. Capture think-aloud commentary, facial reactions, and body language. Probe after tasks for feedback. Supplement with post-task ratings, questionnaires, and interview questions.
8. Analyze Results and Make Recommendations
Identify key issues from notes, recordings, and data. Dig deeper into why problems occurred and how representative they are. Provide actionable recommendations for design iteration. Share compelling clips showing issues firsthand to build stakeholder empathy.
Frequently Asked Questions About Usability Testing
1. Question: How many users do you need for a usability test?
Answer: 5-8 users per test round is ideal. A small, diverse sample uncovers the majority of usability issues while keeping recruitment manageable.
2. Question: What tasks should you include in a usability test?
Answer: Focus on critical tasks aligned to key goals like onboarding, frequent workflows, and pain points. Balance real-world scenarios with targeted manipulations to isolate issues.
3. Question: How long should a usability test session last?
Answer: Plan for 45-60 minute sessions. Include time for pre/post-test interviews, instructions, and debrief. Test sessions themselves often take 20-30 minutes.
4. Question: Where should usability testing take place?
Answer: In a controlled lab or user’s natural environment for in-person testing, Remote moderated testing uses online tools with the user in their home/office.
5. Question: When should you conduct usability testing?
Answer: Early and often! Test prototypes and wireframes early, continue testing regularly throughout development, and keep testing post-launch.
6. Question: How much does usability testing cost?
Answer: $50-150 per test user. Significant savings come from remote moderated testing over in-lab testing.
7. Question: What metrics should you measure in a usability test?
Answer: Task success, completion time, errors, satisfaction ratings (CSAT/NPS), clicks, sentiment, and qualitative feedback like think-alouds.
If you’d like to learn more about usability testing to increase conversion rates, and need expert help, read our ebook for more information.
Is your CRO programme delivering the impact you hoped for?
Benchmark your CRO now for immediate, free report packed with ACTIONABLE insights you and your team can implement today to increase conversion.
Takes only two minutes
If your CRO programme is not delivering the highest ROI of all of your marketing spend, then we should talk.