1.1.2 Usability Testing

Introduction

Usability testing is a key method used to evaluate how user-friendly and effective a system or device is. It helps designers and developers find problems in a system before it reaches the final users. By watching real users interact with a product, teams can identify confusing features, slow processes, and areas where users make mistakes.

The textbook explains that there are several types of usability testing, each with a specific focus and approach. Choosing the right type depends on what the team wants to learn and the resources available.

Definition

Usability Testing is a method used to evaluate how user-friendly and effective a system or device is by observing real users as they interact with it.

In simple words: Usability testing means watching real users use a system to find out what works well and what needs improvement.

Main Idea

Usability testing is not about asking users "Did you like it?" — it is about observing what they actually do. A user might say a system is fine, but watching them struggle reveals the real problems.

• Moderated Usability Testing
• Unmoderated Usability Testing
• Qualitative Usability Testing
• Quantitative Usability Testing
• Remote Usability Testing
• AB Testing

Each type serves a different purpose and provides different kinds of insights.

Core Concept

1. Moderated Usability Testing In this type of testing, a facilitator or moderator is present to guide users through tasks, ask questions, and observe behaviors in real-time. The moderator can interact with users to clarify instructions or probe deeper into specific issues.

Example from the book: A company developing a new software application invites participants to a lab environment. The moderator asks users to perform specific tasks, such as navigating through the app's features or completing a transaction. The moderator observes how easily users can accomplish these tasks and asks questions about their experience, helping to identify any usability issues.

Why use it? Moderated testing gives deep insights because the moderator can ask follow-up questions and explore unexpected problems.

2. Unmoderated Usability Testing In unmoderated usability testing, users complete tasks on their own without a moderator's direct involvement. This type of testing is often conducted remotely, allowing participants to use the system or device in their natural environment. The sessions are typically recorded for later analysis.

Example from the book: An e-commerce website wants to test how easily customers can find and purchase products. The company conducts unmoderated usability testing by sending out tasks for users to complete at home, such as searching for a product, adding it to the cart, and completing the checkout process. The user interactions are recorded, and the company reviews the footage to identify any pain points or areas of confusion.

Why use it? Unmoderated testing is more natural because users are in their own environment, and it can reach more people at a lower cost.

3. Qualitative Usability Testing This approach focuses on understanding the user's experience, thoughts, and feelings while interacting with the system or device. The goal is to gather in-depth insights into user behavior, including what they find intuitive or challenging.

Example from the book: A mobile app developer conducts qualitative usability testing by interviewing users after they use the app. Users are asked open-ended questions about their experience, such as what they liked or disliked and any challenges they encountered. This feedback helps the developer understand the user's perspective and make improvements to the app's design.

Why use it? Qualitative testing explains why users behave the way they do. It reveals the reasons behind problems, not just the problems themselves.

4. Quantitative Usability Testing Quantitative usability testing collects numerical data to measure usability aspects like task completion time, error rates, and success rates. This method is useful for comparing different versions of a system or device or setting benchmarks for usability.

Example from the book: A company tests two versions of a website's navigation menu. In a quantitative usability test, users are timed as they attempt to find specific information using each menu version. The company collects data on how long it takes users to complete the tasks and how often they make errors. This data is then analyzed to determine which menu is more efficient and user-friendly.

Why use it? Quantitative testing provides measurable data that can be compared objectively. It answers questions like "How much faster is Version A than Version B?"

5. Remote Usability Testing Remote usability testing allows users to test a system or device from their own location, using their own devices. This type of testing can be either moderated or unmoderated, and it is particularly useful for reaching a wider audience or observing how users interact with a product in their natural environment.

Example from the book: A global software company wants to test a new feature with users in different countries. Remote usability testing is conducted, with participants using the software on their own computers while being observed through screen-sharing technology. The company gains insights into how the feature performs in different regions and environments without needing to bring users into a lab.

Why use it? Remote testing is convenient for users and allows companies to test with people from different locations, cultures, and device setups.

6. AB Testing AB testing, also known as split testing, involves comparing two versions of a system or device to see which one performs better in terms of usability. This type of testing is commonly used in web design and online services.

Example from the book: An online retailer wants to determine whether a new homepage layout will lead to more sales. They create two versions of the homepage and use AB testing to randomly show each version to different users. By comparing the sales metrics and user engagement data between the two versions, the retailer can determine which layout is more effective.

Why use it? AB testing provides direct, data-driven comparison between two designs. It answers the question "Which version works better?"

Intermediate Level

The six testing methods can be grouped in useful ways:

• Moderated: A facilitator is present (can be remote or in-person).
• Unmoderated: No facilitator; users work alone.

• Qualitative: Focuses on words, thoughts, and feelings (the "why").
• Quantitative: Focuses on numbers and measurements (the "how much").

• Lab-based: Users come to a specific location (often moderated).
• Remote: Users test from their own location (can be moderated or unmoderated).

• **AB testing**: Directly compares two versions.
• Other methods: Explore one version in depth.

A good usability study often combines methods. For example, a team might start with quantitative testing to find a problem, then use qualitative testing to understand why the problem exists.

Important Points

• - Usability testing evaluates how user-friendly and effective a system is.
• - There are six types: moderated, unmoderated, qualitative, quantitative, remote, and AB testing.
• - Moderated: Facilitator present, real-time observation.
• - Unmoderated: No facilitator, users work alone, often recorded.
• - Qualitative: Focus on user experience and feelings.
• - Quantitative: Focus on numerical data (time, errors, success rates).
• - Remote: Users test from their own location.
• - **AB Testing**: Compares two versions to see which performs better.
• - Different methods provide different insights; combining them often gives the best results.

Remember

• - Moderated = With facilitator
• - Unmoderated = Without facilitator
• - Qualitative = Why (thoughts, feelings)
• - Quantitative = How much (numbers, data)
• - Remote = From user's location
• - **AB** = Two versions compared
• - Usability testing = Watching real users to find problems.

Common Mistakes

• 1. Thinking usability testing is the same as asking users for opinions — It is about observing behavior, not just listening to opinions.
• 2. Confusing qualitative and quantitative testing — Qualitative explores thoughts and feelings; quantitative measures numbers.
• 3. Assuming moderated testing is always better — Unmoderated testing is more natural and can reach more users.
• 4. Using only one testing method — Different methods reveal different insights; combining them is often more effective.
• 5. Ignoring AB testing for design decisions — AB testing provides direct comparison data, which is useful for choosing between two designs.
• 6. Believing remote testing is less valid — Remote testing can provide valuable insights because users are in their natural environment.

Quick Revision

• Usability Testing: Evaluating how user-friendly a system is by observing real users.
• Moderated: Facilitator guides users through tasks in real-time.
• Unmoderated: Users work alone; sessions are recorded for later analysis.
• Qualitative: Focuses on understanding user experience and feelings.
• Quantitative: Collects numerical data like time and error rates.
• Remote: Users test from their own location; can be moderated or unmoderated.
• AB Testing: Compares two versions to determine which performs better.

60-Second Revision

• Usability Testing = Observing real users to find problems.
• 6 Types: Moderated, Unmoderated, Qualitative, Quantitative, Remote, AB.
• Moderated = facilitator; Unmoderated = no facilitator.
• Qualitative = why; Quantitative = how much.
• **Remote** = from user's location; **AB** = two versions.
• Warning: Don't confuse qualitative with quantitative, or moderated with unmoderated.

Frequently Asked Questions (FAQs)

Q1. What is usability testing? Answer: Usability testing is a method used to evaluate how user-friendly and effective a system or device is. It involves observing real users as they interact with the system to identify problems, confusing features, and areas for improvement.

• Moderated Usability Testing: A facilitator guides users through tasks.
• Unmoderated Usability Testing: Users complete tasks on their own.
• Qualitative Usability Testing: Focuses on user experience, thoughts, and feelings.
• Quantitative Usability Testing: Collects numerical data like task completion time and error rates.
• Remote Usability Testing: Users test from their own location.
• **AB Testing**: Compares two versions to see which performs better.

Q3. What is the difference between moderated and unmoderated usability testing? Answer: In moderated testing, a facilitator is present to guide users, ask questions, and observe behaviors in real-time. The moderator can clarify instructions and probe deeper into issues. In unmoderated testing, users complete tasks on their own without a moderator's direct involvement, often remotely, and sessions are recorded for later analysis.

Q4. What is the difference between qualitative and quantitative usability testing? Answer: Qualitative testing focuses on understanding the user's experience, thoughts, and feelings. It uses methods like interviews and open-ended questions to gather in-depth insights. Quantitative testing collects numerical data such as task completion time, error rates, and success rates. It is useful for comparing different versions or setting benchmarks.

Q5. What is remote usability testing? Answer: Remote usability testing allows users to test a system or device from their own location using their own devices. It can be either moderated or unmoderated and is particularly useful for reaching a wider audience or observing how users interact with a product in their natural environment.

Q6. What is AB testing? Answer: AB testing, also known as split testing, involves comparing two versions of a system or device to see which one performs better in terms of usability. It is commonly used in web design and online services. Users are randomly shown one of the two versions, and data on their behavior is compared to determine which version is more effective.

Q7. Why is usability testing important? Answer: Usability testing is important because it helps identify problems in a system before it reaches the final users. By observing real users, designers can find confusing features, slow processes, and areas where users make mistakes. This allows them to make improvements that enhance user satisfaction and productivity.

Q8. Which usability testing method should be used to understand why users struggle with a feature? Answer: Qualitative usability testing should be used to understand why users struggle. It focuses on gathering in-depth insights into user behavior through interviews and open-ended questions, revealing the reasons behind problems.

Q9. Which usability testing method is best for comparing two different designs? Answer: AB testing is best for comparing two different designs. It directly compares two versions of a system to determine which one performs better in terms of usability, using data on user behavior and engagement.

Q10. Can usability testing methods be combined? Answer: Yes, usability testing methods can and often should be combined. For example, a team might start with quantitative testing to identify a problem, then use qualitative testing to understand why the problem exists. Combining methods provides a more complete picture of usability.

Practice Questions

Short Response Questions (SRQs)

• Define usability testing and explain its purpose.
• List and briefly describe the six types of usability testing.
• Differentiate between moderated and unmoderated usability testing.
• What is the difference between qualitative and quantitative usability testing?
• Explain the purpose of AB testing.

Extended Response Questions (ERQs)

• Discuss the six types of usability testing. For each type, explain its purpose and provide an example of when it would be most appropriate.
• Compare and contrast moderated and unmoderated usability testing. What are the advantages and disadvantages of each?
• Explain how qualitative and quantitative usability testing complement each other. Provide examples of how they can be used together in a single usability study.
• You are designing a new educational app. Describe how you would use different types of usability testing to evaluate and improve your design.
• Analyze the role of remote usability testing in reaching diverse users. What are the benefits and challenges of remote testing compared to lab-based testing?

Multiple Choice Questions (MCQs)

• Q1. Which type of usability testing involves a facilitator guiding users through tasks?
• A. Unmoderated testing
• B. Moderated testing
• C. AB testing
• D. Quantitative testing
• Correct Answer: B
• Explanation: Moderated testing has a facilitator present to guide users.

• Q2. Which type of usability testing focuses on user experience, thoughts, and feelings?
• A. Quantitative testing
• B. Qualitative testing
• C. AB testing
• D. Remote testing
• Correct Answer: B
• Explanation: Qualitative testing focuses on understanding the user's experience and feelings.

• **Q3. What is the main purpose of AB testing?**
• A. To observe users in a lab environment
• B. To collect numerical data on task completion time
• C. To compare two versions of a system to see which performs better
• D. To interview users about their experience
• Correct Answer: C
• Explanation: AB testing compares two versions to determine which one performs better.

• Q4. In which type of testing do users complete tasks on their own without a moderator's direct involvement?
• A. Moderated testing
• B. Unmoderated testing
• C. Qualitative testing
• D. Lab-based testing
• Correct Answer: B
• Explanation: Unmoderated testing allows users to work alone without a facilitator.

• Q5. Which testing method collects numerical data like task completion time and error rates?
• A. Qualitative testing
• B. Quantitative testing
• C. Moderated testing
• D. Remote testing
• Correct Answer: B
• Explanation: Quantitative testing collects numerical data to measure usability aspects.

• Q6. Which type of testing allows users to test a system from their own location?
• A. Lab-based testing
• B. Remote testing
• C. AB testing
• D. Moderated testing
• Correct Answer: B
• Explanation: Remote testing allows users to test from their own location using their own devices.

• Q7. What is the key advantage of unmoderated usability testing?
• A. It provides deep insights through facilitator interaction
• B. It allows users to work in their natural environment
• C. It directly compares two versions of a system
• D. It collects only numerical data
• Correct Answer: B
• Explanation: Unmoderated testing is often conducted remotely, allowing users to work in their natural environment.

• Q8. Which type of usability testing is commonly used in web design to compare two homepage layouts?
• A. Qualitative testing
• B. Moderated testing
• C. AB testing
• D. Unmoderated testing
• Correct Answer: C
• Explanation: AB testing is commonly used in web design to compare two versions.

• Q9. What does qualitative usability testing aim to gather?
• A. Numerical data and error rates
• B. In-depth insights into user behavior and thoughts
• C. Two versions of a system for comparison
• D. Data from users in different countries
• Correct Answer: B
• Explanation: Qualitative testing aims to gather in-depth insights into user behavior, thoughts, and feelings.

• Q10. Which testing method can be either moderated or unmoderated?
• A. AB testing
• B. Remote testing
• C. Quantitative testing
• D. Lab-based testing
• Correct Answer: B
• Explanation: Remote testing can be either moderated or unmoderated, depending on whether a facilitator is involved.