Gaining Insights through a Usability Study - Analytics for Product Development - Customer Analytics For Dummies (2015)

Customer Analytics For Dummies (2015)

Part IV

Analytics for Product Development

Chapter 14

Gaining Insights through a Usability Study

In This Chapter

arrow Defining tasks

arrow Measuring usability

arrow Finding problems

If you’ve ever struggled to change the time on an alarm clock, couldn’t figure out how to work a remote control, or failed to successfully apply for insurance on a website, you have some idea about how important the ease of use of a product or experience is.

Your product can have all the features a user wants, but if it’s too difficult to use, then the user will move on to another product.

Recognizing the Principles of Usability

If you make your products too difficult to purchase online, or make a form too difficult to fill out, customers will leave or call your customer support line. Both of these are expensive, undesirable outcomes.

Good usability can lead customers to recommend your product; poor usability leads customers to discourage their friends and colleagues from purchasing a product. (See Chapter 12 to find out how important customer loyalty is.)

remember Usability is different than other data collected from market research in that it measures both customer attitudes and customer actions. Usability is as much about what customers do as what they say.

A product or website is usable if

· Customers can complete what they want to do (effectiveness)

· Customers can complete their task quickly (efficiency)

· Customers don’t find the experience frustrating (satisfaction)

There isn’t a usability thermometer that tells you how usable an experience is. Instead you have to rely on the outcome of experiences by measuring the effectiveness, efficiency, and satisfaction of the experience.

A usability problem is anything in the interface, buttons, labels, designs, or layout that prevents users from completing their task, makes the task difficult, or leads to longer task times. In this chapter, I show you how to run a usability test so you can discover and fix your usability problems and provide a quantified better customer experience.

Conducting a Usability Test

One of the best ways to identify usability problems is to conduct a usability test. The basic idea behind a usability test is that you watch users complete (or fail at) tasks with software, hardware, or a website and see what problems they run into.

remember You might be tempted to help participants or just ask if they like software or a website. But the point of a usability test is to observe your participants and not interfere with their process.

Here are the steps to conduct a usability test:

1. Determine what you want to test.

2. Identify your goals.

3. Outline your task scenarios.

4. Recruit users.

5. Test your users.

6. Collect as many metrics as possible.

7. Code your data and analyze it.

8. Summarize and present the results.

The following sections provide the details to complete each of these steps.

Determining what you want to test

You can rarely test everything about a product or website that you want to at once. That would simply overwhelm your participants.

Narrow the scope of your study by focusing on known problem areas, high usage areas, or some combination of the two. See Chapter 13 for ideas on prioritizing features for testing.

tip Eye-tracking usability tests involve special equipment that track where participants look as they use websites and software. Eye-tracking studies are more involved and require special equipment, but can help answer questions about whether customers’ eyes are attracted to features or content.

tip If you’re developing a new product, you don’t have to wait to test its usability. You can conduct usability tests on sketches or partially completed products. The earlier you find problems in the development process, the easier they are to fix. Test early and often to get the most of your usability testing.

Identifying the goals

While a usability test is primarily about observing customers and uncovering the problems they have, the approach you take depends on your goal.

Identify what the main goals of your study are:

· Are you trying to find and fix as many usability problems as possible?

· Do you want a benchmark of usability and performance?

· Are you comparing an application with a competing product or alternative design?

Each of these three testing goals requires slightly different testing parameters and sample size requirements (see Chapter 2). Many studies will also combine the goals. For example, you can have a study to compare your product to a competitor, and find out which tasks take longer to complete.

Outlining task scenarios

The task scenario is the hallmark of the usability test.

A task is made up of the steps a user has to perform to accomplish a goal. A task scenario describes what you want the participant to achieve. Crafting task scenarios is a balance between providing enough information so users aren’t guessing about what they’re supposed to do and not providing too much information (which could overwhelm them). Your goal is to simulate the discovery and nonlinearity of real-world application usage.

Follow these guidelines for creating task scenarios for your usability test:

· Be specific. Give participants a reason or purpose for performing the task. Instead of giving generalities like “find a new kitchen appliance,” ask them to find a blender for under $75 that has high customer ratings.

In the real world, users usually start searching with general ideas of what they want, and then quickly narrow their selection based on price, quality, and recommendations. In the artificial world of usability testing, users encounter problems if what you provide is too vague, and they will look to a moderator (if there is one) for what they should do. Don’t be so vague in your task that users have to guess what you want them to do.

For example, “You need to rent a mid-sized car on July 21 at 10 a.m. and return it on July 23 at noon from Boston’s Logan Airport” is specific.

· Don’t tell the user where to click and what to do. While providing specific details is important, don’t walk the users through every step. Such hand-holding will provide biased and less useful results.

For example, instead of saying “Click the small check box at the bottom of the screen to add GPS,” just say “Add GPS to your rental car.”

· Use the customer’s language, not the company’s language. It’s a common mistake to mirror the internal structure of a company on a website’s navigation or on software screens. It’s also bad practice to ask participants to do things based on internal company jargon or terms.

warning False positive test results or outright confusion may result if users don’t understand the terms used in a scenario. Do users really use the term “asset” when referring to their kids’ college funds? Will a user know what a product “configurator” is or an “item-page” or even the “mega menu”?

· Have a correct solution. If you ask a user to find a rental car location nearest to a hotel address, there should be a correct choice. This makes the task more straightforward for the user and allows you to more easily know if a task was or wasn’t successfully completed.

warning The problem with “Find a product that’s right for you” tasks is that participants are in the state of mind of finding information to solve problems. At the time, there probably isn’t a product that’s right for them; they’re more interested in getting the test done and collecting their honorarium. This can lead to a sense that any product selection is correct and inflate basic metrics like task completion rates.

· Avoid making tasks dependent on each other. It is important to alternate the presentation order of tasks so not all tasks are presented in the same sequence. Participants get “warmed-up” during a usability test and this can make earlier tasks seem harder and later tasks seem easier. When your tasks have dependencies (for example, create a file in one task, then delete the same file in another task), if a user fails one task, he or she fails the other.

warning Avoiding dependent scenarios isn’t always possible if you’re testing something like an installation process, but be cognizant of both the bias and complications introduced by adding dependencies.

· Provide context but keep the scenario short. Get the users thinking as if they actually need to perform the task, but don’t go overboard with details.

For example, “You will be attending a conference in Boston in July and need to rent a car.”

tip While there isn’t a “right” number of tasks for a usability test, try to identify three to five tasks for your test participants to accomplish.

remember Make the tasks realistic and representative of what users actually do. Don’t just ask participants if they like or will use an application or website. Have them register, log in, fill out a form, search for information, answer a question, or purchase a product. See Chapter 13 for more ideas on identifying the top tasks.

Write the task scenario in a way that gives participants enough information to complete a task but not too much that leads them step-by-step down a path. For instance, provide product names, specific price ranges, and brands.

Here’s an example:

· Task Example: Update a saved expense report.

· Earlier you started to create an expense report but ran out of time. You saved your work with the description, “Industry Conference.” You want to add an expense for sending a FedEx package to the report. Find the report and add the FedEx package as a new expense for $26. Submit the report for approval when you are done.

tip If you’re testing different types of customers or users, you may need to set up a different set of test scenarios based on

· User needs. For example, software that both teachers and students use (like Blackboard) has different tasks.

· User skills and experience. For example, international non-English speaking users often have a different perspective than domestic U.S. users when browsing a website.

Recruiting users

You need customers in order to do usability testing. It can be a small-scale, do-it-yourself usability test or a large sample corporate usability test, but finding available (and willing) participants can be difficult. In fact, it’s often cited as one of the reasons usability testing isn’t done more often.

The process by which you find your users will vary depending on what you are testing, the types of users you need, and the stage of testing (early versus late).

Here are six sources to help find users.

· Hallway testing: Grab anyone who is unfamiliar with the application or website you’re testing. These can be coworkers, friends, family, or folks at the local coffee shop.

Hallway testing works well for general purpose websites or apps and when you’re looking to uncover the more obvious problems with the interactions (problems, of course, always seem obvious after you find them).

warning Don’t rely too heavily on this method, especially when the design is more refined or users have specialized skills. When users don’t have an interest in a product or service, they will be happy to give you an opinion on matters — but relying on them may generate false positive results and overlook issues that actual users will encounter.

· Existing users: Your existing users are an obvious wellspring for testing. If you’re offering a brand-new product, you won’t have this source, of course. But, if your company makes similar products, then it makes sense to leverage those customers:

· Finding existing users off your website is an easy place to start. You can use pop-ups (offered by UserZoom and Ethnio) or fixed opt-in boxes to solicit volunteers. Marketing and sales departments usually have customer contact information you can tap.

· You can work with the customer support department and ask customers if they’d be interested in participating in follow-up studies. While existing customers sound like a panacea for finding users, volunteers may not have the time or availability to commit to an hour-long study, so you often have to rely on other sources.

· This website not only delivers audio and video of users using a website or mobile app, but it also has a large panel of users. allows you to recruit based on age, gender, and geography. You can also ask participants to self-select by asking them to participate only if they’ve used a certain website, own a particular product, have health insurance, or have a 401(k) account, among other things. Although it isn’t a solution when you need hundreds of responses or very specific recruiting criteria, or to test software and hardware, getting a source of users and a 15- to 20-minute usability testing for under $40 per participant is often very effective.

Similar services are offered by the websites and

· Craigslist and Facebook: When your recruiting requirements become more specific, you can post ads on Craigslist or Facebook. Marketing agencies have posted ads for focus groups for years in classifieds, so you should have no problem finding willing participants (usability testing is, of course, not a focus group, so be prepared to set expectations). Craigslist works for in-person and remote usability testing. In addition to the cost of the posting, be prepared to spend between $50 and $200, depending on the commitment and type of user you need.

warning Don’t count on Craigslist for finding participants with highly specialized skills or who are high-income earners. You tend to get people who have time on their hands and are looking to make some extra cash. With the right posting and honorarium, you can usually find a great pool of participants for most usability tests.

· Panel agencies: For unmoderated usability testing or surveys where you need hundreds to thousands of respondents, consider using a panel agency. Panels have huge databases of people from around the world. They keep track of all the usual demographic variables but allow you to find people like small-business owners, IT managers, Costco shoppers, or tablet owners.

Opinions for Good (Op4G; helps non-profits raise money by having its members participate in the studies. The participants keep a portion of the honorarium and a portion gets donated. This model allows them to reach many people with specialized job skills and high-income individuals who normally wouldn’t be interested in taking an online survey for a few dollars. You can also try Toluna ( and Research Now ( Plan on paying between $15 and $55 per completed response.

· Market research recruiters: Looking for hardware engineers, lawyers, medical professionals, chief financial officers, or people with $100K+ in two investment accounts? When you need to find very specific skills for in-person testing, you’ll likely need the help of professional recruitment firms. Firms like Fieldwork ( and Plaza Research ( maintain huge local and international databases to match almost any recruitment need you may have.

tip Plan on paying a recruiting company between $150 and $300 per recruited participant plus an honorarium of between $100 and $250 per participant. This approach can get expensive, so be sure you put as much effort into creating effective tasks and asking the right questions as you do in finding the right test participants. Don’t obsess over perfectly matching every demographic variable.

Find a sample of participants who represent the larger customer base you are interested in understanding. The total number of users you’ll need to test will largely depend on your testing goals (see the earlier section “Identifying the goals”).

· If your goal is to find and fix problems, then aim for between five and ten participants for each round of testing. See the later sidebar, “Sample sizes for a usability test.”

· If your goal is to find a benchmark of usability performance, you need a sample size that achieves a tolerable margin of error around your metrics. See Chapter 2 for a table of sample sizes.

For example, if you had 32 participants and 50% of them successfully completed the task, you can be 90% confident the percentage of all customers completing the same task would be between 36% and 64%.

· If your goal is to compare products, the sample size is affected by a number of variables (including the confidence, power, and type of metric).

You can usually approximate the needed sample size based on two factors:

· How large a difference you want to detect (if one exists)

· Whether the same users will attempt tasks on both products (called within-subjects) or if a different set will attempt tasks on each product (called between-subjects)

The following table shows the approximate difference you can detect in metrics, such as the completion rate or any other binary measure (at a 50% completion rate).

Difference to Detect 90% Confidence & 80% Power

Sample Size Within

Sample Size Between














































For example, at a sample size of 426 (213 in each group), you can detect a 12% difference for a between-subjects design (see the row that starts with 12%). So if 50% complete a task on one product and 62% on a competitive product, the difference would be statistically significant.

remember These estimates are conservative but recommended when planning a study without prior data. For continuous measures like perceived difficulty and task time, you can detect smaller differences at the same sample size.

tip If you have the time and budget to test 15 users, it’s more effective to test 5 users, fix the problems you see, then test another 5 users, fix those problems, and then test with a final 5 users.

Testing your users

When you’re ready to test, plan on 60 to 90 minutes per participant. Participants start to get fatigued after 90 minutes. Aim for most tasks to take 10 minutes or less. This leaves time for an introduction, pre-study questions, three to seven tasks, post-study questions, debriefing, and to solve any technical problems that pop up.

tip To uncover problems during a usability test, have your participants think out loud. Tell them to articulate their thoughts: what doesn’t make sense and what confuses them as they attempt the tasks. Don’t stop participants while they are working through details. Instead, wait until they get stuck or after a task to have them explain something they didn’t understand. See the later section, “Finding and Reporting Usability Problems,” for more information.

Collecting metrics

In addition to usability problems, you should collect three core usability metrics when you want to understand the experience a customer has with your product. The following task-based metrics correspond to the three areas of usability: effectiveness, efficiency, and satisfaction.

· Completion Rate: If a participant completed the task, code it Success (1); if the participant didn’t complete the task, code it as Fail (0). Figure 14-1 shows the completion rate for two tasks on two different websites.

· Task Time: Measure how long a user took to complete a task.

· Perceived Difficulty: After a participant attempts a task, ask her to rate how difficult it was. A 7-point scale with 1 being Very Difficult and 7 being Very Easy is common.

tip Record the errors users make, such as clicking on the wrong link, making typos, or not seeing the search results on a page. Keep this data for a findability study (see Chapter 15).

In addition to these task-based measures, capture at least one overall study measure that has participants rate their satisfaction with the overall experience with the product or website. Good questionnaires include the SUS and SUPR-Q (see Chapter 9).


Figure 14-1: The completion rate in a usability test.

Sample sizes for a usability test

Finding the right sample size is a trade-off between the cost of running participants and the capability to detect usability problems. The larger the sample size, the more usability problems that get uncovered. There is, however, a diminishing return as fewer new usability problems get uncovered with each additional user. Not all usability problems uniformly affect all customers. Some problems only affect 30% or 5%. The fewer customers a problem impacts, the larger the sample size you will need to have a good chance of observing the problems in a test.

The recommended way to make a sample size decision is to follow these three steps:

1. Pick a problem occurrence percentage that you want to detect.

For example, 5%, 10%, or 20% of customers will have a problem with a task or element of a design.

2. Pick a percent of these problems you want to observe.

For example, you might decide you want to observe 80%, 85%, or 90% of customers.

3. Use the binomial probability formula to determine the sample size needed.

For example, if you want to identify problems that impact 10% or more of your customers and you want to have an 85% chance of seeing them (if they exist) in a usability test, then you need to plan on observing 18 users. See for calculations and background.

At the sample size of ten, most problems that affect 20% or more of customers will be observed. A sample size of five (for example, using a “new customer” versus an “existing customer” subset) will still uncover 68% of the issues impacting as few as 20% of users. At both sample sizes, virtually all issues experienced by 40% or more of users will be observed (given the tasks performed and type of user recruited).

The following figure shows the percent of usability problems uncovered by testing five and ten customers. For example, around 90% of problems that impact 20% or more of users will likely be seen after testing ten users. That is, if a problem exists that would affect 20% of all users, there’s about a 90% chance of seeing it at least once in a usability testing with ten users.


Coding and analyzing your data

After you observe users attempting tasks, summarize the task and test metrics and include confidence intervals (see Chapter 2). Look for high and low performance across the metrics, and then look to understand “why” by summarizing what problems users encountered and what comments they made.

Summarizing and presenting the results

Correlate your results with graphs and a report or briefing. Report the results of the usability metrics, the usability problems found, and overall findings from the study. Conclude with actionable items that you can accomplish to make the tasks easier for customers to complete.

For example, in a usability test of, the site lacked a total price for the rental car. A clear action item would be to add a total to the confirmation page. If a total can’t be added because of variation in taxes or prices per state, for example, then provide a clear estimate in a prominent location on the page and an explanation of why a precise total can’t be provided.

Considering the Different Types of Usability Tests

Usability testing originally was conducted only in expensive corporate facilities. These usability labs had two-way mirrors and expensive video and audio equipment. Participants had to take the time to come to the lab for testing. Only big-budget firms could afford to conduct these types of usability tests.

In the last 15 years, however, technological improvements have allowed marketers to collect task-oriented data from customers without needing them to come to a physical location.

There are three major approaches to usability testing. Each has its advantages and disadvantages:

· Lab-based: This is the classic approach to usability testing. Customers physically come to a lab, often with a two-way mirror, and are observed by a team of product developers or researchers. This is still the primary method for testing physical products, like remote controls or mobile applications. A facilitator moderates the session, so it’s often referred to as moderated in-person testing.

While many companies may have a dedicated lab for testing customers (our company has two) and specialized software, you don’t need much to conduct a usability test. You just need a conference room, office, or some other space where your product can be used by a customer. You also need adequate time to set up the study.

· Remote moderated: Customers log into screen-sharing software like GoTo Meeting ( or WebEx ( and attempt the same tasks as if they were in a lab. The software allows customers from anywhere in the world to share their screen and even control your computer. You can even have them turn on their webcam to see their faces. A facilitator still guides the participant through the tasks, but the customer and facilitator can be in different cities or countries.

· Remote unmoderated: Software from companies like UserZoom ( and Loop11 ( walk customers through tasks and questions. It’s particularly well suited for testing websites or web applications. Everything about the interaction can be recorded, including a recording of what testers did on-screen, and their facial expressions captured by their web cam. This technology also works for mobile devices so you can have testers from anywhere in the world.

This method can give you stats that you might not be able to replicate in a moderated or lab-based environment. Figure 14-2 shows how UserZoom offers a heat map, showing where testers first start to complete a task.

A variation on this method is a service from has a panel of users who are trained to think out loud as they walk through tasks. You can get feedback from five users the same day you launch a study. It’s called remote unmoderated because you don’t need a test facilitator to walk participants through each task. This allows you to collect a lot of data quicker and with less cost than with other methods.

tip Combine multiple methods to fully understand a customer’s experience with a website or software. For example, you can have 300 users complete tasks using the remote unmoderated method and then have 10 to 15 users come into a lab so you can follow up on interesting interactions or problems.

Table 14-1 shows the pros and cons of each of the three usability testing methods.


Figure 14-2: This heat map shows users had problems with labels and navigation.


Finding and Reporting Usability Problems

When you have a usability test, one of your primary goals will be to find and fix as many usability problems as you can.

Follow these steps to uncover usability problems:

1. Record the undesirable outcome.

Did users make a mistake, not notice an element, take a long time to complete a task, or fail to complete the task altogether? These actions are symptoms, but not guarantees of usability problems. Errors are undesirable, but they aren’t necessarily caused by a problem with the interface (think typos). Both first-time and experienced users commit errors.

2. Identify what in the interface is causing the problem.

Sometimes it’s obvious (maybe users don’t know what an icon means), but other times, knowing the context, what the user was doing, and what he was asked to do is essential for extracting a usability problem.

3. Determine if the issue is global or local.

It can often be helpful to identify whether a problem affects the entire interface or just a section of it. For example, if users have problems turning on filters on a mobile TV app, it’s likely local to the filtering functionality. However, if users don’t understand what a name in the global navigation on a website means, it’ll affect the entire website experience.

remember Just because an issue is local doesn’t make it less important. Rather, it helps you and the design team understand both the scope of the problem and the potential impact of the change.

4. Assign a severity rating.

You want to separate the catastrophic problems from the cosmetic ones. One of the best ways is to assign a severity rating, from minor (1) to major (3). Most usability studies will reveal a lot of problems, many of which can’t be addressed. The severity ratings help prioritize what to fix first.

5. Recommend possible solutions, if appropriate.

Every usability issue doesn’t have to have a proposed solution, but if there is an obvious fix, then suggest it or implement it if you’re in charge of the development. You can often recommend solutions when a new design can fix multiple issues. Usability problem descriptions should be usable and useful. Often just a well-described usability problem can help developers generate ideas for design alternatives.

6. Present usability problems.

While the “right” way to find and present usability problems will depend on the purpose of the study (finding and fixing versus comparing to previously identified problems), it’s ultimately about balancing priorities, design, and business constraints to determine what to fix. Providing a list of usability problems is the simplest and most straightforward approach to presenting usability problems. Include the percent of users who encountered each problem and the assigned severity.

For example, in a usability test of a new email program that tested six users, here are four problems found, with the percent of users who encountered the issues, and the interpretation of each issue’s severity (on a 3-point scale: 1 = Minor, 2 = Moderate, 3 = Severe).

Percentage of Users




2: Moderate

Participants were confused that you can only drag and drop after clicking a check box, not by clicking anywhere within the email cell.


2: Moderate

Participants had trouble locating the “Add Contact” button.


2: Moderate

It was not immediately clear that the envelope icon means send an email.


1: Minor

Participants expected the email to show full screen instead of split screen.

tip Add screen shots, videos, and additional descriptions to help everyone understand the issues — especially if the usability issues are controversial. To aid comprehension, include a simplified problem list at the beginning of a report and a more detailed follow-up section with screen shots to provide the necessary context.

You can also chart your users and problems. In Figure 14-3, the users are numbered on the y-axis (1 to 30) and the issues have each been given a number (1 to 28) and sorted from most to least frequent. Each black square represents which user encountered which problem. You can compute such matrices by task or across tasks for the whole usability test.

Displaying problems in a type of matrix allows you to see three problems from three perspectives:

· Frequency by problem: For example, the most common problem (far left) involved users having trouble adding a GPS navigation to their rental (they had to do it after they entered their personal details). This problem impacted 24 out of 30 users (80%). You can be 95% confident between 62% and 91% of all users would also have this problem. (See Chapter 2 for a refresher on confidence intervals.)

· Frequency by user: The first user in the matrix encountered 9 of the 28 problems (32%), while user 28 encountered only 1 problem (3%), and users 29 and 30 encountered no problems while renting the car.

· Problems that affected only one user (the long tail of usability problems): The last nine problems were encountered by only a single user (3%).


Figure 14-3: Visualize which users encounter which problem.

What 3 users can tell you that 3,000 cannot

Just because you have a small sample size doesn’t mean you can’t identify major problems with a customer’s experience. Software company Autodesk had a problem with a software download. The company noticed a large spike in customer support calls related to a trial version of AutoCAD. A usability study was set up to observe how customers downloaded the software from the website.

During the study, 3 out of 11 customers unknowingly downloaded the wrong version of the software. Not until several minutes into the installation did they encounter a problem — a bizarre error message telling them their operating system wasn’t supported.

This message led customers to call the support line. It turns out that these users were confused about which version of software to choose from the layout of the trial download page.

Using confidence intervals, the company could be 90% confident that somewhere between 11% and 52% of all customers would be also struggling with this design element. In other words, with just 11 users, it could virtually be certain at least 1 out of 10 customers would have a problem and that many of them would call support.

Remember: It’s easier to detect that a product experience is unusable with a small sample size than it is to conclude that the product experience is usable. More common problems will come up in a test with just a few participants, indicating an unusable experience. However, just because you don’t observe a problem with a small sample doesn’t mean that the product is usable. There can be problems but they may affect a smaller percentage of users. You can’t be as confident the product experience is usable with a small sample size.

Facilitating a Usability Study

Facilitating a usability test takes practice, interpersonal skills, and the ability to juggle many things at once in order to collect data from a test participant. Here are some guidelines for facilitating a moderated usability test:

· Be prepared to listen. You need to talk to moderate a session, but don’t let the talking get in the way of discovering.

remember Like in any relationship, you’ve got to know

· When to talk

· When to listen

· When to move on

· Don’t lead the user. Even if a user asks if she “did it right” or is going down the wrong path and asks, “Is this the right way?” try and deflect such questions by asking back, “What would your inclination be?” or “Where would you go to look for that?”

· Don’t put the participant on the defensive. One of your goals as a facilitator is to get into the heads of your customers as they use the product. You need to know why users are clicking in different areas or not understanding a term or concept, for instance. It’s only natural to ask users why they did something. Instead of directly asking, “Why did you click that link?” ask “What about the link led you to click on it?” This will get the participant thinking about her motivations and mental models and not feeling defensive and respond with less helpful “because I thought I was supposed to go there.” Balancing question asking without putting the user on the defensive takes some practice.

· Probe users about interaction problems between tasks. If you wait until after a participant is done, then you get an accurate reading of how long a task takes and you don’t interrupt the user or inadvertently suggest ideas on how to complete a task.

· Have a note taker and separate facilitator, if possible. The facilitator is often kept busy asking follow-up questions, troubleshooting technical issues, answering user questions, and keeping the study on track. It’s easy to miss valuable insights if one person is performing both functions.

· Review the observations and problems after each user. Review the issues when they’re fresh — with another person, such as a note taker or stakeholder. It helps get the problem list out faster and allows you confirm or deny hypotheses about what’s causing problems and what might fix them with your next set of users.

· Record positive issues, suggestions, and usability problems. Don’t just collect the bad news. Collect those suggestions and positive comments and features that go smoothly. Although a development team will often want to get right to the problems, most will also appreciate that users and usability professionals aren’t all gloom and doom.

· Illustrate issues using screen shots and categorize problems. Sorting problems into logical groups such as “buttons,” “navigation,” and “labels,” along with a good picture, can really help with digesting long lists. Figure 14-4 shows responses to open-ended questions about what problems users had while renting a car. The open-ended comments were categorized and then charted. The vertical lines show the 90% confidence intervals around each percentage of comments (See Chapter 2 for a reminder on converting qualitative data to quantitative data and adding confidence intervals). The two biggest issues uncovered in the comments are from participants having problems finding the total price of the rental car (“Totaling”) or finding the GPS and car seat to add on to the rental (“Finding Extras”).

· Use highlight videos. Small clips of the most common usability problems or illustrative examples are helpful for stakeholders who rarely have time to view videos in their entirety.


Figure 14-4: Categorized and charted responses show users have the most problems finding the total price for the rental car.

Comparative usability test of Enterprise and

Here’s how a comparative usability study between two rental car companies, Budget and Enterprise, was conducted.

Study Goals

How well prospective and current customers can use the websites, which websites customers preferred, and what problems customers encountered while trying to find the price of a rental car.


Asked 62 qualified participants (people who had rented a car online in the last year) to complete two tasks on both websites. Half the participants attempted the tasks on Budget first and the other half on Enterprise first.


Unmoderated remote testing setup using software from UserZoom, with audio and video recordings.


Two tasks were created.

· Find the nearest location.

· Find the address of the nearest rental office to the Hilton Hotel located at 921 SW Sixth Avenue, Portland, Oregon, United States 97204. Write down or copy the name of the street of the nearest location.

· Rent a car.

· Location: Logan Airport, Boston, MA

· Rental Period: Friday, April, 13, 2012, at 11:00 a.m. to Sunday, April 15, at 5:00 p.m.

· Class: Intermediate (No SUVs)

· Extras: GPS navigation and car seat for a 2-year-old

· If asked, use the following identification:

· Name: John Smith

· Email:

· Phone: 303-555-1212

· Credit Card: Visa

The total price for the rental determines if participants were successful in picking the right class of car, for the right dates and the add-ons.

Study Metrics

Brand attitude and usage were collected before the study and perceptions of website quality were collected at the end of the study.

Task Metrics

Completion Rate: The participants provided the correct price of the rental car from a list of options. If participants selected the correct price, it was coded as a 1; incorrect answers were coded as a 0.

Task Time: How long participants spent on each website.

Task Difficulty: Participants rated how easy or difficult the experience was; 1 = Very Difficult to 7 = Very Easy.


Task-Based Metrics Results

Completion Rate: If users can’t complete a task successfully, then not much else matters. In both cases, Budget had higher completion rates. The completion rate on finding the price of a rental car was statistically higher with a completion rate over twice as high as Enterprise. (Refer to Figure 14-4.)

Task Time: The data shows that customers took statistically longer to complete both tasks on Enterprise than on Budget.

Task Difficulty: Users found it easier to find a location and to rent a car on Budget. The difference was also statistically significant.

These three task-based metrics were combined into a Single Usability Metrics (SUM), which averages together each attribute of usability, as shown in the following figure. It provides a single view that shows Budget had a more usable experience.


Click Maps

UserZoom provided a heat map showing where users clicked first. The heat map for Enterprise (refer to Figure 14-2) showed only a small fraction of users looked in the correct place — in the Rent a Car menu in the upper-left corner, suggesting problems with labels and navigation. (See Chapter 15 for more on findability.)

The heat map for the Budget website showed a substantial portion of participants correctly clicked the Location tab and most participants (84%) successfully found the nearest location.

Study-Based Metrics Results

SUPR-Q scores summarizing the participants’ perception of the quality of the experience are shown in the following figure. Scores were overwhelmingly more favorable for Budget than for Enterprise. This was also reflected in the Net Promoter Score (NPS). Not surprisingly, participants preferred Budget over Enterprise by more than a 3:1 ratio.


Examining the Verbatims

Here are a few comments users made during this study while on the Enterprise website.

· It didn’t give the whole total with the extras; I had to add that myself.

· Took the price of the car and manually added on 3-day price of car seat ($10.99*3) and same for GPS but still unsure of total because unsure if these items are taxed.

· I saw no area to add the extras.

· I saw no way to add GPS or child seat.

These open-ended comments were categorized into groups and charted in Figure 14-4. The two biggest issues, and areas for improvement, are fixing the total price of the rental and finding the GPS and car seat to add on to the rental. Correcting both problems will likely improve the metrics and the customers’ experience with the website.

For more details of this study, read the online article at