Designing for Behavior Change: Applying Psychology and Behavioral Economics (2013)

Part V. Refining the Product

Chapter 13. Identifying Obstacles to Behavior Change

You’ve built a solid product. You’ve established a baseline for the product’s impact. Now, you want to improve it. This chapter will help you discover problems, develop potential solutions, and generate additional ideas for how to make the product better.

I think of debugging a product’s behavioral impact as a five-step process:

1. Watch how your users are actually using your product, and see what’s stopping them from changing their behavior.

2. Check your data to find other problems in the product and to gauge the severity of the issues identified through observation.

3. Devise potential solutions by understanding the psychology of why the problem occurs and what’s feasible in your product.

4. Prioritize proposed changes to the product and integrate them into the product development process.

5. Test the behavioral impact of high-priority changes.

This chapter walks you through the first three steps; the next chapter covers prioritization and testing the impact of changes.

Watch Real People Using the Product

To understand your product and how it’s impacting people’s lives, there’s no substitute for actually watching people use it. They will use it in ways you won’t expect. They’ll uncover assumptions the team had while designing and building the product. Some of them will also get frustrated, curse a bit, and stop using the product. Reams have been written about the importance of directly observing your users with your product to improve its usability. It’s doubly true when trying to improve the behavioral impact of your product.

When you observe your users with the product, you’re looking for two main things:

§ Where they get stuck, distracted, or discouraged

§ New ways for them to take action that are more natural for them

The first one is straightforward. You’re looking for the places where things go wrong. Where do they give up using the system? Where do they get diverted into other areas or thoughts that aren’t related to the task at hand? Where do they get discouraged or frustrated, putting them in a bad mood that taints their interaction with the rest of the system?

The second one entails revisiting the knowledge gathered in Chapter 4 and Chapter 5: about user motivations, prior experience, relationship to the product and company, and barriers to action. Maybe you can come up with new ways to tailor the product to user needs. You can also generate new ideas for targeting the product based on what users try to do in the system (but the system doesn’t support) or what they say they want to do in interviews.

While I’m a big fan of direct observation, it’s insufficient on its own. Direct observation provides the team with vivid stories and anecdotes about their users and their challenges in the product. These stories can be easy to remember and rally behind, which is vital for coordinating the team’s efforts and getting everyone behind a common vision for improvement. However, the more vivid and the more consistent these stories are with our own experiences and beliefs, the easier they are to remember. Vividness and memorability aren’t the same thing as severity. The team needs quantitative data to assess the importance of various obstacles for users and to systematically discover other obstacles that are more subtle and wouldn’t show up under direct observation.

Check Your Data

Gather Additional Data About Your Users As Needed

Previously, we talked about the quantitative data you need in order to assess the overall impact of the product. To find obstacles along the way, we’ll need to add additional data collection throughout the product.

We’ve outlined the real-world outcome the product seeks to engender, the user of the product that does something to create that outcome (aka the actor), and the action the user undertakes. We’ve also broken the action down into a sequence of steps that leads up to the target action. Building on these concepts, here are the data points we’ll need to find obstacles to impact:

§ Who is using the product? Give each user a unique ID that is tied to all of the user’s data and behavior in the system, if at all possible.

§ Is the user taking one of the steps in the sequence? Link that event record to the user’s ID.

§ Is the user completing the target action? Link that information to the user ID. If the user completes the sequence, tag it.

§ What is the current state of the target outcome (covered in Chapter 12)?

§ If possible: What changes occurred after in the target outcome over time? Link that information to the user ID.

In order to make this explanation clearer, we’re talking about data collection that is used for debugging purposes (this chapter) separately from data collection for impact assessment (previous chapter). But to save yourself time, you should do the actual coding to instrument the product all at once!

In addition, when measuring who is using the product, the more you know, the more you’ll be able to improve the product. That doesn’t mean you should ask lots of unnecessary questions—users will just get annoyed and stop using the product. But if the product can glean useful information about the users while they use it—great; that will make targeting improvements to the product easier. One metric in particular that isn’t essential but will help with targeting is the user’s “persona.” Identify which of the personas identified in Chapter 5 the user best fits into, and store it.

Use the Behavioral Plan to Find Bottlenecks

In previous chapters, we planned out something like this:

Step 1 → Step 2 → Step 3 → Step 4 → Target Action → Target Outcome

The next step in improving the product’s impact is to focus on everything that happens before the target outcome. We want to determine if the sequence that leads up to the target action is actually working. By this point, you should have instrumented the product to measure user behavior at each step in the sequence. Treat the sequence of steps like a conversion funnel in marketing: look at the percentage of users who make it to each step, up to the target action. For example:

Step 1 (100%) → Step 2 (50%) → Step 3 (45%) → Step 4 (12%) → Target Action (11%)

Wherever there is steep drop-off in the number of users completing a step, the product clearly has a problem. In this case, there are serious problems finishing step 2 and step 4. That’s where you would devote additional analysis to understand what’s going on—with user testing, and with a deeper look at the quantitative data to see if there are signs of what’s going on.

If you’re dealing with a target action like “exercising,” you may notice a limitation of this approach. If someone exercises, it could be because of the product, or it could be because of other reasons. If you’re not running an experiment, then those “other reasons” could make it difficult to figure out where the product’s bottlenecks lie. To figure out what’s going on in these cases, it takes a more advanced technique, a causal map, which we’ll talk about shortly.

Check Whether the Action Is Actually Working to Drive Outcomes

Test the fundamental assumption that the target action actually drives the target outcome. In some products, the link between the two is trivial—for example, for a product that helps people keep a regular sleep schedule, the action is waking up on time (like Clocky, the rolling clock). Even then, it behooves us to check that assumption (perhaps users set the morning wake-up time too aggressively, are tired during the day, and have irregular, unhealthy sleeping times).

In the last chapter, we determined the benchmark impact of the product. If there’s no impact, you know there is a problem somewhere. But that only gets us part of the way. We want to narrow in on the relationship between the target action and target outcome.

How can you test this relationship? You can find a way, by hook or by crook, for people to take the action and do nothing else with the product. Pay them to do it. Look them in the eye and ask them to do it. Or, you can look at the overall data you have on people using the product and deploy statistical tools to look for the particular role of the action on the outcome: in an experiment, that’s a mediation analysis (like we used in the last chapter to build a data bridge between behavior within the product and events outside of the product). In a statistical world, structural equation models are one way to do it. Sorry, you need a stats person for that—and honestly, it’s not my expertise either.

Note: this is ideally something that you’ve already done some preliminary testing of, very early on in the process and before the product is actually built. I don’t mean to imply that testing the core assumption of the product should come this late. But the particular way that people take the action, after the product is built, may change the impact. So, double-check the relationship between action and outcome to avoid encouraging an action that doesn’t actually cause what you care about.

Segment the User Population

You may want to (and should) look beyond how the average (or median) individual is doing—and look at the product’s impact on specific subpopulations. The obstacles that users face may be confined to a specific subset of the users, and that knowledge can help the team identify the problem and resolve it.

Segment the user base into groups of people that are relevant for your team. These segments can overlap, and are driven by distinct needs:

§ Segments of users who are of interest to the product’s key constituencies. For example, the marketing team may want to segment users by buying power

§ Segments of users who the team expects will behave differently in the application and respond differently to application’s incentives (family status, age, and income bracket)

§ Segments of users who empirically break the population up according to how well they actually change their behavior (or not)

The first method of segmentation is designed for effective description of what is occurring within the application—using the categories that key stakeholders within the company expect. The latter two methods are used for effective analysis of how the application is successful (or not), and for guiding future changes to the application.

In each case, place individuals into groups that are unambiguous and unchanging—like initial age when enrolling, etc. For this purpose, we do not want segments that depend on user behavior—like “active users versus inactive users”—because it is very difficult to measure changes in engagement-related behavior of those groups.

Advanced Technique: Creating a Causal Map

Our behavioral plan is the ideal path that we want users to take:

Step 1 → Step 2 → Step 3 → Step 4 → Target Action → Target Outcome

However, in reality, often there are lots of other reasons why an individual might, or might not, be changing her behavior. Some of them will be in the application, and some not. To figure out what needs to change, you need to understand what’s happening, both on and off the ideal path.

By drawing out the “other reasons” that user behavior might change, we can get insight into what’s happening into the application. I like to illustrate them using a simple flowchart, then annotate it with the data we have at each step along the way. I refer to it as a causal map, for lack of a better term. I haven’t seen other examples of this approach in product design,^[152^] so I’ll use an example from my own work.

One challenge we faced at HelloWallet was to increase the amount of money that people had set aside for emergencies. The majority of users hadn’t set aside enough; in fact, roughly half of all Americans don’t have enough emergency savings, according to a survey by Bankrate.com ([ref166]). Only half of Americans could come up with $2,000 for emergencies with 30 days’ notice ([ref122]).

The application provided clear guidance on how much, and how, users should have saved. That formed the ideal path:

Enroll → load bank accounts → see advice in our “Guided Approach” sequence on saving → set up an automatic transfer of the money → have more emergency savings

But let’s say someone does put aside money for emergencies, and we don’t know why. Logically, is the application, and that particular path through the application, the only possible cause of that action? Certainly not. The person could have already had a plan to save for emergencies. The user could have received a windfall, like a tax refund, and decided to put some of it away for the future. Or, maybe something else in the application—like budgeting—helped free up the money for savings. One can generate a list of these various scenarios (and estimate how likely they are). But from the perspective of improving the product—many of them are tightly related to one another.

Based on this logic, you can create a map of what can happen within and outside of the system. In particular, look for the discrete actions or bits of data that are measurable that lead up to the person taking the action—on the ideal path or off of it.

In our case, we had a wealth of quantitative data. For example, we could measure whether someone actually went to the budget screen, freed up money to save, etc. We could also measure whether people were already saving money (based on their existing savings balance), and even if it looked like they had a windfall of income.

But even without quantitative data, you can trace out the major ways that a person would come to the same outcome (more saving for emergencies) and get a rough measurement based on user testing and user research, to determine if the unintended paths are something that people normally do.

Either way, the goal is to tell a story of what users do on the ideal path, and beyond. It’s like taking the behavioral plan from Chapter 6 and embedding it in a larger story of the other major things that could be going on in the user’s life, with respect to the target outcome. Figure 13-1 shows what the causal map looked like for this project.

Figure 13-1. A causal map showing the various reasons a person might put aside money for emergency savings (HW stands for HelloWallet; A.S. stands for Action Steps—a guidance module in our system at the time)

In your causal map, you should distinguish between:

§ The ideal path(s). Here, we had two different features that supported the same target outcome, each with its own ideal path. The ideal paths are in light blue.

§ Other actions that user might take within the system that impact the target outcome; that’s in green.

§ Actions that have nothing to do with the product and the company can’t claim to have caused but still cause the target outcome; that’s in red.

Each action within the system should be measurable. In the case of actions completely outside of the product, and not directly measurable, don’t bother mapping out all of the possible things that a user could do that affects the target outcome. Just examine the most immediate prior action that the user takes, right before doing something measurable. The target outcome is, by definition, measurable, so you might just have a single “external” box leading into the target outcome. But try to find earlier points in the map where the external action will have a footprint in the data that you can see and understand. Like, for example, if someone stops saving for other things (and “frees up” money for emergencies), we should know.

Using the Causal Map to Determine Better Ways to Target the Application

The causal map tells us two things. First, it can tell us where outside forces will hinder action. In the previous example, a negative income shock will significantly decrease emergency savings (that’s what emergency savings are there for, in fact). Sometimes the product can be tweaked to help people handle those obstacles; sometimes it can’t.

Second, the map can help reveal other ways to achieve the same outcome. If something other than the ideal path is successfully causing people to take the target action, that’s not a bad thing! Figure out ways that the product can capitalize on it. Maybe the product initially focused on restricting fatty foods to help someone lose weight but found that exercise was more decisive, and the product could encourage that instead.

Using the Causal Map to Help Assess the Impact of the Application and Marginal Changes in the Application

Remember in Chapter 12, where we discussed using statistical models and pre-post analyses to figure out the causal impact of the application? The causal map provides variables to consider in that analysis. If it looks like something outside of the application causes the target action, you’d better control for it when benchmarking the product (this is automatically taken care of with experiments, but is essential otherwise). The same holds true for testing the impact of changes in the product. If you aren’t running an experiment, make sure the factors identified in the causal map have been accounted for.

Using the Causal Map to Forecast the Impact of Future Changes

Finally, the causal map can be used to roughly estimate how important future changes to the product might be on user behavior. For example, with the causal map illustrated in Figure 13-1, we analyzed how much savings contributions would decrease if we took away the goal module, or if we increased the percent of people who loaded their accounts in the system.

Figure Out How to Fix the Obstacles

After you’ve determined where there are problems in the app, try to figure out why there’s a problem—what’s going on with those particular pages or interactions that makes them obstacles and what might resolve them.

In many cases, the problem and solution will be obvious. For example, in your quantitative data, you see a steep drop-off on a particular page, or during qualitative testing, users often complain about that page. When the team takes a fresh look at the page, they themselves can’t figure out what users are supposed to do. There’s a huge block of text with a link in the second to last sentence that they are supposed to click on. Solution: make the call to action clear, and get rid of some of the overwhelming text around it.

When the problem or solution isn’t so clear, there are a couple of paths to take. First, you can try to avoid the problem altogether. Before tackling a nasty behavioral problem head on, see if you can sidestep it. If there is a problem with a particular screen or interaction, is there anything wrong with dumping that content altogether? Is it absolutely necessary for the person to see/do that thing? If not, cut it. Return to the lessons of Chapter 6—simplify, automate, and default where possible to remove user work. You resolve the problem, and get a less complex product at the same time. For example, if users balk at entering their age into your diet tracking app, do you absolutely need to ask them? Can the application function perfectly well without it, or can it use a proxy?

If you can’t avoid the problem, you need to really understand it. Start by asking your users what’s going on. You can often include these questions at the end of a direct observation period you used to initially find the problems. You can also dig into the quantitative data and see if there are signals there. For example, let’s say you have a problem with drop-off over time in your time-management app (like a Nozbe, Asana, or OmniFocus). They use the app for a few days, and then never come back. When you interview users, they may tell you what’s going on (they started using a different program). Or, the data may show that they stopped coming back after hitting an error message in the app and getting frustrated.

If you find solutions with these straightforward, common-sense approaches—avoiding the problem, interviewing users, and reviewing the data—excellent. If not, you can use the Create Action Funnel from Chapter 2 to generate testable hypotheses.

Using the Create Action Funnel to Debug Problems

Let’s continue the example of a time management and organization app. You have a nasty problem, like app abandonment: many people use the app for a bit and then never come back.

Many things could be going on. Users might be turned off by what they saw in the app. They might be perfectly happy with the app, but after an initial period of testing, they determine it’s not relevant for them. They might have switched to a different app. They might love the app but have difficulty forming the habit of regularly managing their tasks after the initial excitement and exploration phase is over. And so on.

Not only are there many possible explanations, but users might say one thing (they were too busy to use the app) and the real reason was something different (the value proposition wasn’t compelling, and so they didn’t make time to use the app).

In these uncertain, confusing situations, you need something to help structure the process of searching for solutions and point you in the right direction. The Create Action Funnel from Chapter 2 is one way to do that. In order for people to take action—like logging back into your task-management application—they need the five prerequisites that make up the funnel.

1. Cue. The idea of logging into the app needs to cross their minds. Are you reminding your users to log in? Have you built a strong association between something they see or interact with in their daily lives and the thought of using your app? For example, when they are prepping for work on Sunday nights, that’s the time to check in and organize the tasks for the week.

2. Reaction. Do your users viscerally dislike something about the app? Did they have a frustrating experience that sticks with them? Do they not trust you with their data?

3. Evaluation. Is the app actually of value to users? Do they see it that way? Have they had first-hand experience of the value that’s relevant to them, or is it buried somewhere and haven’t found it yet?

4. Ability. Do the users know where to log in? Do they have their phones or access to a computer at the time when they want to manage their tasks? What if your app requires Internet access to manage tasks, but users want to use it in places where there is no reliable Internet access (e.g., on the subway)? (I had this problem with a task management app, by the way.)

5. Timing. Do users have a compelling reason to use the app at a particular time? How easy is it to procrastinate?

By going over these five prerequisites, the team can generate a set of hypotheses about what is going wrong. And, with those hypotheses, a way to test whether that potential problem is truly what’s happening. For example, “Hypothesis: users want to use the app, but don’t have reliable Internet access at the right time. Test: log failed access attempts from the mobile app and upload them for analysis to the company’s servers when an Internet connection is available.”

Often, we look for obvious problems: like the app doesn’t do want people want. The Create Action Funnel reminds us that there is much that could be going wrong. I like to generate at least one hypothesis for each of the five prerequisites to make sure that I’m keeping my investigation open-minded. Ideally, the tests for each hypothesis should be quick and simple.

If you’re looking for ideas for potential solutions in each of these areas, check Table 10-1 in Chapter 10. That table, and the sections that follow it, list two dozen techniques from the literature on how to encourage action, organized by where they affect the Create Action Funnel.

Testing: Small, Light, and Incremental

Generating hypotheses about what’s going wrong with the application and testing out potential solutions doesn’t have to be a big, formal affair. Before devoting lots of engineering and design time to changing the application, look for early signals about whether or not you’re on the right track:

Check the data

You’ve hypothesized that people have a frustrating experience, and that’s what’s causing them not to come back. Check the last thing that people do in the system. Do the people who don’t return have errors more often than those who do?

Generate a prototype

If you can quickly generate a graphic design or clickable prototype of the potential solution and try it out with users, great. For example, try field testing a simple email design that helps people plan a regular time to handle their tasks (i.e., use implementation intentions, as discussed inChapter 10 and Table 10-2—they increase the likelihood that people will act at the time they’ve committed to).

Investigate logical implications

Look for other things that would also be true if the hypothesis were true, but are easier to test. This is a common technique in the analysis of formal theoretical models and can be applied here, too. For example, let’s say you believe the problem is a lack of trust. It can be hard to measure directly and takes time to investigate. Instead, if some people didn’t trust the company, what else would they have done? They would have skipped optional data gathering questions, more than other users. They would have been more likely to click on the privacy policy and terms of use. Both are easy enough to find out.

You’re not necessarily looking for the final, definitive answer (if you can get one, excellent). Instead, you’re looking to quickly identify and remove answers that are clearly wrong. Then, you can devote further design and engineering resources to investigating and testing the possibilities that are left.

On a Napkin

Here’s what you’ll need to do

§ Study how your users are actually using the product, to see where they struggle, and especially what obstacles they face on the path to behavior change.

§ Examine the application’s data, and find out whether users are falling off for each step of the process that leads up to the target action.

§ Avoid obstacles by simply removing the problematic functionality, where possible. Otherwise, use the Create Action Funnel to diagnose what’s happening at a particular obstacle.

§ Use small, quick tests to evaluate potential solutions before devoting significant design and engineering resources.

How you’ll know there’s trouble

§ The team doesn’t see any obstacles hindering users from changing their behavior.

§ Qualitative and quantitative data about a potential obstacle strongly disagree.

Deliverables

§ A list of issues to address in the application, their severities, and sketches of potential solutions