Scenario-Focused Engineering (2014)
Part II: The Fast Feedback Cycle
Chapter 10. The importance of iteration
Iterating is the key to delivering great solutions that customers love. Our experience is that the biggest benefits don’t come from any one stage in the Fast Feedback Cycle, or any single “must-do” technique. Rather, the magic comes from the repetition and continual improvement enabled by the iterative cycle itself. The more iterations, the more opportunities for feedback, the more chances to adjust and fine-tune, the better the overall result will be. Fast, lightweight iterations are the key.
Up until now, we’ve dug in deep. We’ve taken a very close and detailed look at the Fast Feedback Cycle and the activities that you might do during each stage, focusing in particular on the first couple of trips around the cycle. Let’s pop up to 50,000 feet now and take a look at iteration from that perspective to see how the rhythm of iteration continues throughout a project and how the activities in each stage shift as you move from identifying the problem to finding a solution to fine-tuning and getting ready to ship.
Interesting patterns emerge when you look at iteration as a cyclical feedback system instead of as a linear sequence of events. Internalizing this cadence allows a team to gain the full benefit of the Fast Feedback Cycle—an iterative rhythm that can be extremely productive, allowing different team member’s strengths to shine at different stages as the team iterates.
What does (good) iteration look like?
We’ve used the word iteration throughout this book, but now it’s time to unpack the term a bit more thoroughly. When we say that you should iterate, what exactly do we mean? What does it look like to iterate well? Is it possible to do it poorly?
At the most basic level, an iteration is a single trip around the Fast Feedback Cycle, doing the work involved in each of the four stages: observe, frame, brainstorm, build. First, observe customers, then develop and check your problem framing, then brainstorm solution ideas, then build or prototype some of those alternatives, which makes you ready to show those ideas to customers in your next trip around. As you make successive iterations, you gradually turn rough ideas into detailed plans, with customer feedback guiding you along the way. As a rule of thumb, teams should strive to iterate at least three times around the Fast Feedback Cycle before committing to a spec, a specific design, or production code, and for more complex projects, you’ll potentially iterate much more. The idea is to continue iterating throughout the entire project, from concept to release.
But the science behind iteration is much more sophisticated and nuanced than the basic concept we’ve used to explain the parts of the Fast Feedback Cycle so far. In fact, you can go through the motions of each stage of the Fast Feedback Cycle yet gain only marginal benefits if you don’t attend to the overall shape, pace, and rhythm of iteration across your project’s lifetime. The next sections describe the characteristics of a healthy iterative process.
Iteration is a feedback system
The Fast Feedback Cycle is a loop. You work around the loop again and again, repeating the stages of the Fast Feedback Cycle over and over during the lifetime of a project. As you finish one iteration, the results of that iteration become the input for the next cycle. Each iteration does not stand on its own. Instead, each depends on the work that’s gone on before and is deeply informed by the learning that happens in previous iterations.
In this way, the Fast Feedback Cycle is a type of feedback system in which you collect both positive and negative feedback from customers on every iteration, and that feedback is used to shape and direct subsequent iterations. Similar to what happens in other feedback systems in biology or economics or engineering, positive feedback reinforces the signal—that is, it encourages you to continue on your current path. Negative feedback causes you to reconsider your approach and perhaps to back up and try some other alternatives. Iterating in a fast feedback loop that is alert to both positive and negative feedback helps you narrow in on a solution that represents the “truth,” or perhaps an “equilibrium,” or (you hope) an “optimal solution” to meet your customer’s needs reliably and efficiently.
However, for this feedback system to work, you have to actually listen to the feedback and be willing to make substantive shifts in your approach, depending on what you hear from customers. Teams that gather customer feedback but pick and choose which areas to listen to are not taking advantage of all the benefits of iteration. Similarly, teams that look at the Fast Feedback Cycle as simply a linear set of steps will never get the full benefit of iteration either. It is crucial to understand iteration as a continuous, cyclical system that requires lots of repetition and constant input from customer feedback to do its job.
This brings us to one of the most important characteristics of a healthy iterative process: keep iterations short so that you can fit in more iterations in a given amount of time. The greater the number of iterations, the more opportunities you have to react to feedback and fine-tune your plans. Our experience shows us that shorter, faster iterations are much more valuable than long ones. The magic is not in the individual activities and artifacts, but in the repetition and constant feedback. For most teams, you should be shooting for iterations of about two weeks, and for small teams and smaller-scale projects, possibly even shorter durations. Yes, that means that you need to be ready to show customers a few very rough prototypes of your first ideas a mere two weeks after you start a project. When we call it the Fast Feedback Cycle, we really do mean fast.
As we said in Chapter 9, “Observing customers: Getting feedback,” you are often going to be wrong. The idea is to find out if you are wrong as quickly as possible so that you can recover as quickly as possible, and this means getting your ideas in front of customers as quickly as you can.
Several teams we’ve worked with made the mistake of initially moving slowly through each stage of the Fast Feedback Cycle, with the goal of doing the work in each step with care and attention to detail. First, they made sure that they did a thorough job of collecting customer research and schooling all team members in that customer data. Then, they spent weeks drafting, polishing, and prioritizing their scenario narratives. With a few chosen scenarios in mind, they involved carefully culled groups of team members in brainstorming and visualizing ideas. Finally, they built some rapid prototypes to show to customers. You know what that very conscientious approach meant? They did one iteration of the Fast Feedback Cycle over the course of a couple months (not weeks). In retrospect, it’s no surprise that when these teams finally showed customers their first round of ideas, many of them did not resonate with the customers, and the teams had to scramble to adjust. That’s better than finding this out after you’ve already written code, but the teams that took this approach still wasted the better part of two months.
The well-intentioned desire to do each step “right” can overwhelm the competing need to keep iterations fast. A funny thing can happen, though. As soon as a team gets customer feedback and starts becoming aware of the bugs in its thinking, the team naturally starts moving a lot faster. Partially that’s because iterating is much easier once you have actual solution alternatives to think about. Also, by that point, the project-management clock is ticking loudly, and the urgency to begin coding becomes very strong. Unfortunately, this urgency can be so great that it tempts a team to move forward and start coding what it knows may be a suboptimal solution, simply to have built something—the details can be tweaked later. If a team’s approach to the customer problem is already on the right track, and the issues in the early builds are minor, this approach can work perfectly well. But often, if not most of the time, the problems discovered in early iterations are much more fundamental and tend to alter the team’s growing understanding of its customers’ actual (versus hypothesized) problems and needs. If you don’t clearly understand the problem you’re trying to solve, tweaking the details and aesthetics of your initial idea will never really satisfy the customer. No matter how much lipstick you put on a pig, it’s still a pig.
Avoid Big Design Up Front. Historically, computer scientists have argued that because bugs are much easier to fix on paper than in code, it’s best to thoroughly think through solutions ahead of time. If you can prevent a problem, that’s better than incurring the expense of coding up a bad solution, finding the bug, filing it, and fixing it. Even worse, for large problems, sometimes a fix isn’t really feasible without substantially rewriting the code.
Agile teams argue strenuously against this Big Design Up Front (BDUF) philosophy. Their primary rationale is that designing a system on paper wastes too much time. In practice, nearly every software project team ends up having to change its initial designs as the code is written. Teams discover new customer requirements or find integration problems between components that invalidate the original design.
We see value in both perspectives, but we fundamentally agree that you want to avoid BDUF and instead start building prototypes and testing them with customers as soon as possible. However, an important nuance we stress is that you don’t launch straight into writing code until you are on solid ground about what problem you are trying to solve. By clearly understanding the customer problem first, you avoid many of the most insidious problems that BDUF was trying to solve in the first place, and you also get the benefit of a more iterative, Agile approach.
A better approach is to iterate quickly from the very start. Get to the first prototype as soon as possible. The immediate customer feedback you get from even a couple of rough, half-baked ideas is almost always more valuable than spending another week trying to identify other unarticulated needs from more customer observation, or trying to write the perfect scenario, or brainstorming a few more ideas. In fact, getting feedback early is so important that some people evaluate a team’s performance level by measuring and tracking its “time to first prototype” and doing whatever is possible to reduce that time.
Let go of perfectionism. To iterate quickly, a team has to let go of a certain amount of its innate desire for perfectionism—a point we’ve stressed before. You have to be comfortable sharing ideas that aren’t fully thought through, showing prototypes that are rough and not picture-perfect, and moving forward with a few well-informed hunches about unarticulated customer needs without having talked to a statistically significant number of people. Relinquishing this desire is one of the most important cultural shifts that a team needs to make to get the highest efficiency from an iterative approach. You have to truly believe that your first ideas will likely be wrong and shift your mindset so that getting early feedback is more important than just about everything else. In fact, we encourage teams that have no ready source of generative customer data to make their best guess and start the iterative process by showing a prototype to a customer and then using that feedback to kick off the iterative cycle.
Time boxing is your best friend
To sustain a rapid pace of iteration, you need to find ways to beat the drum and keep people moving. Many teams find that time boxing is a good approach for managing the iterative process and keeping things moving along.
Time boxing is when you assign a fixed and limited amount of time in which to accomplish a specific goal.
Creating a more structured system to keep a team on a rapid cadence is especially important during the first year of the team’s transition to our approach, when team members are learning new behaviors and trying to unlearn old habits. One way to time box is to create a forcing function, such as a series of regular scenario reviews or checkpoints that force a specific length of time on an iteration. Our favorite approach is to schedule weekly or biweekly user-testing sessions and use each session as the point for finishing an iteration and moving forward. If the team is expected to show three alternative prototypes to a customer on Friday, you won’t make that deadline if you linger too long on any one stage of the iterative cycle. Time boxing and creating forcing functions to achieve small (and reviewable) outputs helps prevent analysis paralysis and keeps the focus on experimentation. Try something, see how it goes, and don’t be afraid to fail: know that the feedback will tell you where to go next. Frequent, regular, and predictable opportunities to put ideas in front of customers are extremely helpful in keeping a team iterating quickly. We’ll discuss these systems in more detail in the next chapter.
But it is possible to overdo it, to focus too much on the speed of iteration that you miss the point of getting feedback. We’ve seen some teams create a cadence in which they iterate so quickly that they don’t bother to get feedback from actual customers before they refine and finalize their ideas. Instead of using customer feedback, these teams use heuristic evaluations, cognitive walk-throughs, dogfooding, and other feedback techniques that use the engineering team, not customers, to provide feedback for fueling iterations. However, if you read the last chapter carefully, you know that these approaches are appropriate only for identifying the more obvious usability issues and don’t reveal much about usefulness or desirability. Don’t be in so much of a hurry to iterate that you forget to talk to customers. Without actual customer feedback to power your iterations, you’re violating the first rule of customer focus: “You are not the customer.”
Iterate throughout the entire project life cycle
It’s easy to imagine how continual iteration works in the context of an online service that a team is designing, building, and deploying to customers with small improvements on a daily or weekly basis. In those cases, it’s clear that you can use a couple of rapid iterations to develop an idea, build the code, and release the service as quickly as possible, often as an A/B test. If the customer feedback is positive, and server performance looks good, the update is made available to the whole customer population.
It’s also easy to see how iterations work at the beginning of a large yearlong project, when you use multiple turns of the Fast Feedback Cycle to identify the right scenarios, find good solutions, and fine-tune those solutions. But once the plan is more or less clear, the specs are locked, and coding in earnest is underway, do you really need to keep iterating?
In our experience, many teams want to stop iterating after they start coding, especially in larger, longer-term engineering projects. They believe that they have everything more or less figured out and that the main focus now is on execution. To some extent this is true—the team has to focus on the more technical functions of writing, debugging, and testing code to ensure the solution will work as planned. But that shouldn’t mean the end of user testing and getting customer feedback. As the different parts of your solution come together, it’s important to remain mindful of the customer’s entire end-to-end experiences and track them using the metrics you defined in your framing. In Chapter 8, “Building prototypes and coding,” we mentioned that some experiences can be tested only with running code. For example, experiences that rely heavily on animations and complex visual transitions, that work with variable kinds of user data (such as managing a personal photo library), or that involve responding to daily email in an active inbox are all situations that probably require that you test with production code.
Also, when several subteams work on different scenarios within a larger project, it’s common to encounter unanticipated confusion that results from bringing together the various scenarios and experiences into the same UI. In your production code you may also discover unexpected latency between client and server components or other complexities that arise in execution. These can throw curve balls into the experience and are discovered only as scenarios begin to function in the codebase. When you bring components together, keep a close eye on how the end-to-end experience is coming together across all the pieces.
Finally, remember tunnel vision? That concept applies here as well. After immersing yourself in a project for any length of time, you (and the team) tend to see the solution through your own eyes, not as customers do—you become blind to the customer-experience issues lurking in your solution. To overcome this blindness you need to continue testing with customers to ferret out these issues as the code is being written and throughout the entire product cycle.
Iterating through the project’s life cycle seems like a straightforward, common-sense idea, but as you will see in Chapter 11, “The way you work,” adopting this habit is possible only when everyone on the team adapts their behavior.
Iterate in a funnel shape
Think back to the example of the ergonomic Microsoft mouse in Chapter 3. In that team’s first iteration, team members used materials such as clay and fiberglass to mock up dozens of rough shapes for mice. In the second iteration, the number was winnowed down to nine ideas that the team worked out in enough detail that the concepts could be drawn in a CAD program and printed on a 3-D printer. In the third iteration, the number of alternatives was winnowed some more, and the team turned these into functional prototypes. Finally, in the fourth and subsequent iterations, the team came down to one final design that it tweaked and fine-tuned and ultimately shipped. This sequence is shown in Figure 10-1.
FIGURE 10-1 The funnel shape of iteration.
To winnow is to reduce the number of items under consideration from a greater number to a smaller number but still more than one.
The mouse example is a great illustration of the “funnel” of iteration, and specifically it shows the shape that you want an iteration funnel to have—an overall cone shape.
A good iteration funnel starts with a lot of rough ideas, and over time the solutions become more detailed, more refined, and less numerous. Eventually you narrow options down to one solution, which you then continue to fine-tune through further iterations until it meets all your metrics. Then you declare it done and release it to customers.
The funnel shape allows the best ideas from multiple prototypes to be combined and recombined, giving you the best chances at finding a globally optimal solution, as we discussed in Chapter 7, “Brainstorming alternatives.” You need multiple ideas to hang around for a while during the early iterations of your project to enable this. Showing multiple ideas to customers enables them to give you the feedback you need to identify the best aspects from each of the ideas you’ve prototyped. You can then combine and recombine those winning ideas into an even better solution. If you decide on a single approach too early, you cut off your chances of finding that magic combination that not only meets your customer experience goals but balances your business and technical goals as well. You might get lucky with an early decision and pick the optimal direction to follow, but statistically that’s unlikely.
Maintaining the shape of the funnel is vital for gaining the most from the iterative process and consistently getting good results. You want a funnel like funnel A, at the left of Figure 10-2—a funnel that has smooth, curved sides, that you use to gradually winnow down the number of ideas, keeping multiple ideas in play long enough to find the right combination of ideas to form an optimal solution. Good iteration follows an asymptotic curve, but practically speaking (just as with an asymptote), you will never reach the truly optimal solution—it’s too time-consuming, and you will get to a point of diminishing returns. At some point you will declare your solution good enough and ready to ship because it solves the scenario and meets the metrics you set out to address.
FIGURE 10-2 Different iteration funnel shapes. A is an ideal shape for iteration. B, C, and D represent flawed approaches to iteration.
In contrast, look at funnel B, which starts out with room for lots of ideas—probably from a robust initial set of customer research or a brainstorming session—but from here the team took a sharp turn and winnowed down too quickly to a single solution. We call this the “martini glass” shape. It’s a suboptimal approach because it doesn’t allow for much cross-pollination of ideas during the iterative process, and therefore you risk landing on a local maximum rather than a more globally optimal solution.
Funnel C is another common variant, where teams iterate a large variety of ideas for an extended period of time and do not winnow as they go. Sometimes this is referred to as “letting a thousand flowers bloom” or “throwing spaghetti against the wall and seeing what sticks.” Often, this shape applies to cases where many branches of a larger team are working on different aspects of a problem but not talking with one another about how to integrate their ideas. The clock is ticking, however, and eventually the pressure to make a decision heats up. Usually, with some pain, the team moves into decision mode, choosing a single solution approach to pursue. Funnel C can lead to good results if a team integrates and mixes ideas along the way, but it results in a lot of waste and is slow, which makes it an expensive and inefficient approach.
Funnel D is the absolutely worst approach, where you consider maybe only one idea or a small set of cousin ideas, narrow in on a single approach right away, and subsequently “iterate” only that one solution concept. Unfortunately, this approach is typically what many engineers call “iterating,” where you pick one reasonably good idea and then get feedback from the team and from customers to make that idea progressively better and better. It’s a great approach for finding a local maximum, but if you are looking for a globally optimal solution, it’s not a reliable technique for attaining consistently optimal results.
How many alternatives are enough?
Up to this point, we’ve talked a lot about the best funnel shape—one that requires you to explore many alternatives at the top of the funnel. But there’s a big question—how many alternatives are we talking about?
The answer depends on how many iterative cycles you have done already, as well as on the depth and scope of the problem you are trying to solve. A good rule of thumb is that in your first trip around the Fast Feedback Cycle, for an average scenario, you should generate a large number of initial solution ideas, at least 50 or 100 or more. These ideas can take many forms and may have been generated in a few different forums—for example, a long list of brainstormed ideas plus a few dozen sketches, diagrams, or rough storyboards. As you continue around the cycle, you should narrow substantially at every iteration.
Require alternatives before making a decision. Some teams take the idea of exploring alternatives very seriously and apply it with rigor at every decision point in their ongoing work, whether they’re considering how the user experience will look, what technology stack to use, or what to do for their next morale event. A team might make a rule not to make a decision of any sort until at least three or five or maybe even as many as nine different, viable alternatives are considered. This is one way to be sure that team members do the due diligence to really think through a problem before making a decision, especially a decision that can be painful to unwind later. The reasoning is that if you cannot generate several alternatives, chances are your understanding of the problem is one-sided or superficial, so this is a way to combat that.
Unpacking the Fast Feedback Cycle
In fact, we can peel back another layer of the onion. While it’s true that some properties emerge from a sequence of iterations that are important to manage, there are also fascinating and elegant patterns within the Fast Feedback Cycle itself. The rhythms of these patterns are vital to embed in the day-to-day functioning of a team.
Understand versus create
As we mentioned in Chapter 3, “Take an experimental approach,” the Fast Feedback Cycle can be split in half, top and bottom. The top of the Fast Feedback Cycle is about understanding—understanding customers, understanding their unarticulated needs, capturing those needs with good framing, and knowing how to measure and track against that framing to stay on course. In this half of the Fast Feedback Cycle, you aren’t actually building anything. In a way, the top half of the cycle is about auditing or governance, continually checking back to be sure that customers still like what you’re doing, that you’re still solving the problems you set out to solve, that you’re meeting the success metrics you laid out—or adjusting your framing based on what you learn.
The bottom half of the Fast Feedback Cycle is all about creating solutions—brainstorming specific solution ideas, visualizing possible solutions with sketches or storyboards, creating prototypes or writing code to make those solutions come to life. The bottom half is about generating specific solutions. Here, you learn by making—you don’t just think about the problem, but you get your hands dirty and build something.
As you travel around the Fast Feedback Cycle, you continually move between the top half and bottom half of the cycle. In the top half, you check in with your customers and verify your framing. Then you shift to the bottom half to do more work on your solutions. Then you shift again and check back with customers to see whether you’re on the right track. Identify the problem, work on the solution, gather feedback, then adjust your understanding of the problem, and repeat. First understand, then create, then check your understanding, then create some more. Problem, solution, problem, solution. Understand, create, understand, create.
External versus internal
You can also split the Fast Feedback Cycle vertically, down the middle, and look at the cycle that way. The left side of the Fast Feedback Cycle is about going outside the team to engage with the external world—building something expressly for customers to use and getting their feedback on it. From a customer’s perspective, all he or she usually sees of your work is the stuff you create when you’re working in the stages on the left side of the cycle—the prototypes and products and services that you build for customers to use, plus the usability tests and conversations and surveys that ask them to tell you what they think of your solutions.
The rest of your work happens inside your offices, internal to the team and invisible to anyone outside your company, during the stages on the right side of the Fast Feedback Cycle. This side is about the team’s internal work—writing down your best understanding of your customers’ needs, tracking metrics, coming up with your best ideas for how you might satisfy those needs, relying on your current understanding of those needs as well as technical feasibility and business constraints. Those are all internal team concerns that you don’t share with customers directly, but they are important to consider and are tools that help guide and structure your work.
You cycle back and forth between building something expressly for customers and seeing how they like it and taking what you learn inside the team and working through it as best you can. Then you’re ready for another shot at showing your latest thinking to the outside world again. You move between doing work internally and building things that you show to customers externally. After that, it’s back to your private, internal team processes, and then back to external customers again. External, internal, external, internal. Outside, inside, outside, inside. Customer, team, customer, team.
Diverge versus converge
The last pattern within the Fast Feedback Cycle is the most interesting—and also the most powerful. Take a look at the diagonals in the Fast Feedback Cycle; they also have something in common.
One diagonal focuses on diverging activities; that is, activities that widen the funnel and create more alternatives to consider. Both observing customers and brainstorming alternatives are fundamentally about creating more options. When you observe customers, you uncover more and more customer needs and possible problems that you could solve. When you are showing customers solution ideas to get their feedback, their responses will cause you to see various places where you need to change your solutions or may even inspire you to notice yet more opportunities for other problems you could tackle in the future. Similarly, when brainstorming solution ideas, you are by definition creating more alternatives, widening your funnel of options to consider.
When you diverge, you create more options and alternatives to consider, broadening your scope.
The other diagonal focuses on converging activities; these activities narrow the funnel and winnow the field to fewer choices. When you frame, you make decisions about which specific problems you’re going to go after and choose the success metrics that are going to best indicate that you’ve solved that problem well. Similarly, in the build stage, whether you are building code or noncode prototypes, you don’t work on every single idea that you came up with. Instead, you pick the handful of the most promising ideas to build and make some judgments about which ones are likely going to work the best. This diagonal is all about making decisions.
When you converge, you winnow from more options to fewer, making decisions about which problems and solutions are the most promising to move forward with.
As you travel around the Fast Feedback Cycle, at every stage in the cycle, you switch from diverging to converging and back to diverging again. Notice that the pace of the diverge-converge rhythm is twice as fast as either the understand-create rhythm or the external-internal rhythm. Practically speaking, diverge-converge is the dominant rhythm that team members notice as they do their day-to-day work, and as such, it is the most important one to integrate into the team’s language, culture, and practices.
Having a team internalize the diverge-converge rhythm is a mark of a team at the top of its game—a team that can smoothly diverge to consider many problems, converge on a few of those problems to focus on them, diverge again to consider possible solution alternatives, converge to pick a few alternatives to get in front of customers for feedback, and then use that feedback to diverge again. Diverge, converge, diverge, converge. Alternatives, winnow, alternatives, winnow.
When you take a closer look at the diverge-converge rhythm, you’ll notice that the iteration funnel isn’t strictly a smooth cone shape that gradually funnels many ideas to fewer. The constant cycle of diverging and converging within the funnel creates jagged edges—“teeth”—in the sides. A more precise drawing of the iteration funnel might look like Figure 10-3. We call this shape the sawtooth. It arises when you repeatedly diverge and converge and at the same time continually narrow your focus from one iteration to the next.
FIGURE 10-3 The sawtooth shape created by diverging and converging within the iteration funnel.
In practice, a team that is diverging looks, acts, and feels very different from how it behaves when the team is converging. If the team is not in sync about what stage it’s in, you can find that some team members are trying to converge while others are trying to diverge. This can lead to a lot of confusion and frustration, as neither job is accomplished.
Aligning the vocabulary across the team is critical here. As the team moves through an iteration, team members need to be verbal and overt with each other about which stage the team is in and what frame of mind that stage brings to the tasks at hand. The team needs to be able to remind itself: “This meeting is about diverging” or “It’s time to stop considering alternatives; time to converge.” People need to instantly understand what that means and be able to align their approach appropriately. Understanding the science behind iteration, and internalizing the rhythms that govern it, helps everyone become comfortable with the idea that there is a time for diverging and a time for converging, but just not at the same time.
In meetings, be clear about whether the primary activity of that meeting is diverging or converging. Meetings designed for diverging might be about brainstorming, generating alternatives, hashing through ideas, investigating possibilities, discussing options. Meetings related to converging might be about making decisions, prioritizing, winnowing, or creating a plan of record. Whenever possible, schedule separate times for activities related to diverging and converging, even if you separate them by only an hour or so. People need time to switch context.
All the activities we’ve discussed in this book fit into the Fast Feedback Cycle as either a divergent or a convergent activity. Figure 10-4 shows another view of some early activities and the artifacts you would likely produce at the convergence points, where you turn the corner from diverging to converging.
FIGURE 10-4 The sawtooth shape, overlaid with the stages of the Fast Feedback Cycle and typical deliverables at each stage.
Figures 10-3 and 10-4 show only a few teeth, representing just one and a half iterations of the Fast Feedback Cycle. But in reality, you’ll iterate many times, and a sawtooth diagram of your entire process will have many more teeth. When you take a bird’s-eye view and consider the sawtooth diagram for an entire project, you’ll notice that the funnel shape becomes clearer. The sawtooth pattern happens throughout the project, but the teeth become smaller and smaller as you progress. When you consider the sawtooth pattern across the whole funnel of a project, it looks something like Figure 10-5.
FIGURE 10-5 The whole iterative cycle follows a sawtooth shape, inscribed inside a smooth, curved funnel.
Turning the corner
We consistently hear from teams that the hardest part of this rhythm is the “turning points” in the sawtooth, when you shift between diverging and converging and back again. Yet, these turning points are essential to making progress—if teams have trouble turning the corner, then they may get stuck in a stage for too long and their iterations will become slow and inefficient.
Some people and teams find that turning the corner from diverging to converging is easier, especially teams that have a natural proclivity toward making decisions, picking a plan, and moving forward. Others find that type of turn difficult, feeling that if they had just a few more days they’d find a better insight or a more revolutionary solution idea. These teams might resist having to make a decision based on what feels like an incomplete set of choices.
On the other hand, some teams and individuals find turning the corner from converging to diverging easier. Typically, these are more creative sorts, who enjoy thinking through various alternatives or hearing the nuances of customer feedback; they look forward to diverging activities. But at the same time, others on the team might encounter difficulty going from what seemed like a solid plan to considering alternatives again, albeit on a smaller scale. This can feel like a step backward without understanding the larger wisdom of the sawtooth funnel.
These preferences highlight the reality that you have diverse individuals on your team, each with different strengths and comfort levels in different parts of the iterative cycle. You can expect that different people on your team will come to the forefront to lead different stages of the iterative process, and you should encourage this to happen. However, this should look less like handing off a baton in a relay race and more like a change in the leader in a V of geese as the flock flies south for the winter. Regardless of who takes the lead at any given moment, the project team needs to remain engaged throughout.
Consider organizing some activities with your team to better understand where different individuals are best suited to contribute in the iterative cycle. One option is the Basadur Creative Problem Solving Profile (www.basadurprofile.com), which is a self-assessment tool that can help identify these different styles and preferences. You may be surprised at how diverse your team’s preferences are, even between people with the same job title. In Chapter 12, “Lessons learned,” you’ll find a sidebar that discusses the Basadur CPSP technique in more detail.
While every team will have a mix of preferences and strengths, a team typically also has a general leaning toward diverging or converging. Most engineering teams lean toward converging, and certainly most managers have a bias in that direction, because converging is fundamentally about deciding a plan for the future and shipping a product. But while shipping is important, so is shipping the right thing, and without a proper balance of diverging and converging, you risk shipping the wrong product. For most engineering teams, particular emphasis needs to be placed on not shortchanging diverging, at the right points in the cycle, to give yourself the best odds at discovering a winning combination. Many designers tend to have a bias toward diverging, so adding staff with that skill and natural preference is one way to help balance a team, as long as that staff is treated as a full-fledged member of the project team and not as a “consultant” that can be easily ignored.
Managers need to change their behaviors, too. Managers of teams have a role to play in making sure that the full iterative process is supported and reinforced. We see managers new to the Fast Feedback Cycle become nervous when their team picks not just one but a few ideas to move forward with, worrying that their team is indecisive or stuck in analysis paralysis. Managers need to realize that, especially in the first few iterations of the Fast Feedback Cycle, working on multiple ideas in parallel is expected and winnowing represents solid progress, even if the final solution hasn’t been chosen yet.
Managers need to get used to the idea of a gradual funneling process, which does not yield an exact plan right away but, in the long view, is a more efficient and reliable approach to finding a solution that will genuinely meet customers’ needs with a minimum of risk. Some time is needed for both teams and leaders to trust an iterative funneling process, so don’t expect this mindshift to happen right away. Having a few projects successfully use this approach helps build confidence and develop a sense for what the funneling progression looks and feels like in practice. Until then, teams and leaders need to keep reminding themselves of the science of iteration and know that it’s important to live with ambiguity a bit longer than they may be used to.
Turning the corner requires a good degree of coordination across the team, as well as mental agility to be able to change your frame of mind. Trouble turning the corner comes in a few forms. Some teams don’t turn soon enough. A feeling of “just a little bit more” keeps the team from switching gears. For other teams, not everyone agrees to turn at the same time, so you end up with part of the team moving on to the next stage and some laggards having trouble letting go. The cure for both problems is the same: get the religion that your first ideas are likely to be wrong, but you just don’t know which parts. The only way to find out what you need to do to improve is to get customer feedback as soon as possible. Teams that truly internalize this frame of mind will have a much easier time turning the corner quickly, because they view the whole iterative process not as a series of irrevocable decisions but as a series of experiments, where any decision that doesn’t pan out can always be reconsidered.
SFE in action: Turning the corner doesn’t happen by itself
Ian Todd, Principal Program Manager, Windows Phone
Our team had a history of success on Windows Phone 7 and 7.5 and was in the zone. We had forty people—program managers mixed with designers—collaborating on scenario-oriented specifications for the communications features in our next release. As usual, we had a strong set of scenarios and a smorgasbord of design concepts—we had enough material to hunker down and write those specifications. The engineers were finishing up a quality milestone and were going to start coding features in a month. We didn’t have any specs yet, but no worries, right? Wrong. That last month came and went, and somehow the team never actually made the detailed design and functional decisions necessary to finish specs and be ready for coding. Deadlines came and went with very little to show for them.
What happened? The team had experience doing this kind of work and hitting deadlines. Yet we were also trying out some new things. We’d read the latest books on the benefits of being design-led and had taken the Scenario-Focused Engineering course. We had established a series of planning milestones to focus first on insights and scenarios, then on concepting, then on final designs, and finally on specs. The problem was that even after hitting the final-designs milestone, the team wanted to keep iterating, to get the design just right. At the same time, not enough pressure was being applied to get the team to turn the corner and get started building the product. That combination resulted in a team that wasn’t motivated to converge. There was no tension in the team culture that made the imperfect prospect of convergence seem like the right place to go. Why not delight your customer just a little bit more with extra design time?
The team recovered over the next few weeks and finished a set of functional specifications. Sure, we probably got to a better set of designs than if we hadn’t spent that extra month, but the incremental increase in the quality of the design was outweighed by a larger incremental reduction in how much time we had to iterate in actual code, with real customers. That’s why regular convergence is so important to strive toward, even when it seems like less fun than spending more time designing.
These turning points are great places to pause and take stock of the work you’ve done. After a diverging activity, stop a minute to take pride in the amount of data or number of insights you’ve collected or in the sheer number of ideas the team generated. After converging activities, you’ll naturally have work that represents the team’s hard-earned decisions and reflects the team’s intentions and focus moving forward. Artifacts from these converging points can and should be saved, discussed, communicated, and celebrated. In addition, focusing on the artifacts you create at converging points can help give managers what they need to know that the team is making progress so that they can confidently support the project over the long haul, especially during the first set of iterations, before the team has narrowed in on a single solution approach.
Phases of iteration
In Chapter 8, we explained how the prototypes you create—and the questions you are trying to answer with your prototypes—are different depending on how far along in development a project is. You saw in Chapter 9 the many different kinds of customer feedback you can gather, from feedback on whether you’ve identified an important customer need, to testing whether the solution works well, to fine-tuning the details. In this section, we follow those thoughts a bit more and explore how the activities in the stages of the Fast Feedback Cycle change as the product matures from concept to rough implementation to detailed production code.
When you put it all together, three distinct phases of iteration emerge across the life cycle of a project. A project starts in the needs phase, which is primarily about identifying customer needs. It continues into the solution-tradeoffs phase to narrow in on a specific solution approach, and finally settles into the details phase to fine-tune that solution. This phase finishes when the solution ships. Production coding typically starts midway into the solution-tradeoffs phase. Figure 10-6 shows how the phases overlay on the sawtooth diagram.
FIGURE 10-6 The phases of a project, and how they relate to the sawtooth funnel.
The activities you do at each stage of the Fast Feedback Cycle can look quite different in each of these phases. For instance, during the needs phase, the Frame stage is mostly about writing scenarios and metrics, but by the time you get to the details phase, the Frame stage has become mostly about assessing how well your solutions are doing, with your scenarios and metrics as a reference point.
The activities you undertake in each stage might also look different depending on whether your project finishes in a matter of days or weeks, versus a much bigger project that continues for months or even years. For example, a short, small-scale project might use only discount usability testing in the Observe stage, whereas a larger-scale project might use a mix of surveys, various levels of usability testing, and site visits to get feedback from customers.
Furthermore, whether you are working on a large project or a small one, the overall goal is still to keep iterations fast, so you will probably only have time to use one or a very small number of techniques during each stage within a single iteration of the Fast Feedback Cycle. Given all these complexities, a natural question is, “Which techniques are the best ones to use at different phases of my project?”
As you’ve seen in the previous chapters, you can consider using many different techniques at each stage of the Fast Feedback Cycle. Even after you focus in on just the techniques that make sense for your specific problem domain (hardware, software, API, user experience, etc.), it can be tricky at first to see which techniques are optimal for which situations and at which stages in the project. Thinking about the main phases of a project and how iterations proceed within those phases can help you select which techniques are most appropriate for each iteration and will also give you a more practical feeling for what a series of iterations might look like over the duration of a full project, especially for a larger-scale project.
The following sections provide more details about each phase, including which techniques are most commonly used in those phases, how long that phase should last, and what each phase is trying to accomplish. These guidelines refer to medium- and large-scale projects. Some special characteristics of small-scale projects are offered in a separate section; significant parts of the iterative cycle and various phases can collapse and overlap for small-scope projects.
What are the most important real-world needs of my customer, and do I understand them fully?
The needs phase entails the first few iterations of the Fast Feedback Cycle, where you focus on identifying customer needs, seeking to deeply understand your customers’ perspective, constraints, desires, and frustrations. The needs phase starts at the top of the iteration funnel, its widest part. You may be simultaneously considering a large number of diverse customer problems or a large number of different aspects of a problem, and possibly even narrowing in on exactly which customer to target. The goal of the needs phase is to figure out which of those areas to focus on, which are the most important aspects to solve from your customers’ point of view, and what factors are going to be important in a truly delightful solution. In this phase, you are more concerned with identifying a problem whose solution would be useful and desirable to a specific target customer; at this point, it’s too early to think about usability or craftsmanship.
During the needs phase you iterate through all four stages of the Fast Feedback Cycle, including showing your proposed solution ideas, but for now you’re most interested in getting confirmation that you’re solving the right problems. In this phase you are not yet concerned with figuring out which solution is the best one. Instead, you use the tactic of showing the customer a wide variety of possible solution ideas to help spur feedback and conversation, with the goal that this will clarify and ferret out the true customer needs. Until you are sure that you’ve identified your customers’ most important problems, specific solutions don’t matter per se. The needs phase is akin to the approach in Lean Startup of testing an MVP to validate your basic value proposition and target customer.
Typically the needs phase involves one to three iterations. It occurs at the widest part of the funnel, where you consider the largest variety of alternatives. You exit the needs phase when you are confident that you have identified your customers’ deep needs, know exactly which problems are most valuable to solve, know how good the solution needs to be to delight customers, and have captured that knowledge in a solid scenario (or other framing device) that can be shared with the entire team. When you finish this phase, the funnel is still fairly broad, as there are many possible solution paths that could be followed, but it is significantly more focused than when you started.
Here are some details about each stage in the Fast Feedback Cycle during the needs phase of a project and which specific techniques make the most sense during this phase:
Observe stage During the needs phase, start with the generative research approaches outlined in Chapter 5, “Observing customers: Building empathy.” Site visits are a particularly good technique for the very first iteration. You can hear pain points and learn about needs directly from customers in the context of their real-world situation. After you have your first round of rough ideas to show, switch to the lighter-weight evaluative approaches found in Chapter 9, such as scenario interviews, concept tests, or informal usability tests using paper prototypes. In the needs phase, you always show multiple alternatives to customers because comparisons lead to understanding customer preferences more clearly. It’s absolutely essential during the needs phase to test with target customers and not proxies.
Frame stage The focus during the needs phase is on writing and editing scenarios (or another framing mechanism) to capture what you’ve learned about your customers. As you write scenarios, you should also begin identifying which success metrics are the most important indicators of deep delight. In your first iterative cycle, you will draft scenarios, and in subsequent iterations you will revise, fine-tune, and finalize your scenarios based on what you learn from your customers, making them more accurately reflect your customers’ real-world situation and your latest insights about their motivations and desires.
Brainstorm stage During the needs phase, you brainstorm many ideas. You look for as much diversity in solution alternatives as possible, explore many diverse directions, and cast a wide net. This breadth of alternatives helps stimulate customers to mention ideas, needs, desires, details, or constraints that have not yet surfaced. Focus on open-ended brainstorming techniques with groups of people to generate many diverse ideas quickly, and be sure to encourage the unusual, wacky, and seemingly impractical ideas. These can lead to breakthroughs in solution concepts and also suggest potential customer needs you might not have noticed in direct research. At this phase, you’re not interested in thoroughness in your brainstorming, you are merely exploring the edges of the space, so the more rigorous techniques such as six thinking hats or SCAMPER are not worth the effort yet. Visualize ideas with sketches, but don’t bother to refine them. Rough storyboards are great to quickly sketch out the flow of how different solution alternatives might play out. The focus on sketching and storyboarding at this stage is not about refining ideas into something practical or optimal, but about getting a rough idea on paper so that it can be shared with others for directional feedback.
Build stage In these first iterations, you should stick to extremely low-fidelity, rapid prototypes, such as rough paper prototypes. Take the storyboards and sketches you generated while brainstorming and pick a half dozen to clean up just enough that you can show them to customers in an initial round of evaluative customer feedback. Your prototypes don’t need to work in any form yet, or even fully make sense, they just need to get the idea across well enough that you can get directional feedback from customers or engage them in a participatory design exercise to have them extend the prototype with their own ideas. Ask them whether it looks interesting and how they would change it to be even better.
In the needs phase, you do something at every stage of the Fast Feedback Cycle, but you spend proportionally more time in the Observe and Frame stages. This emphasis shifts in future phases of iteration.
SFE in action: We forgot to check for usefulness
Tracey Trewin, Partner Program Manager, Microsoft Developer Division
We were on our way to shipping Progression/CodeMap a third time, and yet we were still missing something. The product demoed well. It had even tested well in usability studies for certain scenarios. Everyone agreed that visualizing relationships between code was a compelling concept. It seemed like it had so much potential, but somehow it just wasn’t hitting a chord with our developer audience.
We decided to do another user study. Truthfully, our usability researcher, Paula Bach, and I were preparing the team for a pivot. We were going to conduct a rapid iteration study, where we have the developers on hand make adjustments in real time. After all, there is no point in continuing to test the same prototype over and over when your users are clearly struggling.
We had our first users come in, and we started to walk them through the script. It was a disaster. They struggled and didn’t make it through the tasks without a lot of help.
That is when we regrouped. Paula and I looked at each other and said, “Dang, we are busy testing whether this is usable, but we are not even sure that this is useful.” I know it sounds obvious, but in hindsight, it is a very common mistake. This realization caused us to do a reset of the study and focus on validating the key scenario. We needed to determine whether the problem we were trying to solve was interesting and important to our users in the first place.
Through the study, we determined that our users did find CodeMap useful in understanding and navigating code, but the surprise was that users gained the most value from CodeMap when they were debugging and wanted to jump around in a debugging stack. You can certainly do this in the debugger window, but in general people found it helpful to have a visualization of the code and its relationships.
During this study we had an aha moment where we realized that if we built a CodeMap automatically as users debugged their code, rather than requiring them to manually add elements to the map, that users intuitively understood what they were looking at and were immediately able to navigate the map. Truthfully, this was a relatively small change, but it fundamentally changed how we were thinking about this functionality. Showing the map of the code during debugging dramatically increased the usefulness of the feature.
Looking back, we made the classic mistake of assuming the feature we imagined was useful, jumped ahead to build solutions, and then tested our solutions for usability before we had verified the problem was the right one to solve or fully understood the customer need. So, in short, the moral of the story is always check usefulness first.
Which solution provides the most optimal tradeoff between great customer experience, technical feasibility, and enabling the business plan?
Once you confirm that you’ve chosen the right set of problems, you focus on identifying the best end-to-end solutions to those problems, including making decisions about key tradeoffs. You will get feedback on various solution approaches from customers and then narrow in on a single, specific solution that strikes the right balance between great customer experience, technical feasibility, and enabling your business plan. This phase is characterized by making tradeoffs between what would be ideal and what is practical and balancing multiple, sometimes conflicting, goals.
This phase represents the middle section of the funnel. You start with a reasonably clear description of customer needs, but you still have a broad set of possible solution alternatives. Over multiple iterations, you gradually narrow this set down to a single proposed solution. To accomplish this, you blend multiple solution ideas together, informed by regular doses of customer feedback and ongoing feasibility investigation that provides course corrections and directional guidance.
You should start considering technical feasibility during this phase, providing feedback about the solutions being considered and whether they are reasonable to build in an appropriate time frame with the resources you have available. You will be doing technical design work in this stage, working in parallel to define the underlying architecture as the end-to-end solution is iterated. As you progress through the solution-tradeoffs phase, you make the transition from building rapid, noncode prototypes, to writing prototype or investigatory code, to writing production code. Some projects make the transition to coding faster than others, especially if creating multiple alternatives to test in code is easier for the architecture or platform you chose. However, regardless of whether you are building paper prototypes, functional code, or some of each, you will continue to test your latest ideas with customers at each iteration so that you have a continuous stream of feedback into the iterative process.
You also need to think through the business plan in detail during this phase and confirm with your target customers exactly how your monetization plan is going to work. This customer feedback is needed whether you plan to monetize directly—such as by selling devices, subscriptions, or services to customers—or indirectly through advertisements, developer licensing, partner deals, or other means. Obviously, if customers aren’t willing to click the “Buy” button to pay for your service, you are in trouble. But even for an ad-supported model, if your customers aren’t willing to tolerate and engage with advertisements, you need to know that now and figure out an alternative—it could have a big impact on the overall solution you build. You need to put whichever customer in your ecosystem is going to be financing your business on your list of customers to get feedback from and iterate with, even if they are not a direct consumer. This might include partners, OEMs, developers, device manufacturers, as well as end users.
An often-forgotten aspect of the solution-tradeoffs phase in a larger project is the need to integrate the solutions being iterated in parallel by multiple teams, each of which is focused on solving different scenarios but whose solutions will ultimately appear within the same parts of the user interface or in a set of APIs. Integrating the user experience of these solutions, resolving conflicts, and making tradeoffs between them is an important aspect of winnowing down to a single solution approach. These steps should happen before solutions are locked and before they are implemented with final ship-quality code. Trying to integrate experiences across a product after you see collisions and surprises appear in the build is much more difficult than resolving these problems earlier, when plans are more fluid and adaptable.
An experience architect can be extremely helpful throughout this phase. This team member provides structures to prevent collisions and inconsistencies by establishing visual guidelines, layout templates, interface metaphors, design principles, and other elements of a user experience architecture. An experience architect can also be instrumental in resolving integration collisions when they do happen, and he or she ideally provides oversight throughout the iterative process by proactively checking in with each team to help plans grow together in a complementary way and to ensure that discoveries and ideas are shared broadly. The skills of an experience architect to be able to zoom in to the details as well as zoom out to see a cohesive big picture are rare, but they’re essential to have on any large team.
The solution-tradeoffs phase typically lasts anywhere from two to six iterations, depending on the size and scope of the project and the difficulty or novelty of the problem space, and may vary significantly for individual projects depending on how long the team takes to hit on a solution that meets all the goals and critical constraints.
Allow for the flexibility to iterate. It’s true. Iterating based on feedback and waiting until you get it right makes predicting a development schedule up front more difficult. But in the big picture, it’s much more important to increase your predictability of commercial success than it is to increase the predictability of the development schedule. By providing room in your schedule for iteration, you can avoid the situation where you prematurely decide on a suboptimal solution and ship something that the customer doesn’t understand, need, or like.
Balancing iteration, feedback, and the creative process against real-world pressures to ship is an art, not a science. You need to find a balance that satisfies both the need to delight your customer and the need to bring a product to market. One way to gain more scheduling flexibility is to do fewer things in the first place. And remember, inside the creative process, time boxing is your friend. It is possible to fuel iterations and creativity while still maintaining the respect and discipline for the need to finish.
You are ready to exit the solution-tradeoffs phase when you’ve gotten to a single proposed end-to-end solution that meets experience, business, and technology constraints and all of the related scenarios are well integrated and rationalized. With competing constraints and practical limitations to balance, it’s easy to forget that achieving a delightful user experience is your ultimate goal. If customers aren’t finding your solution deeply delightful by the end of this phase, it’s too late to fix this in the details phase, so be sure you keep your eyes focused on delight throughout the solution-tradeoffs phase.
Here are some details about each stage in the Fast Feedback Cycle during the solution-tradeoffs phase of a project and which specific techniques make the most sense during this phase:
Observe stage By the time you start this phase, you are typically using evaluative research approaches, starting mostly with rapid, lightweight, qualitative approaches, and still evaluating multiple solution alternatives at each iteration. Usability testing with all manner of prototypes or functional code tends to be the workhorse in this phase, often paired with a questionnaire or brief interview at the end of the test to gather preferences, satisfaction, and other attitudinal data. A/B testing may be possible in some situations where infrastructure is readily available. You are listening for which aspects of each solution alternative are working well and which aren’t and for the preferences that people express. This feedback helps direct your efforts to fix those problems in future iterations and recombine the best aspects of multiple alternatives. As you gather feedback, be sure to listen not just for validation about usefulness and usability but also for desirability. If you hear doubts in the solution-tradeoffs phase about the overall usefulness of your solution, instead of trying out a new solution, consider reframing the problem—it’s likely you’re not focused on a valuable customer problem yet. It is still important to use actual target customers when assessing usefulness and desirability, but this is the phase when it is reasonable to use customer proxies during initial rounds of usability tuning or to do heuristic evaluations or cognitive walk-throughs to ferret out bigger usability issues.
Frame stage At this point, your scenario and metrics should be written and fairly well refined, so your first job is simply to briefly check in and remind yourself of your scenario and its key success metrics. Use this checkpoint to make sure that you haven’t drifted away from your initial problem statement and to make small edits to your scenario based on ongoing learnings, when appropriate. Large edits to your scenario at this point will reset the clock and put you back in the needs phase temporarily. For this phase, your new work in the Frame stage is to start reporting on your progress via scorecards or other reports. You need to figure out how you’re going to actually measure and report on your key metrics, start putting telemetry or other infrastructure in place, collect preliminary data as it becomes available, and collate that information into scenario scorecards or reports to track your progress toward shipping. In this phase, your first report will likely contain summaries of qualitative feedback that was heard in user testing and data points from end-of-test questionnaires, but your reports will gradually include more quantitative metrics when they become available. Collecting and reporting on this data at every iteration is important so that you can establish trends over time.
Brainstorm stage The beginning of the solution-tradeoffs phase is your last chance to brainstorm broadly, looking for new approaches and wild ideas. Be sure that you have thoroughly explored the entire problem space before you start to winnow, so include some of the more structured brainstorming techniques, such as the six thinking hats or SCAMPER, or the rigorous use of lateral thinking techniques, especially paired with sketching. Holding a sketchfest is a great way to generate a lot of storyboards that explore multiple end-to-end solution alternatives and to bring fresh brains into the brainstorming. You are looking for as many divergent ideas as possible, as well as cousins of current proposals so that you can compare their relative merits. As feedback on your solution ideas starts coming in during the solution-tradeoffs phase and feasibility and business constraints begin to surface, your brainstorming becomes increasingly constrained. You now generate alternatives for solving a particular problem discovered within the context of a given set of solutions rather than reimagining the solution set from the beginning. The latter part of this phase should have a big focus on blends of ideas, combining and recombining the best aspects of different solutions to fine-tune your solution.
Build stage You start the solution-tradeoffs phase with low-fidelity prototypes of the most promising end-to-end solution ideas generated in the previous stage. As you iterate in this phase, you gradually increase fidelity, moving from rapid prototypes to writing code, as you winnow your ideas and become more confident about which solutions to pursue. Your prototypes should play out the full length of the end-to-end experience, to ensure that you’re delivering a complete solution for your scenario. However, it is rarely worth your time to consider errors or other corner cases in prototypes, unless these are somehow central to the experience. Be careful not to skip to high-fidelity prototypes or code too quickly, before you’ve allowed an iteration or two for the best ideas to be identified by customers and then blended those ideas. Spikes or code prototypes investigate the feasibility of various approaches and can happen in parallel with rapid UX prototypes for a time. Write code as a series of slices of the end-to-end experience, which gives you a double-header of value: insight into feasibility and performance characteristics of the proposed technical architecture, and a slice of functional code that you can use in usability tests to get customer feedback. By the end of this phase, you should certainly have done enough technical investigation to feel confident about the feasibility of your approach, have settled on a technical architecture, and be well underway writing production code.
In the solution-tradeoffs phase, you still do something at each stage of the Fast Feedback Cycle, but you spend proportionally more of your time in the Brainstorm and Build stages.
If you want to super-power your iterations, consider doing a week of co-located iteration with your customers. Bring your development team to live with a customer for a week, do rapid iterations with customer feedback the moment you need it, and maximize your speed and efficiency. A terrific video by Nordstrom’s Innovation Lab demonstrates what this might look like: http://www.nordstrominnovationlab.com/#case_study.
Which details of my solution need to be fine-tuned before I can ship a high-quality release?
Once you have narrowed in on a single solution that meets all your core constraints, you switch your attention to fine-tuning the details. Your goals are to optimize usability, performance, and aesthetic appeal and to ensure flawless execution in the final shipping code. Along with creating a great end-to-end experience for your customers and enabling a viable business plan, the production solution must also be robust, scalable, performant, maintainable, and generally executed with a high degree of craftsmanship. Thankfully, all those aspects of technical craftsmanship (and more) are pretty familiar to most engineers, so we won’t belabor those points. However, that same level of craftsmanship and attention to detail is just as important for fine-tuning the customer experience.
The details phase represents the bottom, straight-down part of the funnel. Here, you take a single proposed solution and fine-tune the details in a fairly linear fashion. You should already have high confidence that your chosen solution is useful for the target customer, reasonably usable, and desirable, so there isn’t much risk of finding surprising, new problems by this point.
You won’t make big changes in your solution from iteration to iteration in this phase, just small adjustments to fix minor problems or confusions or to add small touches that enhance the experience, improving customer delight. As you make those adjustments, you will still try some alternative approaches, but they will be different only in the details. For instance, to enhance the discoverability of a link on a page, you might propose changing its font color, adding a box around it, or underlining it. To improve aesthetics, you might try a couple of different visual motifs and get feedback on which one customers like best.
You don’t have to test everything. It may surprise you that we recommend that you not test every single change. You can overdo it by subjecting every change in color to a usability test or an A/B test, which is clearly overkill and inefficient. A team needs to apply some judgment about which changes are significant enough to merit developing alternatives and running a test. Discount usability is cheap, but it’s not free. Neither are A/B tests. Focus your iterations and data gathering on the places most important for your key success metrics and the places where you have the biggest risks or visible problems.
The details phase can take anywhere from three to as many as dozens of iterations, depending on the size of the project and how long it takes to write the production code and fine-tune the details to meet your success metrics. You are done with the details phase when you’re ready to ship your solution to customers.
Here are some details about each stage in the Fast Feedback Cycle during the details phase of a project and which specific techniques make the most sense during this phase:
Observe stage Usability testing is still the go-to approach for evaluating code. Detailed testing always needs to be done with true customers to be sure you catch any differences with your target population and stay connected to what they find pleasing and desirable. But, from time to time, you can get away with using customer proxies if all you need to know is whether the solution works at all, especially as new code comes online. As your solution stabilizes and is performing well in qualitative usability testing, you may consider adding a formal usability test to capture quantitative metrics with statistical reliability or for benchmarking your solution against a competitor. However, realize that these quantitative tests are expensive and are valid only if you use target customers. If you have the infrastructure, A/B testing is great in this phase to quantitatively test aspects of the design you may need real-world usage data on.
Frame stage You continue to check in on your scenario and metrics to be sure you haven’t drifted off path and to capture the latest round of metrics on the scorecard or scenario report. You should pay particular attention in the details phase to trends you see on your scorecard to ensure that there aren’t any regressions in your key metrics and to identify the areas that need the most work to meet your goals. The Frame stage is when you look at your scorecards and scenario reports and decide whether a solution is good enough to send out to beta testers, to reviewers, or ultimately for release. You’ll also undoubtedly make some hard decisions about whether a scenario is still complete enough to ship when not all the pieces are on track for release, or whether larger cuts need to be made.
Brainstorm stage In this phase, brainstorming is limited to specific targeted fixes to identified problems in your solution. Typically, it’s not worth conducting a big group brainstorm for these types of problems (unless something is really stumping you). However, you should stillgenerate multiple credible alternatives as part of the due diligence of thinking through any problem before you pick which solution to implement. Individual or small-group brainstorming in short bursts when a problem crops up is typical for this phase.
Build stage You should definitely be well into building production code by this point. If a trickier problem comes up, you may perhaps build a few targeted prototypes to explore alternatives, but the vast majority of building is done directly in code at this point. It’s important to build code in slices so that you can continue to test your work in progress with users and ensure that the live code is resonating as well as your initial prototypes did. If there is a problem with how your ideas are being translated into code, you want to find out sooner rather than later, so be sure to sequence your coding so that you can keep testing end-to-end paths as they come online.
In the details phase, although you do something at each stage of the Fast Feedback Cycle, you spend the majority of your time in the Build and Frame stages.
After the project ships, you may do a final usability test to capture final benchmark information, and you certainly will be monitoring new kinds of feedback channels now—live feedback from product support, social media, discussion groups, and any telemetry that might be coming in. This is all great data to inform your next project, whether that is a small update to address some feedback right away or a larger project to figure out the next major release.
What about small projects?
For small projects, especially incremental additions to an existing system, you certainly don’t need to do as many iterations. Also, small projects with small teams typically iterate faster because they have less complexity to manage. Weeklong iterations are not uncommon. Here’s what a three-day iteration might look like:
Day 1 Wallow in available data, draft scenario and metrics.
Day 2 Hold a one-hour to two-hour brainstorming session, create an affinity diagram, and then sketch four to eight storyboards.
Day 3 Build rapid prototypes for two to four of the most promising storyboards, and get ready to start the next iteration by testing the prototypes with customers the next day.
For a small change, where the code revision is trivial, you might go from idea to release in a single iteration and in a single day. However, before you go live to your whole user base, you should probably do an A/B test or partial rollout to confirm that nothing is broken and, ideally, that your key metrics did what you expected when the change was exposed to a smaller set of customers.
Although managing quick iterations for a small project might seem easy, it requires that you already have a firm intuitive grasp of the iterative process and can exercise judgment on which parts you can safely collapse or downplay at each phase in your project. Probably the biggest mistake in small projects is doing too few iterations because the solution seems obvious. Unfortunately, this may well lead you to discover that people didn’t find your change useful or desirable at all, and you end up having to do more work to back it out. Just because the code is easy to write doesn’t mean that you’re doing the right thing. Take enough time to be sure that the change will be well received before you suffer the embarrassment and the brand damage of exposing a worthless feature to your customers.
SFE in action: Tiny iterations
Ed Essey, Group Manager, Microsoft Garage
As a practitioner of design thinking, and more recently as a teacher and coach of Scenario-Focused Engineering, I often come across resistance from engineers who feel that the process is too time-consuming and simply too heavy for the problems they are tackling. When I teach classes, I like to counter that impression and demonstrate how SFE principles can easily be applied to the smallest of situations.
SFE is a fractal process, and it is just as applicable to a problem at the tiniest scope as it is at the grandest scale. Most people can envision using SFE on a large scale, to suss out the biggest customer problems and to prioritize the big business ideas. But it can be elusive to think about how to apply these principles when you’re doing something as small as drafting a convincing piece of email.
I sometimes walk through an example of using SFE concepts in a typical product meeting. We’ve all been in meetings that begin with someone proposing a solution to a problem, and then the rest of the time is spent riffing on that solution until the hour is up. The amazing thing is how often this occurs without anyone actually articulating the problem that is being solved.
To take the class through an example of following SFE in the small, I play out such a meeting. In the meeting, the organizer is talking about building a bridge. As soon as that solution is mentioned, I go up to the whiteboard and write “Build a bridge.” As the conversation and debates about the bridge ensue, I capture those specific notes. After a few minutes, I ask, “Well, what else could we try, if the bridge doesn’t work out? You know, it’s too expensive or we can’t get it done in time?” At the whiteboard I add an “Options” header, showing that building a bridge is just one option, perhaps of many.
I then turn the discussion to divergent thinking and get the room to quickly list a bunch of additional, nonbridge options. After we capture a few, I write two new headings on the whiteboard and ask: “What problem are we solving?” and “Who is the target customer?” Once thesequestions are answered, the class can even capture what we know as a brief scenario, further clarifying the problem, and then solicit more options, now with a specific target in mind.
Incidentally, I’ve led teams down this path many times in actual meetings. I find it a good backdoor technique to get people to think more clearly about framing the precise problem that is being tackled, and to then free themselves to consider a bunch of different options. I’m always amazed to see how often there was little agreement or understanding in the room of what the actual problem was or who it was being solved for. Over time, once teams experience this pattern of getting in sync on the customer and the problem being solved, they can quickly get to the most promising solution and begin working on the details, but this time in the context of a customer, with everyone aligned on the same problem.
After talking about how I use SFE in the context of a one-hour meeting, I’ll ask the people in the class how they might approach writing a particularly sensitive and difficult email communication. And there is an aha moment when the class sees that yes, indeed, it would be valuable to understand who the email is being written to (the customer) and exactly what the desired results are, and then to come up with a handful of different approaches and collaborate with a few colleagues. This can all be done in a matter of minutes; it doesn’t have to be time intensive or process heavy.
Finally, I remind my students that SFE is about applying intention to what you’re solving, and the formula is simple and repeatable.
Crisply articulate the problem you are trying to solve.
Identify who you are solving it for (and, ideally, gather data).
Come up with a few options.
Pick the most promising one and try it out.
The main point is to keep it lightweight—save time solving the problem by focusing on the problem to be solved. It is entirely possible to follow an SFE cycle in just a few minutes. And often, these few minutes will save you a lot of time later on.
Final thoughts on iteration
This chapter has examined some patterns, rhythms, and phases involved in the Fast Feedback Cycle. The key is going around the iterative cycle many times, as quickly as possible, throughout the design process and continuing through development. A sequence of rapid iterations should create a funnel-shaped winnowing of ideas, as you search for the optimal solution to a given scenario. Along the way, there are times to diverge and be creative in the iterative funnel, when you consider more options, as well as times to converge, when you must make difficult decisions and narrow in, and there’s a consistent rhythm that governs both. A progression of iterative phases goes from understanding customer needs, to working through tradeoffs to finding an optimal solution, to fine-tuning the details and getting ready to ship. The really breakthrough ideas come through combinatorial blends of ideas from different lines of thinking, which are iterated with continual customer feedback as you progress through this iterative cycle.
Knowing all this, you might conclude that following the Fast Feedback Cycle produces a very regimented and predictable process. However, this isn’t exactly true. While there is definitely a rigor and science to doing iteration well, it’s also important to be flexible, to roll with the punches, and to allow the process to feel a little messy. Being able to live with ambiguity a bit longer than usual is an important ingredient. In reality, it’s not smooth or orderly, it looks more like the scribble in Figure 10-7.
FIGURE 10-7 The design squiggle by Damien Newman.
Sometimes to go forward, you go backward. And sometimes the nature of the funnel might not be apparent in the moment (“Hey, why are we diverging. I thought we were converging at this point?”). In the middle of it all, it really does feel quite messy. But with experience, you gain confidence and learn that this messy process is quite predictable, and in the end it will lead you to great results.
SFE in action: The design squiggle
Damien Newman, CEO, Central; Lecturer, Stanford University School of Engineering
Years ago I found myself having to explain the design process to a billionaire with an unbelievably short attention span. He had never considered putting design into software, let alone thought it was needed. He was committed to spending a lot of money to build some software and had been told to bring me on to make the software pretty. So I had a split second to capture his attention and then hold it long enough to convince him to spend money on design in the right way, not just the “let’s make it pretty at the end” way. It was a challenge I accepted, and it required me to update my approach to explaining the value of design.
Some 12 years before, I had the great fortune to work as a graphic designer at the Cambridge, UK, office of Xerox PARC (known as EuroPARC). I had started my working life as a graphic illustrator, so some of my work at EuroPARC was to illustrate scenarios found in their research. And it was there that I first learned about the process of design and how to describe it.
For the researchers and scientists at Xerox PARC, their process began with research, from which they created prototypes and then a final iteration of a solution. I learned that there were three distinct phases of design: an abstract phase, when ideas and thinking come from research; a concept phase, when ideas are turned into something physical; and a design phase, when a final prototype or working version is created. Each phase contributed to the next, helping to refine the team’s understanding as a single point of focus.
So for 12 years I successfully pitched the three phases of design as abstract, concept, and design. I explained to my clients at the time that we had to begin with an abstract notion of what our problem might be, and through a phase of exploration we would refine our understanding and generate several possible concepts to solve it. This was a tough sell more than 20 years ago. I didn’t have books like this to help me. Developing empathy for users was quite an alien concept. But at least I was armed with a reasonable-sounding, intellectual way of selling the process of design, which worked until I met the billionaire.
I had three things to communicate about the design process to my easily distracted and impatient new client: what the process is, how you move from uncertainty to focus as you progress, and that the design phase is actually in the process, right there toward the end. And I also had to explain the three fundamental phases of design. An added bonus would be if I could somehow also explain the magic of design, which is synthesis—how one makes sense of the research.
One of the points I wanted to convey was that the process would not always be comfortable because of the nature of how we go about it. One whole phase, when we’re learning and understanding, would feel like trying to clear a fog or untangle a mess of cables and wires. But we would untangle the wires and become more certain of our direction as we proceeded. What the design process makes possible is the space to collect information, either through observations or other methods, and to methodically synthesize that knowledge into meaningful insights. The effort taken to create insights from research before generating ideas for design is what makes this process unique. Any self-respecting illustration of the process of design has to provide the means to explain this. There is, in fact, a noticeable line going backward in the design squiggle, which is to suggest that sometimes you have to go back to earlier stuff to make sense of what you have. In my thinking, if I could explain to my client that the upfront parts of the process would feel uncomfortable, then when the time came and he felt discomfort, he might remember that the feeling would pass as we progress and he didn’t need to panic. This is similar to telling people at the beginning of a workshop that energy will dip, everyone will feel uncomfortable, but that this will pass, and they will leave with far more energy than when they started. This helps prevent anyone from freaking out when they feel that dip.
I use the design squiggle constantly. Even during my days at IDEO. As widely known as the term design thinking is, many people don’t understand the process behind it. If you present the story of a process inside a corporate conference room, many people think of a Gantt chart or a straight-line experience. Instead, by using the squiggle, you’re able to show that the process is a little different. When I display a slide that shows the squiggle along with an image of rampant Post-It notes, that usually conveys the nature of the process quite well.
Back to the billionaire. The squiggle made sense to him. I landed the project and spent a very interesting 10 months or so designing software for him. Doing the upfront work to adequately explain the nature of the design process to the person in charge allowed us the space we needed to proceed during the project.
The squiggle is free for you to use. If it helps you explain the process of design thinking, big D design, or SFE and the Fast Feedback Cycle, then by all means use it. Search for “design squiggle,” and you’ll find many different versions you can download. It’s been adopted widely—printed on a wall of an Australian consultancy, referenced in books and on T-shirts, and seen in a TED Talk. People frequently send me pictures of it where it’s seen in the wild. Do let me know if it works or doesn’t; I’m always interested to hear the stories.
Breaking up the iterative process into patterns and phases and distinct stages within each iteration makes the process of design easier for new practitioners to understand and conceptualize. But the truth is that design is not a linear process and there is no prescriptive stepwise process. Expert teams skip around between the stages of the Fast Feedback Cycle at times, or may collapse some aspects of the phases together when it’s efficient and appropriate to do so. A mature team may split into parts, with some people conducting user tests while the rest of the team starts to work on the next level of detail in the solution as they wait for test results to react to. Experts may do faster iterations at the beginning of a project and then slow down the iteration cadence as they get into the details phase, once there is high confidence that only tweaks remain. They will use judgment about which kinds of changes require them to generate alternatives and do customer testing and which changes aren’t worth that level of work.
So even though the squiggle diagram may seem messy, it actually isn’t that far from the truth, at least when you watch a high-performing team operate. With time and persistence, you will get there.