Introduction to Game Design, Prototyping, and Development (2015)
Part I: Game Design and Paper Prototyping
Chapter 10. Game Testing
Inherent in the concepts of prototyping and iteration is an understanding that high-quality testing is absolutely necessary to good game design. But the question becomes how exactly should this testing be performed?
In this chapter, you learn about various methods of playtesting for games, how to implement them properly, and at what stage in development each method is appropriate.
Once you’ve analyzed your goals, designed a solution, and implemented a prototype, it’s time to test that prototype and get some feedback on your design. I understand that this can be a frightening proposition. Games are difficult to design, and it’s going to take a lot of experience for you to get good at it. Even when you become an experienced designer, you’ll still probably have some trepidation when you think about people playing your game for the first time. That’s okay. The number one thing to keep in mind is that every person who plays your game is making it better; every comment you get, whether positive or negative, can help steer you in a direction that will improve player experience and hone your design.
Refining the design is what it’s all about, and you absolutely must have external feedback to do so. I have served as a judge for several game design festivals, and it always amazes me how easy it is to tell whether a dev team has done sufficient playtesting. Without enough playtesting the goals of the game are often not clearly specified, and the difficulty of the game often ramps up very quickly. These are both common indications that the game was most often played by people who already knew how to play and knew how to get through the difficult parts, so they couldn’t see the ambiguity or the rise in difficulty the way that a naïve tester would have.
This chapter will give you the knowledge and skills to run meaningful playtests and get the information from them that you need to make your games better.
Investigators Versus Playtesters Oftentimes in the game industry, we refer to both the people running the playtests and the participants in the tests as playtesters. For clarity, in this book, I will use these terms as follows:
Investigator: A person administering a playtest
Playtester: A person taking part in a playtest by playing games and giving feedback
Being a Great Playtester Yourself
Before getting into how to run various types of playtests for your games or what to look for in playtesters, let’s examine how you can be a great playtester for other people.
Think out loud: One of the best things you can do as a playtester is to describe your internal thought processes out loud while playing. Doing so will help the investigator running the test to correctly interpret the thoughts behind your actions. This can be especially helpful if it’s the first time that you’ve ever encountered the game.
Reveal your biases: All players are all biased by their experiences and tastes, and it’s often difficult for investigators to know where their playtesters are coming from. As you’re playing, talk about other games, films, books, experiences, etc. that the game reminds you of. This will help the investigators understand the background and biases that you bring with you to the playtest.
Self-analyze: Try to help the investigators understand why you’re experiencing the reactions that you’re having to the game. Instead of just saying something like “I feel happy,” it’s better to say “I feel happy because the jumping mechanic makes me feel powerful and joyful.”
Separate elements: As a playtester, once you’ve given overall feedback on the game experience, try to see each element separately; analyze art, game mechanics, game feel, sound, music, etc. as individual elements. This can be very helpful to investigators and is akin to saying “the cellos sound out of tune” rather than “I didn’t like that symphony.” As a designer, your insight into games can allow you to give more specific feedback than most players, so take advantage of it.
Don’t worry if they don’t like your ideas: As a designer, you should tell the investigators any ideas you have to make their game better, but you also shouldn’t be at all offended if they don’t use them. A lot of game design is about checking your ego at the door; it turns out that playtesting has an element of that as well.
The Circles of Playtesters
The game testing you do will go through several expanding circles of playtesters, starting with you and expanding outward through your friends and acquaintances to eventually encompass many people you have never met. Each circle of people can help with different aspects of your playtesting.
As a game designer, the first and last playtester of the games you design will most likely be you. You will be the first person to try out each of your ideas, and you’ll be the first person to decide whether the game mechanics and interface feel right.
A central theme of this book is that you always want to get a prototype of your game working as soon as possible. Until you have a working prototype, all you have is a jumble of ideas, but after you have a prototype, you have something concrete to react to.
Later in this book, you’ll be making digital game prototypes in Unity. Every time you press the Play button in Unity to run your game, you’re acting as a playtester. Even if you’re working in a team and are not the primary engineer on the project, as a designer, it will be your job to determine whether the game is heading toward the kind of experience your team wants to create. Your skills as a playtester are most useful in the very early stages of prototyping when you may need a great prototype to help other team members understand the design or when you may be still trying to discover the core mechanic or core experience of the game.
Eventually, there is a point at which you need to branch out and show your game to other people. You can never get a first impression of your own game; you know too much about it. Once you feel that your game is anything better than terrible, it’s time to find a few other people and show it to them.
Once you’ve playtested your game, iterated, made improvements, and actually crafted something that approximates the experience that you want, it’s time to show it to others. The first of these should be trusted friends and family members, preferably those either in your target audience or in the game development community. Members of your target audience will give you good feedback from the point of view of your future players (and hopefully future paying customers), and game developers can help by sharing their considerable insight and experience. Game developers will often have the experience to overlook aspects of the game that are obviously unfinished, which can also be very useful for relatively early prototypes.
Tissue playtester is an industry term to describe playtesters that are brought in to play the game and give their feedback once and are then discarded. They are one-use, like a facial tissues. This kind of tester is important because they can give you a naïve reaction to your game. Once anyone has played your game even a single time, they know something about it, and that knowledge biases subsequent playtest sessions. This kind of naïve perspective is critically important when testing:
The tutorial system
The first few levels
The emotional impact of any plot twists or other surprises
The emotional impact of the end of the game
Everyone Is a Tissue Playtester Only Once
Your game never gets a second chance to make a first impression. When Jenova Chen was working on his brilliant game, Journey, he and I were housemates. However, he asked me to wait until more than a year into the development of the game before I playtested it. Later, he expressed to me that he specifically wanted my feedback on the polish level of the game and whether it was achieving its intended emotional arc. As such, it would have ruined the experience for me to have played it in the early stages of development before any of that polish existed. Keep this in mind when playtesting with close friends. Think about the most valuable kinds of feedback that each person can give and make sure to show them the game at the best time for each individual.
That being said, never use that point as an excuse for hiding your game from everyone “until it’s ready.” Hundreds of people playtested Journey before I saw it. You will find that in the initial stages of playtesting, most people will tell you the same things in slightly different ways. You need that feedback, and even very early in the development process, you need tissue playtesters to tell you which of your game mechanics are confusing or need work for a variety of reasons. Just save a couple of trusted people for later when you know that their specific feedback will be most useful.
Acquaintances and Others
After you’ve been iterating on your game for a while and you have something that seems pretty solid, it’s time to take it out into the wild. This isn’t yet the time to post a beta to the Internet and expose your game to the rest of the world, but this is when feedback from others that you don’t normally associate with can be helpful. The people you call your friends and family often share your background and experiences, meaning that they will also often share some of your tastes and biases. If you only test with them, you will get a biased understanding of your game. A corollary to this would be someone in Austin, Texas, being surprised that the state of Texas voted for a Republican presidential candidate. Most people in Austin are liberal, while the rest of the state is primarily conservative. If you only polled people in Austin and didn’t break out of that left-leaning bubble, you’d never know the opinion of the state as a whole. Similarly, you’re going to need to get out of your normal social circles to find more playtesters for your game and to understand a larger audience’s reaction to your game.
So, where do you look for more people to playtest your game? Here are some possibilities:
Local universities: Many college students love playing games. You could try setting up your game in the student center or quad and showing it to groups of people. Of course, you’ll want to check with campus security before doing so.
You could also look into whether your local university has a game development club or a group that meets for weekly game nights and ask if they would mind you bringing a game for them to playtest.
Local game stores/malls: People head to these places to buy games, so it could be a fantastic place to get some playtest feedback. Each of these places will have different corporate policies on these kinds of things, so you need to talk with them first.
Farmers markets/community events/parties: These kinds of public gatherings of people can have incredibly diverse audiences. I’ve gotten some great feedback on games from people I met at parties.
The Internet can be a scary place. Anonymity ensures that there is little or no accountability for actions or statements, and some people online will be mean just for kicks. However, the Internet also contains the largest circle of playtesters that you can possibly get. If you’re developing an online game, you’re eventually going to have to reach out to the Internet and see what happens. However, before you do so, you will need to have considerable data and user tracking in place, which you can read about in the later section, “Online Playtesting.”
Methods of Playtesting
There are several different methods of playtesting, each of which is most appropriate for different phases of your game. The following pages explore various methods of playtesting that I have found to be useful in my design process.
Informal Individual Testing
As an independent developer this is how I tend to do most of my testing. I’ve been focusing on mobile tablet games lately, so it’s easy to carry my device around with me and show my games to people. More often than not, during a break in conversation I’ll ask if the person I’m speaking with would mind taking a look at my game. This is, of course, most useful in the early stages of development or when you have a specific new feature that you want to test. Things to keep in mind during this kind of testing include the following:
Don’t tell the player too much: Even in the early stages, it’s important to learn whether your interface is intuitive and the goals of your game are clear. Try giving your game to players and watching what they do before they’ve had any instruction. This can tell you a lot about what interactions your game implies on its own. Eventually, you’ll learn the specific short sentences you need to say to people to help them understand your game, and these will become the basis for your in-game tutorial.
Don’t lead the playtester: Be sure you don’t ask leading questions that may inadvertently bias your player. Even a simple question like “Did you notice the health items?” informs your playtester that health items exist and implies that it is important for her to collect them. Once your game is released, most players won’t have you there to explain the game to them, and it’s important to let your playtesters struggle a bit to help you learn which aspects of your game are unintuitive.
Don’t argue or make excuses: As with everything in design, your ego has no place in a playtest. Listen to the feedback that playtesters are giving you, even (or possibly especially) if you disagree with it. This isn’t the time to defend your game; it’s the time to learn what you can from the person who is taking time out of her day to help the design improve.
Take notes: Keep a small notebook with you and take notes on the feedback you get, especially if it’s not what you expected or wanted to hear. Later, you can collate these notes and look for statements that you heard multiple times. You shouldn’t really put too much stock in what is said by a single playtester, but you should definitely pay attention if you hear the same feedback from many different people.
Formal Group Testing
For many years, this is how playtesting was done at large studios, and when I was working at Electronic Arts, I took part in many of these playtests for other teams. Several people are brought into a room full of individual stations at which they can play the game. They are given little or no instruction and allowed to play the game for a specific amount of time. After this time, the playtesters are given a written survey to fill out, and investigators sometimes interview them individually. This is a great way to get feedback from a high volume of people, and it can get you a large number of answers to some important questions.
Some example post-playtest survey questions include:
“What were your three favorite and three least favorite parts of the game?”
Provide the playtester with a sequential list of various points in the game (or even better a series of images) and ask them, “How would you describe the way you felt at each of these points in the game?”
“How do you feel about the main character (or other characters) in the game? / Did your feelings about the main character change over the course of the game?”
“How much would you pay for this game? / How much would you charge for this game?”
“What were the three most confusing things about the game?”
Formal group testing is often administered by investigators outside of the core development team, and there are even companies that provide testing services like this.
All Formal Testing Requires a Script
Any time you are doing formal testing, either with investigators from inside or outside the team, you will want to have a script. The script should include the following information:
What should investigators say to the playtesters to set up the game? What instructions should they give?
How should investigators react during the playtest? Should they ask questions if they see a playtester do something interesting or unusual? Should they provide any hints to playtesters during the test?
What should the environment be like for the playtest? How long should the playtester be allowed to play?
What specific survey questions should be asked of the playtester once the playtest is complete?
Formal Individual Testing
Where formal group testing seeks to gather small bits of information from many different people and grant investigators a gestalt understanding of how playtesters react to a game, formal individual testing seeks to understand the fine details of a single playtester’s experience. To accomplish this goal, investigators carefully record the details of a single individual’s experience with the game and then review the recordings later to make sure that they haven’t missed anything. There are several different data streams that you should record when doing formal individual testing:
Record the game screen: You want to see what the player is seeing.
Record the playtester’s actions: You want to see the input attempted by the player. If the game is controlled with mouse and keyboard, place a camera above them. If the game is tested on a touchscreen tablet, you should have a shot of the player’s hands touching the screen.
Record the playtester’s face: You want to see the player’s face so that you can read her emotions.
Record audio of what the playtester says: Even if the player doesn’t vocalize her stream of consciousness, hearing utterances she makes can give you more information about her internal thought process.
Log game data: Your game should also be logging time stamped data about its internal state. This can include information about: input from the player (e.g., button presses on the controller), the player’s success or failure at various tasks, the location of the player, time spent in each area of the game, and so on. See the “Automated Data Logging” sidebar later in this chapter for more information.
All of these various data streams are later synched to each other so that designers can clearly see the relationships between them. This allows you to see the elation or frustration in a player’s face while simultaneously viewing exactly what the player was seeing on-screen at the time and the input her hands were attempting on the controls. Though this is a considerable amount of data, modern technology has actually made it relatively cheap to create a reasonably good lab for individual testing. See the sidebar for more information.
Setting Up a Lab for Formal Individual Playtesting
You can easily spend tens of thousands of dollars setting up a lab for formal individual testing, and many game studios have, but you can also mock up a pretty decent one for not a lot of money.
For most game platforms, you should be able to capture all of the data streams listed in the chapter with just three consumer cameras: one on the screen, one on the player’s face, and one on the player’s hands. All cameras can record audio, which can help you to synchronize them. The game data log should also be time stamped to allow for synchronization.
Many software packages out there enable you to sync several video streams, but often the oldest methods are the easiest, and in this case, you can use a digital version of the slate from the early days of sound in film. In a film, the slate is the little clap board that is shown at the beginning of a take. A member of the crew holds the slate, which shows the name of the film, the scene number, and the take number. She reads these three things out loud and then claps the slate together. This enables the editor to match the visual film frame where the clapper closed with the moment in the audio tape that the sound was made, synching the separate video and audio tracks.
You can do the same thing by making a digital slate part of your game. At the beginning of a playtest session, the game screen can show a digital slate containing a unique ID number for the session. An investigator can read the ID number out loud and then press a button on the controller. Simultaneously, the software can show a digital clapper closing, make a clapper sound, and log game data with the time stamp according to the internal clock on your playtest machine. All of these can be used to sync the various video streams later (with the clapper sound used to sync streams which cannot see the screen), and it’s even possible to sync the game data log. Most even half-decent video editing programs will allow you to put each of these videos into one quarter of the screen and fill the fourth quarter with the date, time, and unique ID of the playtest session. Then you can see all of this data synchronized in a single video.
In modern times, many people are concerned about their personal privacy. You will need to be upfront with your playtesters and let them know that they will be recorded. However, you should also promise them that the video will only be used for internal purposes and will never be shared with anyone outside of the company.
Running a Formal Individual Playtest
Investigators should seek to make the individual playtest as similar as possible to the experience a player who had bought the game would have at home. The player should be comfortable and at ease. You might want to provide snacks or drinks, and if the game is designed for tablet or console, you might want to give the player a couch or other comfortable seat to sit on. (For computer games, a desk and office chair are often more appropriate.)
Start the playtest by telling the playtester how much you appreciate the time she has set aside to test your game and how useful it will be for you to get her feedback. You should also request that she please speak out loud while playing. Few playtesters will actually do so, but it can’t hurt to ask.
Once the playtester has finished the section of the game that you want her to play, an investigator should sit with her and discuss her experience with the game. The questions asked by the investigator should be similar to those that are asked at the end of formal group testing, but the one-on-one format will allow the investigator to frame meaningful follow-up questions and get better information. The post-playtest question and answer sessions should also be recorded, though audio recording is more important than video for the post-play interview.
As with all formal playtesting, it is best if the investigator is not part of the game development team. This helps the investigator’s questions and perceptions to not be biased by personal investment in the game. However, after you have found a good investigator, it is very useful to work with the same investigator throughout the development process so that they can provide you information about the progression of playtesters’ reactions to the game.
As mentioned previously, the largest circle of playtesters is composed of online playtest communities. Your game must be in the beta phase before you attempt this, so these are colloquially known as beta tests, and they come in a few forms:
Closed: An invite-only test with a limited number of people. This is where your online tests should start. Initially, you should have only a few trusted people serve as online playtesters. This gives you a chance to find any bugs with your server architecture or any aspects of your game that are unclear before a larger audience sees it.
For Skyrates,1 our first closed online beta started eight weeks into the project and was composed of the four members of the dev team and only twelve other people, all of whom had offices in the same building as the development team. After two weeks of fixing both game and server issues and adding a few more features, we expanded the playtest group to 25 people, all of whom were still in the same building. Two weeks later, we expanded to 50. Up until this point, a member of the dev team had individually sat down with each player and taught her how to play the game.
1 Skyrates (Airship Studios, 2006) is a game that was introduced in Chapter 8, “Design Goals.” It made extensive use of the concept of sporadic play, where players interacted with the game for only a few minutes at a time throughout their day. Though this is now common behavior for Facebook games, at the time we were developing it, this was an unusual concept, and it required many rounds of playtesting to refine it.
Over the next two weeks, we developed an online game tutorial document and entered the limited beta phase.
Limited: A limited beta is generally open to anyone who signs up, though there are often a few specific limitations in place. The most common limitation is the number of players.
When Skyrates first entered the limited beta phase, we capped the number of players at 125 and told our players that they could invite one friend or family member to join the game. This was a much larger number of concurrent players than we’d had in prior rounds, and we wanted to make sure that the server could handle it. After that, we limited the next round to 250 before moving on to our first open beta.
Open: Open betas will allow anyone online to play. This can be fantastic because you can watch your game gain popularity halfway around the globe, but it can also be terrifying because a spike in players can threaten to overload your server. Generally, you want to make sure that your game is near completion before you do an online, open beta.
Skyrates entered open beta at the end of the first semester of development. We didn’t expect to work on the game for a second semester, so we left our game server running over the summer. To our surprise, even though Skyrates was initially developed as a two-week game experience, several people played the game throughout the summer, and our total numbers for the summer were somewhere between 500 and 1,000 players. However, this all happened in 2006 before Facebook became a game platform and before the ubiquity of gaming on smartphones and tablets. While 99% of all games on these platforms don’t gain much popularity at all, be aware that a game released on any of them has the potential to go from only a few players to millions in just a few days. Be wary of open betas on social platforms, but know that you need to open up the game eventually.
Automated Data Logging
You should include automated data logging (ADL) in your game as early as possible. ADL occurs when your game automatically records information about player performance and events any time someone plays your game. This is often recorded to a server online, but can just as easily be stored as local files and then output by your game later.
At Electronic Arts in 2007, I designed and produced the game Crazy Cakes for Pogo.com. Crazy Cakes was the first Pogo game to ever use ADL, but afterward it became a standard part of production. The ADL for Crazy Cakes was really pretty simple. For each level of the game that was played, we recorded several pieces of data:
Timestamp: The date and time that the round started
Player username: This allowed us to talk to players with very high scores and ask them what strategies they employed or contact them if something unusual happened during gameplay
Difficulty level and round number: We had a total of five difficulty levels, each of which contained four progressively more challenging rounds
Number and type of power-ups items used during the round
Number of tokens earned
Number of patrons served
Number of desserts served to patrons: Some patrons requested multiple desserts, which this helped us track
At the time, Pogo.com had hundreds of beta testers, so three days after releasing Crazy Cakes to our closed beta group, we had recorded data from over 25,000 playtest sessions! I culled this data to 4,000 randomly selected rows and brought it into a spreadsheet application that I used to balance the game based on real data rather than conjecture. Once I thought that the game was well-balanced relative to the data, I selected another 4,000 random rows and confirmed the balance with them.2
2 The data was limited to 4,000 rows because at the time, Excel had diffi culty handling more than 4,000 rows of data.
Other Important Types of Testing
In addition to playtesting, other important types of game testing include the following five types.
Focus testing involves gathering a group of people from your game’s core demographic (a focus group) and getting their reaction to the look, premise, music, or other aesthetic and narrative elements of your game. This is sometimes done by large studios to determine whether there is a good business case for developing a certain game.
It is now possible to use social networks like Facebook or crowdfunding sites like Kickstarter to poll the level of interest that your game could generate in the online public. On these websites, you can post a pitch video for a game and receive feedback, either in the form of likes on a social media site or pledges on a crowdfunding site. If you are an independent developer with limited resources, this may be a way to secure some funding for your game, but of course, the results are incredibly varied.
Many of the techniques now used in formal individual testing grew out of the field of usability testing. At its core, usability testing is about understanding how well testers can understand and use the interface for a piece of software. Because understanding is so important to usability, data gathering of the screen, interaction, and face of the tester are common practices. In addition to the playtesting of your game, it is also important to engage in some individual usability testing that investigates how easily the playtester can interact with and gain critical information from your game. This can include testing of various layouts for on-screen information, several different control configurations, etc.
Quality Assurance (QA) Testing
Quality assurance testing is focused specifically on finding bugs in your game and ways to reliably reproduce them. There is an entire industry devoted to this kind of testing, and it’s largely outside the scope of this book, but the core elements are as follows:
1. Find a bug in the game (a place where the game breaks or doesn’t react properly).
2. Discover and write down the steps required to reliably reproduce the bug.
3. Prioritize the bug. Does it crash the game? How likely is it to occur for a normal player? How noticeable is it?
4. Tell the engineering team so that they can fix it.
QA is most often done by the development team and a group of game testers hired for the final phase of a project. It’s also possible to set up ways for players to submit bugs that they find, but most players don’t have the training to generate really good bug reports that include clear steps for reproducing the bug. Many free bug-tracking tools are available and can be deployed on your project website, including Bugzilla, Mantis Bug Tracker, and Trac.
Automated testing (AT) occurs when a piece of software attempts to find bugs in your game or game server without requiring human input. For a game, AT could simulate rapid user input (like hundreds of clicks per second all over the screen). For a game server, AT could inundate the server with thousands of requests per second to determine the level of load that could cause the server to fail. While AT is complex to implement, it can effectively test your game in ways that are very difficult for human QA testers to accomplish. As with other forms of testing, there are several companies who make their living through automated testing.
The intent of this chapter was to give you a broad understanding of various forms of testing for your games. As a new game designer, you should find the ones that seem most useful to you and try to implement them. I have had success with several different forms of testing, and I believe that all the forms covered in this chapter can provide you with important information that can improve your game.
In the next chapter, you’ll be shown some of the math that lies beneath the surface of the fun in games. You’ll also learn about how to use a spreadsheet application to aid you in game balancing.