Customer Analytics For Dummies (2015)
Part I
Getting Started with Customer Analytics
Chapter 2
Embracing the Science and Art of Metrics
In This Chapter
Quantifying and qualifying data
Identifying types of data
Deciding on the correct sample size
Knowing what to measure
Customer analytics largely involves turning customer actions and attitudes into data. Not all data is the same, and knowing what type of data you’re dealing with guides you through what you can do with it. When you can interpret the data that’s available to you, you can start making better decisions about product features and service experiences.
In this chapter, I help you understand the different types of data, and show you how to work with quantitative and qualitative data. I also cover best practices for identifying the best metrics to manage.
You’ll have to put on your analytical thinking cap (the science of metrics) and your creative thinking cap (the art of metrics). If you aren’t a numbers person and decide to hire someone who is, that’s okay. But this is still a useful chapter because you need to know how to talk to your analytics person. And you still need to know how to interpret the numbers so you can make sound decisions about how to improve your business.
Adding up Quantitative Data
Quantitative data is information that’s broken down by concrete numbers — for example, how many products a customer places in the shopping cart (3) or how much revenue you earn from a specific customer ($2,000).
Quantitative data falls into two categories:
· Discrete (countable items)
· Continuous (measurements)
You encounter a lot of numbers when quantifying customer experience with products and services. Knowing whether the data is discrete or continuous dictates the method you use in your analysis and reporting.
Discrete and continuous data
Discrete data has finite values, or buckets. You can count them. For example, the number of questions correct would be discrete: There are a finite and countable number of questions.
Other examples of discrete data are
· Number of products in your catalog
· Number of employees you have
· Number of customer reviews for a specific product
Continuous data technically has an infinite number of steps, which form a continuum. The time to find a product on a website is continuous because it could take 31.627543 seconds. Time forms an interval from 0 to infinity.
Other examples of continuous data are
· Dimensions of a specific product
· Miles to your retail store from a customer’s location
· Time for a customer to find the information he or she is looking for on your website
· Days until a product ships to a customer
You can usually tell the difference between discrete and continuous data because discrete data can’t be broken into smaller meaningful units. You can’t have half a customer, but you can have half a minute.
Levels of data
Another way customer analytics data gets divided is by the four levels of measurement. They’re levels because they start with data that’s more limiting in the type of analysis you can perform to the least limiting:
· Nominal: This includes discrete data such as the name of your company, type of car you drive, or name of a product. Nominal means essentially “in name only”; if you have a name, it belongs in this category. Nominal data is qualitative data.
· Ordinal: This includes data that has a natural ordering. The ranking of customers by oldest to newest, the order of callers in a queue for a call center, the order of runners finishing a race, or more often, the choice on a rating scale, such as from 1 to 5.
With ordinal data, you cannot know with certainty whether the intervals between each value are equal. In measuring customers’ attitudes toward their experience with products and services, you have to rely heavily on questionnaire data that uses rating scales. For example, on an 11-point rating scale, the difference between a 9 and a 10 is not necessarily the same as the difference between a 6 and a 7.
· Interval: This is data that has equally split intervals between each value. The most common example is temperature in degrees Fahrenheit. The difference between 29 degrees and 30 degrees is the same magnitude as the difference between 78 degrees and 79 degrees (I much prefer the latter of these two examples).
· Ratio: This is interval data with a natural zero point. For example, time to find a product on a website is ratio, because 0 time is meaningful. Degrees Kelvin has a 0 point (absolute zero). The steps in both these scales have the same degree of magnitude.
Whenever you can establish that data is ratio, you can make reasonable deductions, such as “customers are twice as satisfied using a new product version compared to an old version.”
Just because customers’ average rating on one product is a 4 and the rating on another product is a 2 doesn’t mean customers are twice as satisfied. The first rating is definitely twice as high, but unless the scale is both ratio, and calibrated so the numbers correspond to customer behavior, making such claims is risky. It’s best to simply say the rating was twice as high.
Many organizations, statisticians, and even software programs use this hierarchy so it’s important to understand the terms when you encounter them. Some analysts even restrict their analysis based on it (see the sidebar “The history of levels of measurement” for more information).
Figure 2-1 shows how the levels of measurement fit into the broader categorization of qualitative and quantitative data.
Figure 2-1: Break your information into the data types and then the measurement level.
The history of levels of measurement
Where did the levels of measurement come from and why should you care? Well, you should at least care about identifying nominal or categorical data. If it isn’t nominal, then it’s quantitative. So why all the fuss?
In the 1940s, when behavioral science was in its infancy, there was much concern about trying to make the practice as legitimate as possible. Psychology and other social and behavioral sciences are considered soft sciences as opposed to the hard sciences of chemistry and physics. It was thought that applying some of the same thinking from the hard sciences would improve the legitimacy of these soft sciences — as well as the veracity of the claims made.
One approach was to map types of scaling to more natural laws (something akin to the physical laws of gravity and motion). This classification system was proposed in 1946 by S.S. Stevens. In the article, Stevens went so far as to say that you should only take averages on at least interval and ratio data. Nominal and ordinal data should only be counted and described in frequency tables — no means and standard deviations.
One of the more famous articles showing the fallacy of such rigid thinking was by an eminent statistician named Lord, who in his article “On the Statistical Treatment of Football Numbers” showed how the means of nominal data can be meaningful too!
In practice, rating scales are ubiquitous in companies and rarely have they been shown to have interval, much less ratio scales (what is the 0 point of customer satisfaction ?)
In summary, it’s generally OK to take means and apply statistical tests to ordinal data, just be careful about making ratio claims such as “twice as satisfied.”
Variables
A variable is a characteristic of a product or service that varies, which can often be manipulated. For example, price, delivery time, and color are product variables. Customer variables can include gender, income, geography, new customer versus existing customer, and type of industry, to name a few. When you look at product and customer variables, you can understand how different product attributes attract more or less sales and how different customers respond to different products and feature combinations.
There are two types of variables:
· Dependent variables are usually the things you care about but can’t affect directly, such as profitability, customer satisfaction, and customer loyalty. You can influence dependent variables by changing the independent variables. An example of this relationship is shown in Figure 2-2.
· Independent variables can be directly controlled or manipulated.
For example, independent variables include price, features, advertising, and usability.
Often, independent variables correlate with dependent variables, but the correlation doesn’t equal causation. Other variables that you’re not measuring can “mediate” or be responsible for the relationship. For example, higher sales (dependent variable) might be attributed to a new marketing campaign (independent variable) but the increase is actually just due to a growing economy that’s helped all businesses (mediator variable). See the appendix for more discussion on measuring correlation and causation between variables.
You can think of independent variables as the ingredients you use to cook a stew. The soup is the dependent variable (what you care about), but adjusting the ingredients and their combinations is what you can control.
Variables often come in the form of words instead of numbers — for example, new customer or existing customer, male or female, high income or low income. To make analysis of these qualitative values easier, you can code them into dummy variables by assigning them a number (for example, new customers get coded a 1 and existing customers get coded a 0; men get coded as 1 and women coded as 0 [or vice versa]).
With your variables coded as 1s and 0s, you can compute the percentage of customers with each variable.
Figure 2-2: Change the independent variables you can control to affect the dependent variables you can’t.
Quantifying Qualitative Data
Qualitative data is often helpful by itself to explain the “why” behind low satisfaction rates, higher sales, or high customer turnover rates. For example, if you see customers complaining that they don’t know what the total price of their order is in local currency or how to change the currency in a website shopping cart, you know what you can fix.
Comments provide immediate insight and potentially action (improve the currency display on checkout).
Always take the time to sort and count customer comments. Just because customer information is qualitative doesn’t mean you can’t use quantitative methods to interpret qualitative data to make better decisions. If you find that a significant portion of comments revolve around a specific issue (say 20% of the comments center around currency issues), you’ve just turned your qualitative data into quantitative data.
Quantifying iTunes experiences
My company asked 56 customers to rate their experience with iTunes. Customers answered several questions about the ease of use and quality of iTunes and whether they would recommend it to their friends. (See Chapter 12 for more on customer loyalty.) Customers who provided low scores were asked to briefly describe what motivated the rating.
For example, one customer said
I used to use iTunes much more often before they raised their prices from 99 cents to $1.29 per song. I still use the iTunes player to transfer songs to my iPod but I usually buy songs on Amazon.com now because it’s a much better deal.
And another said
I dislike iTunes. It’s unnecessary and can be annoying and difficult to use.
In total, 16 customers were rather dissatisfied with the iTunes experience. To understand the main drivers behind the dissatisfaction and the prevalence of some of these issues in the larger population, similar comments were first grouped, followed by the total number of comments in each of the groups. This table shows the results.
Conclusions from this data indicate that one of the primary reasons users are unlikely to recommend (called detractors) iTunes is because it is difficult to use, with four users making comments about ease.
Usability is often a key driver of product loyalty, and iTunes is no exception. (See Chapter 14 for more on usability.) I was 95% confident that between 9% and 50% of all iTunes detractors feel ease of use is one of the main reasons for not recommending the product to a friend.
I generated a 95% confidence interval around each group of comments using the online calculator (www.measuringu.com/wald.htm) to get an idea about how prevalent these reasons would be in the whole iTunes user population.
Quantifying the frequency of customer comments helps you understand how prevalent a certain attitude may be in your entire customer population. Some examples of open-ended responses (often called verbatim responses) are common for things such as:
· Reasons why customers are not recommending your product
· Observations from customers using a product at their workplace
· Product complaints in customer service calls
Here are three steps you can follow to turn qualitative data into quantitative data to estimate the prevalence of responses:
1. Group similar comments and behaviors.
Customers will use their own words to describe how they feel. Group similar phrases, behaviors, or concepts together. Some comments will be virtually identical and grouped easily. Others will differ and require additional layers of grouping.
There can be a high amount of variability between people grouping items. If possible, consider having multiple people independently categorize comments.
2. Count the frequency.
Count the number that appear in a category and the total number. If 5 out of 50 comments are related to price, for example, then an estimate of how often price is a concern is 10% (5/50).
3. Estimate the frequency.
You can estimate how common an issue is with the entire customer base by using a confidence interval (discussed later in this chapter).
Collecting data from a sample of customers costs a lot less and takes a lot less time than measuring every customer. The level of precision you get from even a small sample is usually sufficient to make decisions from the data.
Determining the Sample Size You Need
It would be good to know what customers think or what product features they like so you can build better products. How do you get this information? By asking your customers to take a poll.
But the problem is you rarely can ask all your customers to take your poll. That would be too time intensive and expensive, and you’d be swimming in data. Instead you collect data from a sample of customers. Your sample size can be small — 5 to 10 customers, or very large — 10,000+ customers. The data you collect from your sample represents all your customers.
Polling sample sizes
In August 2014, Gallup reported the presidential job approval rate was 41% with a margin of error of +/- 3%, using a 95% level of confidence.
Gallup derived this approval rate by asking 1,500 representative voters. Fifteen hundred responses represent some 240 million citizens, or just .001%!
This small proportion has a small margin of error. If Gallup were somehow to ask every voter whether he or she approved of the president, the best estimate of the job approval rating is between 38% and 44% (41% - 3% and 41% + 3%, respectively). This range is created by adding and subtracting the margin of error to the sample percentage.
Surveying a sample of customers is a tried and true method. Polling agencies that measure electoral votes take the same approach: They ask only a subset of the voting population to predict how the entire electorate will vote.
You can achieve a rather low margin of error after just sampling a small fraction of the entire population. There is a diminishing return from increasing sample sizes beyond a certain point. While you want your estimate to be accurate, you have to determine the sample size you can afford based on your budget and company resources.
Table 2-1 shows a table of confidence intervals around different sample sizes. You can see how the margin of error and width of the confidence interval decrease as the sample increases. As a general approximation, you need to quadruple your sample size in order to cut your margin of error in half.
For example, at a sample size of 381, your sample estimate will have a margin of error of approximately plus or minus 5%. That means if a survey to customers indicates that 50% say they would repurchase their subscription in the following year, you can be 95% confident that between 45% and 55% of all customers actually will. The sample size needed when making comparisons (say, between different product designs) will differ, and is covered in Chapters 10, 14 and 15.
Estimating with a confidence interval
Rarely will you be able to survey every customer. Instead you need to take a sample of customers and use this sample to make inferences about all customers. You have to accept that the sample estimate has a risk of being inaccurate.
Even if your sample size is small, you can still use it to make sound decisions, especially when you use techniques like confidence intervals.
You can measure the amount of error with a confidence interval. Confidence intervals tell you how much you can expect sample estimates to fluctuate based on sample size. They provide you with the most likely range for the unknown customer population numbers you’re trying to predict. The larger the sample size, the better the estimate is. By building a 95% confidence interval around a sample, you can expect that 95% of the time your interval will contain the actual customer population average.
Confidence intervals and all data you collect will only accurately predict the customer population if your sample is representative of the actual customer population. If you sell products mostly in Europe but most of your survey data comes from the U.S., it’s unlikely your estimates will be accurate (unless U.S. customers respond similarly to European customers).
Computing a 95% confidence interval
To compute a 95% confidence interval, you need three pieces of data:
· The mean (for continuous data) or proportion (for binary data)
· The standard deviation, which describes how dispersed the data is around the average
· The sample size
In the following sections, I show you how to calculate a confidence interval when you have continuous data or discrete data.
Continuous data example
Imagine you asked 50 customers how satisfied they were with their recent experience with your product on an 7-point scale, with 1 = Not at all satisfied and 7 = Extremely satisfied. These are the steps you would follow to compute a confidence interval around your sample average:
1. Find the mean by adding up the scores for each of the 50 customers and divide by the total number of responses (which is 50).
If you have Microsoft Excel, you can use the function =AVERAGE() for this step. For the purpose of this example, I have an average response of 6.
2. Compute the standard deviation.
You can use the Excel formula = STDEV() for all 50 values. I have a sample standard deviation of 1.2.
3. Compute the standard error by dividing the standard deviation by the square root of the sample size.
1.2 / √(50) = .17
4. Compute the margin of error by multiplying the standard error by 2.
.17 x 2 = .34
5. Compute the confidence interval by adding the margin of error to the mean from Step 1 and then subtracting the margin of error from the mean, like this:
6 +.34 = 6.34
6 -.34 = 5.66
You now know you have a 95% confidence interval of 5.66 to 6.34. The best estimate of what the entire customer population’s average satisfaction is ranges between 5.66 and 6.34.
If you have a smaller sample, you need to use a multiple greater than 2. You can find what multiple you need with the online calculator available at www.measuringu.com/ci-calc.php.
Discrete data example
Imagine you asked 50 customers if they are going to repurchase your service in the future. Using a dummy variable, you can code Yes = 1 and No = 0. If 40 out of 50 report their intent to repurchase, you can use what is called the Adjusted Wald technique to find your confidence interval:
1. Find the average by adding all the 1s and dividing by the number of responses.
40 / 50 =.8
2. Adjust the proportion to make it more accurate.
1. Add 2 to the numerator (the number of 1s).
40 + 2 = 42
2. Find the adjusted sample size by adding 4 to the denominator (total responses).
50 + 4 = 54
3. Divide the result to find the adjusted proportion.
42 / 54 = .78
3. Compute the standard error for proportion data.
1. Multiply the adjusted proportion by 1 minus the adjusted proportion.
.78 × (1 -.78)=.17
2. Divide the result (.17) by the adjusted sample size from Step 2.
.17 / 54 = .0031
3. Determine the square root of the value from the preceding step.
√.0031 = .057
4. Compute the margin of error by multiplying the standard error (result from Step 3) by 2.
.057 × 2 = .11
5. Compute the confidence interval by adding the margin of error from the proportion from Step 1 and subtracting the margin of error from the proportion.
.78 + .11 = .89
.78 - .11 = .67
The 95% confidence interval is .67 to .89. The best estimate of the entire customer population’s intent to repurchase is between 67% and 89%.
Values are rounded in the preceding steps to keep them simple. If you want a more precise confidence interval, use the online calculator available at www.measuringu.com/wald.htm.
Determining What Data to Collect
The art and science of customer analytics means turning metrics into insights. But you need metrics to begin with. Metrics can be found all across an organization and a customer’s journey (see Chapter 7). I discuss in detail the right metric to collect depending on your goals and methods throughout this book. But first it’s a good idea to get a better understanding of problems and opportunities for your customers.
You need to collect data from each of the following four customer analytics data types.
· Descriptive: The descriptive data becomes the template for whom you measure. It includes demographic data like gender, age, geography, and income. It also includes self-described attitudes and preferences toward products, categories, and technology. From this data, you can create meaningful segments (for example, early adopters or value-seekers) and personas (see Chapter 5).
You can collect this data from purchases, registrations, surveys, interviews, and contextual inquiries.
· Behavioral: The behavioral data becomes the framework for testing experiences. It is the general pattern customers exhibit when using your products and services. It includes making purchases, registering, browsing, and using different devices.
For example, customers of certain product categories, like consumer electronics or home furniture, tend to browse products on their tablet at night and make purchases on their desktop during the day.
· Interaction: The interaction data becomes the task scenarios that you simulate and measure during a usability test (see Chapter 14). It includes the clicks, navigation paths, and browsing activities found on websites and software.
The classic usability test typically focuses on this level of granularity by simulating real interactions. You can use real-time data from A/B testing, Google Analytics, and lab-based or unmoderated testing to collect data for this grouping (see Chapter 10).
· Attitudinal: Preference data, opinions, desirability, branding, and sentiments are usually captured in surveys, focus groups, and usability tests. This is where questionnaires like the SUPR-Q (www.suprq.com), System Usability Scale (SUS; www.measuringu.com/sus.php), or the Net Promoter Score (www.netpromoter.com) quantifies how interactions and behaviors affect attitudes. These attitudes will then affect some self-described descriptive attributes quantified in the descriptive grouping (see Chapter 9).
Improvements you make that affect attitudinal data, like increased trust and loyalty, drive further buying behavior.
Managing the Right Measure
If you can’t measure the customer experience, you can’t manage it. Improving the customer experience starts with measuring. But you must be sure you’re getting the right measure (or usually measures) to manage. The right measure(s) will:
· Identify problem areas
· Track improvements over time
· Be meaningful to the customer
The wrong measure(s) can:
· Identify wrong areas of focus
· Miss problems all together
· Lead to unintended consequences
· Alienate customers
Here are some different ways of thinking about measuring experiences:
· Conversion rate versus number of conversions: Conversion rates are the central metric for testing better designs, ads, and campaign effectiveness (covered in Chapter 10). The ratio of total users who purchase, register, or click (convert) to all users who viewed the page is an effective ratio because you can compare low traffic and high traffic pages.
While this is a convenient metric, the total number of conversions likely has a bigger impact on your bottom line than the rate. Wouldn’t you rather have 100 conversions from 100,000 page views than 10 conversions from 100 page views?
· Number of clicks versus time to destination: When you’re trying to make a more efficient experience, reducing the number of clicks to accomplish a goal seems like a good way to measure. Putting all functionality and content on one page would certainly reduce the number of clicks, but that probably is not what your customers have in mind.
Too much functionality or content on one page makes for an overwhelming experience; just make sure the paths customers need to complete a task are clear (see Chapter 15).
· Call time or call satisfactorily resolved: Wonder why those often scorned customer service agents you call to complain to speak so quickly? If you want to reduce call time in a customer support center, you can instruct agents to get off the phone faster, but have you really increased service or quality if customers have to call back?
Often a simple follow-up question sent via email can solve this problem.
· Customer satisfaction as a bonus motivator: Many companies pay bonuses based on achieving and exceeding certain customer satisfaction goals. Unfortunately and not surprisingly, this can lead employees to improve their chances for getting the bonuses in ways that make measures less meaningful.
· Likelihood to recommend or likelihood to repurchase: With the popularity of the Net Promoter Score (see Chapter 12), it may seem like word of mouth is the only measure you should care about. But if everyone already knows about and owns your product or visits your website, likelihood to purchase again might be a better measure of growth.
For measuring customer loyalty, I recommend using both repurchase rates and likelihood to recommend. This provides a mix of both behavior and attitude.
· On-time arrival versus on-time departure: Have you ever been on a plane that pulled away from the Jetway only for it to sit on the tarmac waiting for mechanical issues or other delays? You then arrive at your destination late? It’s likely that the flight segment still counted as an on-time departure. You can’t argue with the measure: The plane did pull away from the Jetway on time and that does mean something. However, that action just doesn’t mean that much to the customer sitting in the idled plane.
Measuring is good. Knowing what to measure is better. Finding the right measure means taking multiple measures and seeing which one best tracks customer sentiments and revenue.