Big Data Bootcamp: What Managers Need to Know to Profit from the Big Data Revolution (2014)

Preface

If it’s March or December, watch out. You may be headed for a break up. Authors David McCandless and Lee Byron, two experts on data visualization, analyzed 10,000 Facebook status updates and plotted them on a graph. They figured out some amazing insights. Breakups spike around spring break and then again two weeks before the winter holidays.

If it’s Christmas Day, on the other hand, you’re in good shape. Fewer breakups happen on Christmas than on any other day of the year. If you’re thinking that Big Data is a far off topic with little relevance to your daily life, think again. Data is changing how dating sites organize user profiles, how marketers target you to get you to buy products, and even how we track our fitness goals so we can lose weight.

My own obsession with Big Data began while I was training for Ironman France. I started tracking every hill I climbed, every mile I ran, and every swim I completed in the icy cold waters of San Francisco’s Aquatic Park. Then I uploaded all that information to the web so that I could review it, visualize it, and analyze it. I didn’t know it at the time, but that was the start of a fascinating exploration into what is now known as Big Data.

Airlines and banks have used data for years to figure out what price to charge and who to give loans to. Credit card companies use data to detect fraud. But it wasn’t until relatively recently that data—Big Data as it is talked about today—really became a part of our daily lives. That’s because even though these companies worked with lots of data, that data was more or less invisible to us.

Then came Facebook and Google and the data game changed forever. You and I and every other user of those services generate a data trail that reflects our behavior. Every time we search for something, “Like” someone, or even just visit a web page, we add to that trail. When Facebook had just a few users, storing all that data about what we were doing was no big deal. But existing technologies soon became unable to meet the needs of a trillion web searches and more than a billion friends.

These companies had to build new technologies for them to store and analyze data. The result was an explosion of innovation called Big Data. Other companies saw what Facebook and Google were doing and wanted to make use of data in the same way to figure out what we wanted to buy so they could sell us more of their products. Entrepreneurs wanted to use that data to offer better access to healthcare. Municipal governments wanted to use it to understand the residents of their cities better and determine what services to provide.

But a huge problem remained.

Most companies have lots of data. But most employees are not data scientists. As a result, the conversation around Big Data remained far too technical to be accessible to a broad audience.

There was an opportunity to take a heavily technical subject—one that had a relatively geeky bent to it—and open it up to everyone, to explain the impact of data on our daily lives. This book is the result. It is the story of how data is changing not only the way we work but also the way we live, love, and learn.

As with any major undertaking, many people provided assistance and support, for which I am deeply grateful. I would particularly like to thank Yuchi Chou, without whom this book would not exist; Jeff Olson at Apress for spearheading this project and for his and his team’s incredible editing skills; Cameron Myhrvold for his enduring mentorship; Houston Jayne at Walmart.com and Scott Morell at AT&T, and their respective teams, for their insights and support; and Jon Feiber, Mark Gorenberg, and Joe and Nancy Schoendorf for their wisdom and advice.