Hacking Happiness: Why Your Personal Data Counts and How Tracking It Can Change the World (2015)
[PART]
2
Be a Provider
9
AUGMENTED REALITY
There is no other future of computing other than [Augmented Reality] which can display information from the real world and control objects with your fingers . . . it’s the keyboard and mouse of the future.1
MERON GRIBETZ, FOUNDER AND CEO, META
I WROTE A SHORT story in 2012 that describes how I see augmented reality working in our very near future, a (Google) Glass half-empty/half-full scenario to whet your appetite regarding the possibilities of how geeky tech will influence our lives.
SELF-SCREENING
I lurched from the train car, elbow to elbow with a thousand other commuters stepping off New Jersey Transit. I jerked my head to the right and heard a chime indicating my CPRS was online. A bright red arrow hovered in the air before me, analyzing the platform leading to the stairs going up to the main platform of Penn Station.
“Go right.” Sean Connery’s brogue sounded in my brain as a red line appeared on top of the horde of pressing flesh, all vying for the same staircase. As I turned my head, the line flashed green when my best virtual path appeared be-fore me.
IBM’s CPRS (Consumer Pattern Recognition Simulator) lets you set the voice that navigates your actions through a virtual commuter game. (Connery’s voice had been chosen for me because I was a fanboy.) The app worked for any major New York transportation hub and was the latest in IBM’s Smarter Cities offerings. It utilized image-recognition-based augmented reality to analyze results of multiple predictive formulas to create algorithms based on commuter behavior. The game played out on my iPhone 8 contact lenses.
“What arrr yoo prepared to dooo?” Nice. Connery’s quote from The Untouchables.
I headed toward the stairs. In my urgency, I bumped a woman next to me, and she grunted. In the upper-right-hand corner of my vision, I saw my points decrease on a small New Jersey Transit con.
“Fuck!” I muttered, apologizing and letting her pass. In my ear I heard the sound of a baby crying and my points dropped even further. The AR in my contact lenses analyzed her past fifty tweets and discovered she was pregnant. Son of a bitch.
Everyone’s actions in the game were tied to real-world penalties and rewards. Early social-based action apps like Recyclebank and DailyFeats were still in use to encourage people to earn free stuff or gain social cred. But apps like GymPact where, by choice, you were penalized by your peers for not going to the gym had become wildly popular. Geek-chic went from craving Klout to demonstrating your accountability, and the craze had caught on with local government and utilities companies. I regularly did my laundry at three in the morning to get a high ABI (accountability-based influence) score from OPower, the leading social network based on the Smart Grid.
In my case, my next month’s commuter pass would cost about fifty cents more because of bumping a pregnant lady. So now I had to make up my points via speed. I started walking fast. A heart-shaped icon appeared in the upper left-hand corner of my vision as the pulse monitor watch grew snug on my wrist. If the heart went from red to purple, my doctor would get a text indicating I was at risk for cardiac arrest. The monitor went all the way to magenta four times one month and my insurance premiums increased.
Once at Penn Station I headed for the stairs, noting the commuter-gamers outside Starbie’s. (Certain sims let you order your coffee mid-play so you could pick it up right away and mobile-pay via NFC.) Distracted by the aroma of fresh-brewed coffee, I stumbled on something large at my feet. I looked and saw a large sack of grain. A money icon appeared in my vision over the bag, so I pulled my eyes to the left, indicating I would take the points for the grain. Frustrated at having to wait for the points to tally, I kicked the sack, hard. Then I ran upstairs.
As I neared the middle of the station, a red stop sign icon filled my vision. I paused, not sure what was happening. Virtual Air Rights codes had deemed it unlawful for advertisements or any game component to trick someone when using augmented reality-based app functionality.
I heard a new voice in my ear. “Chuck, look up.”
This freaked me out, as none of my voice recognition software was programmed to speak unless I spoke first. And most of my sims used eye tracking or Microsoft Kinect to recognize my gestures before I heard external voices in a game.
I looked up as I heard the board clicking, the shifting words forming the following phrase:
All the world’s a screen, and we are merely layers.
I stood for a long minute, gazing at the board and not really comprehending what was happening. I vaguely registered that my contact lenses were in reality mode, meaning the board actually said the words I saw above me.
“You a Shakespeare fan?” the voice came again.
“Sure?” I said, turning to see where the voice was coming from.
“Up here, Chuck.” I looked back at the board, and one side of the screen was shaped like a smiley-face icon. The lips moved when it spoke. “Thought I’d go with a smiley face versus a scary Tron-looking thing. Besides, part two sucked.”
“Agreed.”
“They call me the Bard. A few years back MoMA did a real-time data exhibit thing and they ran Shakespeare quotes on my screen. Somebody got cute and took the ‘o’ out of ‘Board’ and it trended on Twitter, so here we are.”
I looked at the other side of the screen, opposite his “face.” “So what’s with the quote? Am I a ‘mere layer’?”
I assumed the metaphor had to do with the Smart Grid, where the notion of Big Data meant that with networked artificial intelligence, we’d arrived at an Internet of Things mentality. In a sense, everything with a chip in it was alive, or in this case, a layer. Being a geek, I’d felt the whole idea of the singularity was inevitable starting around 2011 or so. I was also sure Bard was hooked to the Internet and dozens of cameras in the station that pumped images he could access anytime from the cloud.
“Sort of,” Bard responded. “Don’t get pissed, but I accessed your e-mail and social channels just now.”
I wasn’t that pissed. “Privacy” had a whole new definition these days. Since a GPS knew where you were at all times and everyone’s virtual games were hooked real-time to the Web, government types simply accessed video or Outernet feeds from citizens any time they wanted. Everyone’s lives were recorded at all times. According to CNN, no event occurred without at least two cameras recording what happened for potential public usage. News and law enforcement had become whole different animals in the past few years.
“Why is my foot wet?” I said, interrupting Bard as warm liquid seeped onto my right foot. I looked down to see my black loafer had a dark stain.
Bard spoke quietly. “That’s what I’m trying to tell you, Chuck. Look.”
His screen switched to an image of me arriving at the platform downstairs from a few minutes ago. I saw my journey from the train and could tell I was in my game, since my eyes appeared glazed and distant. The camera views switched a few times, from commuters to station cameras and back. And then as I turned a corner, I saw myself stumble where I had kicked the sack of grain.
But it wasn’t a sack of grain. It was a homeless guy.
He had blocked my path, and I had stumbled hard into his arm. And then I watched in horror as I pulled my foot back and kicked him squarely in the face, breaking his nose. Blood poured onto my shoe as he clutched his face in agony while I simply took a step back while my points tallied.
“You weren’t supposed to kick the grain, Chuck,” said Bard. “That’s what the money signs are for. The game disguised Tom as a bag of grain so people would avoid him. Kicking good things means you lose points. But you were too fast. And Tom sat up at a bad time.”
I couldn’t move. That image of me kicking the homeless guy—Tom—kept playing over in my brain. It was an image I knew I’d never erase. And it wasn’t a game.
What kind of man am I?
I stood for a long moment, game-blinded commuters rushing by. For once I heard the sound of shoes on pavement—no sound track, no sound effects. This was reality. And it sucked.
“Chuck.”
I looked up. Bard had cleared his screen and the following words appeared slowly, one by one:
What are you prepared to do?
•
About a year later, on my birthday, Bard said he had a surprise for me.
“Turn on your CPRS game.”
I did, and my commuter sim turned on. Bard had hacked it so it was pointing downstairs, and I followed the arrow to the spot where I had kicked Tom. Instead of a sack of grain, I saw a huge virtual package with the words The Impossible Idea written on the side. An icon on the upper-left-hand corner of my vision flashed, indicating it would open if I moved my eyes quickly to the left.
My impossible idea had been fairly simple. I quit my job and volunteered at the Robin Hood Foundation to create an app that rewarded people for kind actions to the homeless in New York City. Gwyneth Paltrow was their spokesperson, and with her avatar in the sim, the game took off. Starbucks joined in, and pretty soon the pay-it-forward mantra went full swing in Penn Station. Within a few months, people donated their Klout perks and accountability bonuses so that actions generated behavior change as well as words.
So now I looked at the spot where I had kicked Tom and my life had transformed by mistake. I moved my eyes to the left, and the virtual package fell open.
And I saw Tom. He was clean-shaven, waving and smiling. He was piped in via Skype and virtual, wearing the Robin Hood T-shirt they’d given him the day we first met. He’d gone from being a volunteer to a full-time staff member and gotten the first assisted-living residence made via profits from my app.
I smiled. “Thanks, Bard.”
Staring at the spot, I blinked three times rapidly to turn off my sim contact lenses. My vision cleared of all icons, layers disappearing between me and the empty pavement where Tom used to lie.
And it was empty.2
From (Cyber) Punk to Possible
I’ve been a science fiction fan since I was a boy, watching Star Trek with my dad and brother, and then seeing Star Wars in the theater when it first came out. Years later I devoured the work of people like Arthur C. Clarke and Robert Heinlein before falling into a geeky bromance with cyberpunk fiction from writers like Philip K. Dick and Neal Stephenson. Dick wrote Minority Report, which was made into one of the greatest cyberpunk films of all time, featuring Tom Cruise as a police officer who utilized augmented reality technology to stop murders that hadn’t yet occurred. Augmented reality, or AR, is a term coined by Tom Caudell from Boeing in 1990, as noted by Brian X. Chen in Wired.3 Caudell created the term based on a head-mounted, hands-free interface used by workers assembling airplanes. Digital data overlaid on the glasses in front of workers’ eyes let them see reality (the plane) with augmentation (instructions in real time on what to fix). Beyond the convenience for workers, the technology set the stage for combining virtual and physical reality.
There are a lot of desktop-based applications of augmented reality utilizing your webcam, but my fascination has always been with mobile AR. Whether you’re using your phone screen, a tablet, Google Glass, or eventually contact lenses, the idea of hands-free mobile computing is transformational. When images appear in front of your vision based on what you’re looking at, the technology simply feels magical.
I first wrote about AR for iMedia Connection in the article “Augmented Reality: What Marketers Need to Know.” Here’s how I described the future of AR:
The seminal promise of AR is as the touchstone technology allowing social networks, geo-based tracking, and the semantic Web to converge. Put less geekily, think of AR as your personalized digital butler, who will get to know your behavior so specifically that it can prethink your choices based on your friends, location, and how you search online. The cyberpunk fictions have come to reality with AR, and the cultural ramifications are as powerful as the marketing opportunities.4
At the time I wrote the piece, most of my marketing friends felt AR was gimmicky and only useful for gaming applications. But I saw AR as a type of wearable computer, an idea that’s pretty simple—you take the mobile phone you’re already carrying (which is a computer) and put it in front of your eyes to access information. The big differentiator for augmented reality, however, is how digital data can be placed over what you’re seeing on your mobile screen based on what you’re looking at. In this way, as I often say, augmented reality’s main staying power comes in the fact that it’s a browser5 versus just an application. It lets you look at digital data you can’t see unless you’re using AR technology.
I call this greater world you look at via augmented reality the Outernet. Instead of having to turn on your computer or look down at your phone, when you’re using an AR-based technology, the Outernet is already all around you. You’re used to this idea, for instance, if you’ve got an Xbox and use the Kinect gaming system. I wrote about this in iMedia: Kinect recognizes your movements using AR with video games as you play in your living room. Using something called haptic technology, you don’t even need a controller; using your hands and body, the system recognizes your movements that characters in the game respond to.
There’s another great AR game created by Georgia Tech called ARhrrrr that lets you kill zombies by using your mobile phone and their augmented reality app. By pointing your phone at a physical map that serves as an image marker for the game, the system recognizes your movements as you move around and try to kill zombies. An intriguing feature of the game also comes when you place a physical object on the map, like a Skittles candy. The creatures in the game react to the candy that acts like a bomb in the augmented environment. A consumer application of this technology would be for a coaster at a restaurant: Kids could play an AR game while waiting for a soda.
Technology oftentimes gets adopted faster for the pragmatic uses it offers versus a more gimmicky application. In that sense, one of my favorite augmented reality apps is New York Nearest Subway from a company called acrossair. I’ve lived in the greater New York City area for over twenty years, and I still get lost in SoHo. When you look through your mobile phone screen using New York Nearest Subway, you see a visual icon based on the nearest subway transit station. So if a floating letter “N” is in the right-hand part of your screen, turn to the right and head to that train. It’s a great way to get a sense of how pervasive AR will become in the future with these types of applications.
The possibilities for AR technologies are limitless in the same way that Internet applications know no bounds. The primary reason the technology hasn’t become ubiquitous yet is that it’s only available on smartphones, and things like phone processor speeds have been slow up until the past few years. But those issues have been diminishing as time wears on. There was also an era when physical markers were needed for augmented reality to work (black X-shaped boxes similar to QR codes). Then image recognition came into play, where you can simply hold your phone up to a photo or object that is recognized by the AR tech and projects a digital image over your screen.
Here are some other examples of how augmented reality is currently being used:
· In cars—General Motors created an AR-enabled windshield.6
· In surgery—the Scopis Surgical Navigation System provides guided visualizations.7
· In retail—German AR company Metaio provided Ikea with an AR-enabled catalog.8
· In dating—the short film Sight shows AR and facial recognition used for dating.9
· In art—projection mapping on walls uses AR light shows to entertain.10
The Future and the Financials
“We believe that physical real estate will become a valuable commodity once augmented reality-capable devices are ubiquitous.” Wedge Martin is a cofounder of GeoPapyrus, an AR company that lets you publish and interact with social content such as photos, videos, audio, or websites by browsing physical elements (frames, windows, buildings, books) from the real environment around you. I asked Wedge why he created the company.
Right now, people pay ten thousand dollars a month to advertise on a billboard when they have no idea how many people will look at it. We want to find ways to increase engagement for these types of environments as well as other physical spaces. In essence, if we increase people’s interaction with the physical side of things, we’ll be bringing “social networking” back into the real world—that place where people walk around and get fresh air we feel has been hugely underrated as of late.11
Wedge and GeoPapyrus are addressing an idea I wrote about in Mashable a few years back called Virtual Air Rights (VAR). Here’s how I explained this concept in the article “Who Owns the Advertising Space in an Augmented Reality World?”:
Look up in Times Square and you’ll see the earliest version of a banner ad. Real estate developers pay massive sums to secure air rights for the empty space above buildings. Monetizing by building up (as opposed to out) in crowded areas like Manhattan, they also get to dictate what advertisements appear in the air that they control. Augmented reality (AR) has made it possible for this same paradigm of advertising to exist via your smartphone. Multiple apps feature the ability for ads to appear on your mobile screen as miniature virtual billboards assigned to GPS coordinates.12
Virtual Air Rights will be a fascinating subject in the coming years. As an opt-in experience like the one GeoPapyrus provides, VARs provide a huge economic opportunity. In essence, the Outernet is a blank canvas ready to paint, and advertisements will come to look more like experiences or content than billboards.
Along with opportunity, there’s going to be controversy. After the BP Deepwater Horizon Oil Spill, Mark Skwarek and Joseph Hocking created an app called The Leak in Your Home Town. Just point your phone at the BP logo after downloading the app and it bursts into flame. It’s an amazing bit of geekish parody, but an area of law we’ll hear more about13 in the years to come.
For instance, in the near future you’ll walk into a grocery store wearing an augmented reality-enabled device and only see the brands you want to buy. You’ll input your grocery list and, if you’re a Pepsi fan, you’ll only see their products in the soda aisle. Items from Coke will essentially disappear, appearing as an empty shelf if your device looks in their direction. The physical real estate at the ends of aisles in the grocery store known as endcaps will also be highly desirable virtual real estate.
Our faces are also billboards, and there’s the possibility that people will screen other faces they don’t want to look at utilizing AR. Irritated by Democrats? Using facial recognition, you’ll only see images of a donkey over people’s faces if they vote Democrat. Taken to the extreme, this could even lead to a form of virtual racism if you don’t want to see people of a certain race or background.
In terms of facial recognition and AR, I asked Senator Al Franken his thoughts on the issue, per his role as chairman of the Senate’s Judiciary Committee on Privacy, Technology, and the Law:
Facial recognition technology is a big deal. It can help us catch dangerous criminals and secure sensitive workplaces. But it also places a tremendous amount of power in the hands of governments, companies, and private individuals. The technology already exists that will allow a stranger to identify you, by name, by simply snapping a photo of you on the street. That’s a problem. At a bare minimum, facial recognition should only be deployed commercially—and on a strictly opt-in basis.
What’s worse is that our privacy laws are utterly unprepared for this technology. Over the past few years, Facebook has used the photos posted to its site to create the world’s largest privately held database of faceprints—without people’s permission. If you have a Facebook account and you haven’t clicked a little button on a little menu, chances are that Facebook has created a unique digital model of your face. That file can be used to identify you in any photo taken anywhere, whether or not that’s posted to Facebook. All of this is 100 percent legal under federal law.14
Rather than try to hinder innovation, however, Franken’s primary focus is on helping consumers be informed about new technologies. He also urges companies to be proactive in having consumers choose to opt into a service, rather than having them automatically be signed up for a service without their knowledge where they have to opt out.15 Regarding Google and Glass for this issue, Franken noted:
In the past, Google has taken a thoughtful approach to facial recognition technology. While facial recognition is on by default on Facebook, it’s opt-in on Google+. I get the impression that the Glass team is also taking the privacy concerns around facial recognition quite seriously. My biggest goal is to make sure that our privacy laws keep up with our technology. I want to make sure that all of the benefits that we see from new technology don’t come at the expense of our privacy and personal freedom.16
The legality around issues of augmented reality are rapidly developing. Brian D. Wassom is the Social, Mobile, and Emerging Media Practice Group chair and a partner at Honigman Miller Schwartz and Cohn LLP. He has an excellent blog on AR and legal issues, and recently wrote about best practices for facial recognition privacy.17 I interviewed him for Hacking H(app)iness and asked him if people would be able to block their images in the future from being recorded by Glass or other such devices.
The actual act of capturing someone’s image doesn’t necessarily infringe their publicity rights. Indeed, the creation, reproduction, and distribution of imagery is the subject matter of copyright law, and typically the person who creates the image owns it. What you do with the image—i.e., whether and how you commercially exploit it—determines whether you’re infringing on publicity rights.
That said, I fully expect publicity rights law to evolve in response to situations like this. Precisely because we don’t have a uniform rule describing what this right even is, let alone what it protects or how it can be infringed, it leaves a lot of room for creative interpretation by opportunistic lawyers. For example, although I have yet to see anyone argue this, I wouldn’t be surprised if we soon hear the argument that individuals’ facial features are a part of their identity, and that they can be exploited in various commercial ways, and so any use of unauthorized facial recognition technology is a violation of that person’s publicity rights. Mark my words—that argument is coming.
Of course, publicity rights and copyrights aren’t the only legal issues implicated by surreptitious recording. The scenarios you describe raises questions of privacy and eavesdropping, first and foremost, which also vary by state. Applying these laws will require fact-specific analysis to determine whether there was an expectation of privacy in the specific circumstances involved.18
A number of companies are helping navigate consumers toward a time when augmented reality technology becomes ubiquitous in a privacy-protected environment. One of the best applications for AR in this regard is in the B2B, or business-to-business, environment. A leading company in this field is APX Labs, which is focused on creating AR environments for “deskless workers” using their smart glasses technology. I interviewed Robert Gordon, the company’s chief strategy officer, to ask him more about this trend.
Deskless workers, enabled by smart glasses (augmented reality-enabled glasses), provide employees with a hands-free environment so they can be more efficient. Doctors and nurses in the medical community, workers in transportation and logistics, even people in the entertainment community are better off when utilizing hands-free, wearable computing. Biometrics can also be a component of the smart glasses experience.19
Biometric integration provides a fascinating use of technology that APX has incorporated in the past with military technology. Outfitted with smart glasses that can utilize a technology like Cardiio to read people’s heart rates by measuring changes in the skin tones of their faces, military guards can determine if someone they’re speaking to is getting nervous.
A medical application of this same technology could allow a doctor to look around a waiting room with smart glasses and be alerted to the patient who most quickly needs attention, based on heart rate or other visual characteristics. APX has also created something called See What I See (SWIS) that lets people wearing smart glasses switch views to what another person is looking at. For health workers in the field, this means they could stare at a sick child and get a consultation from an expert physician anywhere around the world. You can see how this experience looks in a consumer environment by watching a video created using Google Glass by a teacher taking his class on a virtual field trip.20
The Vision’s Our Own
Technology isn’t inherently good or evil. People can use a device or platform as they see fit. Augmented reality used for value-added applications is already transforming the world for good, allowing people to easily see information they haven’t seen before in a hands-free environment. Facial recognition technology can also be utilized to great benefit when people knowingly allow their images to be tagged or utilized for medical or work purposes based on mutual consent. Soon biometric and quantified self data will also let people project a visualization of their own health to the world, perhaps publicly showing something like an icon of a sneaker to be identified as a runner in public settings.
Hacking H(app)iness will also mean we can project our emotions to the world, where augmented reality applications could essentially let our faces or clothes be perceived as a sort of mood ring to others. Or our faces could serve as the permissions portal for economic exchange, where selling your data to trusted sources could provide a new source of income. The visions of the future are limitless, and with augmented reality, we’ll get to see them like we never have before.