Designing and Developing for Google Glass (2015)

Part IV. Appendices

A series of helpful appendices with several pro tips are included to give you background information on how to maximize usage of the system, and how users can configure networking, system settings, and manage Glassware registration and installation. We also dive deep into the architecture of several popular Glassware projects and let the teams that produced them tell you in their own words how they put them together and what lessons they learned from doing so.

Appendix A. Glassware Done Right: Case Studies from the Field

We’re making sure you get the most bang for your buck, since you were so diligent to read this far. Now that you know how to effectively Think for Glass, it’s helpful to see some real-life case studies to see how others are applying the mindset to great Glassware projects, for both the Google Mirror API and the GDK. And to do this, we’ve enlisted some very reliable friends of ours.

In the short time that Google Glass has been around, we’ve made countless connections and have logged immeasurable hours in forums, Hangouts, Google+ Communities groups, chat sessions—and, of course, on Glass—discussing with people their passion, ideas, tips, and criticism about the Glass ecosystem. But, more importantly, we’ve become chummy with several cutting-edge developers who have generously and honestly shared—in their own words—their architecture, triumphs (and horror stories) involved with building great wearable software.

We’re in awe of their creativity and we’re proud to let them share their insight and expertise with you here. Let their work inspire you and feel free to reach out to them and inquire about their projects. These efforts aren’t just great Glassware, they’re backed by good people who truly care about our community.

§ Thuuz Sports: Lets you know what’s hot

§ KitchMe: Glassware that’s simply delicious

§ Fancy / ColorSnap: Leveraging colorful queries

§ LynxFit: A personal trainer strapped to your head

§ Genie: The Swiss Army Knife of Glassware

§ Tits & Glass: Takes Glassware into the bedroom…then pivots to keep it there

§ NameTag: Launching headfirst into facial recognition

§ CrowdOptic: Crowdsourced broadcasting

§ Vodo: Real-time collaboration in Google Drive

§ Preview: The latest movie trailers are just a glance away

§ GlassFrogger: Hybrid Glassware using the browser

Thuuz Sports Lets You Know What’s Hot

Winning features: second-screen experience, custom push architecture integration

Long before it debuted as featured Glassware in MyGlass, Thuuz Sports, a popular online platform that tracks the pulse of games for several international leagues, had already earned a reputation as a vital component for sports fans on iOS, Android, and Google TV (Figure A-1). Thuuz not only keeps track of scores for games in the NFL, NBA, MLB, and NHL, as well as English Premiere League, UEFA, tennis, cricket, and rugby, but the service tracks the velocity of those games by measuring each’s “excitement rating,” an intensity index dictated by several factors, including how close the score is as the game nears its end, fan interest, and other situational signals.

Figure A-1. Thuuz’s Glassware makes the perfect second-screen companion

It makes for a perfect companion to stay abreast of how your favorite teams are doing if you’re unable to watch the game…or if you’re totally hardcore and watching three games at the same time and need alerts from other events as they develop (which is awesome). It’s also an indispensable tool for the fantasy sports crowd, alerting team owners if a player they have is having a monster statistical day. “Thuuz Sports for Glass allows you to get timely updates by alerting you to exciting events as they happen, as well as reminders for upcoming games you might have forgotten about,” said Jordan Toor, a software engineer on the Thuuz team. “We show you a rating on a scale of 0–100 of how exciting a game in your favorite league is and a teaser of what’s going on.”

The Glassware is written on top of the Mirror API, generating timeline cards with game data and leveraging speakable text so as not to distract users from the actual action while they’re glued to their seats, watching the actual contest (Figure A-2).

Figure A-2. Web-based configuration is helpful for nearly every sports league

You’re in command of what you track—either specific teams, conferences, or entire leagues via Thuuz’s web-based control panel (Figure A-3).

Thuuz’s Glassware server is written in Python, using Django for database support and template rendering, and talks to Glass through the Google API’s Client Library for Python. Toor explained that integrating the Palo Alto company’s in-place cloud infrastructure for mobile OSes and connected TVs with Glass sync was a breeze. “It was fairly easy to integrate into our push delivery system we wrote in-house,” he noted. “We just added a new device type and then the logic to deliver content via the Mirror API instead of Google Cloud Messaging or Apple Push Notification Service. This is all powered by a workflow system that is distributed between our backend servers.”

Figure A-3. In-game score updates keep you on top of the action

What became challenging, Toor said, was seamlessly translating Thuuz’s signature UI onto the card paradigm. “Adapting our existing Android and iPhone alerts to look great on Glass, primarily due to differences in formatting,” was a particular hurdle. “And also making sure the speakable text sounds good, as well.”

Even for games you may not be watching or teams you may not be following, Thuuz Sports lets you know if something major is going down. Its Glassware is an extension of the experience that fans have relied on over several seasons, masterfully handling the transition from huge monitors and portable displays to Glass prism—without missing a shot, snap, pitch, or goal.

KitchMe’s Glassware Is Simply Delicious

Winning features: web integration, organized bundles

Foodies, rejoice! Whether you’re a connoisseur of culinary creations or just someone with the munchies, or if you’re an expert chef or more the type that just enjoys the art of cooking, KitchMe, by Coupons.com, is Glassware that you can’t live without (Figure A-4). The Glassware doubles as a search engine for its sizable listing of recipes, as well as a step-by-step cooking assistant to aide you as you prepare ingredients and meals (Figure A-5). It’s a multiplatform extension of KitchMe’s browser content with deep social integration that curates recipe listings from all over the Web and aggregates them for easy use in personalized collections you create.

You can also filter recipes by type of diet, course, or style of cuisine, and then send listings to your Glass headset to use while preparing a meal. It’s really tight integration with clever use across platforms. Chief software architect Gene Reddick says early tests of voice search capability, which returns bundles of cards for matching recipe results, which can then be shared and pinned for later reference, have proven to be a huge hit and will be available publicly with a future native version of the service. The instructions can then be read to users as they prepare the dishes.

While the service does rely on the trackpad at the moment, as swipes are needed to advance through a card bundle to move from one step to the next, Reddick is hopeful for native voice control down the road, creating truly hands-free cooking. “If this is not provided, we will consider building something ourselves,” he speculated. “Either voice control or using the camera to recognize gestures or hand motions.”

KitchMe’s Glassware server is an ASP.NET MVC application, written in C# and running on IIS, developed with the Google Mirror API. “We also looked at a pure JavaScript solution, but felt that dealing with OAuth was easier handled on the server,” recounted Reddick. He said migration of the existing app to Glass wasn’t too much of a stretch, noting, “Because most of the KitchMe services were already built and we had working APIs for all the requests we needed for Glass, it didn’t require much work to build out the search and recipe services we required. I cropped and scaled all our images to fit the Glass screen resolution.”

Figure A-4. Recipes are shown beautifully as background images

Figure A-5. Recipe steps are represented as bundles

Reddick says the UI was the biggest challenge in bringing KitchMe to life as a wearable computing service. “In getting the UI right, we initially built quite a few more features into the product and displayed a lot more data on each card. With each iteration, we stripped out more information and removed features,” he said.

“In retrospect this seems obvious, but it really took a while to sink in just how far back from our starting point we had to go to get to a clean usable design. Partially this results from the device itself—the screen resolution and the touch-based interface, but more importantly I think from the unique way Glass is used,” Reddick aded. “Specifically, interacting with a screen directly in front of your eye feels very different than a screen on your phone held at a greater distance or a computer screen. It feels to me like you have less time to dwell on the screen, less time to notice peripheral detail on a screen that is already at the edge of your peripheral vision.

“Presentation and actions needed to be as simple and clear as we could make them.”

Fancy and ColorSnap Leverage Colorful Queries

Winning features: search-by-color

One of the opportunities that Glass makes possible, like other mobile platforms that preceded it, is the ability to create new ways of achieving proven ideas. Savvy developers can execute complex jobs while maintaining the Glass goal of simplicity and user experiences with minimal use. Two pieces of Glassware that brought to light an emerging space that we find absolutely amazing are Fancy and ColorSnap by Sherwin-Williams. Both are extensions for Glass of existing platforms for the social shopping service and the paint company, respectively, using color recognition technology.

They introduce the innovative approach of “search-by-color,” wherein Glass wearers can take a picture and share that image with their Glassware. In Fancy’s case, a bundle of cards representing a result set of items matching the colors within the captured photo are returned with the option to purchase for people that like a consistent theme to their clothing, furniture, or household items.

ColorSnap examines the colors of objects within the photo and inserts a single card on the user’s timeline of the same image with a color swatch overlay for a pallette of similar shades of paint the company carries (Figure A-6).

Figure A-6. ColorSnap matches images with known shades of paint

Fancy even uses a clever trick to let the user know about long-running operations (which are usually on the order of 30 seconds or more)—inserting temporary search cards into a timeline to let the user know the status of things that aren’t immediate like product searches based on color. This is an interesting visual technique to denote status for operations that don’t return immediately (Figure A-7).

Figure A-7. Fancy’s “search in progress” card

Fancy’s Glassware inserts temporary cards onto a user’s timeline to indicate the status of color searches (Figure A-8).

Figure A-8. Matches from Fancy’s product database

Both examples are absolutely stellar ways to integrate ecommerce and introduce new ways to think about search. The approach of basing product search on color isn’t a snarky retort to the fact that facial recognition technology wasn’t allowed early on…it’s just a really sharp, quick, and convenient way of interacting with a system and getting results.

Both epitomize what it means to Think for Glass and are endlessly fun to play with.

LynxFit: A Personal Trainer Strapped to Your Head

Winning features: sensor access, Chromecast integration

Figure A-9. LynxFit is a one-stop wearable workout solution

Your product has gone through several iterations to get where it’s currently at. Share how the feature set has changed as the Glass ecosystem has expanded with new frameworks, hardware, gradual rollout to a larger audience, etc.

The Glass SDK and API, combined with the Google Glass cloud platform made exploring various features easy to prototype. The ease of development made rolling out functionality based on feedback very nimble and seamless.

How would you position LynxFit in the grander scope of fitness apps?

LynxFit is a better way to work out, using wearable computing to bring a more fun, immersive experience that motivates users on their path to health and fitness. Working out hands-free is a new level of independence for the user, bringing their fitness fix wherever they are, whenever they need it, without the need of checking a cellphone or tablet, which completely disrupts the flow.

LynxFit was among the first that called for programmatic access to the Glass sensors. Describe the genesis of your concept to have workout software for wearables, and how this matured when you first tried to implement it as Glassware with the Mirror API.

The first implementation of what was then GlassFit, was pushing content to Glass, and even when it had some animated GIFs (in lieu of actually animating graphics in the display) which were pioneering at the time, it was not as interactive as we’d like it to be. The natural evolution for GlassFit was LynxFit, an experience that is voice coached and immersive counting and cheering for the user as they go through their daily fix of fitness, as we like to call it.

What were some of the challenges you experienced translating your idea from essentially a RESTful service to one that was a fully native experience (Figure A-10)? Have you in any way merged the two frameworks for hybrid functionality?

In a way, with LynxFit we found out that we were pushing the limits of Glass and the GDK as we were stressing the battery and processor through extreme use of video, sensors, and processing, which lead to overheat among other issues. We had to carefully craft the app around those limitations playing with the screen wakelife and pulling some other tricks to make it viable.

We are using a hybrid between GDK and Mirror API as there is a server-side process that at the scheduled time sends a card to the timeline to remind the user of their workout time. From these reminder cards the GDK app can be launched. Also, at the end of each workout we send statistics and PIE (Pace, Intensity, and Endurance indexes) through Mirror API as well.

The web admin interface is extremely slick, but the app still features lots of input controls. How might other Glassware developers use this model to provide user administration for their own projects, rather than directly on the device?

We think of it as driving any complexity into the website (and in the future to the mobile app) versus Google Glass. Glass is designed for microinteractions and even when parts of the LynxFit experience are immersive, we try to go with simpler first. The website on the other hand has all the full-blown possibilities for the user to choose their content and schedule it to show up on Glass.

Figure A-10. Set your exercise schedule and progress

You also broke new ground for being the first Glassware to integrate with Chromecast. Talk about the idea behind this and how the architecture and APIs supported it.

The idea is to use any resource available to get the best possible user experience to the user. On that thought trend, Chromecast became a natural extension of the Glass experience when the user has a TV at reach allowing for gamified shared quantified workouts that were before hard to capture. Because we are having requests to implement the Chromecast experience on its own from mobile devices (with the possibility of adding external sensors, Plantronics, Jarvis, Glass) we are reviewing the architecture in order to give mobile a more central role and sharing most of the core code at the same time.

What have some of the challenges been in developing software that’s whose very existence is contingent on tracking motion?

There’s also a clever gamification angle to the workouts, which makes using the software fun. You’ve picked up on the hint that this might also have bigger implications for doing extended workouts.

What’s on the horizon for LynxFit, as far as applying new ideas, adding new features, maybe premium membership services, integrating other platforms, or even adding new products to the Byte an Atom line?

The play with Byte an Atom Research is to build innovative software (bytes) and kickass hardware (atoms). Right now the primary focus is to enable healthy lifestyles through wearable computers so don’t be surprised to see a Lynx-related hardware device on Kickstarter to help that dream along someday soon.

Genie: The Swiss Army Knife of Glassware

Winning features: beautiful UI, multiple uses, location data

Genie is heralded as one of the most well-designed Glassware services available. The UI is simply beautiful—simple and effective. It really works for the medium. Share your approach for good design strategies when working with wearables.

We at 33Labs have always had a strong focus on usability and user experience. Mauro Canzian, our cofounder, has been influenced by his past studies on architecture and has a special talent to bring together graphic design and UX design to get the best experience possible for the user. Mauro actually designed for Google Glass from South America, months before he ever tried it.

On our Google Glass approach the main things we focus on are:

§ Following Google’s guidelines and standards.

§ Keeping it simple, clean, minimalistic in a way, showing only what the user really needs to see on each microinteraction.

§ Using a visual color-coded reference for icons, bars, etc. to identify each key section.

§ UX using as much as we can on each platform to enrich the experience. In the new GDK development for Genie, we are experimenting with the use of text-to-speech, speech-to-text, two-finger swipes, two-fingers taps, etc.

Early on, Genie made waves by being THE one-stop shopping Glassware service, essentially being the first software suite for Glass. What are you most proud of as far as Genie’s feature set, and what aspect of the initial program was the most challenging to build?

Coming out of the Glass Foundry in February 2013, 33Labs spent numerous brainstorming sessions to decide what to build. Such a new platform, so many options. One of the things that we realized early on is that accessing apps would be complicated on Glass as the number of apps grew. That thought pointed us in the direction to make one app that could be highly usable on a daily basis. Together with a longstanding obsession to have computers, and better yet wearable computers, help us expand and augment our faulty human memory the project was born.

You use Mirror API location data by querying for the user’s position to mark their parking spot for later reference. How did you enjoy working with the location scope?

It was early on and we found out that some features were not working as we expected. Our initial idea was for the user to tap and instantaneously get location information, but we were not getting the geolocation as a custom menu option was being pressed. We had to trick the Mirror API by using voice recognition in order to get a location out of the system. Now, with all the new possibilities brought up with the GDK this is changing.

Figure A-11. Genie has lots of cards for lots of situations

Genie also applies an interesting spin on the traditional concept of identity, using a calling card metaphor within the service to store public contact information about the user. Talk about how this plays into its broader applications.

This was an idea from our friend David Lorenzini. Given that facial recognition was out of the picture early on, why not use simple voice passphrases to identify a user and be able to exchange their public information, a problem most users want resolved yesterday? It was a lot of fun to experiment with this concept. In the end there were not enough Glass users on the streets for this to work and now it’s a feature we’re walking off from at this time.

One of the initial challenges was making sure that Glassware had some sort of persistent presence within the timeline. Without using custom voice commands or menu items, Genie achieved this by pinning itself. However, Explorers discovered that when Glass would do housekeeping and purge old cards, it did so for items left of the home screen, too, which would then force users to have to resubscribe to the Glassware to get it back. How did you resolve this problem from a UI standpoint?

That was an interesting surprise to see that the pinned cards were being recycled. At that point we realized we were pushing the limits and using pinned cards in a way they weren’t intended (isn’t that what developers and users just do?). As a funny anecdote, my friend and Explorer Noble Ackerson told me recently that he was intentionally unpinning and pinning the Genie main menu weekly to refresh it and prevent it from vanishing. In the end the solution came with the “Take a note” voice command addition to the “OK, Glass” main menu, which we are using now.

Genie was created using the Mirror API—any plans to integrate native functionality, either as a rewrite or as a hybrid using the GDK with functionality between the two frameworks?

At this point we’re moving into developing with the GDK as the main platform for Genie and using Mirror API in a hybrid style to allow navigation and sharing lists from the timeline.

The main reason we decided to move to the GDK is access to gestures to allow users to handle lists in an intuitive and faster way. Also, we get better response times when notes need to be remembered or recalled. We still love Mirror API for broadcasting and other features, but we feel its just not agile enough for the main purpose of the application.

In the short term we expect to be able to satisfy one of our users’ main requests, which is to be able to take notes using offline voice recognition. This should be resolved in the level of Glass Android OS mainly as it upgrades to fresher versions of Android, but also it involves syncing offline/online data, which will be a bit challenging.

MiKandi Takes Glassware into the Bedroom…Then Pivots to Keep it There

Winning features: scalability

Jennifer McEwen, cofounder of Seattle-based MiKandi, a leading innovation shop in the adult industry and producer of the largest third-party adult app market for Android, bore the burden in June 2013 of having to be the first Glassware publisher to have her creation banned—mere hours after it went live. Tits & Glass, which lets users browse and vote up sexy imagery, was deemed inappropriate after Google modified its policy governing what type of content would be allowed for its still-in-beta developer platform.

Rather than sit and pout—or worse, just give up—McEwen wasted no time in modifying the service by toning things down considerably and featuring tamer content that didn’t violate the clause prohibiting sexually explicit material. Similarly, she gave members the ability to discretely share their own first-person perspective pictures using Glass, which are then viewed on the Web. And like many of its contemporaries, MiKandi has also had to deal with the challenge of managing software with overwhelming demand against a rate-limited API.

As a result, McEwen and her three-person team have had to be extremely creative and forward-thinking to ensure MiKandi maintains its presence throughout Google’s ecosystem (Figure A-12). Here, McEwen recalls her first-hand experience of putting the system together and keeping it running amid some interesting challenges with user demand and Google terms.

Figure A-12. MiKandi has handled pretty impressive scalability issues gracefully

The MiKandi Experience

First and foremost, Glass is a communications device, so naturally intimate conversations will happen through it. Tits & Glass provides an outlet for adults to share their intimate POV photos with like-minded individuals in a safe, fun, adults-only environment. Originally, the Glassware delivered 18+ content to users but after Google updated their policies to prohibit adult material, we revised it to deliver only SFW content to the device. Users can still upload their adult Glass photos, but they’ll only be viewable on the website for now.

Installation/Configuration

Tits & Glass is very simple to install and use—simply link the app with your Google account and we start sending updates, including an introductory greeting with several of our top cards. At the moment, while Mirror API requests are so scarce, we have considered having a welcome card, which requires confirmation to weed out those users who are linking their Google account even though they don’t have Glass. We essentially hit our API limits every morning before the majority of our users can use the app.

Home-Rolled Development

As very early Mirror API developers, we didn’t have many options available with the Mirror framework, so we had to roll our own. We used our in-house PHP framework, Propellant (similar to CodeIgniter, an open source framework), and added Mirror API modules to our standard library. We backed the Tits & Glass app using Redis. We may well consider open-sourcing a PSR-0 version of our module.

Facilitating Requests: Challenges in Building

One of the biggest challenges is the fact that sharing notifications and other actions do not clear the newly created action card from the timeline. This requires additional API calls to fetch data from the share and clear the extra card.

In addition, the Mirror API effectively requires that developers maintain their own version of app state (which cards are shown, which are hidden, etc). Our first implementation relied on Mirror to keep state, and polled it when updates were required. API limits and expressed best policies on the Mirror API docs pushed us to keep a complete local copy of state of each user’s Glass. For the relatively small number of Glass users at the moment, keeping this extra state is a simple problem (and Redis’ sorted sets worked really, really well to manage this in a simple and efficient manner). As the number of Glass users grows, however, this will present a nontrivial database/datastore problem to Mirror developers.

On another note, the Mirror API Playground as a card previewer is a long way from being suitable for designers to build and thoroughly test layouts and designs. We ended up building a small card previewer that our designers could use to preview card designs using Chrome but pulling from our live database data.

More broadly, we hope a solid toolkit of wrappers and helpers springs up for the Mirror API. For our experienced devs, working with the API at a wire level was difficult, but doable. I think that many people who just want to tinker will be put off by the difficulty of interacting with the service.

Scalability Issues

The limited number of API requests made Tits & Glass hard to develop. On the upside, it made us build a really frugal app. By keeping local state, we managed to avoid a number of API calls. Image size is obviously a big issue, which is why we compressed the Glass version of images using SendFaster’s CRUNCH compression software.

Due to the popularity of Glass, an app can get a huge amount of traffic in a short period of time—a cheap shared hosting account will almost certainly fall over under the load that even a modest Glass app will generate with some modest press. We ended up hosting our Glassware in a load balanced auto-scaling scenario.

Then there’s the issue of Google’s policy changes. When Glass was announced at Google I/O in 2012, the running joke was that it would be perfect for POV adult content. News broke that we were developing an adult Glass app two weeks before we publicly released it. Google quietly updated their terms over the weekend without giving developers notification, then punished us for violating what they considered “clear” policies. These kind of quiet, last-minute policy changes cause the same frustration as the arbitrary policies changes of Apple’s App Store.

Google’s trend toward closing their beta technologies and their recent closed policy toward the delivery of Chrome Apps has raised some alarms. By all accounts, Chrome Apps should be more open than Android, but we see the opposite. It’s too early to tell whether or not these are signs of an overall shift in attitude from Google, but we’re watching them very closely and will continue to push for openness and innovation.

Platform Expansion: Plans on Having an Installed App?

I hope to see Glassware that can facilitate two-way interaction between adults—couples, cam models and fans, or total strangers. In regards to our Glassware in particular, our hands are somewhat tied behind our backs until Google agrees to increase our API calls.

With respect to possibly extending our reach using native code on Glass, we expect that Google will provide a terms of use in the GDK compiler that disallows compilation against the codebase without their permission, as they did with the Chromecast SDK.

Going Forward, Pushing the Envelope

Wearable devices are a brave new world. There’s definitely going to be a learning curve for the community. To compare to literature, in I, Robot Asimov talked about “early days of robots” where robots were required to have human riders, because people were so scared of the new technology that they hobbled it to feel safe. In the same way, the prohibition on apps like Winky and features like facial recognition cannot and will not stand in the long run.

Such natural and obvious uses of the technology may be delayed for a time, but will not be denied.

NameTag Launches Headfirst into Facial Recognition

Winning features: real time facial recognition

Six months after Google announced they’d be shipping Glass without any sort of facial recognition ability, FacialRecognition.com, which promotes itself as having some of the most accurate facial recognition software in the world, announced a beta of NameTag, its native Glass application that lets wearers scan and identify people with the Glass camera by comparing them to known public systems in real time (Figure A-13). It seeks to provide a deeper way of connecting people in the real world by liberating social media from traditional desktop and handheld devices.

Figure A-13. NameTag compares scanned faces to known, public data sources

While not sanctioned by Google, the app does some very remarkable things and hints at what might be a big market for Glass.

What are the core use cases NameTag was built to address?

Since launching the beta program we have had requests from doctors with lots of patients, professors with lots of students, and sufferers of face blindness, to name just a few. Anywhere it is important to remember a correct name in a timely fashion we can provide a useful service. There has been particular interest from the dating industry to provide a safety background check against a database of serious sexual offenders to warn you of potential dangers. Ultimately we would like to see a situation where you can safely meet people based on mutual interests, so if I capture your face I will instantly see things we are both interested in and this enables a much more effortless conversation.

With respect to usability on Google Glass, some of the initial natural limitations with the device are framing a shot, holding the camera still, and shooting a subject at the right angle to legibly scan them. How might NameTag improve upon this?

NameTag is built with some advanced capture features to frame and crop a face automatically when the user positions the viewfinder reticule over someone’s face. The most important thing is to create an intuitive interface that engages the user. Over time the hardware and camera devices in smart glasses will improve and this will allow us to do even more.

What signals does the system look for when not only running tests on 2D digital imagery or printed photos, but actual people in the real world?

People are often moving, standing in poor light, or wearing something on their face, which can frustrate the recognition algorithm. The NameTag app has been extremely successful at overcoming these challenges. We work hard to optimize for both 2D and 3D recognition situations.

NameTag is a fully immersive experience that doesn’t implement the default Glass UX like the timeline or cards. Describe building it as an Android app with the GDK libraries and constructing its user interface for Glass.

It does need some forethought to consider the smaller size of a Google Glass screen. We actually built an initial prototype using a web stack and constrained the proportions to the Google Glass dimensions so we could see how components in the UI would fit together.

Detail your approach for the complex task of not only algorithmically recognizing a user’s face, but performing object detection and being able to discern what a face is from digital imagery being received by a camera in real time.

The high-level process happens in a specific order: analyze the video stream for face objects, when a face is detected frame and crop that face, convert the face to biometric data, run a comparison search for similar faces in our database, and finally return data for top matches in our database. We continue to optimize our techniques to achieve better results for the user.

NameTag is a perfect example of network effects—the more systems the app hooks into, the smarter and more valuable it becomes. In addition to the sources it currently taps, what other systems do you see as key to making the service applicable to a wider range of situations, therefore increasing the accuracy of scans?

NameTag operates as a search engine for people. We spider the web accessing publicly accessible data, then compiling it into useful profiles. Our servers will continue to link to new public records to create the best possible results for our users.

From a big data standpoint, describe the process of indexing information from all the disparate systems you use, and keeping such in the cloud.

Future developers need to consider classic spidering and data collection techniques, partnered with Hadoop-style analysis, and running on one of the major cloud infrastructure systems.

What were some of the challenges in building for Glass? Any difficulties working with the Explorer Edition hardware?

We ran into challenges throughout the development of this app because of the limitations of the Explorer Edition hardware. We expect that Google Glass will continue to evolve with better hardware even before being released as a consumer product. This evolution will allow development teams like ours to continue to push the boundaries of what is possible. In the meantime we will continue to make our software more efficient to address issues like overheating.

It’s important to note that NameTag is just one application of FacialNetwork.com’s innovative approach to working with detection/identification. How might some of your other ideas establish a presence on Glass?

Smart wearable technology will continue to evolve allowing more efficient use of facial recognition. Everything from looking up a business contact at a tradeshow, to a doctor who wants to confirm the identity of the patient they are diagnosing in relation to the chart, or even a university professor who wants to access reports for the student standing in front of them. Many of our beta testers are interested in and are even developing useful applications; we would love to support these apps with the power of our platform and create the maximum value out of this revolutionary technology.

CrowdOptic: Crowdsourced Broadcasting

Winning features: crowdsourced broadcasting model

One of the best things about Glass development is that in a few amazing cases, it’s not just an application that arises. Such is the case of CrowdOptic, which is using the wearable computing revolution to transform broadcasting, effectively building its own platform to do so. Using an innovative approach, the software is able to detect people using devices like Glass and allow them to share their video feeds with each other, or with other broadcast equipment for an incredible experience and powerful analytical tool.

CrowdOptic works bidirectionally—giving users the ability to ingest live content from other wearers, and also for wearers to distribute their video dramatically onto mammoth digital displays, to television, and to the Web. The merged integrations are controlled via a browser-based dashboard. It’s an amazing system that perfectly demonstrates how Glass can be used as a client within other platforms.

Describe your concept of crowdsourcing video content in real time based on proximity and how other Glass users can partake of each other’s feeds.

CrowdOptic’s patented focal clustering technology (U.S. Patent No. 8,527,340) senses where multiple computing devices, including Google Glass and smartphones, are aimed in real time and enables instant, live applications based on where people are looking. CrowdOptic’s Broadcast-In feature leverages this technology to allow fans in luxury suites and select seating to aim their Google Glass at any broadcasting device, or in the same direction as any broadcasting device, including another pair of Google Glass, to inherit the video feed from that device.

How difficult or liberating was applying the particularities of Google Glass to your existing software? What aspect of the hardware were you able to exploit to make your platform shine?

Using Google Glass—a computer that sits in your line of sight—with CrowdOptic’s focal clustering technology is such a natural combination that developing for the Glass platform was a no-brainer from our perspective. Initially, there were challenges, partially due to the challenges Google itself has faced with balancing form factor against things such as processing power, battery life, heat, etc. These considerations placed limits on the device’s capabilities from a hardware perspective and made it more challenging for us initially to do cutting-edge things using Glass. We had to employ various workarounds, including rooting the device, in order to access the sensor data needed to enable our clustering technology.

Later, updates to the Glass OS allowed for easier access to these sensors.

Detail your backend architecture and how you were able to integrate Glass into it.

The CrowdOptic Glass Broadcast Platform networks multiple Glass units to identify the best views out of all possible video feeds. Our platform requires a CrowdOptic Video Proxy Server (VPS), local WiFi network, Internet connectivity to access the CrowdOptic Cluster Detection Server (CDS), and n units of Google Glass running CrowdOptic software (Figure A-14).

The Glass units (any smart devices) broadcast video streams to the VPS, which connects to the CDS to analyze the quality and perspective of each stream. The best video streams are instantly made available for broadcast-in and broadcast-out integrations that can be managed through a web-based dashboard.

Figure A-14. You can’t not be impressed by CrowdOptic’s system architecture

You’ve made big waves with athletic communities. What have the major real-world use cases been so far?

Sports, media, and entertainment are a natural fit for our technology, and adoption of the CrowdOptic platform has rapidly taken off in this vertical. The first professional sports deployment was by the Sacramento Kings, who used our platform to obtain live crowdsourced video content from Glass units and broadcast the footage in real time to their Jumbotron. The Kings were able to offer a one-of-a-kind fan experience using CrowdOptic as a tool to crowdsource live video content directly from Glass, because our technology allowed them to select which video streams to broadcast in real time from among multiple Glass units recording simultaneously.

How do you see CrowdOptic also being used throughout other industries?

CrowdOptic shines in any environment where there is desire for spectators to see something being filmed and broadcast digitally from multiple angles or points of view. The applications are vast. For example, we have been deployed across industries from consumer packaged goods (L’Oreal) to security (Metlife Stadium). In that latter example, the end users were security agents, who were able to instantly view live video footage of the locations of possible incidents. Importantly, CrowdOptic provided both the location of the camera/witness and the location of the video subject (incident location).

Your software not only has great impacts on personal content sharing, but for professional network-level broadcasting. How could your technology be implemented with TV networks?

We are very involved with TV networks and are working with them to enable electronic news gathering and real-time Glass reporting applications. It is also worth mentioning that any video content that is generated at a televised event can be broadcast to television via the CrowdOptic platform dashboard just as easily as it can be sent to a Jumbotron or other in-stadium digital media or to the Web. This is considered phase two by many of our customers.

Some major deployments are in the works. Stay tuned for announcements.

You’ve done a lot of custom development to achieve the experience you’re after. What about the Mirror API and the GDK did you enjoy using, and where did you find them to come up short?

We have begun using the GDK. The Mirror API isn’t as useful when you’re actively running an application like ours. It mainly allows applications to interact with the user when the application isn’t actively running. In the future, I’m sure we’ll use the Mirror API to alert the user when they’ve wandered into an area where our app could be used. The full GDK hasn’t been released yet so all that we’ve used the GDK so far for is to add a voice launcher for our application. That was a big deal because launching the application before that required some third-party stuff to be installed on Glass.

We’re eagerly awaiting the full GDK release to see what functionality it will offer that we can leverage to enhance the capabilities of our system.

Vodo: Real-Time Collaboration for the Enterprise

Winning features: Google Drive integration, real time collaboration

One of the main ways that Glass could be used is as a tool to help you work better. Vodo, which is Allen’s first Glassware service, was created for that very reason. His is also the first sanctioned Glassware to incorporate features from the Google Drive API, letting workers collaborate on cloud documents in real time (Figure A-15).

Figure A-15. Vodo ties the collaborative nature of files in Google Drive with the immediacy of Glass

Describe the application hosting environment on which Vodo was built and its components. How long did it take to code?

The latest version of Vodo runs using Node.js on a Unix platform. We use Google’s library to communicate with both the Mirror API and the Google Drive API and Mongoose to talk to a Mongo DB data store. The core function for it was written in about two days, and based on work done at the Glass Foundry hackathon, which gave us about 24 hours to develop.

Why was Node.js chosen as Vodo’s platform? Any advantages to using it over other server-side frameworks?

Node.js has a reputation of very fast threadless operation, which seemed like it would scale quite well at high load levels. This seemed like an important requirement for Glassware as we got started, and has proven to be very useful in the long-term. Some of the design constraints of Node.js and Mongo DB prompted us to think about good approaches for general Glassware design in any language.

What were most difficult parts of making Google Apps Script talk to Glass?

When the project started at the hackathon, there was no Glass support for Google Apps Script and there were some issues with processing HTTP POST bodies. Since then, both of these issues have been dealt with by the Apps Script team.

The service has always been Mirror API Glassware. At any point did you consider reengineering it as a GDK app, or are there plans to merge its features as a hybrid service between the two frameworks?

It was considered a couple of times, but there are no real advantages to doing so right now. The Mirror API manages all of the network issues that would otherwise occupy much of the code and probably cause most of the problems a user might experience. Leveraging all the resources that Mirror gives enables us to focus on presenting data in the form most suited for people to use with Glass and not having to worry about features that should “just work.”

How important was it to keep all admin features web-based and not within the Glassware itself?

Using the Web has allowed our team to make the configuration flexible and let users tailor Vodo to their specific needs. Right now, the configuration is fairly simple, mostly allowing users to pick what Google Drive folder they want to work with. But even that would be overwhelming to work with on Glass. It lets us present a very simple Glass interface that is perfectly suited to the concepts of creating or receiving content, while leaving the more complex options to be made through the Web.

What have been the use cases that you’ve seen that have impressed you most?

What has been most impressive are how many different ways people use Vodo as part of their regular workflow. Many people use it to edit task lists on the desktop, where it is easier to do so, but view the contents from Glass. We have a person who produces videos use it to get their recordings off Glass and quickly into their production system.

Most exciting, however, is the building inspector who uses Glass with Vodo to capture pictures and notes about his site surveys; when he returns back to the office, all his field notes and pictures are saved in one folder and he can quickly cut and paste them into his formal report using the documents format in Google Drive. Cases like these prove that Glass is ready for business today, and will only improve as we identify other file types that people use and the best way to represent them on Glass.

Preview: The Latest Movie Trailers Are Just a Glance Away

Winning features: image recognition, real-time video stream processing

Glass goes Hollywood! One of the major areas that Glass and Glassware are going to make a major impact is in media/entertainment. Software engineer Takahiro Horikawa came up with the notion of letting a Glass wearer gaze at a movie poster, recognizing what film the marquee was promoting, and playing back its trailer video. The result is Preview, a GDK application that uses real-time processing of the camera’s inbound video feed and cloud-based image recognition algorithms to trigger playback of the official promotional clip (Figure A-16). It’s so obvious, logical, and simple, and the implementation is brilliant.

And it’s endlessly fun to use when you’re at the moviehouse, at your desktop, or out on the town and want to see what all the buzz is about.

Figure A-16. Intelligent object recognition is at the heart of Preview

Preview’s typical use case is fairly well defined, but talk about the various ways users have told you they’re putting the app into action in live settings.

The use case that I and most people are thinking is going to the theater and looking at posters to see the movie trailers to decide which movie to see. While some people have already used it at the real theater, a lot of users tried the app in front of their PC or mobile device and just enjoy how our image recognition works so well. I heard some feature requests: some users said it would be great if Preview notifies them about popular movies in a nearby theater. It will be possibly supported in the future, but I want to focus on image recognition and visual input first, which I believe will be very common sometime soon.

Another interesting feature request is the ability to screencast a trailer or send a link to it from Glass to a nearby smartphone. That way instead of just one person with Glass figuring out which movie to watch, a group could do the same. I think this is a good idea and will consider implementing it…though I still need to figure out the feasibility of it.

Describe Preview’s data store hierarchy—where’s the content coming from and what’s the size of the corpus that you’re searching against?

Our backend collects movie data from IMDb, Rotten Tomatoes, and TasteKid using web APIs and integrates them. Preview is currently focusing on the movies that are playing in the theater right now, so the number of movies is several hundred. But the database will grow to accommodate new movies as they come because the database exists on the server.

How does data flow when a Glass wearer begins looking at a movie poster? What’s the distribution of cloud versus local computation?

Preview recognizes the movie poster by comparing it to images in the cloud. Preview starts the camera, takes a photo, and sends it to our server to recognize it. This process happens several times until it succeeds. There are some reasons for doing it in the cloud—first, with cloud recognition, we can save a lot of CPU/memory/disk resources on Glass. Second, we can accommodate new movies as they come.

For playing a trailer, our system simply returns the YouTube Video ID associated with the movie’s title. After receiving that ID, Preview streams the clip with the built-in Glass media player.

The processing model—analyzing the camera’s inbound video stream—pretty much mandates that it needs to be done as a native application. In the interest of having the app consume less battery, do you have any desire to try to implement this as a Mirror API service?

We are just sending a still image for image recognition, so technically it can be replaced with Mirror API. However I don’t want to implement Preview as a Mirror API service because of our UX design. Preview is designed for users to access movie trailers easily, quickly, and completely hands-free. If we employ the Mirror API, it may require some touch gestures, which we don’t want to do.

Preview also does not consume the battery very much, because Preview is designed for short lifetime—it exits after playing a trailer, so the lifetime is at most two minutes or so; and all recognition is running server side, not on Glass. Hence there would be no strong battery life benefit by using Mirror.

What’s the secret sauce to how Preview actually recognizes an object as a movie poster, and then compares it against your database?

Our backend preprocesses all movie poster images in advance, stores “features” for every image, and runs through them against the camera picture at runtime. The feature descriptor that I am using is robust in terms of allowing slight image rotation, scale changes, brightness changes, and blurring, which makes it possible to recognize a movie poster in any situation.

What are some of the challenges of detecting digital images on a computer screen or mobile device as opposed to a real-life poster in a marquee?

There is not a very big difference between them, but if any, the size of movie poster is likely to be different—when people are looking at the movie poster digitally, its size is likely to be small, but in real life, it is probably bigger (and sometimes cropped). Small images are not good for recognition accuracy, so we still need to come up with a way to educate users to zoom into a movie poster when they look at it.

In testing Preview, we scanned the movie poster for Need for Speed, but I hilariously got the trailer for the Need for Speed video game. What other funny outliers have you noted due to matching ambiguities?

Oh, thank you for reporting a bug! :) Yes, this kind of issue could happen. As I said, we collect data from several web resources, and since the data is somehow generated by machine, the association between movie title and YouTube Video ID is sometimes wrong. Another funny case we noticed early after launching was that when a user tried to recognize the Bollywood film Queen, they would get the a live performance from the band Queen. We fixed that one!

GlassFrogger: Hybrid Glassware Using the Browser

Winning features: use of remote HTTP server to run a game, sensor access via JavaScript

This notion of web applications working with Glass is powerful. How can your site integrate with Glass for sensor access or make use of JavaScript? Hybrid Glassware provides some suggestions at what might be possible.

As a testament to its potential for Glassware development, the team behind GlassFrogger (Figure A-17), which won 1st prize in the 2013 Breaking Glass hackathon in San Francisco, used a Dart codebase to create an homage to the classic 8-bit arcade game where an ambitious amphibian attempts to cross a busy highway to reach the safety of his beloved lily pad. The app is hybrid Glassware, using the Mirror API to insert a static card into the user’s timeline as a splash screen, with a web app running in the Glass browser as the “native” end.

It avoids the need for time-consuming Android development, using the browser as a stage for a game running on an HTTP server. JavaScript listeners measure the Glass accelerometer to handle a player’s real-life movement to dictate how the character advances on the screen. As a Glass wearer jumps up and down, the frog animates fluidly. The game’s impressively responsive and really fun to play.

And perhaps most impressively, this app was built in less than 48 hours.

The entire game’s code is open source and available on GitHub so you can study the implementation for yourself.

Figure A-17. GlassFrogger’s program flow

This is architecture you may want to consider in your own projects, especially if your strong suit is in web development and not coding native apps in Java. You can write a complex web app and run it through the browser—just be mindful of performance issues due to JavaScript execution and graphics, the battery charge consumed by the projector being on for extended periods of time, and how the attention demands deviate from the microinteractions. It’s worth looking into.