On February 22, the New York Times reported that Google plans to unveil augmented reality (“AR”) glasses by the end of the year. Despite a healthy buzz about the new product on the Internet, solid information about the capabilities of the potentially groundbreaking device has been hard to come by. In an attempt to get more information about the product, I conducted a search of the U.S.P.T.O.’s published patent applications. Below are my findings. In the spirit of full disclosure, I am unable to say with certainty that the AR glasses described here are, in fact, associated with Google.
The patent applications below, have all been assigned to the “Osterhout Group Inc.” The Osterhout Group is a San Francisco based design house that “specializes in integrating new and emerging technologies to create disruptive product lines in Consumer, Industrial, Defense and Toy Markets.” See (www.osterhoutgroup.com). Obviously, the assertion that the AR glasses described here describe the Google product, depends on a connection between Google and the Osterhout Group. It should also be noted that a patent application is a technical disclosure not a product spec sheet. Therefore it is likely that the released product will not contain every feature described in the patent application.
Features of The Glasses:
The essential feature of augmented reality is the ability to project an image onto a transparent or translucent lens such that users can see their environment overlaid with a computer generated image. The augmented reality glasses described here project a high-resolution image using micro-projectors mounted just behind the hinge on the glasses. The projectors emit light through a series of optics and onto a 1280 x 720p full-color display.
The glasses have a forward facing camera capable of capturing video from the user’s point-of-view and projecting it into the user’s display. In order to capture and present high quality images to the wearer, the glasses incorporate various forms of image correction and stabilization, including optical and digital techniques. In addition, the glasses may have a rear-facing camera and integrated, removable ear-buds.
The glasses have wireless connectivity and will include a navigation system such as a GPS processor, an accelerometer, a gyroscope, an altimeter, an inclinometer, a speedometer, a laser range finder and magnetometer.
In some versions the AR glasses may include sensor arrays capable of measuring concentrations of gasses, toxins or other compounds of interest in the environment. Health sensor arrays may also be included, such as sensors capable of monitoring the electrical activity of the heart. Cataloging data from the AR glasses’ would allow for later analysis for tracking or trending purposes.
The AR glasses as described, are capable of receiving content from an abundance of sources such as an Internet browser, local businesses, the onboard camera or other sensors as well as from cellular or other wireless networks. The inventors described specifics of a preferred embodiment of the AR glasses as containing OMAP4 dual 1GHz processor cores, an onboad camera capable of capturing 1.3MP color video, a communications system-on-a-chip (SOC) capable of operating with WLAN, Bluetooth, GPS receiver or FM radio, a micro projector illumination engine and a 3.6 VDC litium ion battery. The weight of the device is listed as 3.75 Oz (95g).
Operation and Uses:
The AR glasses have various “modes” of operation such as Tourist Mode, Educational Mode, Internet Mode, TV Mode, Gaming Mode, Exercise Mode and the like. The AR glasses may contain suitable software such that the eyepiece will automatically display certain content based upon sensing some visual or other environmental cue such as a building or painting. The applications describe integration of the “SRE engine” which is able to identify objects such as, architecture, natural structures, pictures, and rooms.
The AR glasses will have substantial uses for social networking. For example, a Wi-Fi enabled user may opt-in to a geo-location application in which the application would identify other users and send information such as pictures, Facebook posts, “tweets,” “blips” or other greetings into the view of nearby users. Additionally, users may leave notes, comments, reviews, or other messages, even “virtual graffiti” viewable to other augmented reality users.
Gaming applications run on an AR platform will be a mobile, online, gaming experience that integrates software-generated as well as real-world content. Games will take advantage of the location of each player, for example as in a game where points are awarded when a player reaches a particular location. Clues or other information could also be communicated to a player at given GPS coordinates as part of a game.
The glasses described will also be able to enhance the visual capabilities of the wearer. With night vision powered by black silicone short wave infrared (SWIR), the user will be able to see in the dark as well as through, fog, smoke and particulate matter. Lest the user become bored with simply augmenting his own point-of-view, the video feed may come from other sources such as a panoramic camera, a camera mounted on a vehicle, drone or helicopter.
Of course the glasses carry out many of the functions typically associated with traditional eye wear such as providing for vision correction, “sunglass mode” where the lenses appear somewhat dark and “blackout mode” where the lenses become opaque. To achieve this, the inventors contemplate using eclectrochromic or suspended particle materials that reversibly change appearance in response to an applied electrical charge.
While traditional control means such as buttons, dials and the like may integrated into the AR glasses themselves, traditional controls will be secondary to more innovative control mechanisms. For example, users of the AR glasses will control the applications running on the device with hand, finger, head or eye movements, or tapping against a hard surface. Other control mechanisms may include sensors worn on the user’s body allowing the wearer to transmit complex body movements to the glasses. The use of audio commands may also be enabled.
One innovative input technique described for the AR glasses is the “virtual keyboard.” The virtual keyboard is projected, in a fixed position, in the user’s display. The user interacts with the virtual keyboard normally and the AG glasses translate the user’s motion into alpha-numeric input. The virtual keyboard may be projected onto a desk, wall or the user’s arm or hand.
The AR glasses may also use a cursor, controlled through a virtual computer mouse worn on the wrist of the user or a wireless trackpad, handheld remote, or a remote mounted on a firearm. Other control means include the use of facial movements such as tensing a facial muscle or clicking one’s teeth. The glasses may also be able to track and utilize the user’s gaze to make selections.
Advertising and Surveillance:
Since it’s Google (maybe), it should be no surprise that the “[a]pplications of the present disclosure may provide a way for advertisements to be delivered to the wearer.” For example, ads may be displayed to users who are going about their day, browsing the Internet, conducting a web search, or walking through a store. Advertisers may direct their offerings to people who are within a certain geographic area, or other selection criteria, to better target customers.
While traditional advertising techniques such as banner ads and text-based ads may be employed, other more innovative advertising techniques are enabled. For example an advertisement such as a the coffee cup seen in figure 19, is associated with a particular building and appears large and prominent in the user’s field of vision. In addition to visual ads, the system contemplates the use of audio messages to communicate advertising content. For example, an advertisement for beer may include an image of a beer bottle as well as the sound of a bottle being opened and beer being poured into a glass.
A slightly concerning capability of the AR glasses comes in the field of surveillance. The inventors describe that the AR glasses may be equipped with powerful biometric data collection abilities. Specifically, they describe the use of audio recording combined with speech recognition capabilities to identify individuals based on their speech. Also, the optical features of the AR glasses may allow for the identification of people based on captured images of their faces or the iris of their eye.
Whether the AR glasses described here are actually Google’s product, the world of augmented reality seems to be upon us. It remains to be seen how many of the features described here will show up in the consumer version of the glasses. Personally, I eagerly look forward to the release of Google’s first augmented reality offering and I am excited to see how AR works.