What is augmented reality? Only the technology that is getting Mark Zuckerberg and Tim Cook super stoked, that's what. Like virtual reality, AR is a technology that's been around for years in some form or another but it's only recently taken off. Thanks to powerful smartphone hardware and software, AR can be taken out of your pocket and used in the blink of an eye after downloading an app.
It sounds a lot like VR, which can also be used with a mobile device, but the two are very different experiences and technologies. VR requires immersion in a virtual environment without sensory knowledge of the world around you. AR instead overlays virtual 3D objects over the real world to create a sense that they're in front of you, requiring you to be aware of your surroundings. The picture you see with your eyes is a composite of what is really around you and what the app/game/experience adds to it.
To make it even more confusing, there are also several ways AR can be produced. Marker based implementation uses QR/2D codes that can be read by mobile devices and markerless AR (or location based AR/position based AR) which uses GPS. There also mobile apps with software that can 'read' images to show you AR.
Heads up displays (HUD) and head mounted displays (HMD) are hardware focused, and have seen the greatest amount of interest recently thanks to big names throwing their hats into the AR ring (think Microsoft and Google).
The term mixed reality also pops up for certain AR HMDs as well. MR is virtual reality overlaid on the real world where the goggle portion is semitransparent, allowing you to see what's actually around you. There are those who are especially picky about separating the three terms - AR, VR, MR - but generally, the industry isn't too worried about fretting over small things like names. MR is simply another feature of AR technology.
Some popular examples of AR come from films like Iron Man or Minority Report where characters interact with displays of data or objects that are projected onto surfaces or simply pulled up in front of them.
Of course, it's all CG in the movies. In reality, it will take some time to arrive at that level of 'true' AR. The tech behind it is much more complicated to create and has been greatly simplified for brevity. Let's just say that right now, we're really left with smartphone apps and the promise of heavy duty head mounted displays with the hope that the latter won't end up costing an arm and a leg.
A brief history of AR
To understand where AR is going and to see its potential, we have to take a quick look at the history of the tech and ask why it failed to take off the first time around. The trajectory closely resembles VR's origin story and starts earlier than you'd expect. The first AR device was born in the late 60s. Called the Sword of Damocles and created by Ivan Sutherland at Harvard University, the contraption basically superimposed a geometric grid over the user's view of a room. It was huge, unwieldy and impractical beyond the experiment but showed that despite being the early days of computer sciences, AR could be done.
Research into AR continued well into the 90s where it really flourished and much experimentation was done in the military and in space programs like NASA. It wasn't until phones could handle the processing power required by AR that it started to really show up in the consumer space with QR code scanning and apps.
In the 2000s, German researchers Daniel Wagner and Dieter Schmalstieg, now lauded as pioneers in augmented reality, were the first to create a framework to run AR on a mobile device. With the new fangled 'smartphone' devices growing in popularity and advancing in hardware and software, AR was becoming the latest fad for companies to combine with products.
Ralph Osterhout, CEO and founder of the Osterhout Design Group (ODG), has been in the business of AR since the early days and has seen it all in flux. He notes that Apple's iPhones really put AR on the map - mostly because the components were finally of the right quality.
AR all the time
"Until the iPhone really came out in 2007, there was no phenomenally high volume in cellular devices, or what you call a true smartphone. The more simplistic, cute very functional phones like the analog (were around), then it went digital - which Nokia did a very good job at making consumer phones - but until you got into a real smartphone with complexity, that has a lot more processing power and more memory, and everything - you didn't have an ability to have a market that would drive the component cost down.
"Take eight years ago, how much was 4GB of memory? How much would a dual-core or quad core processor cost? You're talking big money...overall component costs wouldn't allow you to come out with a consumer product."
But even though the power was there, AR still didn't really take off. It was all too contrived on the marketing side and components were still too expensive to pursue a fully fledged head mounted display.
Ori Inbar, who heads up popular AR conference Augmented World Expo and has founded various AR ventures thinks the early days of AR were actually too focused on consumers. "Most of the focus back then was advertising, on consumer oriented gimmicks and gaming," he says. "If we look at the initial areas when the iPhone and Android phones came out, that time was the first time you could experience AR with a mass market device."
None of it really stuck and AR for the people generally died down. However the enterprise side began to stick its toes in the AR waters and that's why certain enterprise focused AR companies have been so successful for the past few years.
What AR is now
Cut to 2014 when Google Glass arrived on the scene. All of a sudden, AR was relevant again for the masses. But just as Osterhout noted, the price wasn't right. Not only that, people weren't comfortable with the Glass features and those donning the so called smart glasses were deemed 'Glassholes.' Again, another bout of AR fever quickly died.
A year later, Microsoft HoloLens was announced, a head mounted display that captured the world's attention once more but one that suffered from a small field of view. You can currently pay $3,000 for a developer version but the next, perhaps consumer version isn't due until 2019.
For the following two to three years, companies like Meta, ODG and Epson have also continued to refine their AR glasses hardware. There have been some interesting uses and apps, not to mention the shrinking of the hardware to glasses-like form factors. Still, none have managed to cut through to mainstream consciousness in the same way as HoloLens.
In 2017, there are two big AR trends to keep an eye on. We did expect a Magic Leap device to be the big story of 2017 but this has still yet to materialise, despite some promising signs that a launch could finally be on the cards this year. It's a Florida startup, with $1.5 billion in investment from Google, Alibaba, Warner Bros, Qualcomm & more, that has been building a platform and some sort of goggles or glasses since 2015. When launched, it could be an iconic new device or simply a disappointment after years of hype.
Magic Leap, and Microsoft, actually refer to their technology as mixed reality, that subset of augmented reality we mentioned earlier. Opinions vary as to the actual definition but it tends to refer to next-gen AR in which the virtual objects/characters are aware of and interact with real physical environments.
The second big trend is what the social networks are doing. Both Facebook and SnapChat are pushing AR filters, on smartphones, as big deals for 2017. In future, we expect to see AR smartglasses from both these companies but for now, Facebook's Camera Effects Platform and SnapChat's World Lenses need to get developers on board and people regularly using the features.
The hardware is, of course, only half the story. Gamifying AR failed before, but it seems that with the right game, AR could work for the public. HoloLens is trying to garner attention with Minecraft, and various other titles, which definitely has some people readying themselves for AR. So too Magic Leap has been bringing in special effects companies like Weta Digital and Industrial Light & Magic at a fairly early stage to get some sort of entertainment experience in the works pre-launch.
As others have pointed out, now that Facebook has come out with a strong, initial platform, we really need to see a strong AR content creation studio, whether that's stickers, characters, games or film-like experiences. Studios like Within, Penrose & Baobab and (the about to close) Oculus Story Studio can provide a template, though augmented reality will, of course, develop its own norms and ways of working. Either way, now is the time to start thinking about AR applications outside the realms of industry, healthcare, education and enterprise.
What's next for AR
What's next for AR? In a word, Apple. If all the rumours around job hires, patents and well, Tim Cook's actual words are to be believed we should be seeing a big AR push with the next iPhone in September. It has also been confirmed by anonymous sources that Apple has an Augmented Reality Group which is working on some kind of glasses or goggles. It's Apple so glasses then and the device is slated for a 2018 launch.
As ever, if/when Apple gets involved it will change everything for AR and MR. One question is whether say Facebook's AR filters and content would be compatible with an Apple AR device or if we'd all have to choose as with smartphone apps etc. What's also interesting is that with smartphones, smartwatches and a bunch of other categories, Apple has been late to the game. But if it does launch its AR glasses next year, it will be one of the first to release a consumer product in this area.
Alongside Apple, there is of course everyone mentioned above to keep an eye on: Facebook, SnapChat, Microsoft and Magic Leap. We'd add to that list Avegant, the makers of the Glyph, who are showing off a mixed reality helmet prototype that looks to compete with Magic Leap right now. Very promising.
For AR to truly succeed, it will take vision, patience, tech innovation and resources. Its counterpart VR is already starting to seep into everyday life after its own long, storied past but it still needs improvements to get to a second generation. Once AR hardware arrives, though, it won't be the blip that Google Glass ultimately was.
Sophisticated AR requires more finesse on the part of technology. The sensors, tracking and visuals (light, resolution, depth, placement) must match up to what you're seeing in front of you and your interactions with it can't stutter. AR also requires lighter, untethered, more glasses-like devices than VR because you could be using it for day-to-day activities.
Simply put, the applications for AR are pretty much endless. Inbar says that enterprise is very much where AR has lived for the past few years but the consumer side of things will ultimately catch up: "Translation of text in the real world, whether it's for travel or try before you buy like furniture to see how it looks in your living room - there's applications that let you do it in a really interactive way."
Again, AR will have to remain, in some configuration, on the mobile platform, which makes sense considering everyone has a smartphone so accessibility isn't a problem. They're also capable of handling the AR apps - at least for now. But the benefits of a heads-up display that won't embarrass you in the street are clear. In fact, in the search for the next interface, AR may replace the screen as we know it altogether, aided by gaze tracking, gesture interactions and voice controls.
Whatever the use and form augmented reality takes - headsets, smartglasses, apps, even contact lenses - it's clear there's a real future in it. Now the race begins to see whose reality we'll be living in.