Everything you need to know about augmented reality: Then, now & next

From HoloLens to Pokémon Go, we look at the past, present and future of AR
Everything you need to know about AR

What is augmented reality? Like virtual reality, AR is a technology that's been around for years in some form or another but it's only recently taken off. Thanks to powerful smartphone hardware and software, AR can be taken out of your pocket and used in the blink of an eye after downloading an app.

It sounds a lot like VR, which can also be used with a mobile device, but the two are very different experiences and technologies. VR requires immersion in a virtual environment without sensory knowledge of the world around you. AR instead overlays virtual 3D objects over the real world to create a sense that they're in front of you, requiring you to be aware of your surroundings.

To make it even more confusing, there are also several ways AR can be produced. Marker based implementation uses QR/2D codes that can be read by mobile devices and markerless AR (or location based AR/position based AR) which uses GPS. There also mobile apps with software that can 'read' images to show you AR.

Heads up displays (HUD) and head mounted displays (HMD) are hardware focused, and have seen the greatest amount of interest recently thanks to big names throwing their hats into the AR ring (think Microsoft and Google).

The term mixed reality also pops up for certain AR HMDs as well. MR is virtual reality overlaid on the real world where the goggle portion is semitransparent, allowing you to see what's actually around you. There are those who are especially picky about separating the three terms - AR, VR, MR - but generally, the industry isn't too worried about fretting over small things like names. MR is simply another feature of AR technology.

Some popular examples of AR come from films like Iron Man or Minority Report where characters interact with displays of data or objects that are projected onto surfaces or simply pulled up in front of them.

Of course, it's all CG in the movies. In reality, it will take some time to arrive at that level of 'true' AR. The tech behind it is much more complicated to create and has been greatly simplified for brevity. Let's just say that right now, we're really left with smartphone apps and the promise of heavy duty head mounted displays with the hope that the latter won't end up costing an arm and a leg.

A brief history of AR

To understand where AR is going and to see its potential, we have to take a quick look at the history of the tech and ask why it failed to take off the first time around. The trajectory closely resembles VR's origin story and starts earlier than you'd expect. The first AR device was born in the late 60s. Called the Sword of Damocles and created by Ivan Sutherland at Harvard University, the contraption basically superimposed a geometric grid over the user's view of a room. It was huge, unwieldy and impractical beyond the experiment but showed that despite being the early days of computer sciences, AR could be done.

Research into AR continued well into the 90s where it really flourished and much experimentation was done in the military and in space programs like NASA. It wasn't until phones could handle the processing power required by AR that it started to really show up in the consumer space with QR code scanning and apps.

In the 2000s, German researchers Daniel Wagner and Dieter Schmalstieg, now lauded as pioneers in augmented reality, were the first to create a framework to run AR on a mobile device. With the new fangled 'smartphone' devices growing in popularity and advancing in hardware and software, AR was becoming the latest fad for companies to combine with products.

Ralph Osterhout, CEO and founder of the Osterhout Design Group, has been in the business of AR since the early days and has seen it all in flux. He notes that Apple's iPhones really put AR on the map - mostly because the components were finally of the right quality.

"Until the iPhone really came out in 2007, there was no phenomenally high volume in cellular devices, or what you call a true smartphone. The more simplistic, cute very functional phones like the analog (were around), then it went digital - which Nokia did a very good job at making consumer phones - but until you got into a real smartphone with complexity, that has a lot more processing power and more memory, and everything - you didn't have an ability to have a market that would drive the component cost down.

"Take eight years ago, how much was 4GB of memory? How much would a dual-core or quad core processor cost? You're talking big money...overall component costs wouldn't allow you to come out with a consumer product."

Read next: Best smartglasses of 2016

But even though the power was there, AR still didn't really take off. It was all too contrived on the marketing side and components were still too expensive to pursue a fully fledged head mounted display.

Ori Inbar, who heads up popular AR conference Augmented World Expo and has founded various AR ventures thinks the early days of AR were actually too focused on consumers.

"Most of the focus back then was advertising, on consumer oriented gimmicks and gaming," he says. "If we look at the initial areas when the iPhone and Android phones came out, that time was the first time you could experience AR with a mass market device."

None of it really stuck and AR for the people generally died down. However the enterprise side began to stick its toes in the AR waters and Inbar says that's why certain enterprise focused AR companies are so successful today.

What AR is now

Cut to 2014 when Google Glass arrived on the scene. All of a sudden, AR was relevant again for the masses. But just as Osterhout noted, the price wasn't right. Not only that, people weren't comfortable with the Glass features and those donning the so called smart glasses were deemed 'Glassholes.' Again, another bout of AR fever quickly died.

A year later, Microsoft HoloLens announces a head mounted display that captured the world's attention once more. During all this time though, other companies like Meta, Epson and ODG have been working on their own pieces of hardware too.

Magic Leap is also one of the groups that's been quietly tinkering away on its own AR device - so quietly that no one knows what they're doing. Aside from short 'demo' videos of the software, we have no clue what the hardware is or what it's actually capable of doing. Generally, it's more of a mixed reality concept - where elements of AR and VR (also what Microsoft and Meta are doing) combine to give you a taste of both tech. To date, Magic Leap has generated $1.4 billion from investors and that's all without showing the public any alpha or beta devices. It's safe to say, this Florida based company has got something up its sleeves that will hopefully show us what AR (or MR) can really do.

With the big names officially in the running and smaller names making a big splash, augmented reality looks like it'll stick around. Recent events around games make this especially apparent.

Gamifying AR failed before, but it seems that with the right game, AR will work for the public. HoloLens is trying to garner attention with Minecraft, and various other titles, which definitely has some people readying themselves for AR. But it's still not ready for prime time and at $3,000 a pop for just a developer kit, we shudder at the thought of a consumer price.

What's really prepared and excited the general public about AR is a little game called Pokémon Go. It's a form of AR that uses a mobile device and good 'ol location based AR (remember that from earlier?) to zone in on you causing your favorite creatures to pop up.

While it's a simpler form of AR, the minds behind the more sophisticated tech absolutely love the attention Pokémon Go is creating for AR as a whole.

ODG vice president Nima Shams on the game: "It's really brought the conversation of AR forward. It's really moved that conversation of 'is AR going to happen in the future' to something that's actually happening right now."

Soren Harner, chief product officer at Meta, also told us that he could have conversations with people he didn't think would know much about AR thanks to the popular game. "It drives awareness and it gives people a glimpse of why they want it (AR)."

So it seems right now, AR is staging a major comeback in terms of mobile apps but there's still a while to go for the consumer HMDs. Even enterprise-grade HMDs are still being tested in the field.

What's next for AR

For AR to truly succeed, it will take a great amount of patience. Its counterpart VR is already starting to seep into everyday life after its own long, storied past but still needs improvements here and there. Luckily, like Pokémon Go, VR is paving a good path for people to walk on. Once AR hardware arrives, it won't be the blip that Google Glass ultimately was.

Still, sophisticated AR requires more finesse on the part of technology. The sensors, tracking and visuals must match up to what you're seeing in front of you and your interactions with it can't stutter. AR also requires lighter, untethered devices than VR because you could be using it for day-to-day activities.

Simply put, the applications for AR are pretty much endless. Inbar says that enterprise is very much where AR lives and the consumer side of things will ultimately catch up.

"Translation of text in the real world, whether it's for travel or try before you buy like furniture to see how it looks in your living room - there's applications that let you do it in a really interactive way. That's really happening now but it's not as big as what's happening in enterprise.

"When you look at smart glasses and AR, like Meta for example, we're probably going to need another debate on whether it will be 18 months or three years until all these devices will shape up and be ready for the consumer space. The design, comfort, fashion - all these different aspects are going to make a big return eventually."

Again, AR will have to remain on the mobile platform, which makes sense considering everyone has a smartphone so accessibility isn't a problem. They're also capable of handling the AR apps - at least for now. Shams believes that AR on mobile, in reference to Pokémon Go, is mostly a taster for what's to come. Of course being part of a smart glasses company, he says ODG's devices are the future - though from his musings, we can glean that smart glasses do have a lot of potential compared to head mounted displays.

Relevant read: 8 augmented reality experiences to look forward to

"This is just the tip of the iceberg. Because of the fast adoption (of Pokémon Go) and people learning about it (AR), it's really good to educate the public on what AR can be.

"There's still limitations with sunlight readable screens, holding a device in your hands, looking down at your device versus actually being immersed in it.

"This is 0.1 of what an AR game or interaction can potentially be. I think what people are going to quickly realize is that phones may not be the best medium or platform to have such an experience. Head worn devices can be pass through, see through. There's body position and independence all of which are the biggest differentiators between us and some of the others (headset makers) we're not tethered, we're extremely mobile and we have cinema quality displays."

Meta, on the other hand thinks its head mounted display is the future of computer screens and that AR will in fact replace the screen altogether. Harner says that hopefully by the time the next headset is ready, everyone in the office will be working from Meta headsets.

Inbar agrees but says there's a variety of factors working together to create the new AR interface, beyond screens.

"The search for the new interface is a big thing. Once you don't have a keyboard and mouse, how do you interact with the real world? We've seen a lot of attempts - everything from controllers to gaze tracking, gesture interactions, voice of course. And there's a battle on which interface will be the prevailing one and the answer is probably a combination of all of the above depending on the use case."

In general, there are already 16-20 different apps for Meta in the works, and completed. Harner says it won't stop there, and that the potential for AR is massive, particularly for schools.

"There are a lot of things like remote assistance, visualizing 3D information, targeting architects, a lot on education too. There's a lot of interest in creating immersive environments. Classrooms have not changed much in, I don't know how many years, and computers technologies are poorly integrated in classrooms.

"We believe by being able to put things into the real world, it becomes much more collaborative. You can teach anatomy, chemistry with reactions, models of the solar system. There's huge applications in education."

Similarly, Osterhout Group has its own array of uses for smart glasses that range from vision aids like NuEyes to Vital Enterprises, where doctors at hospitals like Johns Hopkins and Stanford can use ODG's glasses to assist with procedures like heart surgery. Other use cases include telepresence for in-flight emergencies - where crew members become the 'virtual hands' of the doctor millions of miles away.

Whatever the use and form augmented reality takes - HMDs, smart glasses, apps, even contact lenses - it's clear there's a use for it. Now, it's a matter of keeping the hardware accessible and affordable while catching some of the spark from games like Pokémon Go.

Shop for VR headsets on Amazon

Oculus Rift
Oculus Rift
PlayStation VR
PlayStation VR
HTC Vive
HTC Vive
Samsung Gear VR
Samsung Gear VR

Wareable may get a commission



  • kelvin44 says:

    I'm curious as to why a little company called Immy is never mentioned as a headset in these AR/VR articles?  I have heard from people that have worn the ic60's and they say this headset is the best thing out there when it comes to low distortion, FOV, weight, compfort, etc.  There are stories about Immy, but these guys seem to want to stay under the radar unlike companies like Magic Leap who like to toot their horn but can never show a headset.  Check out the Immy website... www.immyinc.com The headset looks slick.

  • rise_ar says:

    Interesting read, and we broadly agree. The technology to distribute AR to the masses is already here, but it's our belief that it's the marketing of AR that's not working properly yet. Simple, marker-based mobile AR can be incredibly powerful (such as the CBS products demonstrator we produced recently) yet everyone wants to forge ahead with the next generation of AR - are we running before we can walk? Mobile VR  is extremely marketable. There's a captive audience in billions of smartphone users, and a wealth of opportunity for businesses. We just need to convince them it's worth the effort.

What do you think?

Connect with Facebook, Twitter, or just enter your email to sign in and comment.