Augmented reality visions through the ages: Movies, concepts and experiments

AR Week: The scenes and stills that have inspired AR designers and creators
AR visions through the ages

There's a specific sense of being one part badass and one part smug that we associate with augmented reality, by which we don't mean you pointing your phone at a billboard five years ago, we mean AR in the movies. By which we mean - just picture Tony Stark and you'll get it.

Starting with some of the first AR experiments, which brought the tech into the public consciousness, let us take you on a trip through past AR visions in tech, movies, TV, games and books right up to the creative predictions we're seeing in 2017.

Read this: Check out our AR Week hub for more stories

We'll include a few of the most famous instances of holograms and gesture control interfaces in pop culture, too, which are tied up with the idea of augmented reality even if there's different hardware alluded to in the sci-fi. Because let's face it, designing visors and helmets is fun but designing how the AR will look and work from the human point of view is much more tantalising and just as important. Plus there's naturally some crossover with VR too.

We don't know yet if we'll soon be interacting with virtual objects, media and characters which are projected or ones that only we can see through glasses, contact lenses or implants.

1975-: Videoplace

An early pioneer of VR and AR, Myron Krueger's interactive art exhibition Videoplace used monochrome, then coloured projections instead of a headset and data gloves. Over the next decade or so, versions of the project - whose influence can also be seen in Microsoft's Kinect - toured the world.

Participants interacted with onscreen objects with gesture controls: they could select letters to 'type', draw in 2D, manipulate 2D objects, play with patterns and interact with 'critters' - animal characters or a human silhouette who copied their movements. Krueger called his technology and lab, "artificial reality".

1977 -: Star Wars

In A New Hope, R2-D2 projects a 3D, glowing-blue hologram of Princess Leia recording a message including the now famous line "Help me, Obi-Wan Kenobi. You're my only hope." This might have been based on the first use of a hologram effect in a live-action film - the conversation between Dr Morbius and his daughter in the 1956 film Forbidden Planet.

Since then, we've seen holograms used to communicate in all the Star Wars episodes as well as larger projections of virtual elements like the Millenium Falcon-filling holographic star map leading to Luke Skywalker in The Force Awakens.

1991: Terminator 2: Judgment Day

James Cameron's Termo Vision from the Terminator's heads up display (HUD) has been hugely influential in how we think about AR. Shot from the military robot's point of view, we see lines of code and stats overlaid onto scene like the one where Arnie scans the patrons of a bar.

The Terminator is able to analyse everything from the models of cars, bikes, tech, weapons, even DNA to advanced recognition of targets and civilians; digitising handwriting and enhancing images. Not to make something crazy cool too nerdy but remember, this is contextual information presented over the real people and environment. Aesthetically, the UI is flatter in the earlier films, more dynamic later but purists prefer the first iconic scenes like this one.

1992: Virtual Fixtures

Augmented reality visions through the ages: Movies, experiments and more

Back to early AR experiments and two years after the term 'augmented reality' was coined by Tom Caudell, Louis Rosenberg's Virtual Fixtures system for the US Air Force was one of the first practical helmet and controls setups.

It was designed to provide audio and visual guides to the wearer, completing tasks such as remotely operating a vehicle, using AR overlays but to do so in the early 90s, Rosenberg used binocular magnifiers to make robot exoskeleton arms appear as if they were in the exact same spot as the user's arms.

1993: Star Trek: Deep Space Nine

Augmented reality visions through the ages: Movies, experiments and more

Star Trek is another show teeming with bold ideas around augmented and virtual reality. In the Deep Space Nine era, we got a taste of what was to come in headgear, particularly from Google Glass, from the Jem'Hadar and their virtual display devices which create a virtual field in front of one eye, capable of looking through the hull of the ship. Another neat trick - if anyone other than the Jem'Hadar units or the Vorta tries to use the tech, they get terrible headaches. Future biometric security feature? Probably not.

1994: Dancing in Cyberspace

Augmented reality visions through the ages: Movies, experiments and more

Julie Martin's Dancing in Cyberspace was billed the first augmented reality theatre production. The Australian art project saw dancers and acrobats controlling life sized virtual objects on a physical stage. Sadly, there doesn't seem to be any footage online.

1997: Ace Combat 2

Whether they're blocky beat'em'up scores or a million RPG panels, game HUDs, whether based on helmets or cockpits, have helped to make virtual, visual overlays a thing of desire over the years.

We're not going to list every single one for obvious reasons - though Ars Technica has a nice gallery showing its evolution - so we've picked out Ace Combat 2 for the PlayStation. It allowed gamers to see the sort of stats, guides and maps that a fighter pilot might see, in realistic, semi-transparent green text.

2002: Minority Report

Here we go with looking smug but badass while manipulating augmented reality graphics. Spielberg is said to have held an Idea Summit on futuristic technologies while he was developing Philip K Dick's short story.

In the film, Tom Cruise's Chief of PreCrime John Anderton uses gesture control gloves - and no glasses - to swipe, grab, point, wave and tap his way around multiple, curved, transparent displays and slides. As he "scrubs the image" looking for clues, viewers made mental notes on the user interface which - again - isn't augmented reality but is referenced a hell of a lot by the people building it. (Though we could do without the obvious sound effects).

Also on show in Minority Report are the VR/AR booths in which we see customers, encased in full-on bodysuits and wearing visors, living out their fantasies whether it's steamy sex with a soap star or just being praised by a room full of (semi-transparent, glowing, holographic) people. Plus don't forget the retina scanning, hologram adverts that know your name in the Gap - it's all go in 2054.

2002: Firefly

Augmented reality visions through the ages: Movies, experiments and more

It's nice to watch episode 9 (Ariel) of Joss Whedon's Firefly TV series having seen the Case Western HoloLens biology demos. Medical scanning and imagery is something of an obvious fit for AR and in Firefly, the Serenity team are able to see a skeleton floating above their patient as well as inspect a virtual model of her brain, turning it with two handed gesture controls.

2006: Rainbow's End

Augmented reality visions through the ages: Movies, experiments and more

Vernor Vinge's Hugo Award winning novel Rainbow's End is set in San Diego in 2025 when everyone interacts with augmented reality 24/7. No glasses are needed as smart contact lenses handle the visuals, smart clothing takes care of gesture controls and physical objects like robots can provide tangible haptic feedback. It all sounds very believable doesn't it? For Vinge, AR comes to dominate how we communicate - video calls, obviously - play games, work in industry (with AR instructions) and get treated.

The mid naughts were a pretty good time for AR visions in sci-fi novels. See also William Gibson's 2007 Spook Country, in which virtual media is everywhere to the extent that it comprises the world,and Halting State by Charles Stross (also 2007) which is set in a fictional 2012 where everyone wears Bluetooth connected AR 'Specs'.

2008-: Iron Man

Another AR touchstone from the movies, Iron Man's Tony Stark - and his buddies and enemies - use both a heads up display within the various suits' helmet as well as hologram-style projections. Stark tells J.A.R.V.I.S, his conversational AI assistant to "engage heads up display" and "port all preferences from home interface" in the first movie, showing that these are part of the same system.

As well as Terminator, military-style features that use cameras like target lock and facial recognition to identify civilians, Tony also uses his AR, both 2D and 3D, interface to video call Pepper Potts and track his own vitals via suit sensors. When power is low, the whole UI flickers. When he's moving around projections of globes and discovering new elements, gestures include clicking, spinning, flicking and manipulating the graphics with two hands.

Check out the supercut above of all the HUD scenes in Marvel flicks, including all characters who wear a helmet, up to Captain America: Civil War.

2011 - 2016: Black Mirror

Ideas around tinkering with our reality come up again and again in Charlie Brooker's must-see dystopian anthology TV series, Black Mirror. And often Brooker and his collaborators blend ideas of the internet, social media, VR and AR in episodes like Fifteen Million Merits, The Entire History of You and Be Right Back.

From last year's season four (on Netflix), check out the episode Playtest if you haven't already. Not only does it feature an adorable whack-a-gofer mini game that looks like near-future, interactive AR but it also offers up questions around how we might agree to have not just our visions but our minds manipulated for entertainment via a temporary implant. Spoiler alert: it gets dark.

2014: Kingsman: The Secret Service

Proving filmmakers haven't run out of AR ideas yet, especially when it comes to violence, Matthew Vaughan's spy movie Kingsman makes Bond's gadgets look stuffy. AR glasses are on the scene, sure, but one of our favourite moments is the AR umbrella as superbly wielded by Colin Firth's agent Harry Hart. Which which everyday objects will we see imbued with AR superpowers next? Stunning.

2015: Magic Leap

We're classing Magic Leap's first 'Just another day at the office' video from March 2015 as an AR vision/concept more than a demo as we're not expecting to see something like this any time soon. It shows off the productivity and gaming options in 'mixed reality' as Magic Leap calls its tech.

It's been viewed 3.8 million times, the effects were supplied by Weta workshop and it's now labelled as an 'original concept video' after some controversy over how real it was. Take the ideas, interfaces and interactions - which are fluid, creative and intuitive - as more of a statement of what the Florida startup is working towards. It might be years before this "mind-blowing" experience (our hot take words) makes it in front of our eyes. Regardless, there's no doubt it jumpstarted the industry after years of throwing shade at Google Glass.

2015: Windows Holographic

In April 2015, we got our first pitch from Microsoft about why we should care about HoloLens at its Build conference. As with many of its demos since, the footage has no field of view issue - unlike the actual helmet - but it shows Microsoft's vision for what its platform, then Holographic, now Mixed Reality, could become.

Even though it's only two years old, the graphics already look quite dated and very, well, Microsoft. To keep itself relevant, it needs to keep updating its AR concepts as well as the real demos and apps that developers are building now.

2016: Hyper-Reality

Ready for augmented reality overload? Then try Keiichi Matsuda'sHyper-Reality on for size. The years-long project took the form of a six minute short in 2016, showing a "kaleidoscopic" point of view.

Taking on familiar themes in a visual way, a city is "saturated in media" to the point that wherever you look, ads and suggestions and options for everything from searches to shopping pop up over the real world. It's bright, it's zany and it's energetic - until the system detects that the device is under attack and everything goes dull for a moment...

2017: Strange Beasts

This five minute short film by Magali Barbé starts out as an advert for a new AR app called Strange Beasts that lets you hang out with virtual pets which you design yourself. In this scenario, the digital critters are projected onto your retinas, natch, and can play with other people's pets. Then, you guessed it, everything gets a bit weird.

What did we miss? Let us know in the comments.




What do you think?

Connect with Facebook, Twitter, or just enter your email to sign in and comment.