The race to mixed reality: Facebook, Apple & the new light field upstarts

Who's who & what all the fuss is about
The race to mixed reality

Here's a weird prediction. Microsoft will be the only company to own mixed reality (or next-gen augmented reality) in 2017. Everyone is chasing it and the ambition sure is bolder than virtual reality.

You put on a pair of glasses, a visor, a helmet, some contact lenses, and your world is transformed, part real and part virtual. And it's not just the crappy AR you had on your phone five years ago either - the new "mixed reality" term this is hi-res, interactive, realistic-looking, reacts to the environment. Or it will do when it arrives.

So back to 2017. Sure, just last December we named mixed reality one of the biggest trends of the year in our Wareable 50 list but as the year rolls on, the more we here from Facebook, from Apple, from Snap and the longer we wait for something concrete from Magic Leap, the more it seems like we'll have to be patient until at least 2018 for a big release.

Facebook &Apple &Snap &Google

The race to mixed reality: Facebook, Apple & the new light field upstarts

If that is disappointing, just know that everyone - and we mean everyone - is working on this stuff. Google Glass was too ahead of its time but it's not dead and dusted. Plus, Google is one of the big investors in everyone's favourite Florida startup Magic Leap, leading some to speculate that future Google faceware could use into the platform.

Meanwhile Tim Cook can't stop talking about AR and, along with talent hires, patents and occasional leaks like that recent accident report which detailed eye pain "after working with a new prototype", the industry has one eye on what Apple is working on. When it arrives, it will be much more of a gamble than the Watch so Cupertino needs to get it spot on.

Augmented reality has been around for years but the technological advancements, combined with the tantalising prospect of putting it all in glasses you might actually wear, mean we've been hearing a lot from Facebook and Snap lately. Snap has new World Lenses filters for smartphones which map 3D virtual objects onto the world, promising for its (probably) inevitable AR camera glasses. It also acquired Cimagine, an AR startup based out of Israel last year. The pieces are falling into place for just one example of a software company getting into hardware.

The race to mixed reality: Facebook, Apple & the new light field upstarts

In terms of futuristic wearables we can try, though, it's still really all chatter. "It's quite vague right now because lots of people are using the words mixed reality to refer to different things," says Amandine Flachs, a VR/MR specialist at the Realities Centre. "Snapchat is a good example; people don't care if it's AR or MR, they're having fun on their phones and the glasses are cool."

So right now, there's a lot of talk and a good number to have in your head is 2022. That's when Facebook's chief scientist for Oculus Research, Michael Abrash, predicts that AR smartglasses will begin to take over. And we mean begin. At Facebook's F8 conference this week, he said: "Despite all the attention focused on AR today it will be five years at best before we're really at the start of the ramp to widespread, glasses-based augmented reality, before AR has its Macintosh moment."

Which brings us back to Microsoft. You can buy a HoloLens now if you're willing to put up the cash and make do with its field of view. It'll be interesting to see how its Windows Mixed Reality headsets will be used and what type of apps will be built for them. It's the same Holographic platform as HoloLens but these headsets might have inside out tracking to move around but they're far from everyday glasses too. It'd be a shame if they are treated as purely budget VR devices when they start to be released this summer.

L-l-l-light field, baby

If you follow the money in mixed reality it leads you to Magic Leap, which has raised over $1.5 billion, partly based on the hype over its gamechanging light field display technology.

As the years have ticked by since its demo videos, though, Rony Abovitz' operation is no longer the only contender promising to make 3D virtual images look real. Last November at Web Summit, Edward Tang, the co-founder and CTO of California-based Avegant - makers of the Glyph - was waxing lyrical to me about the future of mixed reality.

The race to mixed reality: Facebook, Apple & the new light field upstarts

Now, Avegant is ready to show off what its 45-strong team has been working on - a Light Field Technology platform and prototype which uses the retinal projection tech from the Glyph too. A light field allows you to focus on multiple planes ahead of you as it works with data on the amount and direction of light flowing through every point in the space.

"We've been working on the transparent mixed reality type devices, the display technologies, for a couple of years now," says Tang. "Light field is real and it's awesome. In the VR space, the way they're showing us 3D means your eyes point inward when it's close to you but they can't change the focus of that image. We need to solve the problem of objects close to your face and light field is the way to do it. Once you see it, you can't unsee it."

So, for instance you could have a face to face conversation, hold a virtual object in your hand or use an app that overlays something onto the table in front of you, not two metres away as with something like HoloLens. The tech can also help virtual characters and objects look "correct" in the lighting of the real room you're in with dynamic, virtual lighting adding to the illusion. This is one of the challenges brought up by Facebook and Oculus when explaining why AR/MR is a few years behind VR.

The race to mixed reality: Facebook, Apple & the new light field upstarts

Avegant's mixed reality device prototype

Now, Avegant hasn't announced that it will be building a mixed reality helmet itself, though Tang does stress that it has experience with manufacturing a device with retinal projection displays from the Glyph.

One more name to keep an eye on is Light Field Lab. I first came across the tech when playing around with the Lytro camera which uses light field to re-focus on any point after you take the shot.

It's since pivoted into the world of VR but a few alumni, led by Jon Karafin, have broken away to work on technology to beam mixed reality visuals onto the users' eyes, possibly even without smartglasses or a visor. Light Field Lab is appearing at the NAB expo in Las Vegas from 22 April so we'll be keeping an eye on what's in store from them.

How we compute

Magic Leap already has partners, collaborators and celebrity fans (Ricky Gervais loved it, Beyonce was meh) but it will be interesting to see how the rest of the industry shakes out. Who buys who for what tech, which platforms open up and which stay closed and when each tech giant decides enough is enough and releases a product.

There's a lot to think about in the meantime. Even the most impressive progress is still a long way from slimming down the tech into a pair of Spectacles-style glasses. Schemes like Avegant's Light Field SDK that works with the Unity game engine will speed up development of apps, experiences and games.

The race to mixed reality: Facebook, Apple & the new light field upstarts

Commentators have already started to warn about competing corporate filters over our world - actually viewing the mixed reality world that Facebook/Google/Apple curates for you. Plus, Amandine Flachs says: "We need to think about privacy now. Big companies are getting into AR but not changing their policies. If I walk down the street will someone see my name and Facebook profile on their glasses?"

Edward Tang thinks we need to focus on working out what the initial applications are, aside from enterprise: "What application is going to justify putting down several hundred dollars and putting something on your head? VR is struggling with that, right now they're focused on gaming and media. Is it something else for mixed reality?"

The Mark Zuckerberg answer, and the Tim Cook answer, to the question of what mixed or advanced augmented reality is for is, of course, everything. Mixed reality will be a new computing platform to replace our phones. Zuck is fond of making promises like: "We want glasses, or eventually contact lenses, that look and feel normal, but let us overlay all kinds of information and digital objects on top of the real world."

It's exciting to hear the CEOs of Facebook and Apple getting pumped up about something we know Google has been working on for years. As Tang says, "it's not a question of if but when".

The tech - light field, retinal projection, inside-out tracking - is ready; the appetite is there thanks in part to Snap, Oakley & more deconstructing smartglasses. The eye-watering investment and the bullish CEOs are there. Now, we wait patiently for something real.


Shop for VR headsets on Amazon

Oculus Rift
Oculus Rift
$499.99
PlayStation VR
PlayStation VR
$399.99
HTC Vive
HTC Vive
$799
Samsung Gear VR
Samsung Gear VR
$129.99

Wareable may get a commission



What do you think?

Connect with Facebook, Twitter, or just enter your email to sign in and comment.