Apple mixed reality investigation: Why connected glasses are on the cards

We build the case for the folks at Cupertino releasing smart specs
Apple AR/VR headset investigation

2016 will be remembered as the year that real VR landed. The Oculus Rift, HTC Vive and PlayStation VR have grabbed the headlines, and Google Daydream is now here too. Plus with Pokémon Go's summer smash, Magic Leap and HoloLens on the way, AR has never looked a hotter proposition.

With big talk lately from Tim Cook and co, it seems Apple is investing, researching and hiring in both AR and VR. Piecing together the quotes, speculation, job listings, patents and acquisitions, here's our look into Apple and its augmented, mixed and virtual reality plans.

Apple on AR and mixed reality

apple ar smartglasses
ODG's R7 smartglasses

The latest Cook quotes, internal hires and patents suggest that it's more likely we will see an augmented reality (AR) wearable from Apple first.

The latest report is that we should expect some sort of AR Apple smartglasses. It has reportedly ordered small quantities of a near-eye displays and anonymous sources say it will display images and other info from your iPhone. That's all we know for now but it sounds very Glass-like to us.

Intriguingly, at Web Summit 2016, Robert Scoble also teased that Apple is working on mixed reality tech that it will incorporate into a future Apple TV.

"They are building the PrimeSense sensor right into the television, into the iPad and into the iPhone… so you can do this kind of mixed reality," Scoble said, indicating that the system would launch within a year.

Read this: #Trending: AR and mixed reality

Over the summer Tim Cook really stepped up his AR chat, calling it a "core technology". He even went so far as to compare the two saying that AR will be a bigger business in the long term.

"There's virtual reality and there's augmented reality - both of these are incredibly interesting," Cook said on ABC News' Good Morning America. "But my own view is that augmented reality is the larger of the two, probably by far."

"Virtual reality sort of encloses and immerses the person into an experience that can be really cool," he said, "but probably has a lower commercial interest over time. Less people will be interested in that."

The success of Pokémon Go seemed to have helped Apple's AR cause – and Cook namechecked it in a July Apple earnings call, continuing to say: "... we have been and continue to invest a lot in this. We are high on AR for the long run."

Apple on VR

apple vr ar headset

An Apple VR headset looks increasingly unlikely, at least in the near term. But what does Tim Cook actually think of VR? Judging by early comments he made last year, he finds the immersive tech an intriguing prospect. "In terms of virtual reality, I don't think it's a niche. I think it can be… it's really cool, and has some interesting applications," he said.

Read this: The best VR headset for iPhone users

More recently he seems much more taken with AR but has noted that Apple is very interested in new forms of education and gaming, both areas where VR is making waves.

It might not have its own headset but Apple is selling a virtual reality headset on the Apple Store. The retro View-Master iPhone VR headset popped up online in February, and could be the company's first attempt to test the waters with fanboys and fangirls to see how much they really love virtual reality.

It's also not like Apple hasn't dipped its feet in VR already. Back in October last year, Apple teamed up with U2, The Weeknd and virtual reality app VRSE to produce 360-degree videos to tie in with the new Apple Music service.

The patents

apple ar smartglasses

Patents are often a pretty strong indication of what a company has up its sleeve. Sometimes though, they are reminders of ideas that were shelved.

With the latest rumours of Apple AR specs, there are no new patents to pore over.

Browsing through the older patents, though, you can see that Apple was thinking about headset displays as far back as 2009. The patent filed for a new iPod video headset display would apparently have let users record video and zoom in and out of the footage. Apple also thought it would be great for viewing live events, letting you get closer to the action when you're way out in the cheap seats.

Evidence that VR is on the agenda arrived in February last year. Apple was awarded a patent (image above) for a head-mounted virtual reality headset. It uses the iPhone as the display, much like the Gear VR does with Samsung's smartphones or Google Cardboard. The patent even talks about supporting additional accessories like spare batteries, physical control inputs and a cooling system.

apple ar smartglasses

More recently, people have been digging up patents around Apple's AR plans. Apple Insider has patents including the one above which shows a system for mapping a room using sensors and computer vision as well as transparent displays and augmented reality apps.

Building the Apple dream team

Of course, Apple is a company that likes to recruit the best of the best. When it was building the Apple Watch, the company picked up one of the top designers that had worked on the Nike Fuelband and also snapped up Paul Deneve, the former CEO of Yves Saint Laurent.

According to a report in the FT, Apple has a "secret team… of experts in virtual and augmented reality" who have "built prototypes of headsets" and that "includes hundreds of staff from a series of carefully targeted acquisitions, as well as employees poached from companies that are working on next-generation headset technologies."

Read this: How does VR actually work?

Apple's decision to appoint Doug Bowman in January, a leading virtual reality (and augmented reality) researcher is seen as a clear signal of intent. Bowman moved from his post as director of its centre for human interaction at Virginia Tech. He's spent time designing user interfaces for fully immersive VR experiences and also won a $100,000 grant from Microsoft last year to work on a mixed reality data study centred around its HoloLens holographic headset.

More recently, Apple has hired top VR and AR talent including former Magic Leap employee Zeyu Li as a senior computer vision engineer on the AR side and ex-Oculus research scientist Yury Petrov to focus on physiological studies of VR headsets. Apple is also reported to have sent employees to test out the empathy VR apps at Stanford's Virtual Human Interaction lab - three times in three months.

There's even a job listing from as far back as 2014 showing that Apple was looking for a virtual reality developer to design "next generation software experiences" and "create high performance apps that integrate with Virtual Reality systems for prototyping and user testing."

Splashing the cash

apple ar smartglasses

Why do the work when you can just buy the companies that have done the work already? Apple is not the first and certainly won't be the last company to snap up the expertise it needs to help build its hardware. And Apple has certainly been busy on that front in regards to its VR ambitions.

"Apple buys smaller technology companies from time to time, and we generally do not discuss our purpose or plans," is the go-to line when responding to queries about these acquisitions. But it is possible to join the dots to get an idea of what Apple might have planned.

Explained: The difference between VR and AR

First up, Emotient. Its tech analyses facial expressions to capture data on emotions and moods - it's been used for everything from marketing to medical cases looking at pain responses.

Another fairly recent acquisition was Flyby Media, an augmented reality-focused startup that has expertise in 3D tracking and has worked with Google in the past. It built an app that lets you scan real world objects with your phone's camera and uses image recognition to group objects into a collection. The key here is tracking, which would be pretty useful for building a headset, and relates to the patents we've seen.

There's also Metaio. The German startup is best known for its augmented reality apps and has already worked with the likes of Ferrari to take advantage of its 3D tracking tech. So why is that a big deal? Well, Apple could well be looking at augmented reality as well as virtual reality. Both are set to be big and this could be Apple covering its bases.

Next is Faceshift, a Swiss startup that specialises in generating CGI avatars and capturing expressions using real-time motion capture. Faceshift has previously used Microsoft's Kinect camera and Intel's RealSense 3D camera to demo its innovative software. The technology was reportedly used in the making of Star Wars: The Force Awakens as well. Motion capture is a big deal for gaming and current headsets like the HTC Vive are starting to get to grips with it for VR to help inject some realism.

PrimeSense, the Israeli startup that built the 3D sensor for Microsoft's Kinect camera on the Xbox 360 and namechecked by Scoble in terms of mixed reality, was picked up for a cool $360 million. The motion sensing tech could of course be used for a range of hardware currently under the Apple umbrella, but it would certainly be a good fit for VR. Motion sensing is vital for control and movement so this could offer the same kind of slick 3D movement that software like Leap Motion promises.

When will we see it?

That's the big question. This is a company that doesn't tend to rush things out until they're ready. The original iPhone was apparently in the works for around five years and with so much still to learn about bringing VR and AR into the home, we could be looking at a similar roadmap with a launch as far away as 2018 - 2019.

And which will it be? A Rift-like device or a pair of AR smartglasses? There's plenty of evidence to suggest good money should be on augmented or mixed reality smart spectacles sooner rather than later.

What do you want to see from Apple? AR specs or a VR headset?


Shop for VR headsets on Amazon

Oculus Rift
Oculus Rift
$599.99
PlayStation VR
PlayStation VR
$399
HTC Vive
HTC Vive
$799
Samsung Gear VR
Samsung Gear VR
$97.98

Wareable may get a commission


What do you think?

Connect with Facebook, Twitter, or just enter your email to sign in and comment.