2016 will be remembered as the year that real VR landed. The Oculus Rift, HTC Vive and PlayStation VR grabbed the headlines, and Google Daydream arrived too. Plus with Pokémon Go's summer smash, Magic Leap and HoloLens, AR has never looked a hotter proposition.
With big talk from Tim Cook and co not to mention leaks out of Carl Zeiss, it seems Apple is investing, researching and hiring in both AR and VR. Piecing together the quotes, speculation, job listings, patents and acquisitions, here's our look into Apple and its augmented, mixed and virtual reality plans.
Apple on AR smartglasses
ODG's R7 smartglasses (not Apple's)
The Apple AR smartglasses rumours on the web have been coming thick and fast through the first half of 2017.
Most interestingly, veteran tech blogger Robert Scoble says he has been told by an anonymous Carl Zeiss employee that Apple is working on a "light" pair of AR/MR smartglasses (rather than a helmet/visor/headset) in partnership with the optics experts.
These Carl Zeiss-made smart Apple specs are said to be due for launch either by the end of 2017 or next year, according to Scoble's secret source.
It's the best evidence yet that this is really happening and our bet is on 2018, possibly with a reveal in late 2017.
Aside from that, Cook quotes, internal hires and patents also suggest that it's more likely we will see an augmented reality (AR) wearable from Apple first.
One report that backs up the Carl Zeiss murmurs is that Apple has reportedly ordered small quantities of a near-eye displays and anonymous sources say it will display images and other info from your iPhone. That's all we know for now but it sounds very Glass-like to us.
Intriguingly, at Web Summit 2016, Scoble also teased that Apple is working on mixed reality tech that it will incorporate into a future Apple TV. "They are building the PrimeSense sensor right into the television, into the iPad and into the iPhone… so you can do this kind of mixed reality," Scoble said, indicating that the system would launch within a year.
Read this: #Trending: AR and mixed reality
Over the summer of 2016, Tim Cook really stepped up his AR chat, calling it a "core technology". He even went so far as to compare the two saying that AR will be a bigger business in the long term.
"There's virtual reality and there's augmented reality - both of these are incredibly interesting," Cook said on ABC News' Good Morning America. "But my own view is that augmented reality is the larger of the two, probably by far."
"Virtual reality sort of encloses and immerses the person into an experience that can be really cool," he said, "but probably has a lower commercial interest over time. Less people will be interested in that."
The success of Pokémon Go seemed to have helped Apple's AR cause – and Cook namechecked it in a July 2016 Apple earnings call. In fact, Cook has talked about AR so much at this point that he simply said "AR, as I've said in the past, is something we're really excited about" when asked about the tech by Mad Money's Jim Cramer.
Apple on VR headsets
An Apple VR headset looks increasingly unlikely, at least in the near term. But what does Tim Cook actually think of VR? Judging by early comments he made last year, he finds the immersive tech an intriguing prospect.
"In terms of virtual reality, I don't think it's a niche. I think it can be… it's really cool, and has some interesting applications," he said.
Read this: The best VR headset for iPhone users
More recently he seems much more taken with AR but has noted that Apple is very interested in new forms of education and gaming, both areas where VR is making waves.
It might not have its own headset but Apple is selling a virtual reality headset on the Apple Store. The retro View-Master iPhone VR headset popped up online in February 2016, and could be seen as the company's first attempt to test the waters with fanboys and fangirls to see how much they really love virtual reality.
It's also not like Apple hasn't dipped its feet in VR already. Back in October 2015, Apple teamed up with U2, The Weeknd and virtual reality app VRSE (now Within) to produce 360-degree videos to tie in with the new Apple Music service.
Patents are often a pretty strong indication of what a company has up its sleeve. Sometimes though, they are reminders of ideas that were shelved.
With the latest rumours of Apple AR specs swirling, there are no new patents to pore over. Browsing through the older patents, though, you can see that Apple was thinking about headset displays as far back as 2009. A patent filed for a new iPod video headset display would apparently have let users record video and zoom in and out of the footage. Apple also thought it would be great for viewing live events, letting you get closer to the action when you're way out in the cheap seats.
Evidence that VR is on the agenda arrived in February last year. Apple was awarded a patent for a head-mounted virtual reality headset. It uses the iPhone as the display, much like the Gear VR does with Samsung's smartphones or Google Cardboard. The patent even talks about supporting additional accessories like spare batteries, physical control inputs and a cooling system.
Go down a patent rabbit hole
- The patented history and future of… Google GlassQuite the spectacle
- The patented history and future of… the Apple WatchHow we got to here - and where we go next
- In patents: Samsung smartwatchesGears for years: We look at Samsung's R&D from September 2013 up to now
- In patents: Android WearThe paper trail of Google's smartwatch ambitions
More recently, people have been digging up patents around Apple's AR plans. Apple Insider has patents including the one above which shows a system for mapping a room using sensors and computer vision as well as transparent displays and augmented reality apps. In late April 2017, Apple was granted a patent detailing "a method for representing virtual objects in a real environment," also known as augmented reality. There would also be a "recording device" to capture a 2D image of said environment.
The Cupertino company is also updating patents for existing products with references to AR, like the upcoming Mac Pro. Apple, of course, uses Macs to make its own products. It's a little difficult to work on AR when you don't have a computer with powerful enough graphics capabilities to work on AR, hence a new Mac Pro with enough power to work on AR, and a patent to prove it.
Building the Apple dream team
Of course, Apple is a company that likes to recruit the best of the best. When it was building the Apple Watch, the company picked up one of the top designers that had worked on the Nike Fuelband and also snapped up Paul Deneve, the former CEO of Yves Saint Laurent.
According to a report in the FT, Apple has a "secret team… of experts in virtual and augmented reality" who have "built prototypes of headsets" and that "includes hundreds of staff from a series of carefully targeted acquisitions, as well as employees poached from companies that are working on next-generation headset technologies."
Read this: How does VR actually work?
Apple's decision to hire Doug Bowman in January, a leading virtual reality (and augmented reality) researcher is seen as a clear signal of intent. Bowman moved from his post as director of its centre for human interaction at Virginia Tech. He's spent time designing user interfaces for fully immersive VR experiences and also won a $100,000 grant from Microsoft last year to work on a mixed reality data study centred around its HoloLens holographic headset.
Also, Apple has hired top VR and AR talent including former Magic Leap employee Zeyu Li as a senior computer vision engineer on the AR side and ex-Oculus research scientist Yury Petrov to focus on physiological studies of VR headsets. Apple is also reported to have sent employees to test out the empathy VR apps at Stanford's Virtual Human Interaction lab - three times in three months.
There's even a job listing from as far back as 2014 showing that Apple was looking for a virtual reality developer to design "next generation software experiences" and "create high performance apps that integrate with Virtual Reality systems for prototyping and user testing."
All in all, Apple's AR team will be led former Dolby head Mike Rowell, who advised a small AR glasses company called Meta. His team is comprised of former employees of Magic Leap, Oculus, Microsoft's HoloLens team, THX, Amazon's Lumberyard VR platform, Google Earth and Weta Digital. On top of that, Apple has been poaching talent from Imagination, the company that used to provide graphics technology for iPhones, iPads and more. Why? Apple plans to make their own GPU, and it's not a leap to imagine its because it's moving to AR and needs something a little more proprietary.
Splashing the cash
Why do the work when you can just buy the companies that have done the work already? Apple is not the first and certainly won't be the last company to snap up the expertise it needs to help build its hardware. And Apple has certainly been busy on that front in regards to its VR ambitions.
"Apple buys smaller technology companies from time to time, and we generally do not discuss our purpose or plans," is the go-to line when responding to queries about these acquisitions. But it is possible to join the dots to get an idea of what Apple might have planned.
Explained: The difference between VR and AR
First up, Emotient. Its tech analyses facial expressions to capture data on emotions and moods - it's been used for everything from marketing to medical cases looking at pain responses.
Another fairly recent acquisition was Flyby Media, an augmented reality-focused startup that has expertise in 3D tracking and has worked with Google in the past. It built an app that lets you scan real world objects with your phone's camera and uses image recognition to group objects into a collection. The key here is tracking, which would be pretty useful for building a headset, and relates to the patents we've seen.
There's also Metaio. The German startup is best known for its augmented reality apps and has already worked with the likes of Ferrari to take advantage of its 3D tracking tech. So why is that a big deal? Well, Apple could well be looking at augmented reality as well as virtual reality. Both are set to be big and this could be Apple covering its bases.
Next is Faceshift, a Swiss startup that specialises in generating CGI avatars and capturing expressions using real-time motion capture. Faceshift has previously used Microsoft's Kinect camera and Intel's RealSense 3D camera to demo its innovative software. The technology was reportedly used in the making of Star Wars: The Force Awakens as well. Motion capture is a big deal for gaming and current headsets like the HTC Vive are starting to get to grips with it for VR to help inject some realism.
PrimeSense, the Israeli startup that built the 3D sensor for Microsoft's Kinect camera on the Xbox 360 and namechecked by Scoble in terms of mixed reality, was picked up for a cool $360 million. The motion sensing tech could of course be used for a range of hardware currently under the Apple umbrella, but it would certainly be a good fit for VR. Motion sensing is vital for control and movement so this could offer the same kind of slick 3D movement that software like Leap Motion promises.
When will we see it?
That's the big question. This is a company that doesn't tend to rush things out until they're ready. The original iPhone was apparently in the works for around five years and with so much still to learn about bringing VR and AR into the home, we could be looking at a similar roadmap with a launch as far away as 2018 - 2019.
And which will it be? A Rift-like device or a pair of AR smartglasses? Don't rule out an iPhone VR viewer in the next few years. But there's now plenty of evidence to suggest good money should be on augmented/mixed reality smart spectacles sooner rather than later.
What do you want to see from Apple? AR specs or a VR headset?