2016 will be remembered as the year that real VR landed. The Oculus Rift, HTC Vive and PlayStation VR have grabbed the headlines, and don't forget that Google is gearing up to launch its new Daydream platform. There's plenty to look forward to.
Plus with Pokémon Go's summer smash, Magic Leap and HoloLens on the way, AR has never looked a hotter proposition. With big talk lately from Tim Cook, it seems Apple is investing and researching and hiring in both VR and AR. Piecing together the quotes, speculation, job listings, patents and acquisitions, here's our look into Apple and its virtual and augmented reality plans.
Apple on VR
So what does Tim Cook actually think of VR? Judging by early comments he made last year, he finds the immersive tech an intriguing prospect. "In terms of virtual reality, I don't think it's a niche. I think it can be… it's really cool, and has some interesting applications," he said.
More recently he seems much more taken with AR (see below) but has noted that Apple is very interested in new forms of education and gaming, both areas where VR is making waves.
It might not have its own headset but Apple is selling a virtual reality headset on the Apple Store. The retro View-Master iPhone VR headset popped up online in February, and could be the company's first attempt to test the waters with fanboys and fangirls to see how much they really love virtual reality.
It's also not like Apple hasn't dipped its feet in VR already. Back in October last year, Apple teamed up with U2, The Weeknd and virtual reality app VRSE to produce 360-degree videos to tie in with the new Apple Music service.
Apple on AR
ODG's R7 smartglasses
Despite the evidence of VR intentions, the latest Cook quotes, internal hires and patents suggest that it's more likely we will see an augmented reality (AR) wearable for Apple first.
Over the summer Tim Cook really stepped up his AR chat, calling it a "core technology". He even went so far as to compare the two saying that AR will be a bigger business in the long term.
"There's virtual reality and there's augmented reality -- both of these are incredibly interesting," Cook said on ABC News' Good Morning America. "But my own view is that augmented reality is the larger of the two, probably by far."
"Virtual reality sort of encloses and immerses the person into an experience that can be really cool," he said, "but probably has a lower commercial interest over time. Less people will be interested in that."
The success of Pokémon Go seemed to have helped Apple's AR cause – and Cook namechecked it in a July Apple earnings call, continuing to say: "... we have been and continue to invest a lot in this. We are high on AR for the long run."
Patents are often a pretty strong indication of what a company has up its sleeve. Sometimes though, they are reminders of ideas that were shelved.
Browsing through the older patents, you can see that Apple was thinking about headset displays as far back as 2009. The patent filed for a new iPod video headset display would apparently have let users record video and zoom in and out of the footage. Apple also thought it would be great for viewing live events, letting you get closer to the action when you're way out in the cheap seats.
Evidence that VR is on the agenda arrived in February last year. Apple was awarded a patent (image above) for a head mounted virtual reality headset. It uses the iPhone as the display, much like the Gear VR does with Samsung's smartphones or Google Cardboard. The patent even talks about supporting additional accessories like spare batteries, physical control inputs and a cooling system.
More recently, people have been digging up patents around Apple's AR plans. Apple Insider has patents including the one above which shows a system for mapping a room using sensors and computer vision as well as transparent displays and augmented reality apps.
Building the Apple dream team
Of course, Apple is a company that likes to recruit the best of the best. When it was building the Apple Watch, the company picked up one of the top designers that had worked on the Nike Fuelband and also snapped up Paul Deneve, the former CEO of Yves Saint Laurent.
According to a report in the FT, Apple has a "secret team… of experts in virtual and augmented reality" who have "built prototypes of headsets" and that "includes hundreds of staff from a series of carefully targeted acquisitions, as well as employees poached from companies that are working on next-generation headset technologies."
Read this: How does VR actually work?
Apple's decision to appoint Doug Bowman in January, a leading virtual reality (and augmented reality) researcher is seen as a clear signal of intent. Bowman moved from his post as director of its centre for human interaction at Virginia Tech. He's spent time designing user interfaces for fully immersive VR experiences and also won a $100,000 grant from Microsoft last year to work on a mixed reality data study centred around its HoloLens holographic headset.
More recently, Apple has hired top VR and AR talent including former Magic Leap employee Zeyu Li as a senior computer vision engineer on the AR side and ex-Oculus research scientist Yury Petrov to focus on physiological studies of VR headsets. Apple is also reported to have sent employees to test out the empathy VR apps at Stanford's Virtual Human Interaction lab - three times in three months.
There's even a job listing from as far back as 2014 showing that Apple was looking for a virtual reality developer to design "next generation software experiences" and "create high performance apps that integrate with Virtual Reality systems for prototyping and user testing."
Splashing the cash
Why do the work when you can just buy the companies that have done the work already? Apple is not the first and certainly won't be the last company to snap up the expertise it needs to help build its hardware. And Apple has certainly been busy on that front in regards to its VR ambitions.
"Apple buys smaller technology companies from time to time, and we generally do not discuss our purpose or plans," is the go-to line when responding to queries about these acquisitions. But it is possible to join the dots to get an idea of what Apple might have planned.
Explained: The difference between VR and AR
First up, Emotient. Its tech analyses facial expressions to capture data on emotions and moods - it's been used for everything from marketing to medical cases looking at pain responses.
Another fairly recent acquisition was Flyby Media, an augmented reality-focused startup that has expertise in 3D tracking and has worked with Google in the past. It built an app that lets you scan real world objects with your phone's camera and uses image recognition to group objects into a collection. The key here is tracking, which would be pretty useful for building a headset, and relates to the patents we've seen.
There's also Metaio. The German startup is best known for its augmented reality apps and has already worked with the likes of Ferrari to take advantage of its 3D tracking tech. So why is that a big deal? Well, Apple could well be looking at augmented reality as well as virtual reality. Both are set to be big and this could be Apple covering its bases.
Next is Faceshift, a Swiss startup that specialises in generating CGI avatars and capturing expressions using real-time motion capture. Faceshift has previously used Microsoft's Kinect camera and Intel's RealSense 3D camera to demo its innovative software. The technology was reportedly used in the making of Star Wars: The Force Awakens as well. Motion capture is a big deal for gaming and current headsets like the HTC Vive are starting to get to grips with it for VR to help inject some realism.
PrimeSense, the Israeli startup that built the 3D sensor for Microsoft's Kinect camera on the Xbox 360, was picked up for a cool $360 million. The motion sensing tech could of course be used for a range of hardware currently under the Apple umbrella, but it would certainly be a good fit for VR. Motion sensing is vital for control and movement so this could offer the same kind of slick 3D movement that software like Leap Motion promises
When will we see it?
That's the big question. This is a company that doesn't tend to rush things out until they're ready. The original iPhone was apparently in the works for around five years and with so much still to learn about bringing VR and AR into the home, we could be looking at a similar roadmap with a launch as far away as 2018 - 2019. And which will it be? A Rift-like device or an AR helmet to rival Microsoft's HoloLens? There's plenty of evidence here to suggest that both are ultimately on the agenda.
Shop for VR headsets on Amazon
Wareable may get a commission