Face tracking is coming to mobile VR

Let's take eyebrow waggling virtual
Wareable is reader-powered. If you click through using links on the site, we may earn an affiliate commission. Learn more

We could conduct a whole conversation in virtual reality and I could shake my avatar head or turn it towards you to show I'm listening. By the end of the year, though, I'll be winking, smiling and frowning, with the whole gamut of emoji faces at my disposal.

Specifically, if I'm wearing Veeso, a new mobile VR headset for Android and iOS that's raising funds on Kickstarter now. Starting at $70 for early bird units, and $80 for dev kits, Veeso adds two infrared cameras to a fairly standard device.

Read this: Where VR tourism is heading next

One camera tracks your mouth, lips and jaw and the other tracks your eyelids, eyebrows and pupils, more like the experiments in eye tracking that we've seen before. These are then recreated in real time by your 3D digital avatar, in multiplayer VR apps, games and social networks.

It is designed to work with Cardboard apps - not sadly (generally better) Gear VR ones - for now, though we presume Veeso is keeping one eye on Daydream. Plus, Veeso plans to release 10 apps and games of its own, over six months.

The headset is currently a pretty ridiculous-looking prototype in the demo videos but from the artwork, the New York based startup reckons it can ship a polished product by the end of the year. If the campaign doesn't reach its $80,000 target - it's still sub $1,000 - the tech itself might get snapped up.

Feel my face

Face tracking is coming to mobile VR

Veeso isn't the only company interested in trying to make face tracking a reality, of course. FOVE has been working on eye tracking in mobile VR - both for controls and the ability to 'look' at other avatars in VR.

In PC-based VR, last year Oculus partnered with researchers at the University of Southern California to track facial expressions. They did this using a camera pointed down at the mouth and strain gauges for the upper part of the face which are added to the headset's foam padding. Hao Lin, who led the project, said the experiment could be put into production quickly.

Social VR apps like Altspace VR and Oculus Social are the obvious case for this type of tracking. Ebbe Altberg, CEO of Linden Labs, which is building a virtual world called Project Sansar, has also alluded to the potential for realising facial expressions in multi-player VR.

"We are working a lot on facial tracking to make it very natural; mouth and lip syncing so that it's a very natural feeling when you interact with other people," he said at Web Summit last year.

If you can't wait and fancy pledging for a Veeso headset on Kickstarter, the team aims to ship in December 2016.

Face tracking is coming to mobile VR


How we test


Sophie was Wareable's associate editor. She joined the team from Stuff magazine where she was an in-house reviewer. For three and a half years, she tested every smartphone, tablet, and robot vacuum that mattered. 

A fan of thoughtful design, innovative apps, and that Spike Jonze film, she is currently wondering how many fitness tracker reviews it will take to get her fit. Current bet: 19.

Sophie has also written for a host of sites, including Metro, the Evening Standard, the Times, the Telegraph, Little White Lies, the Press Association and the Debrief.

She now works for Wired.

Related stories