Microsoft Research's hand tracking tech will let you feel objects in VR

Handpose mixes gesture recognition with haptics
Microsoft Gesture research , Jamie Shotton and team at the Microsoft Centre, 21 Station Road, Cambridge. CB1 2FB  13 June 2016
Wareable is reader-powered. If you click through using links on the site, we may earn an affiliate commission. Learn more

Microsoft Research's latest project will come as welcome news to Oculus Rift and HTC Vive owners looking for a touch more immersion. Handpose is a futuristic, precise 3D hand-tracking technology which uses Microsoft's Kinect sensor and computer vision. It's designed to let you use your hands to naturally interact with computers and yes, VR headsets.

Accuracy, including for finger tracking, has been improved by the team with the work based on algorithms dating back to the 1940s and research carried out since then. The aim is to get to the point where an avatar hand feels like your hand.

Read this: Oculus Rift review

The virtual controls could also be so impressive that if you touch your fingers together you could get the sensation of touching something hard or else something soft and pliant. That's all thanks to haptics, which will become more important as VR gets more sophisticated.

According to Microsoft Research's blog post, Hrvoje Benko, a senior researcher in the natural interaction group at the Redmond lab, thinks it will become harder and harder to trick our body into immersing itself in a virtual experience without anything to touch.

Haptics could be combined with limited real world objects and accessories such as one building block to be used over and over in a building game or a couple of knobs and buttons for a flight sim. The project is called haptic retargeting and it could really open up VR controls.

"A single physical object can now simulate multiple instances in the virtual world," said Eyal Ofek, another senior researcher.

The system, presented at SIGGRAPH 2016, is still a research project within Microsoft Research for now. But it's easy to see how this could be applied to its augmented reality device, the HoloLens, as well as VR headsets. Watch the video above for more details.

Microsoft Research's hand tracking tech will let you feel objects in VR


How we test


Sophie was Wareable's associate editor. She joined the team from Stuff magazine where she was an in-house reviewer. For three and a half years, she tested every smartphone, tablet, and robot vacuum that mattered. 

A fan of thoughtful design, innovative apps, and that Spike Jonze film, she is currently wondering how many fitness tracker reviews it will take to get her fit. Current bet: 19.

Sophie has also written for a host of sites, including Metro, the Evening Standard, the Times, the Telegraph, Little White Lies, the Press Association and the Debrief.

She now works for Wired.

Related stories