Our eyes play a vital role in social interactions. The expression on a person's face, from the slightest furrowing of a brow to the twitch of a mouth, can tell you a lot about what they're thinking, or whether they're really listening.
We need to convert these social cues into virtual reality, but it's hard. Our faces compose endless combinations of expressions, making the dream of a proper VR game of poker difficult to realise. The good news is that there's encouraging progress being made, and the proof was embedded in a HTC Vive headset at this year's GDC. Swedish company Tobii has been developing eye-tracking technology for PC gaming for several years, but it's announced it's now moving to VR and is collaborating with Valve.
Read this: Eyes-on with LG's SteamVR headset
It's not building a peripheral though; it's looking to embed the tech into the next generation of virtual reality headsets. It has to, as for the tracking to be precise it must scan your eyes from behind the lens. I tried it on a retrofitted Vive, and from the outside it was hard to tell that anything about the headset was different. The only giveaway were the illuminators, a ring of infrared lights around each lens.
Rather than letting me blast aliens with my eyes like X-Men's Cyclops, which would be cool, Tobii put me in a demo that was about demonstrating the more subtle use cases of eye tracking. To begin with, I was put in a room with a mirror so I could see my cartoon avatar up close, and sure enough, wherever I moved my gaze my avatar copied with zero latency. When I blinked, it blinked; when I winked, it did the same. While some of my more subtle eyebrow movements weren't detected, it was good at picking up when I did more dramatic eyebrow raises.
In the next room, I could only pick up objects that I was looking directly at. This was meant to demonstrate how we usually do, in fact, look at something as we reach for it. In another section of the game I met two robots who would only acknowledge me when I looked them right in their cold, man-made eyes. A different mini game involved knocking over bottles by throwing rocks at them. In this one, there was a special rock that was more likely to hit the bottle if it was directly in my eye-line.
"One can say that eye tracking is to VR as touch was to mobile phones," said Patrik von Bergen, Tobii's head of marketing. "Eye tracking is the natural next step for virtual reality."
Eye tracking will be essential for social VR if our virtual avatars are ever to be truly convincing, but it can also be used for foveated rendering, where you reduce the workload on the PC by only fully rendering the bits you're looking at. That would allow lower-specced computers to run better VR. It's also just more... natural. In VR you often have move your head to focus on things where, in the real world, you'd just move your eyes. But it's only this way because it's the best way for today's VR headsets to know where we're looking.
"What makes us unique is that we've already done this in games," said Bergen. Tobii is providing its development kit to a "limited number" of game studios right now, and the plan is eventually to make it buyable for all developers. As it needs to integrate its trackers into the headsets, we can expect it to start showing up in the next generation of devices, but Tobii won't yet say which ones.
Given it's working close with Valve here, I wouldn't be surprised to see it in a SteamVR headset down the line. It still needs work, but Tobii's eye tracking is far more smooth and accurate than I went in expecting. I don't doubt developers will find interesting ways to apply this technology, I just hope it's not long before this is the norm.
How we test