Made you look: How to tell stories, hold attention and trick brains with VR

What comes after the eye-popping demos?
Made you look: Telling stories in VR

A train arrives into La Ciotat station. A shark pops out of the screen. A virtual helicopter takes you flying over New York City. Whether it's 1895, 1995 or 2015, new technologies - from the invention of cinema to 3D movies to 360 degree video - guarantee one thing. People will rush to create wow moments to show off what the medium can do. Then these get cheesy.

VR is no different, we can't move for eye-popping, pant-wetting demos but what happens when the novelty wears thin?

Read this: How virtual reality storytelling is taking flight in 2015

It will take time to get there but real, constructed VR movies, series and games with actual stories and narratives are in development. This is how storytellers and game designers are going to make the leap to getting viewers to invest in the stories they are creating.

Fresh but familiar

What better place for this futuristic medium to start than the past. The demos and shorts published so far are essentially recreating the experience of going to the theatre. That's according to Tim Smith, senior lecturer in cognitive psychology at Birkbeck, University of London, who has conducted studies into how filmmakers manipulate viewers' perception and attention using eye-tracking cameras and glasses.

"The issue I see with a lot of the current experiments in VR is that they're creating fully immersive theatre," he told Wareable. "You're basically sat in an optimal seating position in an auditorium in front of a stage. You're looking through the fourth wall at a theatrical performance in front of you. It has to be something more than that. Why would we go back in time? Why would we regress back to theatrical modes of storytelling when cinema has advanced so far beyond that?"

Working within the constraints of VR to create something that's fresh but more than a 3D-style novelty is the challenge facing directors, publishers and game designers interested in 360 degree video.

It's something Edward Miller, head of visuals at VR news app, tried to thrash out in his Digital Shoreditch talk on the future of broadcast television. Miller created one of the first pieces of journalistic 360 degree storytelling, a seven minute documentary on the 2014 Hong Kong protests.

"It seems that we're at the stage with VR that cinema was at when it was first released in terms of trying to wow," said Miller. "So you might remember this scene, this story of some filmmakers capturing a scene of a train arriving at a station. Everybody inside the theatre threw their hands up because they thought the train was coming towards them, they thought it was going to hit them.

"We're in a very similar stage - it's easy to put a headset on somebody and for them to say this is the future. That's different to someone saying this is what I want to experience again and again and again the same way we do with TV. So we still have a lot of development to do. Do we need this 360, spherical view around us or can we make do with 180 degrees for immersive broadcast?"

Who is in control?

EVE: Valkyrie ready to define VR

There's a new word for a short film or minimally interactive game and it's experience. Plenty of people creating work for VR now refer to them as experiences. Why? Because a lot of the time, as the viewer or player we're not sure how interactive what we are about to watch/play/experience will be.

If you've tried out Oculus Rift or HTC Vive demos, you'll be familiar with this feeling of exploration. We played a Vive demo based on the excellent mobile game The Room and barely noticed the lack of actual gameplay, it was so gorgeous to explore. There aren't really any rules yet and that's exciting.

There's a number of reasons for this. Firstly, though the platform is maturely rapidly, it will take time before the content being made for VR headsets settles down into a canonical form that works. Secondly, directors are used to being directors, game designers are used to designing games.

"It's always easy for artists to take baby steps from their existing comfort zone into a new domain," said Smith. "They're saying let's take a film and make it minimally interactive. We don't want to go too far because then we lose control. Or we take games and port them into VR and all that we're really giving you is fully immersive 3D vision and sound. VR has great potential for creating whole new experiences which weren't possible before."

It all comes down to who is control - the creator or the viewer-player. For game designers, this can depend on the style of game - shooting, driving, space exploration - and the input from controllers to head tracking to voice.

For Project Nemesis, an EVE spin-off and Samsung Gear VR space shooter title, CCP Games' Tom Farrer told Wareable that his team of designers "plan the waves of incoming attacks" so the player isn't moving their head constantly. And since you're in a fixed cockpit defending a mining platform, there's nothing to see behind you.

Read this: The problems facing VR game designers

EVE: Valkyrie, on the other hand, is a multiplayer dogfighting game for Oculus Rift and Morpheus so where the action is - above you, below you, at odd angles - will depend on your teammates and rival players as it would in a traditional console game. Still, the designers added in bright fighter trails to try to bring a sense of story to the chaos.

"These long, looping bright lights are almost like a narrative of what's been happening for the last five or six seconds," said VR Experience Designer Andrew Robinson. "That painted, visual narrative was so strong and it was something we wanted to sustain in the new versions of the game."

Cinema's bag of tricks

For directors, relinquishing control is new territory. If a viewer really wants to spend the entire time looking up above them regardless of what is happening in terms of a narrative in front of them, the director of a VR movie has to sigh and get over it. But cinema has plenty to offer in terms of tricks to tell stories and curate our attention in this new relationship.

The most obvious methods include using 3D audio to alert us to the direction of action happening in another part of the VR world as well as sudden movement or dialogue. But it gets much more interesting than that.

"One technique for VR would be to basically violate the laws of physics and scramble up when things happen," explained Smith. "It could be that I'm looking over the left hand side at Bob and Margaret who are having an argument but in actual fact Phyllis is picking up a knife behind me. I need to know that to infer that later Phyllis is going to use it in a fight. You can delay that reveal until I've voluntarily looked to the right part of the screen.

"And that's similar to what a filmmaker would traditionally do. They have to somewhat warp space and time in the way they show you events happening either by cross cutting or actually moving backwards and forwards in time."

By using traditional cinematic techniques, it also gives VR filmmakers the opportunity to stealthily craft scenes and manipulate our emotions without us realising too. The simple way to do this is to use social cues - we are social oriented creatures so we assume new information is more likely to come from human-like faces than anywhere else.

"When I first see the establishing shot in a film, my eyes go all over the place looking at where the location is, who the characters are," said Smith. "Then very quickly my attention goes to the characters' faces. When I come into a VR scene first, the layout of the scene is established from my static viewpoint. When someone suddenly moves, strikes a person or starts speaking, my attention is only going to focused towards the source of that sound or that act not elsewhere.

"At that moment it gives me, the director, an opportunity to do something, maybe change the players positions, change the lighting, change the details. By knowing where the viewer is looking, I can then start to craft the visual experience in a similar way to the filmmaker without having to get them locked down to the physical space and a single viewpoint that seems to be inherent in VR."

Gaze grabbing

What's next for VR animation

Dan Ferguson is head of digital at Reel FX, the studio behind The Book of Life CG animated film, and there's more animated VR to follow the 'experience' Reel FX created for the Oculus Rift. What's coming will be snackable but it won't be marketing content and most of all it will tell a story.

"Having a chair that rotates cannot be the solution to holding viewers' attention in VR," he told Wareable. "You have to tell a story. Even cutting from scene to scene is actually OK as long as you have some consistency in that so you're not cutting all the time."

Embodying a CG character is one way to draw viewers into a story whether or not the VR system allows for motion tracking or limb tracking. "For The Book of Life experience, you became one of the main characters in the movie Manolo," said Ferguson. "When I look down, I don't see my neck because you can't see your neck but you see the chest and arms of a character. And it feels very natural. It's a great way of bringing people into the story that doesn't feel strange."

As for the question of control of where we are focusing our attention, Ferguson notes that of all the people who have tried the VR Book of Life experience, older people are most likely to sit down and just look straight in front of them, disregarding most of the 360 degree environment.

It's certainly possible that exploring 360, virtual worlds will become just as natural for future generations as toddlers prodding anything that looks like a touchscreen with stubby fingers. The old habits of sitting down to view a movie in a window on the living room wall will seem quaint.

VR storytelling takes flight

Even if our heads are looking straight ahead, new headsets such as Fove are popping up that offer not just head tracking but also gaze tracking. This opens up new techniques such as blurring out portions of the 3D scene that we are not focusing on. This produces a sense of depth of field for VR that's similar to how our eyes operate and more comfortable for longer sessions.

"The added benefit on top of that is that the gaze becomes an interface device," said Smith. "Either deliberately via a gaze cursor for shooting or implicitly in the way that other agents (for instance, fictional CG characters) know where you're looking and they can be a lot more socially authentic in their interaction.

"Where we're at now, the technology is in the uncanny valley of human computer interaction. As is the case with the animation version of the uncanny valley, it's not good enough for it to quite work, we want it to be better. Better audio, capture of motion or facial expressions. But it's better than what we had before. We're too close and not close enough."

We might be too close but not close enough to truly fool our eyes and brains into inhabiting a VR story. But the tech is good enough and the immersion entertaining enough that within 12 months we'll have VR games from publishers like CCP and snackable, animated series from studios like Reel FX. Maybe even feature length movies. Now we just need some good stories to tell.

What do you think?

Connect with Facebook, Twitter, or just enter your email to sign in and comment.