In 2016, Tribeca Film Festival's Immersive Hub was full of wonky, experimental, five-minute demos of what VR films could be capable of. By 2017, the Manhattan-based festival was teeming with Hollywood directors and actors, all injecting their talent into exciting, but truncated, VR demos.
You could see the beginnings of exciting new VR cinematography and techniques, but inside frustratingly short films cut down by budget and skill limitations. It was unclear if the VR medium could ever evolve past being too experimental and expensive to live up to regular films.
In my third year at TFF, I have seen virtual reality come into its own: with more than 40 VR and AR films and experiences available, and most running between 10 and 30 minutes, even two straight days inside VR headsets wasn't enough to try everything.
Read this: VR filmmaking matures at Sundance 2018
What I did manage to see showed off why Tribeca is one of the premier VR festivals in the world. From holograms on Hololens to an AI controlling a VR film's story, powerful 360-degree camera tech to virtual activism, Tribeca Immersive 2018 showed off how the VR/AR industry could grow in the next couple of years.
Voice-commanding the scene
VR films tend to feel more passive and less immersive than VR games. Even though you're supposedly in the middle of a film scene, you usually can't interact with your environment, communicate with the people around you, or affect the outcome.
But Terminal 3, an AR experience by Asad Malik, uses voice commands to create an interactive narrative and make the user feel in control. As a customs agent, you speak to 3D, volumetric holograms of immigrants looking to visit the United States, dictating whether the conversation is friendly or interrogative and, ultimately, whether or not they will be allowed to enter.
Malik, a Pakistani immigrant, created Terminal 3 as a response to being shouted at by customs agents following the events of 9/11. He chose AR because he wanted viewers to "feel [the immigrants'] presence in real space… forcing them to share presence with ideologies and humans that they otherwise wouldn't cross paths with".
Relying on a Hololens devkit and Alexa voice functionality, Terminal 3 had the most intriguing tech of the show. And it's easy to predict that other AR and VR creators will want to use voice commands as well. VR headsets could eventually incorporate Alexa Skills or Google Actions so that users' words can trigger branching story moments and make the world feel more real.
Speaking with my own voice to a young immigrant about her life – even if the responses were pre-recorded – honestly felt more real than most other VR experiences I've tried.
Gaze-based storytelling
At previous Tribecas, directors like Doug Liman and Eric Darnell told me how important it was to guide the viewer's gaze with cuts, lighting, and other standard Hollywood techniques. They hated when viewers became disengaged and looked at anything that wasn't the main action – which one could argue defeats the purpose of an interactive, 360-degree experience.
With #Wargames, a VR series about a group of hacktivists unravelling a conspiracy, it explicitly does the opposite by giving you several potential protagonists and storylines to follow, changing the footage based on who or what most interests you.
Read this: Bourne director Doug Liman talks VR films
Narratives could eventually respond to biometrics like heart rate and "unconscious gestures" like eye tracking
Sam Barlow, creator of the award-winning game Her Story, spoke to me about his goal of making #Wargames and other VR films as a "comedy standup"-like experience. "You read the room, and if an adult joke doesn't land, you drop those sketches and focus on politics instead".
In the same way, the series gauges which characters you empathise with most and makes them more impactful to the story, or changes characters' personalities to match yours. Barlow told me he believes narratives could even eventually respond to tracked biometrics like heart rate and "unconscious gestures" like eye tracking.
#Wargames' branching narrative experience was much more passive and subtle than, say, Telltale's The Walking Dead, and requires multiple viewings to see how much the story can deviate. But it hints at the role that AI tech could play in future interactive VR or gaming, where an experience senses its audience's boredom or excitement and, like any good standup artist, adjusts itself accordingly.
Hunting for Easter eggs
When VR filmmakers give up on making viewers stick to their mental storyboard, and instead embrace the medium's potential for hiding details for us to hunt down, it makes for some highly entertaining experiences.
Sam's Surreal Gems is a VR series that places the viewer smack dab in the middle of a choreographed single take of random hilarity. You look in all directions for Easter eggs, spotting everything from a woman fishing for a man's toupee to a fugitive prisoner to a blimp blasting a "HELP ME" message.
Future VR films could and should copy this kind of format, and not just for comedic slapstick. Imagine a masked man killing someone, and then the viewer must look for clues around them and decide at the end which witness looks the most guilty.
But AR might be an even better fit for searching for hidden knowledge. At an exhibit called Objects In Mirror AR Closer Than They Appear, I wandered around a recreation of an off-Broadway set, armed with an AR-enabled smartphone inside a stereoscope. When I scanned a random object, it would trigger movie clips, commercials, scenes from the play, or other cool content.
AR could be used for everything from virtual scavenger hunts to educational augmentations at museums to a means of immortalising people's memories in certain objects.
Instant virtual editing
Sam's Surreal Gems was just one of six projects made as part of Samsung VR's Pilot Season program announced at Tribeca. Samsung gave indie filmmakers funding and their $10,000 360 Round cameras to develop long-running VR series, and showed the press how the cameras worked in person for the first time.
360 Round broadcasts eight left- and right-eye 4K streams simultaneously for both live streams and on-demand, and it's both semi-waterproof and extremely durable. One filmmaker, Rick Graves, stuck his 360 Round on top of and inside of a sportscar racing around a track; watching his VR experience felt like a surreal real-life version of a Forza game.
A Samsung technician showed off how the tech worked. While they live-streamed a nearby room, they adjusted the colour volume, frame rate and stitch lines in real time.
Stitch lines appear when objects get too close (within 5 feet) to the camera and the two lenses can't properly connect their streams. But Samsung's editing software lets you shift the lines around frame by frame. So if an actor gets too close to the camera at the wrong angle, the editor can move the stitch off their face – live or in post-production – until they back away.
Beyond that 5-foot window, though, you're looking at multiple simultaneous 4K camera angles. Graves said that outside of his VR filming, he actually uses his 360 Round for traditional 2D filming, because it gives him front, side and back angles of a scene in just one take.
Tourism + activism
For all the technical innovations at Tribeca this year, many of the experiences aren't paving new ground or portending future trends. Instead, they followed Kathryn Bigelow's lead from last year's anti-poaching VR film The Protectors, combining Hollywood production values and exotic locales with tragic stories to support a charitable cause.
Into The Now lures viewers in with sharks and deep-sea-diving, then delivers its anti-overfishing, anti-climate change message. Lupita Nyong'o narrates My Africa, another anti-poaching video with beautiful nature shots and a baby-elephant-feeding minigame. Coral Compass and Pale Blue Dot take you coral reef diving and into space to support conservation and clean energy.
Last year we spoke with several VR experts who said that VR is a "memorable and persuasive tool" for improving empathy in viewers for the marginalised and oppressed. These sorts of activist films disguised as tourism videos could have a powerful impact, convincing viewers to take a greater interest in the struggles of people halfway across the world – though VR is a while away from being a perfect empathy machine.
The best of the bunch
I waited a full year to see the conclusion to Penrose Studios' Arden's Wake: Tide's Fall, after last year's demo ended on a painful cliffhanger. Without any spoilers, it was worth the wait. The story of Meena, now voiced by Oscar winner Alicia Vikander, as she quests to save her father and confront her tragic childhood is a standout of this year's festival.
It was one of the most affecting "real world" VR experiences I've encountered
Another excellent experience in the lineup is Hero, in which you're a tourist in a war zone, putting you in Syria after a bomb goes off. The experience at Tribeca used a wireless VR rig and a fully-built prop set matching the virtual world, so I could walk around and touch my surroundings. And project creator Navid Khonsari, who previously worked on motion capture at Rockstar and Resident Evil 7 VR, told me that he hired Syrian actors for the voices to preserve realism. Overall, it was one of the most affecting "real world" VR experiences I've encountered.
By complete contrast, LAMBCHILD SUPERSTAR: Making Music In The Menagerie Of The Holy Cow, eschews any realism for a hilariously entertaining co-op music game, where you partner up to compose music. With or without any musical talent, with a few friends this could be a really great party.
But these "best of the festival" choices are pretty subjective. What's truly exciting is that nearly all of Tribeca's VR films achieved a level of skillful professionalism and technical proficiency that had been lacking before. Indie filmmakers made these experiences, but they felt anything but amateur.
How we test