Sometime between exploring Allumette's city in the clouds in Penrose Studios's swanky Airbnb studio and interviewing the director of the Madagascar movies at Tribeca's red carpet, I realized that VR filmmaking has already achieved an aura of legitimacy and profitability rivaling many traditional films. I was observing studios at the forefront of a budding VR film movement.
Both Penrose and Baobab Studios have raised millions in seed money for their VR films, on the strength of pitches by talented filmmakers and designers like former Pixar and Oculus Studios creator Eugene Chung and Hollywood director Eric Darnell.
Read next: VR experiments at Sundance
After months of critically acclaimed tech demos, both studios are showcasing the final versions of their films to the public at the Tribeca Film Festival's Virtual Arcade. Based on my first impressions of these films, Allumette (about 20 minutes) and Invasion! (about seven minutes) should be hugely popular: the combination of graphical beauty and storytelling talent was on full display in both releases.
I had the opportunity to speak at length with both creative teams on their journey to Tribeca, the difficulty of storyboarding in VR and the future of the medium.
Jumping from traditional animation to VR
We have traditional filmmaking down to a science, but no one knows VR film language yet.
Eugene Chung left his cushy Pixar gig to pursue VR filmmaking in 2012. "I strapped on an Oculus DK1 and was blown away by the demo. But then the Oculus team told me they were all 'game people.' They knew the Rift could have a huge impact on filmmaking but had no idea how to do it themselves. That's when I started up Oculus Story Studios."
Chung's team would release seminal VR films like Henry and Lost before he left Oculus to found his own VR company, Penrose Studios.
Eric Darnell, director of the popular Madagascar films, also left his Hollywood studio to co-found Baobab Studios with Maureen Fan, the former VP of Games at Zynga.
Must read list: Best VR and AR headsets
"We have traditional filmmaking down to a science, but no one knows VR film language yet," Darnell explained. "It's exciting, frontier territory. I worked on computer generation in the Wild West of the 80s, so being at the forefront of VR is just as exciting and challenging."
Darnell and Fan joined forces over their shared vision of animated films as being singularly suited for virtual immersion. "My first experience with live-action VR was pixelated and underwhelming," Fan confessed. "But my first animated VR experience had me fully immersed in what felt like art in motion. I knew then VR could put someone into different worlds and make you feel they're real."
Graphical challenges and 2D-to-3D transitions
We've come to expect Pixar-level graphics to our animated films these days. Unfortunately for the filmmakers at Penrose and Baobab, immersive, graphically intensive worlds are much more difficult to render in VR.
Jimmy Maidens, the Lead Technical Director at Penrose, said that Allumette uses "90 frames-per-second (fps) to two separate eye streams - one image every ninetieth of a second." Traditional animation standards used to be 24fps, and it could take a studio four days to render a single image.
By comparison, the Baobab team rendered Invasion! at 30fps because Darnell claimed 90fps looks unnatural to viewers. But the slower rendering led to fast-moving objects noticeably strobing and stuttering, so they used motion blur and moderate amounts of 90fps on the action scenes to compensate.
VR filmmakers also can't cheat on backgrounds the way Hollywood graphical teams do. Maciej Gilwa, animation supervisor at Baobab, talked about how Hollywood creates 2D background art that looks 3D from one camera perspective. But with VR he says, "You never know where your viewer is going to look or walk. Everything has to have sculpted dimensions at all times."
Explained: The origins of virtual reality
Maidens made a similar point, noting that every single cloud, building, road and lampshade was painstakingly rendered and placed in their environment to preserve the immersion. This became especially difficult when they realized how poorly 3D art drawn on a 2D screen translates to VR.
"You spend hours graphically designing something on your computer, but when you look at it on your headset you realize the proportions are all wrong. We kept going back and forth making edits without knowing how things would eventually look. Eventually we just started drawing assets directly in the VR world."
Baobab also follows this strategy. "We have our story artists draw directly inside the 3D stage, so the art becomes a guide for animators," Darnell said.
Right now, Baobab is using Unreal Engine to design its VR films. The platform is currently more reliable for video game development than VR film production, but Fan believes Unreal and Unity are working hard to create dedicated VR platforms designed for working in a virtual space.
Commanding your viewers' gaze
We didn't stop making changes until 99% watched the story the way we wanted them to.
One of the thrills of a VR movie is the freedom to choose where to focus your attention. But for VR filmmakers, the excitement comes from pushing viewers to ignore that freedom to focus solely on your story.
"If a viewer is bored with a 2D movie they have nowhere else to look. They're stuck," joked Darnell. "But in VR if they're bored or distracted they can look away from the narrative."
To overcome this, Darnell adopted the misdirection philosophy of magicians: "If you want the audience to look at something, look at it; if you want the audience to look at you, look at the audience." Watching Invasion! at the Baobab release party on Saturday, I saw this in action.
As the adorable bunny above ran up to me, I found I didn't want to break eye contact. And when she turned to look at an approaching spaceship, I naturally followed her gaze to the next part of the story rather than looking elsewhere.
"The only way to know you're telling the story right is to test," Darnell said. "We showed Invasion! to 1,700 different people and we didn't stop making changes until 99% watched the story the way we wanted them to."
If Darnell's philosophy was more controlling and exacting, Allumette's creators had a more relaxed, trusting view of their viewers' attention spans.
Up next: How does VR work?
"Our first focus was to build a transcendent world," explained Chung. "Allumette (the protagonist) is just one character in this world, and our goal is for people to explore that setting and meet many different people while following along with the main stories of that world."
Without giving away the plot of Allumette's poignant emotional arc, I will say that the characters' storylines were compelling but almost overshadowed at times by the joy of exploring the beautiful city in the clouds. I wanted to follow Allumette's story, but I also wanted to stick my head in every building to see what was there.
Maidens described that precarious balance between encouraging exploration and telling a compelling narrative. "We wanted to create incentives to explore our world, to try and find all of its secrets. But you can't place too many Easter eggs out there, or else you distract from the core story."
Making VR universally accessible
During our discussion, Darnell stressed several times that VR should be something viewers of all ages can enjoy, not just tech gurus. That means respecting just how affecting the VR experience can be for young viewers.
"For an old Invasion! demo we had a hawk kill the bunny to save on animation time. A mother who watched it said she loved it but she couldn't show it to her kid. In the theater that kind of dark humor is okay, because the adults will laugh and the kid could turn to mom for comfort. But immersed in a headset, alone, with no social comfort, her kid could be traumatized."
Your next VR fix
Another notorious barrier for universal VR enjoyment is simulation sickness, triggered when a VR world moves around you but your body stays still. Penrose's team takes special care to tackle this problem.
"Eugene is incredibly susceptible to simulation sickness," Maidens said, laughing. "Practically any artificial movement will get him queasy. So we made sure that in Allumette the overall world remains static." Instead, viewers can control their own perspective and movement.
The future of VR filmmaking
Penrose's team believes virtual reality is just the first step in a larger technological movement, and plan to stay on top of these trends.
"I don't think either virtual reality or augmented reality will solely dominate the market," said Maidens. "What will dominate it probably hasn't even been invented yet. Some kind of mixed reality."
"Virtual and augmented reality will be the next ubiquitous computing platform after smartphones," says Chung. "But when iPhones came out no one knew what to do with them until Apple created example apps. Now mobile apps are a huge platform.
"Penrose is one of the few VR studios that actually got paid to make VR content last year, but we think the alternate reality ecosystem is going to grow immensely in the next few years, and we hope other studios will follow our example and grow the industry."
Baobab, meanwhile, has big plans for continuing the Invasion! story to please its growing fan base.
Headset head-to-head: HTC Vive versus Oculus Rift
"Invasion! is just the first chapter of a longer story," says Fan. "We have more episodes written already. But the story will change based on how audiences react."
"For example, we didn't anticipate just how much everyone would love the bunny. We planned on focusing on the aliens for the sequels, but the bunny might make a reappearance."
Darnell wants to make Invasion! his next big franchise. "Our hope is that you might see our bunny on a kid's lunchbox one day." We'll see if VR films will end up having that kind of cultural impact. I suspect if any of them do, these two films might just lead the way.