If you've ever held your hand out and imagined yourself moving objects like a superhero, you know how lustful it is to want to control things with your hand. I saw that power when I tried out the AR Sandbox built by UC Davis researchers, holding my hand above it and making it rain like a god, watching the newly spawned water flow down hills, running through trenches and resting in valleys.
The incredibly addictive Sandbox is an actual box of sand with a topographical map projected onto it and its part of the team's pursuit of ways to create 3D experiences that make sense of work related to as AR, VR and mixed reality. Also in development are Keck CAVEs which are best described as a sort of all-immersive HoloDeck in which you wear 3D glasses. Here's how I got on during my day at the labs. Spoiler: it was awesome.
Enter the Sandbox
The Sandbox is easy to work. The only thing you really need to know is that opening your hand above it makes it rain, with your hand "pretending" to be a cloud. Everything else comes naturally.
You make a mountain of sand, the projection changes to recognize that you've created a mountain. You build a valley, the map changes to show you a valley. If you make it rain, the water acts like water. It'll run down that mountain and settle in that valley. If you turn that valley into a mountain, the water will roll elsewhere. If you create a trench and want the water to go from the valley to somewhere around the mountain, go ahead.
It all works seamlessly, and it's hard to believe it's not magic. Oliver Kreylos, research scientist at UC Davis, sadly reveals that it isn't. Instead, it's a system of technology that fades into the background. It works like this: A Kinect sensor captures what it sees in the sandbox and makes a 3D model, which is then projected back onto the sand. The projection mapping has been calibrated down to about a millimeter, so you can't notice that it's being projected unless you're looking for differences.
It's hard to believe it's not magic
That magic is both incredibly addictive to play with, while still being informative; it's a superb showcase of what future AR in education, in our headsets, could be. While the Sandbox currently works without any facewear, Kreylos said the system - alongside the KeckCAVE - can easily be transferred into something like the Microsoft HoloLens or even a VR headset. The problem, however, is that the tech just isn't there right now. HoloLens' field of view is far too small for the wide immersion of both the projects.
From VR to AR
The AR Sandbox originally started out as a VR project, Kreylos says. UC Davis' Earth and Planetary Sciences Department teamed up with three museums: ECHO Lake Aquarium and Science Center in Burlington, Vermont; UC Davis Tahoe Environmental Research Center in Nevada; and the Lawrence Hall of Science in Berkeley, California. The original plan was to make big stereoscopic screens and use its VR software to create informative experiences.
VR and AR are really two sides of the same coin
However, one day in 2011 a member of the team was browsing YouTube and happened to watch a video of an augmented reality sandbox created by two students from the Czech Republic. Kreylos saw it and knew they immediately needed to replicate it, adding the water simulation. He had worked with both the Kinect sensor and projection mapping before, so he knew what they needed to do. However, the biggest aid to the project was Davis' software.
The future is AR
- Magic Leap: Holographic hype or the future of AR?AR Week: We're wondering if we'll ever get to play with that light field
- We need to get AR ethics, ownership and safety rightAR Week: What happens when virtual and real worlds collide?
- From ARKit to HoloLens: Watch the best AR demosWe explore which developers are making AR magic
- The future of AR is...AR Week: Editors, experts and industry insiders cast their predictions
- Can VR and AR make you fit?AR Week: We chat to FITAR about burning calories in virtual and augmented worlds
"The reason we could pull it off so quickly, and why it worked out so well, is because we had a giant stack of software that was available for us that we had been developing for honestly the last 20 years primarily to drive VR work," Kreylos explains. "But if you think about it VR and AR are really two sides of the same coin so all of the software we had was just directly usable for that."
The team made three AR Sandboxes, one for each museum. The software was open sourced and uploaded onto the internet. All of a sudden, the project exploded in popularity. People were watching the videos Kreylos uploaded of the Sandbox to YouTube. People were also using the source code to make their own AR Sandboxes.
Kreylos says every two days they receive a new email about someone in the world who has made their own Sandbox. Davis tries to keep track of all of them via a map, which currently has 300 different Sandboxes accounted for, but they haven't been able to keep tabs on all of them.
It turns out the technology behind the AR Sandbox isn't just for creating cool ways to learn about geology and water dynamics. Kreylos says third-party groups have approached him and Davis about creating customized versions of the technology for different uses - like the US Forestry Service and AutoDesk.
"So they would just go and get a chunk of the forest terrain that's going to be burning, and then use the existing means the Sandbox has to recreate that terrain in the sandbox," Kreylos says. "Instead of a water simulation we would have replaced it with a fire simulation. Then they would have been able to recreate the fire situation and use the Sandbox to plan the fire fighting approach. Where to dig the fire trenches and where to set a counter fire or where to drop the firefighters."
The engine in the Cave
The tech behind AR Sandbox is birthed from Keck CAVE, which was developed back in 2003 when Louise Kellogg, former chair of Davis' Earth and Planetary Sciences division and current head of W.M. Keck Center for Active Visualization in the Earth Sciences (KeckCAVES), needed something to help her make sense of her advanced Earth simulations. She got some colleagues together, they applied for funding, hired Kreylos and made Keck CAVE.
So what is the Keck CAVE? It looks like a HoloDeck. There are three screen walls with projections on them, as well as a floor. Up above, there are sensors and blinking lights keeping track of everything within the Keck CAVE. There's also a giant mirror and projector projecting images into the middle of the Keck CAVE. When you first walk in, it all feels like a projection show of some kind. But then you put on the 3D glasses, and the CAVE turns into the most immersive mixed reality experience you could ever have.
Read next: Augmented reality v mixed reality
There is no limited field of view here. In my demo, Kreylos brought up a map of Earth keeping track of earthquakes. When he zoomed into the map, it felt like I had left the room and gone inside a virtual Earth. The 3D glasses give everything an extra dimension, and you're able to use wand-like controllers to easily move and manipulate things in the CAVE. Also, it's social. Because the CAVE was designed for researchers, who often work in groups, it needs researchers to be able to see each other and the content at the same time.
The CAVE isn't VR. You're not wearing a headset that blocks you off from the real world. At the same time, it's not AR either. It's an artificial environment that isn't laying out things over your reality. It's a bit of both, but there is a catch for all that immersion: it's expensive to buy and maintain. It cost the group $500,000 back in 2003, and prices haven't gone down. The lifespan of the system is supposed to be five years, but Kreylos and researchers have managed to stretch it out to 12 years.
One of the problems with UC Davis' VR and AR efforts is that it's limited by the number of computer scientists that work on them. The school doesn't have a VR program, which makes it hard to get computer scientists to further the programming side of things.
One thing is for sure though. UC Davis has two pieces of technology that are totally focused on educating in the most addictive way possible. Both technologies are so wildly immersive they feel magical. Even crazier, they were both created with technology that is older than many of the high-end AR and VR systems we have today.
There are still a ton of questions about AR but UC Davis' technology feels like a sign of things to come, whether its the realization of the AR visions we've seen over the years or the dream AR glasses that are so incredibly hard to make. They feel like the verge of something bigger and better on the way.