RIT Student Combines HoloLens Glasses with HTC Vive System to Make New Shared Reality App

Whether you’re reaching out and grabbing items in a game using the Dexma Exoskeleton, or taking an up close and personal look at the human brain, what’s not to love about entering a completely immersive environment from the comfort of your couch? Virtual reality (VR) is especially magical if you’re using it to create art in thin air. But Drew Gottlieb, a software developer studying computer science at the Rochester Institute of Technology, realized that if one person is creating 3D art using virtual reality, it’s not much fun for the other friends in the room: they’re only able to see what’s being created by taking a look at a rather distorted view, on a computer monitor, of the other person’s perspective.

Gottlieb was recently with a friend who was painting a 3D submarine using the Google Tilt Brush, which is basically a hand-held high-tech handle, used to paint in the air while wearing a VR headset. When Gottlieb’s friend told him to look at a periscope she’d drawn with the Tilt Brush, the only way he could see it was for his friend to lean in closer to the periscope while wearing her VR headset, as he leaned in closer to his monitor. His opinion? “We can do better than this.”

Gottlieb says he is on “a mission to empower creativity with computing that feels human,” and has created projects using PHP, Java, Python, and C++, among others. Until now, it would have been impossible for a person not using a Tilt Brush and wearing a VR headset to see the created artwork without getting up and going to the computer. But Microsoft recently released development kits of their mixed-reality HoloLens glasses, which we’ve written about before. Gottlieb has access to a few of these units, and decided to make a proof of concept for people to use mixed reality to share in another person’s VR experience. He notes on his website that he recently started an internship with Google working on the Tilt Brush, and that this project was his own work before the internship began and has no Google association.

His shared reality system, called the HoloViveObserver, runs on a VR system called the HTC Vive, which comes with two postionally-tracked controllers and was recently on the Top Tech List at CES 2017. Instead of wasting time trying make something like Tilt Brush, Gottlieb instead put together an app where the player using VR can draw simple shapes like cubes in thin air with their controller.

According to Gottlieb, “The interesting part is that when the same app runs on a HoloLens, it automatically connects to the VR session using Unity‘s built-in networking and matchmaking service.”

Unity is a game development platform, where users can make a highly-optimized 2D or 3D game and utilize the site’s integrated services to speed up the game development process. Gottlieb’s new app uses the HoloLens to search for the VR environment, and aligns it with the Vive environment, using both a physical Vive controller and a virtual one. Thus, a shared reality environment is born.

“A challenge in this experiment was not just getting the Vive and HoloLens to talk to each other, but to bring them to a shared understanding of space,” Gottlieb wrote. “How do we align the virtual and room spaces? Ideally this would happen automatically, but I couldn’t think of a way to do this just using the technology offered by the Vive and HoloLens. Perhaps you could attach tracking symbols to the VR base stations?”

Gottlieb came up with a quicker solution. The game enters what is called “alignment mode” when the HoloLens app connects to the VR app: the HoloLens will actually speak, and prompt the user to pick up a physical Vive controller and intersect it in the air with a virtual controller. This can be a little tricky, as there are a total of four degrees of alignment to match up: one degree of rotation around the room’s vertical axis, and three position degrees. Gottlieb explained that the biggest error when trying to enter alignment mode is when there’s a slight rotation inaccuracy when you’re trying to align the controllers: if you’re just one degree off, users will start to see increasing misalignment as they move away from the original point. Gottlieb says he plans to prompt the the alignment of three points, instead of just one, in the future, to reduce this degree of error. This fix will use the three points’ position information to set the rotation, and ignore the controller’s rotations.

When the two controllers are aligned, the HoloLens user can pull the controller’s trigger, and the HoloLens will announce, “You are now aligned.” Gottlieb says he’s successfully tested two HoloLens devices in the same session, and that there’s no limit to how many people can join in on the same session. One of the cool takeaways from his project is that people are now able to work together in the same space, across both holographic and virtual environments. If you’re the spectator wearing the HoloLens, and you have a cool idea, you can ask the VR player in the room to hand over a controller, and you’re able to interact with the art as well. The other option is both people wearing their own HoloLens: each person can take a controller, and you can build larger shapes together.

“I have no doubt this kind of mixed space will be a big part of the future, especially for creative industries,” Gottlieb wrote. “As virtual and mixed reality become stronger platforms for content creation, it’s only inevitable that they’ll be able to interact on a whim.”

If you’d like to try this project out yourself, and happen to have both a $3,000 HoloLens and the $800 HTC Vive, Gottlieb made the app open source; you can get the source code from GitHub here. Watch this video Gottlieb posted to see how the HoloViveObserver works:

Discuss in the Shared Reality forum at 3DPB.com.