MIT’s CSAIL Researchers Develop a Glasses-Free Way to Watch 3D Movies at Home: the Home3D System

“Star Wars: The Force Awakens” was worth 3D glasses

Going to see a movie in 3D is a cool experience, but it comes with an essential tool – 3D glasses. As I already wear glasses, not contact lenses, it has to be a really good movie for me to pay extra to see it in 3D and spend two-plus hours wearing two pairs of glasses. While 3D movies are a pretty common occurrence these days, a 3D television in your home is not, and it has a lot to do with those glasses. Movie theaters either project a pair of images onto the screen to create a simulated sense of depth in the movie, or they use a special polarized light, so 3D glasses are pretty much a necessary evil…unless you ask the brilliant researchers at MIT’s Computer Science and Artificial Intelligence Lab (CSAIL). Most recently, CSAIL was busy creating programmable soft 3D printing materials and developing a new modeling system for multi-material 3D prints, but the researchers have since turned their eyes to our TV screens with the new Home3D system.

The system, which builds off CSAIL’s 2016 work on glasses-free 3D for the movies, lets you watch 3D movies from the comfort of your couch without 3D glasses. Home3D actually converts your traditional 3D movies into a different format – one that’s compatible with what’s known as an automultiscopic display, which has a lot of potential for use in home theater systems.

Petr Kellnhofer

Petr Kellnhofer, CSAIL postdoctoral associate, said, “Automultiscopic displays aren’t as popular as they could be because they can’t actually play the stereo formats that traditional 3D movies use in theaters. By converting existing 3D movies to this format, our system helps open the door to bringing 3D TVs into people’s homes.”

Kellnhofer was the lead author on a paper, titled “3DTV at Home: Eulerian-Lagrangian Stereo-to-Multiview Conversion,” that he will be presenting later this month at the SIGGRAPH computer graphics conference in Los Angeles; co-authors include Chulalongkorn University postdoc Pitchaya Sitthi-Amorn; MIT professors Fredo Durand, William Freeman and Wojciech Matusik; former master’s student Szo-Po Wang; and former CSAIL postdoc Piotr Didyk, who’s now at Saarland University and the Max-Planck Institute.

The Home3D system converts 3D movies from stereoscopic to multiview video, which means that instead of a pair of images, the screen will display three or more images that simulate what the scene looks like from different locations. A person’s eye will take these images and perceive what it would actually see at the location, which lets the brain “naturally compute the depth in the image.”

The two existing techniques that are used to convert 3D movies are limited in what they can offer. Phase-based rendering offers high-resolution, and is fast and largely accurate, but when the right-eye and left-eye images differ too greatly from each other, the technique can’t handle it. Depth image-based rendering can manage the differences better, but only because it runs at low resolution, which means that small details can be missed.

The CSAIL research team came up with a new algorithm, which combines elements from both of these rendering techniques: it’s able to handle bigger right and left differences, and it can resolve things like reflections and depths of focus. In addition, the Home3D algorithms allow users to customize their viewing experience by raising or lowering the desired 3D level for movies. A user study borrowed clips from Big Buck Bunny and The Avengers, and 60% of the time, participants rated the Home3D movies as having higher quality than the existing movies.

According to Gordon Wetzstein, an assistant professor of electrical engineering at Stanford University who was not involved with the CSAIL project, “The researchers have used several clever algorithmic tricks to reduce a lot of the artifacts that previous algorithms suffered from and they made it work in real-time. This is the first paper that produces extremely high-quality multiview content from existing stereoscopic footage.”

However, one of the issues that comes from converting traditional 3D video to multiview TVs is ghosting, and I’m not talking about that thing when you don’t want to date someone anymore and stop communicating with them, hoping they’ll take the hint, rather than just ending the relationship. In 3D TV terms, this is when the limited resolution can cause images to have duplicates near them. The CSAIL team hopes to continue perfecting their algorithm to minimize this phenomenon.

With all of the high-tech viewing options out there, most modern TVs have a high enough resolution that it can even be hard to tell the difference for 2D content.

Didyk says, “But using it for glasses-free 3D is a compelling application, because it makes great use of the additional pixels they can provide.”

The team is excited that the Home3D conversion system has the potential to bring 3D movies out of the theater and into your home – the system can run in real-time on a GPU, so if you have a PlayStation or Xbox, you’re set.

“Glasses-free 3D TV is often considered a chicken-and-egg problem. Without the content, who needs good 3D TV display technology? Without the technology, who would produce high-quality content? This research partly solves the lack-of-content problem, which is really exciting,” Wetzstein said.

In the future, Home3D could maybe even take the form of a chip, which could be put into TVs or media players, such as Google Chromecast. Discuss in the Glasses-Free 3D forum at 3DPB.com.