BitDrones Allow Users to Program Matter and Interact with 3D Computer Displays

A group of researchers from Queens University’s Human Media Lab think that the future of computing will be a mixture of the gesture based interactivity of Virtual Reality technology with interactive self-levitating programmable matter. The group created a concept project called BitDrones, which is an interactive swarm of flying 3D pixels – or voxels if you want to be technical – that respond to the needs of the user. BitDrones can be used to 3D model in real space, create displays, interface with computers and view virtual 3D information and files. They were unveiled this week at the ACM Symposium on User Interface Software and Technology in Charlotte, North Carolina and the researchers published a paper outlining their concept.

While the eventual goal is to reduce the size and versatility of the drones, creating literal swarms of nano components, for the research team’s working concept the floating interface is represented by three different types of small-scale quadcopters. A ShapeDrone is intended to be the building blocks of a 3D model, it is covered in a mesh material and a 3D printed geometric frame that is easy to manipulate in 3D space. The PixelDrone will include an LED light and a simple dot matrix display. While the DisplayDrones come equipped with a forward-facing camera, an Android smartphone board and a curved flexible high-resolution touchscreen. The touchscreen can be interacted with just as any, but it will also act as a video screen for communicating with other users via video chat. All of the BitDrones themselves have reflective markers and will be tracked in real-time with modern motion capture technology. And it turns out that the team’s research is based off of early research into the same technology dating as far back as the 1960s.

Below, left to right: DisplayDrone, ShapeDrone, PixelDrone

“In 1965, [computer scientist Ivan] Sutherland envisioned the ‘Ultimate Display’ as a room in which the computer controlled the existence of matter. Such programmable matter would consist of small, parallel, cellular automata nodes capable of geometrically shaping themselves in 3D space to create any kind of material structure. Since then, there has been a significant amount of research conducted towards this goal under various monikers, such as Claytronics, Organic User Interfaces, and Radical Atoms,” the Queens engineers explained in their BitDrone paper outlining the concept.

The current BitDrone concept, while still quite remarkable, is at the moment limited and still extremely early in the development cycle, probably years or deades too early to be viable. But the concept that the team put together is still a fantastic working model of how it would function once technology allows it to be scaled down. The three different varieties of drones are controlled with MultiWii software running from a standard iMac, and the system is programmed to track several markers on the user’s hands which allows for the gesture-based inputs. Turbulence in the environment, which tends to limit how well the drones are capable of interacting with each other, seems to be the primary problem uncovered by working with the prototype.

Clearly a more reliable method of maintaining stable connections between drones is one of the many hurdles that needs to be overcome before their technology is ready for real world applications. Most of the work that the team used as preliminary research was primarily theoretical, and even with today’s technology is years away from being build-able. Coping with gravity remains the main problem, as the levitation methods envisioned for controlling the Catoms, the nano-scale robotic components that would make up the swarm of drones, in 3D space are extremely limited. Specifically ultrasound technology and magnetic levitation are currently unsuitable to produce the type of complex movements required for the system to work as intended.

“The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked,” wrote Ivan E. Sutherland in a 1965 paper used as inspiration by the Queens team of researchers.

For their prototype the Queens research team built the larger-scale nano quadcopter drones to represent how an eventual working programmable matter system will function. Each of the BitDrones represents a Catom that can hover within an area of 4m x 4m x 3m in size. Because they maintain their configuration in 3D space, users can walk around them or interact with each individual drone by touch. A single drone can be repositioned simply by moving it into the desired placement, and because the drones are all linked together, the entire swarm will respond and shift orientations as a unit if two of the drones are moved. Essentially the real world equivalent of using a mouse to move a 3D object within a CAD program. The prototype also allows communication and interaction between users of two different linked swarms via Skype.

Take a look at this demonstration video of the working BitDrones concept: