PARAMOUR on Broadway – A Cirque du Soleil Musical. Credit: Richard Termine

Last year, Intel partnered with Lady Gaga on the Super Bowl Halftime Show to showcase its latest aerial technology called “Shooting Star.” Intel did a reprise performance of its Shooting Star technology for Singapore’s 52nd birthday this past week. Instead of fireworks, the tech-savvy country celebrated its National Day Parade with a swarm of 300 LED drones animating the night sky with shapes, logos, and even a map of the country.

Intel’s global drone chief, Anil Nanduri, explained, “There’s considerably more operational complexity in handling a 300 drone fleet, compared with 100 drones in a show. It’s like juggling balls in your hand. You may be able to juggle three, but if you juggle nine, you may have to throw them higher and faster to get more time.” Earlier this year, Intel first showcased its 300 drone show at Coachella music festival on the heels of claiming the Guinness World Record of a 500 drone performance.

Choreographed drones are winning the hearts of Cirque du Soleil theatergoers with a fleet of flying acrobatic vehicles dancing around its human performers. These drones are the brain child of Professor Raffaello D’Andrea of the ETH Zurich, Switzerland and his new startup Verity Studios. D’Andrea is probably best known as one of the three founders of Kiva Systems and now he is taking the same machine intelligence that sorts and delivers goods within Amazon’s warehouses to safely wow audiences worldwide.  The flying lampshades (shown in the video below) are actually autonomous drones that magically synchronize with the dancers, without safety nets or human operators.

Verity’s customer, Cirque du Soleil’s Chief Creative Officer Jean-Francois Bouchard, said D’Andrea’s “flying machines are unquestionably one of the most important statements of the PARAMOUR show.” The key to the flying machines’ success over 7,000 autonomous flights on Broadway is its proprietary technologies that enable multiple self-piloted drones to be synchronized within flight. Verity’s drone is part of a larger performance system called “Stage Flyers.”

The Stage Flyer platform has proven itself in the field, flying next to thousands of people each evening by having built-in redundancy to any single failure. According to Verity’s website, the system is “capable of continuing operation in spite of a failed battery, a failed motor, a failed connector, a failed propeller, a failed sensor, or a failure of any other component. This is achieved through the duplication of critical components and the use of proprietary algorithms, which enable safe emergency responses to component failures.” This means that the drones can operate safely around audiences and performers alike, carrying payloads of cameras, mirrors, and special lighting effects. As shown in the above diagram, the drone system includes a fleet of self-piloted drones that utilize one positioning system and control unit. The company boasts that its system only takes a few hours to install, calibrate and learn how to operate.

Drone swarms are not just for entertainment, as today there are a number of upstarts and established players utilizing these mechanics for e-commerce fulfillment centers. Last June, Amazon was issued a patent for a “Multi-Level Fulfillment Center for Unmanned Aerial Vehicles.” The beehive-looking distribution center (above) is designed to facilitate traffic between inbound and outbound delivery drones. The patent illustration details “multiple levels with multiple landing and take-off locations, depending on local zoning regulations.”

This is all part of Amazon’s larger plan to grow its robotic workforce over the next two to three years. Instead of human truck drivers, the patent displays delivery bays that open and close automatically based on the direction of the drones and interior platforms that cycle around the hive. CB Insights reports that the patent describes “impact dampeners,” such as nets, for receiving inbound drones and “launch assist mechanisms,” such as fans, for launching outbound drones. It appears that Amazon will be looking again to technology like D’Andrea’s research to revolutionize its global network of warehouses with synchronized swarms of drones that safely soar above human workers.

Amazon’s competitor Walmart announced last summer its plans to utilize swarms of indoor drones for inventory management, replacing the need for people climbing dangerous ladders to manually scan labels. The New York Times first reported last July that the retailer applied for a FAA exemption to begin testing drones inside its massive distribution centers. Shekar Natarajan, the vice president of last mile and emerging science for Walmart, demonstrated for the newspaper how swarms of drones could easily move up and down aisles, from floor to ceiling, to scan 30 barcode images a second (an efficiency that would be impossible for even the most agile humans). Walmart has publicly boasted that it will spend close to $3 billion on new technology and other cost-saving investments to bolster its e-commerce business which is growing, but at a slower pace than its nemesis Amazon.

The race to dominate the warehouse has led to increased investments in the logistics sector and even an accelerator dedicated solely to technology around distribution centers. Chattanooga, Tennessee-based Dynamo Accelerator showcased last May its second cohort of startups. One of the most successful showings was Chicago-based Corvus Robotics, a software company that uses indoor aerial drones to scan inventory (similar to the Walmart example above). According to Dynamo’s managing directors, Corvus is building enabling tools that allow operators to fly drones autonomously, scan & sync barcodes, and enter the SKU data into the existing warehouse management system.

Santosh Sankar, the director at Dynamo, explained his accelerator’s mission succinctly in a recent blog post: “We believe our focus and hands-on approach is one of our value-adds. As such, we’re leaning into being seed investors and upholding our commitment to transforming our industry by focusing on our founders and our corporate partners. We’ve opted to not hold a quota for our programs and hone in on companies we can truly help because that ultimately makes for good seed investments.” Sankar added that several of the program participants “are already well on their way to generate ($1 million or more) in annual revenue and/or have raised their initial round of capital.”

Corvus may be the latest indoor drone startup to enter an already crowded warehouse market, which includes established players like the Hardis Group, Smartx, and DJI. Drones continue to amuse, amaze and evolve as the growing need for more unmanned systems in our lives appears to be almost insatiable. Next month, we plan to dig deeper into the drone market with our RobotLabNYC event series on September 19th at 6pm in WeWork Grand Central. Joining us in the discussions include thought-leaders from NASA, AUVSI, and Genius NY, RSVP today, as space is limited.

Photo: Cirque du Soleil – PARAMOUR on Broadway – at the Lyric Theatre. Featuring Flying Machine Design and Choreography by Verity Studios. Pictured: Ruby Lewis as ‘Indigo’ Ryan Vona as ‘Joey’. ©, Cirque du Soleil Theatrical. Photo by: Richard Termine

Over the past year, 398 audiences of up to 2,000 people witnessed an octet of colorful lampshades perform an airborne choreography during Cirque du Soleil’s Broadway show Paramour, which ran until April 20^th. The work behind the design and choreography of the flying lampshades, which turn out to be self-piloted show drones, bears the signature of the Swiss high-tech company Verity Studios.

But how novel is it really that robots have appeared in theater? Since Karel Capek’s science fiction play R.U.R. (short for Rossum’s Universal Robots) introduced the word “robot” to the English language and to science fiction almost 100 years ago, the technical challenges of incorporating robots into live performance and theater have been difficult to master. Before these Broadway drones, nearly all theater robots were remote-controlled puppets, relying on humans hidden off-scene to steer their movements and provide their intelligence.

Photo: © Verity Studios. Photographer: Federico Augugliaro

Placing intelligent, autonomous flying machines in live theater presents a multi-faceted challenge: creating a compelling performance with safety, reliability, and ease of operation. While the compelling performance translates into shaping a convincing creative concept around the drones’ choreography, the latter chiefly points to designing computerized systems in lieu of making use of human pilots: Verity Studios’ show drones are flying mobile robots and navigate autonomously, piloting themselves, only supervised by a human operator. To navigate autonomously, these drones require a reliable method for determining their position in space. Since GPS is usually not available indoors, Verity Studios has built on more than a decade of research and development to provide a novel localization method.

The first groups of autonomous robots are now meeting the live event industries’ high requirements: Initially, Verity Studios set the stage for widely recognized flying machines in entertainment in 2014, by collaborating with Cirque de Soleil on Sparked, a short film in which 10 siblings of the similarly dressed 8 quadcopters from Paramour dance, hover, and flicker around a stunned human actor. Sparked was named a Winner of the 2016 New York City Drone Film Festival. However, it is one thing to feature drones in a 4-minute video, where the makers have the luxury to shoot it 100 times and then choose the best shot. Live performances on Broadway running 8 times per week in front of up to 2,000 people every time, by contrast, require the drone shows to work every single time. Verity Studios has met these requirements in the following sense: During their one-year run, the drones on Broadway safely completed more than 7,000 autonomous take-offs, flights, and landings. How was that possible?

Performance and reliability are key to Paramour’s dancing drones which are custom-made and hand-crafted in Switzerland. Each of the eight flying machines featured in Paramour uses 80 sensors to fly and performs roughly 1.5 billion calculations per second. Each critical calculation is double-checked every time it is performed in case any of the concerned processors makes a mistake. In other words, accidents are prevented by self-test and monitoring. And even if something does happen, a back-up is in place. For example, on the show of 2017-02-08, the battery from one of the eight vehicles could not provide sufficient power during the takeoff phase. The quadrocopter detected the problem and safely returned to the ground on only two propellers. The other seven continued the show unaffected.

Photo: © Verity Studios. Photographer: Federico Augugliaro

The failure of a single system component has never resulted in an unmanaged loss of control, but was handled appropriately. Each show drone is programmed to land in a controlled fashion despite a failure of any one of its motors, any part of its electronics, any one of its cables or connectors, and any one of its batteries. Similarly, the system has been designed in a way that a loss of communication between a drone and its ground station has created a dangerous situation. This is achieved by creating a fully redundant system, i.e., a system that can continue operation, or trigger a safe emergency behavior, if any one of its components fails. While such designs can be more expensive to create than designs without duplication, they provide much higher safety (and help reduce insurance costs). Their decades-long history in the manned aviation industry has shown this approach’s effectiveness and provides a treasure trove of experience and evidence for the emerging drone industry.

One way to design such a redundant system is to first design a simple flight-capable system, to then duplicate all its components (e.g. sensors, processors, actuators) to achieve the necessary redundancy, and to carefully design switch-over from a failed system to its backup. Importantly, the design of such a system cannot stop with the design of fully redundant drones, but needs to extend to all other critical components of the drone show system, including the positioning system, communications architecture, e-stop systems, and control stations. In the end, this uncompromising approach allows for a degree of safety and robustness that makes those intelligent and autonomous robots suitable for performance next to theater audiences.

At Paramour, the result was a one-of-a-kind performance that extended the traditional palette of light, sound, stage effects, and interaction with human performers by translating an intimate character beat into an unexpected visual motif. The enticing reality is that there are many more areas to explore, ranging from live concerts to active scenography, that offer dramatic possibilities for performing robots on a much broader scale. For more information, have a look at Verity Studios’ drone show systems, the work of Verity Studios’ drone costume designer or subscribe to Verity Studios’ newsletter.

Photo: © Verity Studios. Photographer: Federico Augugliaro

Photo: Cirque du Soleil – PARAMOUR on Broadway – at the Lyric Theatre. Featuring Flying Machine Design and Choreography by Verity Studios. Pictured: Ruby Lewis as ‘Indigo’ Ryan Vona as ‘Joey’. ©, Cirque du Soleil Theatrical. Photo by: Richard Termine

Since April, a troupe of eight flying machines has been performing in a Cirque du Soleil Broadway show called Paramour. This group of quadcopters has now completed its first 100 shows in front of a live theater audience, without a single incident. Given the string of recent safety incidents with drones (there’s more), this begs the question: How was this accomplished?

The Paramour quadcopters were designed and created by Verity Studios of Zurich and were transformed into flying lampshades in collaboration with Cirque du Soleil. Some readers may recognize them as the direct siblings of the similarly dressed quadcopters in Sparked: A Live Interaction Between Humans and Quadcopters; a short movie shot over the span of two days in the controlled environment of a research lab (see previous Robohub article and Quadrotors vs. CGI Q&A.) What sets the Paramour quadcopters apart, and the key to their ongoing success on Broadway, are a new localization technology and fail-safe algorithms, that keep the fully redundant quadcopters safely on course, performing 8 times a week, in front of a live audience of up to 2,000 people, without nets.

The people behind Verity Studios are no strangers to performing quadcopters. Founder Raffaello D’Andrea is a professor of control systems at the Swiss Federal Institute of Technology (ETH Zurich), from which Verity Studios was spun off. From a lab aptly named the Flying Machine Arena he has been conducting research for over 10 years on complex, autonomous flight, using a high-precision motion capture system to guide the vehicles. The ETH team’s first dancing drones performed in 2009 as part of research by Prof. D’Andrea’s former PhD students, Federico Augugliaro and Angela Schoellig. The group first stepped into the public performance space in 2011 with Flight Assembled Architecture, the world’s first multi-quadcopter performance in front of a large audience. In all, more than 200 live research demonstrations have taken place so far in the Zurich lab with occasional appearances given abroad, such as Google IO 2012.

While the team used motion capture technology extensively over this period, this type of localization technology proved inadequate for a long-running Broadway production when taken out of the research laboratory. The fundamental issue is that motion capture systems rely on a centralized architecture whereby all commands are forwarded to a single computer. A failure in that computer’s hardware or software may cause a sudden loss of control for the complete fleet.

Whereas an occasional failure of this type may be inconsequential in a research environment like the Flying Machine Arena, it is unacceptable for most commercial and industrial applications. The live events industry, in particular, has high safety standards owing in part to the amplification of safety risks when performing close to crowds of people. In addition to the safety risks, the consequences of a mishap entailing personal injury can be severe in terms of negative publicity, brand damage, and litigation.

Nevertheless, motion capture based performances near a live audience have been attempted, such as in this recent America’s Got Talent performance.

https://www.youtube.com/watch?v=1CjX1r-gh8U

To address these shortcomings, Verity Studios moved away from vision-based localization and developed a new system specifically for industrial applications that must meet strict safety and reliability standards. As stated by Raffaello D’Andrea at a recent TED talk, both the new radio-frequency based proprietary localization system and the flying machines are redundant and use fail-safe algorithms such that every vehicle can withstand any single point of failure.

Another key factor in the system’s success is its interoperability within a theatrical environment. Any theatrical system must function within a complex ecosystem of stage automation where stringent industry standards prevail.

“These flying machines have met or exceeded all expectations in terms of safety, reliability, maintenance, and ease of operation,” states Michael Shepp, Automation Carpenter for Paramour.

The technological achievement of creating a safe and robust system for flying a multitude of vehicles before a live audience introduces a rich palette for artistic expression. Creative staff at Cirque du Soleil recognized this potential and have worked side-by-side with Verity Studios to achieve this milestone on Broadway. Cirque du Soleil Chief Creative Officer, Jean-Francois Bouchard, states: “The flying machines are unquestionably one of the most important statements of the Paramour show.”

Cirque du Soleil and Verity Studios are currently exploring the artistic use of flying machines in other upcoming productions.

PARAMOUR runs at the Lyric Theatre (213 West 42nd Street) in New York.

If you enjoyed this article, you may also want to read:

• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• New quadrocopter video points to a future for flying machines in entertainment
• Quadrotors vs CGI: A Q&A with the makers of Sparked
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”
• Video: Throwing and catching an inverted pendulum – with quadrocopters
• Surrounded by his quadrocopter drones on stage, Raffaello D’Andrea explains feedback control and talks about the coming Machine Revolution

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.