Photo: Cirque du Soleil – PARAMOUR on Broadway – at the Lyric Theatre. Featuring Flying Machine Design and Choreography by Verity Studios. Pictured: Ruby Lewis as ‘Indigo’ Ryan Vona as ‘Joey’. ©, Cirque du Soleil Theatrical. Photo by: Richard Termine

Over the past year, 398 audiences of up to 2,000 people witnessed an octet of colorful lampshades perform an airborne choreography during Cirque du Soleil’s Broadway show Paramour, which ran until April 20^th. The work behind the design and choreography of the flying lampshades, which turn out to be self-piloted show drones, bears the signature of the Swiss high-tech company Verity Studios.

But how novel is it really that robots have appeared in theater? Since Karel Capek’s science fiction play R.U.R. (short for Rossum’s Universal Robots) introduced the word “robot” to the English language and to science fiction almost 100 years ago, the technical challenges of incorporating robots into live performance and theater have been difficult to master. Before these Broadway drones, nearly all theater robots were remote-controlled puppets, relying on humans hidden off-scene to steer their movements and provide their intelligence.

Photo: © Verity Studios. Photographer: Federico Augugliaro

Placing intelligent, autonomous flying machines in live theater presents a multi-faceted challenge: creating a compelling performance with safety, reliability, and ease of operation. While the compelling performance translates into shaping a convincing creative concept around the drones’ choreography, the latter chiefly points to designing computerized systems in lieu of making use of human pilots: Verity Studios’ show drones are flying mobile robots and navigate autonomously, piloting themselves, only supervised by a human operator. To navigate autonomously, these drones require a reliable method for determining their position in space. Since GPS is usually not available indoors, Verity Studios has built on more than a decade of research and development to provide a novel localization method.

The first groups of autonomous robots are now meeting the live event industries’ high requirements: Initially, Verity Studios set the stage for widely recognized flying machines in entertainment in 2014, by collaborating with Cirque de Soleil on Sparked, a short film in which 10 siblings of the similarly dressed 8 quadcopters from Paramour dance, hover, and flicker around a stunned human actor. Sparked was named a Winner of the 2016 New York City Drone Film Festival. However, it is one thing to feature drones in a 4-minute video, where the makers have the luxury to shoot it 100 times and then choose the best shot. Live performances on Broadway running 8 times per week in front of up to 2,000 people every time, by contrast, require the drone shows to work every single time. Verity Studios has met these requirements in the following sense: During their one-year run, the drones on Broadway safely completed more than 7,000 autonomous take-offs, flights, and landings. How was that possible?

Performance and reliability are key to Paramour’s dancing drones which are custom-made and hand-crafted in Switzerland. Each of the eight flying machines featured in Paramour uses 80 sensors to fly and performs roughly 1.5 billion calculations per second. Each critical calculation is double-checked every time it is performed in case any of the concerned processors makes a mistake. In other words, accidents are prevented by self-test and monitoring. And even if something does happen, a back-up is in place. For example, on the show of 2017-02-08, the battery from one of the eight vehicles could not provide sufficient power during the takeoff phase. The quadrocopter detected the problem and safely returned to the ground on only two propellers. The other seven continued the show unaffected.

Photo: © Verity Studios. Photographer: Federico Augugliaro

The failure of a single system component has never resulted in an unmanaged loss of control, but was handled appropriately. Each show drone is programmed to land in a controlled fashion despite a failure of any one of its motors, any part of its electronics, any one of its cables or connectors, and any one of its batteries. Similarly, the system has been designed in a way that a loss of communication between a drone and its ground station has created a dangerous situation. This is achieved by creating a fully redundant system, i.e., a system that can continue operation, or trigger a safe emergency behavior, if any one of its components fails. While such designs can be more expensive to create than designs without duplication, they provide much higher safety (and help reduce insurance costs). Their decades-long history in the manned aviation industry has shown this approach’s effectiveness and provides a treasure trove of experience and evidence for the emerging drone industry.

One way to design such a redundant system is to first design a simple flight-capable system, to then duplicate all its components (e.g. sensors, processors, actuators) to achieve the necessary redundancy, and to carefully design switch-over from a failed system to its backup. Importantly, the design of such a system cannot stop with the design of fully redundant drones, but needs to extend to all other critical components of the drone show system, including the positioning system, communications architecture, e-stop systems, and control stations. In the end, this uncompromising approach allows for a degree of safety and robustness that makes those intelligent and autonomous robots suitable for performance next to theater audiences.

At Paramour, the result was a one-of-a-kind performance that extended the traditional palette of light, sound, stage effects, and interaction with human performers by translating an intimate character beat into an unexpected visual motif. The enticing reality is that there are many more areas to explore, ranging from live concerts to active scenography, that offer dramatic possibilities for performing robots on a much broader scale. For more information, have a look at Verity Studios’ drone show systems, the work of Verity Studios’ drone costume designer or subscribe to Verity Studios’ newsletter.

Photo: © Verity Studios. Photographer: Federico Augugliaro

Photo: Cirque du Soleil – PARAMOUR on Broadway – at the Lyric Theatre. Featuring Flying Machine Design and Choreography by Verity Studios. Pictured: Ruby Lewis as ‘Indigo’ Ryan Vona as ‘Joey’. ©, Cirque du Soleil Theatrical. Photo by: Richard Termine

Since April, a troupe of eight flying machines has been performing in a Cirque du Soleil Broadway show called Paramour. This group of quadcopters has now completed its first 100 shows in front of a live theater audience, without a single incident. Given the string of recent safety incidents with drones (there’s more), this begs the question: How was this accomplished?

The Paramour quadcopters were designed and created by Verity Studios of Zurich and were transformed into flying lampshades in collaboration with Cirque du Soleil. Some readers may recognize them as the direct siblings of the similarly dressed quadcopters in Sparked: A Live Interaction Between Humans and Quadcopters; a short movie shot over the span of two days in the controlled environment of a research lab (see previous Robohub article and Quadrotors vs. CGI Q&A.) What sets the Paramour quadcopters apart, and the key to their ongoing success on Broadway, are a new localization technology and fail-safe algorithms, that keep the fully redundant quadcopters safely on course, performing 8 times a week, in front of a live audience of up to 2,000 people, without nets.

The people behind Verity Studios are no strangers to performing quadcopters. Founder Raffaello D’Andrea is a professor of control systems at the Swiss Federal Institute of Technology (ETH Zurich), from which Verity Studios was spun off. From a lab aptly named the Flying Machine Arena he has been conducting research for over 10 years on complex, autonomous flight, using a high-precision motion capture system to guide the vehicles. The ETH team’s first dancing drones performed in 2009 as part of research by Prof. D’Andrea’s former PhD students, Federico Augugliaro and Angela Schoellig. The group first stepped into the public performance space in 2011 with Flight Assembled Architecture, the world’s first multi-quadcopter performance in front of a large audience. In all, more than 200 live research demonstrations have taken place so far in the Zurich lab with occasional appearances given abroad, such as Google IO 2012.

While the team used motion capture technology extensively over this period, this type of localization technology proved inadequate for a long-running Broadway production when taken out of the research laboratory. The fundamental issue is that motion capture systems rely on a centralized architecture whereby all commands are forwarded to a single computer. A failure in that computer’s hardware or software may cause a sudden loss of control for the complete fleet.

Whereas an occasional failure of this type may be inconsequential in a research environment like the Flying Machine Arena, it is unacceptable for most commercial and industrial applications. The live events industry, in particular, has high safety standards owing in part to the amplification of safety risks when performing close to crowds of people. In addition to the safety risks, the consequences of a mishap entailing personal injury can be severe in terms of negative publicity, brand damage, and litigation.

Nevertheless, motion capture based performances near a live audience have been attempted, such as in this recent America’s Got Talent performance.

https://www.youtube.com/watch?v=1CjX1r-gh8U

To address these shortcomings, Verity Studios moved away from vision-based localization and developed a new system specifically for industrial applications that must meet strict safety and reliability standards. As stated by Raffaello D’Andrea at a recent TED talk, both the new radio-frequency based proprietary localization system and the flying machines are redundant and use fail-safe algorithms such that every vehicle can withstand any single point of failure.

Another key factor in the system’s success is its interoperability within a theatrical environment. Any theatrical system must function within a complex ecosystem of stage automation where stringent industry standards prevail.

“These flying machines have met or exceeded all expectations in terms of safety, reliability, maintenance, and ease of operation,” states Michael Shepp, Automation Carpenter for Paramour.

The technological achievement of creating a safe and robust system for flying a multitude of vehicles before a live audience introduces a rich palette for artistic expression. Creative staff at Cirque du Soleil recognized this potential and have worked side-by-side with Verity Studios to achieve this milestone on Broadway. Cirque du Soleil Chief Creative Officer, Jean-Francois Bouchard, states: “The flying machines are unquestionably one of the most important statements of the Paramour show.”

Cirque du Soleil and Verity Studios are currently exploring the artistic use of flying machines in other upcoming productions.

PARAMOUR runs at the Lyric Theatre (213 West 42nd Street) in New York.

If you enjoyed this article, you may also want to read:

• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• New quadrocopter video points to a future for flying machines in entertainment
• Quadrotors vs CGI: A Q&A with the makers of Sparked
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”
• Video: Throwing and catching an inverted pendulum – with quadrocopters
• Surrounded by his quadrocopter drones on stage, Raffaello D’Andrea explains feedback control and talks about the coming Machine Revolution

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

A camera man gets up, close, and personal with the human and machine actors on the “Sparked” set.

If you thought “SPARKED” – the new short film by Cirque du Soleil, ETH Zurich, and Verity Studios – is too real-looking to be CGI, you were right. But why go to all the trouble of using quadrotors to get those lampshades dancing in the air for real? We asked Bill Keays, Science and Technology Advisor at Cirque du Soleil, to give us an insider’s perspective about Cirque’s motivation for the film and how it came together.

Bill Keays on the SPARKED film set.

[Robohub] What was the inspiration for the film?
[Keays] The combination of autonomous flying machines with Cirque du Soleil magic was so evidently bursting with potential we committed ourselves to make something original together not knowing what it would be at first. An iterative creative process ensued involving all participants, and what emerged was a wonderful script that carried the talent from both sides of the formula to a whole new level.

Who are the main partners on the project? 

It was a highly democratic and highly collaborative process. Raffaello D’Andrea of ETH Zurich and Verity Studios, and Welby Altidor of Cirque du Soleil were the project leaders, but the true collaborative nature of this project is reflected in the fact that no participant roles are specified in the credits.

Who initiated the project? 

The very first discussions were held between Prof. Raffaello D’Andrea at ETH Zurich and myself as Science and Technology Advisor at Cirque du Soleil in March 2012. After ongoing discussions and considering various options, the two organizations eventually committed to a creation project in January 2014.

The “Sparked” film set. The shoot took place in the Flying Machine Arena at ETH Zurich, a portable space devoted to autonomous flight.

Why was the film made?

For Cirque the project demonstrates a whole new avenue for channeling creativity. It allows Cirque to reach a new audience externally and provides inspiration internally to push the creative envelope in new directions.

As such is a highly effective, multifaceted communications vehicle, and it will certainly be submitted for competition.

What were the production costs? 

The large bulk of the cost came in terms of human resources split between three organizations. The exact figures cannot be known.

How long did the project take from first concept to finished film? What were the major steps/milestones?

The whole project, from go-ahead to YouTube release, took under 8 months. It was a back and forth process whereby content was driven by discussions between Cirque, Verity, and the film director. Through this method we progressed through initial concept design, script development, set and prop design, and finally the shoot, with intense hardware and software development taking place throughout. The product was then handed off to post production and then to distribution.

Setting up before a flight scene.

Who designed the lampshades? The photos show researchers from ETH and Verity fitting the quads with them. Did you provide dimensions to props experts and iterate? How did that process work?

It was indeed an iterative process. The lampshades were designed and produced by the creative team at Cirque in Montreal who used their own quadrocopters for flight tests to ensure that they would be compatible with the flight environment at ETH Zurich. Guidance from the ETH and Verity teams was indispensable of course.

Was the actor controlling the quads with hand motions (via Kinect)? Or was this pre-programmed and choreographed? At times it looks like he is ‘conducting’ their movements …

The Flying Machine Arena fully possesses the ability to control the flying machines with a Kinect or other similar devices, or even hand gestures. But given that this is a movie production, the movements of the actor and the flying machines were choreographed as is the convention in film practice.

Was there any CGI at all in the film? How much is real?

None whatsoever, it’s all real; and even though it looks terrific in the movie, this pales in comparison to seeing it in the round. It was mindboggling to watch live.

Has Cirque worked with robots before?

Yes, Cirque has experimented with robots in the past, but never flying ones.

What were the biggest challenges of putting live actors together with the quads on stage?

Given that this is such an unusual circumstance for making a movie, we didn’t know how our lead actor would react to the flight environment. But the high precision of the flying machines and terrific expertise of ETH Zurich’s Flying Machine Arena staff quickly inspired confidence. By the second day, remarkably, our actors, both human and mechanical, were moving about the set with terrific ease, paying little attention to each other.

Was safety ever an issue?

The design of the lampshades was convenient in that it shielded the blades of the quadrocopters, thus making them far less likely to strike anyone or anything. There were a couple of crashes and a few low passes, but at no point was there any real apparent danger.

What about lighting?

Believe it or not, all the scenes with the flying lampshades are completely lit by the lampshades themselves! Another fortuitous design feature.

Changing batteries.

Were there any other challenges?

The rapid power consumption of the flying machines meant that battery changes would be frequent. With time, the switch-overs began looking like Formula 1 pit stops, performed with great speed and precision.

What was it like for the researchers from the lab to have to ‘perform’ for the camera? How long did they have to rehearse before you were ready?

The lampshades arrived in the weeks prior to the shoot so there was time for Verity Studios and ETH Zurich to fully test their aerodynamic abilities, and to run through the choreography. In the days preceding the shoot we performed flight tests according to the story board and had one practice day with the lead actor. However, due to the extreme time constraints imposed by the film crew, the control software had to be designed such that flight plans could be altered at a moment’s notice to meet specific requests from the director. This proved to be an invaluable tool and was instrumental in completing the shoot in the allotted time.

Had your previous work (e.g. demos, presentations) prepared you for working with quadrotors on a film set?

Absolutely, the Flying Machine Arena team had performed extensively with flying machines in public spaces and conferences (including TED), so much of this experience transferred well onto the movie set.

An early flight test in the Flying Machine Arena.

The quadrotors peek through in the film at times, as do the retro reflective markers for the motion capture system – was this intentional?

It is true that the quadrotors are sometimes visible, but the visual quality of the movie is so overwhelming that it is difficult to remain focused on technical bits. We actually liked the fact that they were occasionally visible; as sort of a wink to those of us who are knowledgeable about them.

With previous demos, the quadrotors played a leading role – but in this film, they are literally hidden behind lampshades. How does it feel to see them step back into a supportive role for this film?

But the flying machines are not hidden. They are in dress like all actors in this movie; they have character, motivation and certainly no lack of presence. As the devil’s advocate, I would be inclined to challenge the assumption that they are playing a supporting role.

Why not just do it with CGI?

There are at least two good reasons:

First of all, because the Flying Machine Arena is not about CGI. It’s about control systems, aerodynamics, and sophisticated algorithms that yield a physical manifestation of uncommon beauty.

The second reason is because you can’t do CGI in front of a live audience and Cirque du Soleil is in the business of live entertainment. Even though this extraordinary little movie is a gem in its own right, it is primarily the proof of concept for something far more powerful; something that must be seen live.

All images credited to the “Sparked” team. For more insight on the film see also New quadrocopter video points to a future for flying machines in entertainment by Markus Waibel of Verity Studios. 

We’ve seen robots on stage before: Humanoids have been featured in theater productions in Switzerland, Austria, and Japan, and industrial robots have played parts in dance performances and other staged events. Flying robots have also played a theatrical role, for example in Shakespeare’s A Midsummer Night’s Dream at Texas A&M, a dance set to a Schubert piano trio at MIT, as well as in advertising. Some robots have even gone pro: Robothespian was designed for acting and is rented out for performances.

But the dance between human and machine has never looked quite like this .

The film’s main plot is straight-forward: Lamps in a repair shop magically come alive, rising from their stands to sneak up on an unsuspecting repairman. As the magic unfolds, and the repairman discovers that the everyday objects of his trade have become animated and full of character, viewers become witness to the evolving dance between man and technology.

Everyone on the team, including myself, was impressed by how well the film captures the stunning, at times hypnotic visuals of the flying lampshades. Everything you see is real — no CGI, wires, or speed-ups/slow-downs were used in the film. 

So how did the lampshades come alive?

This short film, titled “Sparked”, was produced in collaboration between researchers at ETH Zurich’s Flying Machine Arena, their spin-off company Verity Studios, and entertainment company Cirque du Soleil; in other words, it’s quadrocopter magic that makes the lampshades fly.

The film marks a step change from previous work by ETH Zurich’s Flying Machine Arena team, catalyzed by their new spin-off company Verity Studios, which also gets credit for designing the quadrocopters’ choreography. Here, in spite of their computer-controlled precision flights, the flying machines become robot actors that are individuals with their own personalities: Pomponette, with her tassels bustling in the propellers’ airflow; the Medusa with its bizarre tentacles waving in the wind; the Runt, sleek and fast but still glowing and twitching from the electric shock. And Lady Purple, absent from most of the film because of her intractable aerodynamics.

Flying machines up, close, and personal

One of the most striking aspects of this film shoot was watching our actor, Nicolas Leresche, interact with the flying machines. Much of this interaction was captured on camera — and you can see the results in the film, in particular the moment of ‘first contact’ — but, for me, it was what happened off-camera that was most revealing.

Naively unjaded by their joys and dangers, this was Nicolas’ first experience with quadrocopters. As he began interacting with them, tentatively at first, and then — in a matter of seconds — more confidently, even playfully, it became crystal clear that I was not merely observing a human using a robotic tool.

As the quadrocopter’s onboard controller pushed back against Nicolas’ hand, immediately reacting to match his actions, it became apparent that the two interacted in a way we don’t usually get to interact with technology. The interaction here was physical and dynamic, and clearly intuitive for Nicolas. There was no noticeable learning curve, no need for manuals or explanations.

But what was most striking about this interaction was the result. Within seconds, Nicolas had established a deep, almost animate connection with the flying robot actors. As their dance developed, driven by the autonomous machines’ choreography and Nicolas’ script, and became ever more fluid and natural, it became clear that what we were witnessing was a relationship in its making; one of mutual respect and, at the same time, confidence and trust. And once there was trust, magic happened.

In his own words:

Actors think that they are the ones who make objects move. I think that, on the contrary, it’s the objects that make us move. In the case of drones, even more so! They are companions (in an etymological sense), confrères, brothers.

Within hours the entire film crew had become comfortable with the flying machines hovering nearby or brushing past them as they swooshed through the air. On one occasion, a mis-timed dolly motion led to a flying machine hovering just centimeters above the camera man, who continued with his work unfazed. On another occasion, when Nicolas missed his cue, he bumped into one of the flying machines. But they just jostled past each other and continued on with their choreography as if nothing exceptional had happened. On yet another occasion, a drone collided with the moving camera without disturbance.

The only damage during the entire two-day shoot was a single broken lamp, knocked over by one of the quadrocopters’ airflow.

Safety

I have no doubt that this level of confidence and comfort with the technology can in no small part be attributed to the system’s reliability. In spite of the packed shooting schedule (a total of 40 shots over two days) and the high complexity of the project (for example, designing lighting conditions using more than 60 light sources including some moving through space), production went smoothly, and importantly for the quadrocopter team, the flying machines did not add any significant delay. Battery swaps and on-the-spot changes to the choreography were efficient and quick, never exceeding a few minutes. Kudos to the excellent film team from who’s mcqueen for brilliantly doing their part to keep the production on schedule.

Collaboration

One aspect that made this project particularly enjoyable was the collaboration with creatives at Cirque du Soleil. From the start there was a lot of excitement about the potential of this technology for entertainment not just at ETH and Verity Studios here in Zurich but also at Cirque in Montreal. Everyone shared the view that the flying machines could be used to amaze audiences and to tell wonderful stories. What ensued was a collaborative, iterative creation process that culminated in a script and a set design that were imaginative to Cirque du Soleil standards, while exhibiting the abilities of the quadrocopters in their full splendor.

As it turned out, figuring out what is easy and what is difficult to achieve with autonomous quadrocopters can be quite counter-intuitive. Having researchers and creatives on both sides collaborate on the key creative elements of the story from the early phases of the project therefore proved invaluable to its success.

Next steps

Why go to all this trouble? We’ve all seen the amazing things CGI can do on film. I love films and movies, how they can condense experience and add life to our most impossible dreams. But nothing compares to the real experience of getting up, close, and personal with robot actors yourself.

What’s next? Our audiences over the past years have had more ideas than we can count. At this point, only one thing’s for sure: They all want to see more flying robot actors.

Some more snapshots from the film shoot below for your viewing pleasure:

All images credited to the “Sparked” team. Markus Waibel is a former member of ETH Zurich, a co-founder of Verity Studios, and one of the contributors to “Sparked”.