Metallica’s European WorldWired tour, which opened to an ecstatic crowd of 15,000 in Copenhagen’s sold-out Royal Arena this Saturday, features a swarm of micro drones flying above the band. Shortly after the band breaks into their hit single “Moth Into Flame”, dozens of micro drones start emerging from the stage, forming a large rotating circle above the stage. As the music builds, more and more drones emerge and join the formation, creating increasingly complex patterns, culminating in a choreography of three interlocking rings that rotate in position.

This show’s debut marks the world’s first autonomous drone swarm performance in a major touring act. Unlike previous drone shows, this performance features indoor drones, flying above performers and right next to throngs of concert viewers in a live event setting. Flying immediately next to audiences creates a more intimate effect than outdoor drone shows. The same closeness also allows the creation of moving, three-dimensional sculptures like the ones seen in the video — an effect further enhanced by Metallica’s 360-degree stage setup, with concert viewers on all sides.

Flying drones close to and around people in such a setting is challenging. Unlike outdoors, indoor drones cannot rely on GPS signals, which are severely degraded in indoor settings and do not offer the required accuracy for autonomous drone navigation on stage. The safety aspects of flying dozens of drones close to crowds in the high-pressure, live-event environment impose further challenges. Robustness to the uncertainties caused by changing show conditions in a touring setting as well as variation in the drone systems’ components and sensors, including the hundreds of motors powering the drones, is another necessary condition for this drone show system.

“It’s all about safety and reliability first”, says Raffaello D’Andrea, founder of the company behind the drones used in the Metallica show, Verity Studios (full disclosure: I’m a co-founder). D’Andrea knows what he is talking about: In work with his previous company, which was snatched up by e-commerce giant Amazon for an eye-watering 775M USD in 2012, D’Andrea and his team created fleets of autonomous warehousing robots, moving inventory through the warehouse around the clock. That company, which has since been renamed Amazon Robotics, now operates up to 10,000 robots — in a single warehouse.

How was this achieved?

In a nutshell: Verity Studios’ drone show system is an advanced show automation system that uses distributed AI, robotics, and sophisticated algorithms to achieve the level of robust performance and safety required by the live entertainment industry. With a track record of >7,000 autonomous flights on Broadway, achieved with its larger Stage Flyer drones during 398 live shows, Verity Studios is no newcomer to this industry.

Many elements are needed to create a touring drone show; the drones themselves are just one aspect. Verity’s drones are autonomous, supervised by a human operator, who does not control drone motions individually. Instead, the operator only issues high-level commands such as “takeoff” or “land”, monitors the motions of multiple drones at a time, and reacts to anomalies. In other words, Verity’s advanced automation system takes over the role of multiple human pilots that would be required with standard, remote-controlled drones. The drones are flying mobile robots that navigate autonomously, piloting themselves, under human supervision. The autonomous drones’ motions and their lighting design are choreographed by Verity’s creative staff.

To navigate autonomously, drones require a reliable method for determining their position in space. As mentioned above, while drones can use GPS for their autonomous navigation in an outdoor setting, GPS is not a viable option indoors: GPS signals degrade close to large structures (e.g., tall buildings) and are usually not available, or severely degraded, in indoor environments. Since degraded GPS may result in unreliable or unsafe conditions for autonomous flight, the Verity drones use proprietary indoor localization technology.

System architecture of Verity Studios’ drone show system used in Metallica’s WorldWired tour, comprising positioning modules part of Verity’s indoor localization technology, autonomous drones, and an operator control station.

It is the combination of a reliable indoor positioning system with intelligent autonomous drones and a suitable operator interface that allows the single operator of the Metallica show to simultaneously control the coordinated movement of many drones. This pilot-less approach is not merely a matter of increasing efficiency and effectiveness (who wants to have dozens of pilots on staff), but also a key safety requirement: Pilot errors have been an important contributing factor in dozens of documented drone accidents at live events. Safety risks rapidly increase as the number of drones increases, resulting in more complex flight plans and higher risks of mid-air collisions. Autonomous control allows safer operation of multiple drones than remote control by human pilots, especially when operating in a reduced airspace envelope.

Verity’s system also had to be engineered for safety in spite of other potential failures, including wireless interference, hardware or software component failures, power outages, or malicious disruption/hacking attacks. In its 398-show run on Broadway, the biggest challenge to safety turned out to be another factor: Human error. While operated by theater staff on Broadway, Verity’s system correctly identified human errors on five occasions and prevented the concerned drones from taking flight (on these occasions, the show continued with six or seven instead of the show’s planned eight drones; only one show proceeded without any drones as a safety precaution, i.e., the drone show’s “uptime” was 99.7%). As my colleagues and I have outlined in a recently published overview document on best practices for drone shows, when using drones at live event safety is a hard requirement.

Another key element for Verity’s show creation process are drone authoring tools. Planning shows like the Metallica performance requires tools for the efficient creation of trajectories for large numbers of drones. The trajectories must account for the drones’ actual flight dynamics, considering actuator limitations, as well as for aerodynamic effects, such as air turbulence or lift. Drone motions generated by these tools need to be collision-free and allow for emergency maneuvers. To create compelling effects, drone authoring tools also need to allow extracting all of the dynamic performance the drones are capable of — another area that D’Andrea’s team has gained considerable experience with prior to founding Verity Studios, in this case as part of research at the Swiss Federal Institute of Technology’s Flying Machine Arena.

Creating a compelling drone show requires more than the drone show system itself. For this tour, Verity Studios partnered with the world’s leading stage automation company TAIT Towers to integrate the drones into the stage floor as well as tackling a series of other technical challenges related to this touring show.

While technology is the key enabler, the starting point and the key driver of Verity’s shows are non-technological. Instead, the show is driven by the show designers’ creative intent. This comprises defining the role of show drones for the performance at hand as well as determining their integration into the visual and musical motifs of the show’s creative concept. For Metallica, the drones’ flight trajectories and lighting were created by Verity’s choreography team, incorporating feedback from Metallica’s production team and the band.

Metallica’s WorldWired tour
Metallica’s WorldWired Tour is their first worldwide tour after the World Magnetic Tour six years ago. The tour’s currently published European leg runs until 11 May 2018, with all general tickets sold out.

Further Robohub reading

• Drone shows: Creative potential and best practices
• Ben Einstein’s great tear-down of D’Andrea’s Amazon robots (although I disagree with Ben’s conclusion that the bots are “cooler than a flying drone” … maybe Ben hasn’t seen Verity’s Lucie drones yet? You be the judge.)
• Verity’s 7,000 autonomous drone flights on Broadway

Some images for your viewing pleasure

James Hetfield with Verity’s drones

Verity’s drones swarming below TAIT’s LED cubes

A glimpse at the 15,000-strong audience of the sold-out concert

PARAMOUR on Broadway – A Cirque du Soleil Musical. Credit: Richard Termine

Last year, Intel partnered with Lady Gaga on the Super Bowl Halftime Show to showcase its latest aerial technology called “Shooting Star.” Intel did a reprise performance of its Shooting Star technology for Singapore’s 52nd birthday this past week. Instead of fireworks, the tech-savvy country celebrated its National Day Parade with a swarm of 300 LED drones animating the night sky with shapes, logos, and even a map of the country.

Intel’s global drone chief, Anil Nanduri, explained, “There’s considerably more operational complexity in handling a 300 drone fleet, compared with 100 drones in a show. It’s like juggling balls in your hand. You may be able to juggle three, but if you juggle nine, you may have to throw them higher and faster to get more time.” Earlier this year, Intel first showcased its 300 drone show at Coachella music festival on the heels of claiming the Guinness World Record of a 500 drone performance.

Choreographed drones are winning the hearts of Cirque du Soleil theatergoers with a fleet of flying acrobatic vehicles dancing around its human performers. These drones are the brain child of Professor Raffaello D’Andrea of the ETH Zurich, Switzerland and his new startup Verity Studios. D’Andrea is probably best known as one of the three founders of Kiva Systems and now he is taking the same machine intelligence that sorts and delivers goods within Amazon’s warehouses to safely wow audiences worldwide.  The flying lampshades (shown in the video below) are actually autonomous drones that magically synchronize with the dancers, without safety nets or human operators.

Verity’s customer, Cirque du Soleil’s Chief Creative Officer Jean-Francois Bouchard, said D’Andrea’s “flying machines are unquestionably one of the most important statements of the PARAMOUR show.” The key to the flying machines’ success over 7,000 autonomous flights on Broadway is its proprietary technologies that enable multiple self-piloted drones to be synchronized within flight. Verity’s drone is part of a larger performance system called “Stage Flyers.”

The Stage Flyer platform has proven itself in the field, flying next to thousands of people each evening by having built-in redundancy to any single failure. According to Verity’s website, the system is “capable of continuing operation in spite of a failed battery, a failed motor, a failed connector, a failed propeller, a failed sensor, or a failure of any other component. This is achieved through the duplication of critical components and the use of proprietary algorithms, which enable safe emergency responses to component failures.” This means that the drones can operate safely around audiences and performers alike, carrying payloads of cameras, mirrors, and special lighting effects. As shown in the above diagram, the drone system includes a fleet of self-piloted drones that utilize one positioning system and control unit. The company boasts that its system only takes a few hours to install, calibrate and learn how to operate.

Drone swarms are not just for entertainment, as today there are a number of upstarts and established players utilizing these mechanics for e-commerce fulfillment centers. Last June, Amazon was issued a patent for a “Multi-Level Fulfillment Center for Unmanned Aerial Vehicles.” The beehive-looking distribution center (above) is designed to facilitate traffic between inbound and outbound delivery drones. The patent illustration details “multiple levels with multiple landing and take-off locations, depending on local zoning regulations.”

This is all part of Amazon’s larger plan to grow its robotic workforce over the next two to three years. Instead of human truck drivers, the patent displays delivery bays that open and close automatically based on the direction of the drones and interior platforms that cycle around the hive. CB Insights reports that the patent describes “impact dampeners,” such as nets, for receiving inbound drones and “launch assist mechanisms,” such as fans, for launching outbound drones. It appears that Amazon will be looking again to technology like D’Andrea’s research to revolutionize its global network of warehouses with synchronized swarms of drones that safely soar above human workers.

Amazon’s competitor Walmart announced last summer its plans to utilize swarms of indoor drones for inventory management, replacing the need for people climbing dangerous ladders to manually scan labels. The New York Times first reported last July that the retailer applied for a FAA exemption to begin testing drones inside its massive distribution centers. Shekar Natarajan, the vice president of last mile and emerging science for Walmart, demonstrated for the newspaper how swarms of drones could easily move up and down aisles, from floor to ceiling, to scan 30 barcode images a second (an efficiency that would be impossible for even the most agile humans). Walmart has publicly boasted that it will spend close to $3 billion on new technology and other cost-saving investments to bolster its e-commerce business which is growing, but at a slower pace than its nemesis Amazon.

The race to dominate the warehouse has led to increased investments in the logistics sector and even an accelerator dedicated solely to technology around distribution centers. Chattanooga, Tennessee-based Dynamo Accelerator showcased last May its second cohort of startups. One of the most successful showings was Chicago-based Corvus Robotics, a software company that uses indoor aerial drones to scan inventory (similar to the Walmart example above). According to Dynamo’s managing directors, Corvus is building enabling tools that allow operators to fly drones autonomously, scan & sync barcodes, and enter the SKU data into the existing warehouse management system.

Santosh Sankar, the director at Dynamo, explained his accelerator’s mission succinctly in a recent blog post: “We believe our focus and hands-on approach is one of our value-adds. As such, we’re leaning into being seed investors and upholding our commitment to transforming our industry by focusing on our founders and our corporate partners. We’ve opted to not hold a quota for our programs and hone in on companies we can truly help because that ultimately makes for good seed investments.” Sankar added that several of the program participants “are already well on their way to generate ($1 million or more) in annual revenue and/or have raised their initial round of capital.”

Corvus may be the latest indoor drone startup to enter an already crowded warehouse market, which includes established players like the Hardis Group, Smartx, and DJI. Drones continue to amuse, amaze and evolve as the growing need for more unmanned systems in our lives appears to be almost insatiable. Next month, we plan to dig deeper into the drone market with our RobotLabNYC event series on September 19th at 6pm in WeWork Grand Central. Joining us in the discussions include thought-leaders from NASA, AUVSI, and Genius NY, RSVP today, as space is limited.

Photo: Cirque du Soleil – PARAMOUR on Broadway – at the Lyric Theatre. Featuring Flying Machine Design and Choreography by Verity Studios. Pictured: Ruby Lewis as ‘Indigo’ Ryan Vona as ‘Joey’. ©, Cirque du Soleil Theatrical. Photo by: Richard Termine

Over the past year, 398 audiences of up to 2,000 people witnessed an octet of colorful lampshades perform an airborne choreography during Cirque du Soleil’s Broadway show Paramour, which ran until April 20^th. The work behind the design and choreography of the flying lampshades, which turn out to be self-piloted show drones, bears the signature of the Swiss high-tech company Verity Studios.

But how novel is it really that robots have appeared in theater? Since Karel Capek’s science fiction play R.U.R. (short for Rossum’s Universal Robots) introduced the word “robot” to the English language and to science fiction almost 100 years ago, the technical challenges of incorporating robots into live performance and theater have been difficult to master. Before these Broadway drones, nearly all theater robots were remote-controlled puppets, relying on humans hidden off-scene to steer their movements and provide their intelligence.

Photo: © Verity Studios. Photographer: Federico Augugliaro

Placing intelligent, autonomous flying machines in live theater presents a multi-faceted challenge: creating a compelling performance with safety, reliability, and ease of operation. While the compelling performance translates into shaping a convincing creative concept around the drones’ choreography, the latter chiefly points to designing computerized systems in lieu of making use of human pilots: Verity Studios’ show drones are flying mobile robots and navigate autonomously, piloting themselves, only supervised by a human operator. To navigate autonomously, these drones require a reliable method for determining their position in space. Since GPS is usually not available indoors, Verity Studios has built on more than a decade of research and development to provide a novel localization method.

The first groups of autonomous robots are now meeting the live event industries’ high requirements: Initially, Verity Studios set the stage for widely recognized flying machines in entertainment in 2014, by collaborating with Cirque de Soleil on Sparked, a short film in which 10 siblings of the similarly dressed 8 quadcopters from Paramour dance, hover, and flicker around a stunned human actor. Sparked was named a Winner of the 2016 New York City Drone Film Festival. However, it is one thing to feature drones in a 4-minute video, where the makers have the luxury to shoot it 100 times and then choose the best shot. Live performances on Broadway running 8 times per week in front of up to 2,000 people every time, by contrast, require the drone shows to work every single time. Verity Studios has met these requirements in the following sense: During their one-year run, the drones on Broadway safely completed more than 7,000 autonomous take-offs, flights, and landings. How was that possible?

Performance and reliability are key to Paramour’s dancing drones which are custom-made and hand-crafted in Switzerland. Each of the eight flying machines featured in Paramour uses 80 sensors to fly and performs roughly 1.5 billion calculations per second. Each critical calculation is double-checked every time it is performed in case any of the concerned processors makes a mistake. In other words, accidents are prevented by self-test and monitoring. And even if something does happen, a back-up is in place. For example, on the show of 2017-02-08, the battery from one of the eight vehicles could not provide sufficient power during the takeoff phase. The quadrocopter detected the problem and safely returned to the ground on only two propellers. The other seven continued the show unaffected.

Photo: © Verity Studios. Photographer: Federico Augugliaro

The failure of a single system component has never resulted in an unmanaged loss of control, but was handled appropriately. Each show drone is programmed to land in a controlled fashion despite a failure of any one of its motors, any part of its electronics, any one of its cables or connectors, and any one of its batteries. Similarly, the system has been designed in a way that a loss of communication between a drone and its ground station has created a dangerous situation. This is achieved by creating a fully redundant system, i.e., a system that can continue operation, or trigger a safe emergency behavior, if any one of its components fails. While such designs can be more expensive to create than designs without duplication, they provide much higher safety (and help reduce insurance costs). Their decades-long history in the manned aviation industry has shown this approach’s effectiveness and provides a treasure trove of experience and evidence for the emerging drone industry.

One way to design such a redundant system is to first design a simple flight-capable system, to then duplicate all its components (e.g. sensors, processors, actuators) to achieve the necessary redundancy, and to carefully design switch-over from a failed system to its backup. Importantly, the design of such a system cannot stop with the design of fully redundant drones, but needs to extend to all other critical components of the drone show system, including the positioning system, communications architecture, e-stop systems, and control stations. In the end, this uncompromising approach allows for a degree of safety and robustness that makes those intelligent and autonomous robots suitable for performance next to theater audiences.

At Paramour, the result was a one-of-a-kind performance that extended the traditional palette of light, sound, stage effects, and interaction with human performers by translating an intimate character beat into an unexpected visual motif. The enticing reality is that there are many more areas to explore, ranging from live concerts to active scenography, that offer dramatic possibilities for performing robots on a much broader scale. For more information, have a look at Verity Studios’ drone show systems, the work of Verity Studios’ drone costume designer or subscribe to Verity Studios’ newsletter.

Photo: © Verity Studios. Photographer: Federico Augugliaro

Léa Pereyre creating new drone costumes. Credit: Verity Studios

Drone costume designer Léa Pereyre showcases several of her imaginative designs and describes her experience with dreaming up new concepts for this emerging art form.

Check out more designs on their Drone Costume Design Instagram, find it here.

The video below shows clips from the Broadway show PARAMOUR, created by Cirque du Soleil. The flying lampshades soaring over the actors are real. In fact, it’s a special type of magic we do here at Verity Studios. We develop drone shows for entertainment industries.

Once we knew lampshades can fly, we asked ourselves: so what else can fly? This is how I became a drone costume designer. When I arrived, I was given a blank slate. Except for the lampshades, little work existed in this type of field. I needed to conduct research, tests, and experiments, and that meant a number of failures. But with each failure, working designs were right around the corner. In 9 months, we developed lots of costumes. I tried to explore all kinds of varying designs, while at the same time learning about drone technology and its limitations and challenges.

Below, you can find a selection of costume designs.

This first costume is based on a resonance effect. The quad, by flying at the right oscillations frequency, excites the costume movement, giving it a distinct personality. Resonance is often encountered in engineering but usually with negative connotations.

When we create a costume, we have several physical constraints. One of them is the weight. For an 80 cm quad like the one we work with, the maximum costume weight is 250gr. This one (above) is built with a very lightweight and flexible polystyrene foam fixed on a carbon structure.

Each costume is related or will be related to a choreography. So we can decide to design a costume for a specific movement we would like to play with.

Reverse effect on the costume using the drone movement.

This costume is very lightweight: only 150 gr. The white parts are made to catch air. That way, when the quad is flying with large up and down movements the costume reverses. They are all connected with thin carbon rods braided together, providing a nice pattern.

In the next experiment, we aimed to create playful light interaction. We thought it would be interesting, as an alternative, to make use of reflected light.

The result is a simple costume built with mirror surfaces.

And then when you project light onto it, it creates reflections on the wall and ceiling.

And also on the costume itself: it becomes the screen of random graphic effects, giving it an eerie look.

The downward draft created by the drone poses major design challenges for all of the costumes. Since it’s impossible to ignore, why not use it to actuate something on the costume itself? That way we can use it to our advantage. Looking at the GIF below, the blue surface under the quad catches air — as it’s related by wires to the costume — so it creates a “pop-up” movement.

Every time the quad takes off the costume will change shape:

This costume was explored the concept “opening flowers”.

We imagined it at the Singapore airport, in baggage claim. There are giant flowers flying over people, landing on specific platforms to recharge, allowing people to observe the flowers/costumes. Each time the quads take off the costumes would blossom, like flowers.

We are only scratching the surface for this art form, there is still so much terrain to explore. We look forward to continuing this research and I hope for the moment you are inspired to imagine the infinite possibilities within this area.

All images credit Verity Studios.

Check out more designs on their Drone Costume Design Instagram, find it here.

If you enjoyed this article, you may also want to read:

• Drone shows: Creative potential and best practices
• New quadrocopter video points to a future for flying machines in entertainment
• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• SPARKED wins at New York City Drone Film Festival
• Flying robots perform 100th show on Broadway, using new localization technology and algorithms
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Examples of live drone performances (starting top left): Starbright Holiday Drone Show (Intel-Disney), ABB’s 125th Anniversary Celebration (Verity Studios), Meet Your Creator (Saatchi & Saatchi/KMel Robotics) acquired by Qualcomm, Dance with drones on America’s Got Talent (Elevenplay).

Drone shows, such as the ones shown above, are a new and rapidly evolving sector of the entertainment industry.

Preface

At Verity Studios, we’ve published a document outlining this emerging field. For the past 8 years, our group has designed drone systems for shows and live events. With Verity Studios, we are now creating shows and delivering turnkey products for that purpose. Verity Studios is a spinoff company of the Flying Machine Arena, at the Swiss Federal Institute of Technology (ETH Zurich).

Our intent with this paper is to contribute to the emerging ecosystem of drone shows and help accelerate this industry’s safe development. The paper gives a broad overview, including creative, technical, and operational considerations. It also includes a section on best practice, which highlights the main safety concerns related to drone shows.

Further examples of live drone performances (starting top left): Muse Drone Tour, Ars Electronica’s Spaxels, Verity Studios’ drones in Cirque du Soleil’s Paramour on Broadway, Raffaello D’Andrea & Verity Studios’ show at TED 2016.

Abstract

Drone show systems are advanced show automation systems designed for live performances that enable the control of one to potentially thousands of autonomous multirotors or other small flying machines. These novel systems enable the use of drones to extend the traditional palette of light, sound, stage effects, and human performers by choreographing the simultaneous movements of a multitude of flying objects in 3D space. This gives this technology the potential to fundamentally transform the live events experience. Emergent use cases for this technology include animation of flying characters, flying lighting displays, movement of stage props and scenographic elements, and the creation of flying robot actors and synthetic swarms. The key challenges for this new technology are safety and reliability. First solutions for the safe deployment of this technology exist and avant-garde creators are embracing the possibilities offered by autonomous drones.

The full paper can be downloaded here: Drone shows: Creative potential and best practices

Your feedback is welcome at droneshows@veritystudios.com.

Examples of drone costumes (starting top left): Verity Studios, SPARKED; Otto Dieffenbach, Superman Drone; Ctrl.me, Performance Drones; Alan Kwan, Flying Umbrellas.

If you enjoyed this article, you may also want to read:

• New quadrocopter video points to a future for flying machines in entertainment
• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• SPARKED wins at New York City Drone Film Festival
• Flying robots perform 100th show on Broadway, using new localization technology and algorithms
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Photo: Cirque du Soleil – PARAMOUR on Broadway – at the Lyric Theatre. Featuring Flying Machine Design and Choreography by Verity Studios. Pictured: Ruby Lewis as ‘Indigo’ Ryan Vona as ‘Joey’. ©, Cirque du Soleil Theatrical. Photo by: Richard Termine

Since April, a troupe of eight flying machines has been performing in a Cirque du Soleil Broadway show called Paramour. This group of quadcopters has now completed its first 100 shows in front of a live theater audience, without a single incident. Given the string of recent safety incidents with drones (there’s more), this begs the question: How was this accomplished?

The Paramour quadcopters were designed and created by Verity Studios of Zurich and were transformed into flying lampshades in collaboration with Cirque du Soleil. Some readers may recognize them as the direct siblings of the similarly dressed quadcopters in Sparked: A Live Interaction Between Humans and Quadcopters; a short movie shot over the span of two days in the controlled environment of a research lab (see previous Robohub article and Quadrotors vs. CGI Q&A.) What sets the Paramour quadcopters apart, and the key to their ongoing success on Broadway, are a new localization technology and fail-safe algorithms, that keep the fully redundant quadcopters safely on course, performing 8 times a week, in front of a live audience of up to 2,000 people, without nets.

The people behind Verity Studios are no strangers to performing quadcopters. Founder Raffaello D’Andrea is a professor of control systems at the Swiss Federal Institute of Technology (ETH Zurich), from which Verity Studios was spun off. From a lab aptly named the Flying Machine Arena he has been conducting research for over 10 years on complex, autonomous flight, using a high-precision motion capture system to guide the vehicles. The ETH team’s first dancing drones performed in 2009 as part of research by Prof. D’Andrea’s former PhD students, Federico Augugliaro and Angela Schoellig. The group first stepped into the public performance space in 2011 with Flight Assembled Architecture, the world’s first multi-quadcopter performance in front of a large audience. In all, more than 200 live research demonstrations have taken place so far in the Zurich lab with occasional appearances given abroad, such as Google IO 2012.

While the team used motion capture technology extensively over this period, this type of localization technology proved inadequate for a long-running Broadway production when taken out of the research laboratory. The fundamental issue is that motion capture systems rely on a centralized architecture whereby all commands are forwarded to a single computer. A failure in that computer’s hardware or software may cause a sudden loss of control for the complete fleet.

Whereas an occasional failure of this type may be inconsequential in a research environment like the Flying Machine Arena, it is unacceptable for most commercial and industrial applications. The live events industry, in particular, has high safety standards owing in part to the amplification of safety risks when performing close to crowds of people. In addition to the safety risks, the consequences of a mishap entailing personal injury can be severe in terms of negative publicity, brand damage, and litigation.

Nevertheless, motion capture based performances near a live audience have been attempted, such as in this recent America’s Got Talent performance.

https://www.youtube.com/watch?v=1CjX1r-gh8U

To address these shortcomings, Verity Studios moved away from vision-based localization and developed a new system specifically for industrial applications that must meet strict safety and reliability standards. As stated by Raffaello D’Andrea at a recent TED talk, both the new radio-frequency based proprietary localization system and the flying machines are redundant and use fail-safe algorithms such that every vehicle can withstand any single point of failure.

Another key factor in the system’s success is its interoperability within a theatrical environment. Any theatrical system must function within a complex ecosystem of stage automation where stringent industry standards prevail.

“These flying machines have met or exceeded all expectations in terms of safety, reliability, maintenance, and ease of operation,” states Michael Shepp, Automation Carpenter for Paramour.

The technological achievement of creating a safe and robust system for flying a multitude of vehicles before a live audience introduces a rich palette for artistic expression. Creative staff at Cirque du Soleil recognized this potential and have worked side-by-side with Verity Studios to achieve this milestone on Broadway. Cirque du Soleil Chief Creative Officer, Jean-Francois Bouchard, states: “The flying machines are unquestionably one of the most important statements of the Paramour show.”

Cirque du Soleil and Verity Studios are currently exploring the artistic use of flying machines in other upcoming productions.

PARAMOUR runs at the Lyric Theatre (213 West 42nd Street) in New York.

If you enjoyed this article, you may also want to read:

• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• New quadrocopter video points to a future for flying machines in entertainment
• Quadrotors vs CGI: A Q&A with the makers of Sparked
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”
• Video: Throwing and catching an inverted pendulum – with quadrocopters
• Surrounded by his quadrocopter drones on stage, Raffaello D’Andrea explains feedback control and talks about the coming Machine Revolution

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Source: Robin Ritz, Institute for Dynamic Systems and Control, ETH Zurich.

The IDSC Tailsitter has been designed at the Institute for Dynamic Systems and Control, ETH Zurich, as a testbed for novel control algorithms for tailsitter vehicles. The goal of the project is to develop controllers that enable agile and robust flight for all flight regimes, such that the full potential of these vehicles can be exploited.

Airframe based on a Clark Y profile. Source: Robin Ritz, Institute for Dynamic Systems and Control, ETH Zurich.

The airframe of the vehicle is based on a Clark Y profile and designed such that for regular flight the pitching moment vanishes for zero flap angle, and such that the aerodynamic neutral point coincides with the vehicle’s center of gravity. The objective of this design is to allow agile maneuvers for all flight regimes and to avoid flap angle saturation problems caused by large trim angles.

Internal structure. Source: Robin Ritz, Institute for Dynamic Systems and Control, ETH Zurich.

The internal structure of the vehicle is built upon a core of a carbon sandwich composite material. The motors and servos are attached to the core using 3D-printed mountings, where the servos are placed such that they can be directly connected to the flaps. Further, a set of wing profile ribs and two carbon fiber tubes on the side for protection are mounted to the core structure. The electronics are placed at the center of the vehicle, and the battery at the front such that the center of gravity is at the desired position. All parts of the vehicle are screwed together (no glue is used), in order to simplify the replacement of broken components. The tailsitter’s wingspan is 48cm and the total weight is 150g.

The vehicle is controlled by a global controller enabling recovery to hover from any initial condition. A cascaded control architecture is used: Based on position and velocity errors an outer control loop computes a desired attitude keeping the vehicle in coordinated flight, while an inner control loop tracks the desired attitude using a lookup table with precomputed optimal attitude trajectories. The controller runs with an update rate of 500Hz on a Cortex M4F microprocessor. The vehicle is able to communicate with a ground station using an nRF51822 chip from the company Nordic Semiconductor. Communication can be established for recording telemetry data or for sending high-level commands to the vehicle, such as “take-off” or “land”. However, the vehicle is able to fly autonomously, the trajectories that are being tracked are stored on-board.

The on-board state estimation algorithms and technology are provided by the ETH spin-off company Verity Studios.

The attitude control algorithm is presented in the research paper “A Global Strategy for Tailsitter Hover Control”, International Symposium on Robotics Research (ISRR), 2015.

Researchers

Robin Ritz and Raffaello D’Andrea
Institute for Dynamic Systems and Control (IDSC), ETH Zurich, Switzerland
Flying Machine Arena

Technical details

Airframe: Custom design based on a Clark Y profile
Electronics: Custom design using a Cortex M4F microprocessor
Battery: Thunder Power RC G8 Pro Lite 25C LiPo 480mAh 2S
Motors: Hacker A05-13S
Motor controllers: Dys SN20A Mini ESCs with SimonK firmware
Propellers: 5×3 GWS EP-5030
Flap servos: MKS DS65K
Infrastructure: Flying Machine Arena

If you enjoyed this article, you may also want to read:

• ‘Tailsitter’ flying robot hovers and recovers easily thanks to new algorithm
• Multiple tethered quadrotors whip around a pole together at high speeds 
• Quadrocopter failsafe algorithm: Recovery after propeller loss
• Three new quadrotor videos demonstrate agile control and the power of machine learning
• Video: Quadrocopter learns from its mistakes, perfects slalom air racing

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

A camera man gets up, close, and personal with the human and machine actors on the “Sparked” set.

If you thought “SPARKED” – the new short film by Cirque du Soleil, ETH Zurich, and Verity Studios – is too real-looking to be CGI, you were right. But why go to all the trouble of using quadrotors to get those lampshades dancing in the air for real? We asked Bill Keays, Science and Technology Advisor at Cirque du Soleil, to give us an insider’s perspective about Cirque’s motivation for the film and how it came together.

Bill Keays on the SPARKED film set.

[Robohub] What was the inspiration for the film?
[Keays] The combination of autonomous flying machines with Cirque du Soleil magic was so evidently bursting with potential we committed ourselves to make something original together not knowing what it would be at first. An iterative creative process ensued involving all participants, and what emerged was a wonderful script that carried the talent from both sides of the formula to a whole new level.

Who are the main partners on the project? 

It was a highly democratic and highly collaborative process. Raffaello D’Andrea of ETH Zurich and Verity Studios, and Welby Altidor of Cirque du Soleil were the project leaders, but the true collaborative nature of this project is reflected in the fact that no participant roles are specified in the credits.

Who initiated the project? 

The very first discussions were held between Prof. Raffaello D’Andrea at ETH Zurich and myself as Science and Technology Advisor at Cirque du Soleil in March 2012. After ongoing discussions and considering various options, the two organizations eventually committed to a creation project in January 2014.

The “Sparked” film set. The shoot took place in the Flying Machine Arena at ETH Zurich, a portable space devoted to autonomous flight.

Why was the film made?

For Cirque the project demonstrates a whole new avenue for channeling creativity. It allows Cirque to reach a new audience externally and provides inspiration internally to push the creative envelope in new directions.

As such is a highly effective, multifaceted communications vehicle, and it will certainly be submitted for competition.

What were the production costs? 

The large bulk of the cost came in terms of human resources split between three organizations. The exact figures cannot be known.

How long did the project take from first concept to finished film? What were the major steps/milestones?

The whole project, from go-ahead to YouTube release, took under 8 months. It was a back and forth process whereby content was driven by discussions between Cirque, Verity, and the film director. Through this method we progressed through initial concept design, script development, set and prop design, and finally the shoot, with intense hardware and software development taking place throughout. The product was then handed off to post production and then to distribution.

Setting up before a flight scene.

Who designed the lampshades? The photos show researchers from ETH and Verity fitting the quads with them. Did you provide dimensions to props experts and iterate? How did that process work?

It was indeed an iterative process. The lampshades were designed and produced by the creative team at Cirque in Montreal who used their own quadrocopters for flight tests to ensure that they would be compatible with the flight environment at ETH Zurich. Guidance from the ETH and Verity teams was indispensable of course.

Was the actor controlling the quads with hand motions (via Kinect)? Or was this pre-programmed and choreographed? At times it looks like he is ‘conducting’ their movements …

The Flying Machine Arena fully possesses the ability to control the flying machines with a Kinect or other similar devices, or even hand gestures. But given that this is a movie production, the movements of the actor and the flying machines were choreographed as is the convention in film practice.

Was there any CGI at all in the film? How much is real?

None whatsoever, it’s all real; and even though it looks terrific in the movie, this pales in comparison to seeing it in the round. It was mindboggling to watch live.

Has Cirque worked with robots before?

Yes, Cirque has experimented with robots in the past, but never flying ones.

What were the biggest challenges of putting live actors together with the quads on stage?

Given that this is such an unusual circumstance for making a movie, we didn’t know how our lead actor would react to the flight environment. But the high precision of the flying machines and terrific expertise of ETH Zurich’s Flying Machine Arena staff quickly inspired confidence. By the second day, remarkably, our actors, both human and mechanical, were moving about the set with terrific ease, paying little attention to each other.

Was safety ever an issue?

The design of the lampshades was convenient in that it shielded the blades of the quadrocopters, thus making them far less likely to strike anyone or anything. There were a couple of crashes and a few low passes, but at no point was there any real apparent danger.

What about lighting?

Believe it or not, all the scenes with the flying lampshades are completely lit by the lampshades themselves! Another fortuitous design feature.

Changing batteries.

Were there any other challenges?

The rapid power consumption of the flying machines meant that battery changes would be frequent. With time, the switch-overs began looking like Formula 1 pit stops, performed with great speed and precision.

What was it like for the researchers from the lab to have to ‘perform’ for the camera? How long did they have to rehearse before you were ready?

The lampshades arrived in the weeks prior to the shoot so there was time for Verity Studios and ETH Zurich to fully test their aerodynamic abilities, and to run through the choreography. In the days preceding the shoot we performed flight tests according to the story board and had one practice day with the lead actor. However, due to the extreme time constraints imposed by the film crew, the control software had to be designed such that flight plans could be altered at a moment’s notice to meet specific requests from the director. This proved to be an invaluable tool and was instrumental in completing the shoot in the allotted time.

Had your previous work (e.g. demos, presentations) prepared you for working with quadrotors on a film set?

Absolutely, the Flying Machine Arena team had performed extensively with flying machines in public spaces and conferences (including TED), so much of this experience transferred well onto the movie set.

An early flight test in the Flying Machine Arena.

The quadrotors peek through in the film at times, as do the retro reflective markers for the motion capture system – was this intentional?

It is true that the quadrotors are sometimes visible, but the visual quality of the movie is so overwhelming that it is difficult to remain focused on technical bits. We actually liked the fact that they were occasionally visible; as sort of a wink to those of us who are knowledgeable about them.

With previous demos, the quadrotors played a leading role – but in this film, they are literally hidden behind lampshades. How does it feel to see them step back into a supportive role for this film?

But the flying machines are not hidden. They are in dress like all actors in this movie; they have character, motivation and certainly no lack of presence. As the devil’s advocate, I would be inclined to challenge the assumption that they are playing a supporting role.

Why not just do it with CGI?

There are at least two good reasons:

First of all, because the Flying Machine Arena is not about CGI. It’s about control systems, aerodynamics, and sophisticated algorithms that yield a physical manifestation of uncommon beauty.

The second reason is because you can’t do CGI in front of a live audience and Cirque du Soleil is in the business of live entertainment. Even though this extraordinary little movie is a gem in its own right, it is primarily the proof of concept for something far more powerful; something that must be seen live.

All images credited to the “Sparked” team. For more insight on the film see also New quadrocopter video points to a future for flying machines in entertainment by Markus Waibel of Verity Studios. 

We’ve seen robots on stage before: Humanoids have been featured in theater productions in Switzerland, Austria, and Japan, and industrial robots have played parts in dance performances and other staged events. Flying robots have also played a theatrical role, for example in Shakespeare’s A Midsummer Night’s Dream at Texas A&M, a dance set to a Schubert piano trio at MIT, as well as in advertising. Some robots have even gone pro: Robothespian was designed for acting and is rented out for performances.

But the dance between human and machine has never looked quite like this .

The film’s main plot is straight-forward: Lamps in a repair shop magically come alive, rising from their stands to sneak up on an unsuspecting repairman. As the magic unfolds, and the repairman discovers that the everyday objects of his trade have become animated and full of character, viewers become witness to the evolving dance between man and technology.

Everyone on the team, including myself, was impressed by how well the film captures the stunning, at times hypnotic visuals of the flying lampshades. Everything you see is real — no CGI, wires, or speed-ups/slow-downs were used in the film. 

So how did the lampshades come alive?

This short film, titled “Sparked”, was produced in collaboration between researchers at ETH Zurich’s Flying Machine Arena, their spin-off company Verity Studios, and entertainment company Cirque du Soleil; in other words, it’s quadrocopter magic that makes the lampshades fly.

The film marks a step change from previous work by ETH Zurich’s Flying Machine Arena team, catalyzed by their new spin-off company Verity Studios, which also gets credit for designing the quadrocopters’ choreography. Here, in spite of their computer-controlled precision flights, the flying machines become robot actors that are individuals with their own personalities: Pomponette, with her tassels bustling in the propellers’ airflow; the Medusa with its bizarre tentacles waving in the wind; the Runt, sleek and fast but still glowing and twitching from the electric shock. And Lady Purple, absent from most of the film because of her intractable aerodynamics.

Flying machines up, close, and personal

One of the most striking aspects of this film shoot was watching our actor, Nicolas Leresche, interact with the flying machines. Much of this interaction was captured on camera — and you can see the results in the film, in particular the moment of ‘first contact’ — but, for me, it was what happened off-camera that was most revealing.

Naively unjaded by their joys and dangers, this was Nicolas’ first experience with quadrocopters. As he began interacting with them, tentatively at first, and then — in a matter of seconds — more confidently, even playfully, it became crystal clear that I was not merely observing a human using a robotic tool.

As the quadrocopter’s onboard controller pushed back against Nicolas’ hand, immediately reacting to match his actions, it became apparent that the two interacted in a way we don’t usually get to interact with technology. The interaction here was physical and dynamic, and clearly intuitive for Nicolas. There was no noticeable learning curve, no need for manuals or explanations.

But what was most striking about this interaction was the result. Within seconds, Nicolas had established a deep, almost animate connection with the flying robot actors. As their dance developed, driven by the autonomous machines’ choreography and Nicolas’ script, and became ever more fluid and natural, it became clear that what we were witnessing was a relationship in its making; one of mutual respect and, at the same time, confidence and trust. And once there was trust, magic happened.

In his own words:

Actors think that they are the ones who make objects move. I think that, on the contrary, it’s the objects that make us move. In the case of drones, even more so! They are companions (in an etymological sense), confrères, brothers.

Within hours the entire film crew had become comfortable with the flying machines hovering nearby or brushing past them as they swooshed through the air. On one occasion, a mis-timed dolly motion led to a flying machine hovering just centimeters above the camera man, who continued with his work unfazed. On another occasion, when Nicolas missed his cue, he bumped into one of the flying machines. But they just jostled past each other and continued on with their choreography as if nothing exceptional had happened. On yet another occasion, a drone collided with the moving camera without disturbance.

The only damage during the entire two-day shoot was a single broken lamp, knocked over by one of the quadrocopters’ airflow.

Safety

I have no doubt that this level of confidence and comfort with the technology can in no small part be attributed to the system’s reliability. In spite of the packed shooting schedule (a total of 40 shots over two days) and the high complexity of the project (for example, designing lighting conditions using more than 60 light sources including some moving through space), production went smoothly, and importantly for the quadrocopter team, the flying machines did not add any significant delay. Battery swaps and on-the-spot changes to the choreography were efficient and quick, never exceeding a few minutes. Kudos to the excellent film team from who’s mcqueen for brilliantly doing their part to keep the production on schedule.

Collaboration

One aspect that made this project particularly enjoyable was the collaboration with creatives at Cirque du Soleil. From the start there was a lot of excitement about the potential of this technology for entertainment not just at ETH and Verity Studios here in Zurich but also at Cirque in Montreal. Everyone shared the view that the flying machines could be used to amaze audiences and to tell wonderful stories. What ensued was a collaborative, iterative creation process that culminated in a script and a set design that were imaginative to Cirque du Soleil standards, while exhibiting the abilities of the quadrocopters in their full splendor.

As it turned out, figuring out what is easy and what is difficult to achieve with autonomous quadrocopters can be quite counter-intuitive. Having researchers and creatives on both sides collaborate on the key creative elements of the story from the early phases of the project therefore proved invaluable to its success.

Next steps

Why go to all this trouble? We’ve all seen the amazing things CGI can do on film. I love films and movies, how they can condense experience and add life to our most impossible dreams. But nothing compares to the real experience of getting up, close, and personal with robot actors yourself.

What’s next? Our audiences over the past years have had more ideas than we can count. At this point, only one thing’s for sure: They all want to see more flying robot actors.

Some more snapshots from the film shoot below for your viewing pleasure:

All images credited to the “Sparked” team. Markus Waibel is a former member of ETH Zurich, a co-founder of Verity Studios, and one of the contributors to “Sparked”.