Examples of live drone performances (starting top left): Starbright Holiday Drone Show (Intel-Disney), ABB’s 125th Anniversary Celebration (Verity Studios), Meet Your Creator (Saatchi & Saatchi/KMel Robotics) acquired by Qualcomm, Dance with drones on America’s Got Talent (Elevenplay).

Drone shows, such as the ones shown above, are a new and rapidly evolving sector of the entertainment industry.

Preface

At Verity Studios, we’ve published a document outlining this emerging field. For the past 8 years, our group has designed drone systems for shows and live events. With Verity Studios, we are now creating shows and delivering turnkey products for that purpose. Verity Studios is a spinoff company of the Flying Machine Arena, at the Swiss Federal Institute of Technology (ETH Zurich).

Our intent with this paper is to contribute to the emerging ecosystem of drone shows and help accelerate this industry’s safe development. The paper gives a broad overview, including creative, technical, and operational considerations. It also includes a section on best practice, which highlights the main safety concerns related to drone shows.

Further examples of live drone performances (starting top left): Muse Drone Tour, Ars Electronica’s Spaxels, Verity Studios’ drones in Cirque du Soleil’s Paramour on Broadway, Raffaello D’Andrea & Verity Studios’ show at TED 2016.

Abstract

Drone show systems are advanced show automation systems designed for live performances that enable the control of one to potentially thousands of autonomous multirotors or other small flying machines. These novel systems enable the use of drones to extend the traditional palette of light, sound, stage effects, and human performers by choreographing the simultaneous movements of a multitude of flying objects in 3D space. This gives this technology the potential to fundamentally transform the live events experience. Emergent use cases for this technology include animation of flying characters, flying lighting displays, movement of stage props and scenographic elements, and the creation of flying robot actors and synthetic swarms. The key challenges for this new technology are safety and reliability. First solutions for the safe deployment of this technology exist and avant-garde creators are embracing the possibilities offered by autonomous drones.

The full paper can be downloaded here: Drone shows: Creative potential and best practices

Your feedback is welcome at droneshows@veritystudios.com.

Examples of drone costumes (starting top left): Verity Studios, SPARKED; Otto Dieffenbach, Superman Drone; Ctrl.me, Performance Drones; Alan Kwan, Flying Umbrellas.

If you enjoyed this article, you may also want to read:

• New quadrocopter video points to a future for flying machines in entertainment
• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• SPARKED wins at New York City Drone Film Festival
• Flying robots perform 100th show on Broadway, using new localization technology and algorithms
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

The Flying Ring in action. Source: Rajan Gill/YouTube

The Flying Ring is a new flying vehicle being developed at the Institute for Dynamic Systems and Control, ETH Zurich. The goal of the project is to fully characterize all aerodynamic properties of the vehicle. While traditional quadcopters are agile and carry high payloads they are not efficient in forward flight, with traditional lift to drag ratios comparable to a fruit fly. The Flying Ring vehicle, however, can fly on its side, allowing the blades to propel it forward faster than a typical quadcopter.

The video depicts the first prototype flying tethered. The annular wing (or ring) has a flat airfoil shape, which also covers the propellers and enhances human safety. These autonomous controlled flights help extract aerodynamic properties of the vehicle. A lift to drag ratio (which is a metric for aerodynamic efficiency) of 12 is achieved for the ring only. The total vehicle lift to drag ratio is lower, but can be substantially improved upon with an optimized design. Further details will be submitted to a future conference or research journal.

Why is the Flying Ring tethered? Flying tethered is an important part of the test, as it is used to characterize the steady state operating conditions at various flight speeds in a tight space, namely:

• Thrust of the propeller in forward flight
• Analysing annular wing lift and drag
• Evaluating body drag

In terms of structure, the vehicle is a standard quadrotor configuration. It has depron foam sheet attached via zip ties to the four motor mount arms, with black carbon slab wrapped around the foam.

Solidworks render of the vehicle, Flying Ring. Photo credit: Rajan Gill

Links to other videos shown:
Quadrotor pole acrobatics
Cooperative quadrotor ball throwing and catching
Onboard quadrocopter failsafe: flight after actuator failure
Flying Robots, Builders of tomorrow

Researchers
Rajan Gill and Raffaello D’Andrea
Institute for Dynamic Systems and Control (IDSC), ETH Zurich, Switzerland
ETH Zurich, Flying Machine Arena

Acknowledgments
This work is supported by and builds upon prior contributions by numerous collaborators in the Flying Machine Arena project.

This research was funded in part by the National Research Council of Canada (NSERC) and the Swiss National Science Foundation (SNSF).

If you enjoyed this article, you may also want to read:

• ‘IDSC Tailsitter’ flying robot performs vertical loops and easily transitions between hover and forward flight
• ‘Tailsitter’ flying robot hovers and recovers easily thanks to new algorithm
• Multiple tethered quadrotors whip around a pole together at high speeds 
• Quadrocopter failsafe algorithm: Recovery after propeller loss
• Three new quadrotor videos demonstrate agile control and the power of machine learning
• Video: Quadrocopter learns from its mistakes, perfects slalom air racing

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Photo: Cirque du Soleil – PARAMOUR on Broadway – at the Lyric Theatre. Featuring Flying Machine Design and Choreography by Verity Studios. Pictured: Ruby Lewis as ‘Indigo’ Ryan Vona as ‘Joey’. ©, Cirque du Soleil Theatrical. Photo by: Richard Termine

Since April, a troupe of eight flying machines has been performing in a Cirque du Soleil Broadway show called Paramour. This group of quadcopters has now completed its first 100 shows in front of a live theater audience, without a single incident. Given the string of recent safety incidents with drones (there’s more), this begs the question: How was this accomplished?

The Paramour quadcopters were designed and created by Verity Studios of Zurich and were transformed into flying lampshades in collaboration with Cirque du Soleil. Some readers may recognize them as the direct siblings of the similarly dressed quadcopters in Sparked: A Live Interaction Between Humans and Quadcopters; a short movie shot over the span of two days in the controlled environment of a research lab (see previous Robohub article and Quadrotors vs. CGI Q&A.) What sets the Paramour quadcopters apart, and the key to their ongoing success on Broadway, are a new localization technology and fail-safe algorithms, that keep the fully redundant quadcopters safely on course, performing 8 times a week, in front of a live audience of up to 2,000 people, without nets.

The people behind Verity Studios are no strangers to performing quadcopters. Founder Raffaello D’Andrea is a professor of control systems at the Swiss Federal Institute of Technology (ETH Zurich), from which Verity Studios was spun off. From a lab aptly named the Flying Machine Arena he has been conducting research for over 10 years on complex, autonomous flight, using a high-precision motion capture system to guide the vehicles. The ETH team’s first dancing drones performed in 2009 as part of research by Prof. D’Andrea’s former PhD students, Federico Augugliaro and Angela Schoellig. The group first stepped into the public performance space in 2011 with Flight Assembled Architecture, the world’s first multi-quadcopter performance in front of a large audience. In all, more than 200 live research demonstrations have taken place so far in the Zurich lab with occasional appearances given abroad, such as Google IO 2012.

While the team used motion capture technology extensively over this period, this type of localization technology proved inadequate for a long-running Broadway production when taken out of the research laboratory. The fundamental issue is that motion capture systems rely on a centralized architecture whereby all commands are forwarded to a single computer. A failure in that computer’s hardware or software may cause a sudden loss of control for the complete fleet.

Whereas an occasional failure of this type may be inconsequential in a research environment like the Flying Machine Arena, it is unacceptable for most commercial and industrial applications. The live events industry, in particular, has high safety standards owing in part to the amplification of safety risks when performing close to crowds of people. In addition to the safety risks, the consequences of a mishap entailing personal injury can be severe in terms of negative publicity, brand damage, and litigation.

Nevertheless, motion capture based performances near a live audience have been attempted, such as in this recent America’s Got Talent performance.

https://www.youtube.com/watch?v=1CjX1r-gh8U

To address these shortcomings, Verity Studios moved away from vision-based localization and developed a new system specifically for industrial applications that must meet strict safety and reliability standards. As stated by Raffaello D’Andrea at a recent TED talk, both the new radio-frequency based proprietary localization system and the flying machines are redundant and use fail-safe algorithms such that every vehicle can withstand any single point of failure.

Another key factor in the system’s success is its interoperability within a theatrical environment. Any theatrical system must function within a complex ecosystem of stage automation where stringent industry standards prevail.

“These flying machines have met or exceeded all expectations in terms of safety, reliability, maintenance, and ease of operation,” states Michael Shepp, Automation Carpenter for Paramour.

The technological achievement of creating a safe and robust system for flying a multitude of vehicles before a live audience introduces a rich palette for artistic expression. Creative staff at Cirque du Soleil recognized this potential and have worked side-by-side with Verity Studios to achieve this milestone on Broadway. Cirque du Soleil Chief Creative Officer, Jean-Francois Bouchard, states: “The flying machines are unquestionably one of the most important statements of the Paramour show.”

Cirque du Soleil and Verity Studios are currently exploring the artistic use of flying machines in other upcoming productions.

PARAMOUR runs at the Lyric Theatre (213 West 42nd Street) in New York.

If you enjoyed this article, you may also want to read:

• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• New quadrocopter video points to a future for flying machines in entertainment
• Quadrotors vs CGI: A Q&A with the makers of Sparked
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”
• Video: Throwing and catching an inverted pendulum – with quadrocopters
• Surrounded by his quadrocopter drones on stage, Raffaello D’Andrea explains feedback control and talks about the coming Machine Revolution

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Source: Robin Ritz, Institute for Dynamic Systems and Control, ETH Zurich.

The IDSC Tailsitter has been designed at the Institute for Dynamic Systems and Control, ETH Zurich, as a testbed for novel control algorithms for tailsitter vehicles. The goal of the project is to develop controllers that enable agile and robust flight for all flight regimes, such that the full potential of these vehicles can be exploited.

Airframe based on a Clark Y profile. Source: Robin Ritz, Institute for Dynamic Systems and Control, ETH Zurich.

The airframe of the vehicle is based on a Clark Y profile and designed such that for regular flight the pitching moment vanishes for zero flap angle, and such that the aerodynamic neutral point coincides with the vehicle’s center of gravity. The objective of this design is to allow agile maneuvers for all flight regimes and to avoid flap angle saturation problems caused by large trim angles.

Internal structure. Source: Robin Ritz, Institute for Dynamic Systems and Control, ETH Zurich.

The internal structure of the vehicle is built upon a core of a carbon sandwich composite material. The motors and servos are attached to the core using 3D-printed mountings, where the servos are placed such that they can be directly connected to the flaps. Further, a set of wing profile ribs and two carbon fiber tubes on the side for protection are mounted to the core structure. The electronics are placed at the center of the vehicle, and the battery at the front such that the center of gravity is at the desired position. All parts of the vehicle are screwed together (no glue is used), in order to simplify the replacement of broken components. The tailsitter’s wingspan is 48cm and the total weight is 150g.

The vehicle is controlled by a global controller enabling recovery to hover from any initial condition. A cascaded control architecture is used: Based on position and velocity errors an outer control loop computes a desired attitude keeping the vehicle in coordinated flight, while an inner control loop tracks the desired attitude using a lookup table with precomputed optimal attitude trajectories. The controller runs with an update rate of 500Hz on a Cortex M4F microprocessor. The vehicle is able to communicate with a ground station using an nRF51822 chip from the company Nordic Semiconductor. Communication can be established for recording telemetry data or for sending high-level commands to the vehicle, such as “take-off” or “land”. However, the vehicle is able to fly autonomously, the trajectories that are being tracked are stored on-board.

The on-board state estimation algorithms and technology are provided by the ETH spin-off company Verity Studios.

The attitude control algorithm is presented in the research paper “A Global Strategy for Tailsitter Hover Control”, International Symposium on Robotics Research (ISRR), 2015.

Researchers

Robin Ritz and Raffaello D’Andrea
Institute for Dynamic Systems and Control (IDSC), ETH Zurich, Switzerland
Flying Machine Arena

Technical details

Airframe: Custom design based on a Clark Y profile
Electronics: Custom design using a Cortex M4F microprocessor
Battery: Thunder Power RC G8 Pro Lite 25C LiPo 480mAh 2S
Motors: Hacker A05-13S
Motor controllers: Dys SN20A Mini ESCs with SimonK firmware
Propellers: 5×3 GWS EP-5030
Flap servos: MKS DS65K
Infrastructure: Flying Machine Arena

If you enjoyed this article, you may also want to read:

• ‘Tailsitter’ flying robot hovers and recovers easily thanks to new algorithm
• Multiple tethered quadrotors whip around a pole together at high speeds 
• Quadrocopter failsafe algorithm: Recovery after propeller loss
• Three new quadrotor videos demonstrate agile control and the power of machine learning
• Video: Quadrocopter learns from its mistakes, perfects slalom air racing

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Design process

As a starting point we derived a physical model of the flying machine, which helped us to understand the behavior of the system. For instance, as the electric ducted fans are placed farther away from the center of the vehicle, the lever arm of the thrust vectoring system increases – but so does the inertia of the system. These are conflicting requirements for high maneuverability, where one seeks to maximize the lever arm, but still keep the inertia small. The physical model helped us to find a suitable tradeoff.

In parallel, the electric ducted fan and the thrust vectoring mechanism were characterized quantitatively by doing measurements on a separate test bench. In this way, the properties of the actuation – e.g. the total thrust, the time constants of the fan, and the thrust vectoring mechanism – could all be determined experimentally.

Figure 1: The test bench for characterizing the electric ducted fan and the thrust vectoring system.

The measurements on the test bench were used to update the physical model of the flying machine and to assess its flight performance in simulations.

Figure 2: Electric ducted fan with exit nozzle and control flaps.

Key Facts

Flying Platform Website

Researchers
Michael Muehlebach
Raffaello D’Andrea

Mechanical and electrical design
Michael Egli
Marc-Andrè Corzillius

Status
Ongoing research project

Hardware design and specifications

The analysis based on the physical model was used to determine the specifications of the hardware design. The three electric ducted fans (Schübeler DS-51-DIA-HST) were aligned on an equilateral triangle of 30cm side length. A carbon fiber honeycomb structure was used to connect the fans, and a 3D printed exit nozzle was mounted on each fan (see Fig. 2) in order to increase the airflow velocity and augment the thrust vectoring capabilities. The exit nozzle also integrated the two servos that actuate the control flaps. The flaps were optimized for a high stall angle using the XFoil software package (see https://en.wikipedia.org/wiki/XFOIL). Three aluminum feet support the structure when the Flying Platform is on the ground. In total it weighs 7.6kg.

Power is delivered by six 2-cell (or alternatively three 4-cell) battery packs that have a capacity of approximately 6.6Ah. The power consumption at hover is roughly 5.3kW at 120A. A PX4 flight computer is used to run the control algorithms. Each fan delivers a maximum thrust of 4.2kg, which results in a payload of approximately 3kg.

Research on control algorithms

The Flying Platform is used for benchmarking novel control strategies that account for actuation limits. In particular, the flaps are limited to a range of approximately 18°. Control algorithms explicitly accounting for this limitation can provide larger stability margins and other performance enhancements.

We are currently investigating model predictive control algorithms. The idea is to explicitly exploit the physical model of the system – including input and state constraints – for making model-based predictions. Since these methods are computationally expensive, we seek to trade off the prediction accuracy with computational cost, and quantify the consequences of such approximations.

If you liked this article, you may also be interested in:

• Watch flying machines weave a rope bridge you can walk on
• Quadrocopter failsafe algorithm: Recovery after propeller loss
• Quadrocopters learn from prior experience to improve slalom flying

   

• Multiple tethered quadrotors performing high-g, high-speed formation maneuvers 

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

ETH Zurich, 2015.

Using quadrocopters and some rope, researchers have woven together a bridge strong enough to walk across.  Made at the ETH Zurich Flying Machine Arena in Switzerland, the bridge joins two scaffolds, and is the first full-scale load-bearing structure autonomously built by flying machines. The feat represents one more step in the field of robotic aerial construction [1, 2].

Except for the metal scaffolding at either end of the structure, the bridge itself consists exclusively of Dyneema rope, a tensile material with a low weight-to-strength ratio that makes it ideal for aerial construction. Weighing just 7 grams per meter, a 4 mm diameter Dyneema rope can sustain up to 1300 kg. The 7.4 m long bridge uses various rope techniques – knots, links and braids [3] – and has a total rope length of about 120 m.

How it works

The project was developed at the Flying Machine Arena [4], a research and demonstration platform for aerial robotics. The arena is equipped with a motion capture system that provides vehicle position and attitude measurements for the small custom quadrocopters. An offboard computer runs the algorithms and sends commands out to the flying machines via a customized wireless infrastructure.

Flying machines cooperate to create a braid and brace the structure together.

Before construction begins, the location of the scaffolding is measured and input to the system. Once the anchor point locations for the rope are known, all the connections and links of the bridge’s primary and bracing sections are — without human intervention — autonomously woven and secured into place by the quadrocopters, which are each equipped with a motorized spool that allows them to control the tension of the rope.

A quadrocopter equipped with a motorized spool for rope deployment.

A plastic tube guides the rope to a release point located between two propellers. The system estimates the external forces and torques exerted on the quadrocopter by the rope during deployment, and takes this into account in order to ensure that the quadrocopters behave as desired [5].

The researchers then walk across the bridge to demonstrate its load-bearing ability.

Once the bridge itself is constructed, stabilizers are added to make it easier for a person to cross. Because there are no sensors to detect the rope’s configuration, the locations of the narrow openings of the bridge must be measured and input to the system, which then adapts the quadrocopters’ trajectories in order to weave the stabilizers.

The team also developed a series of computational tools to help identify and design the kinds of novel structures that are buildable using flying robots. Using these tools, the researchers can simulate their designs, determine feasible construction sequences, and evaluate the structure before they begin building [6].

More information and related publications can be found on the project websites:

• http://www.idsc.ethz.ch/research-dandrea/research-projects/aerial-construction.html
• http://www.gramaziokohler.arch.ethz.ch/web/e/forschung/240.html

Credits

Aerial Construction is a collaboration between the Institute for Dynamic Systems and Control and Gramazio Kohler Research at ETH Zurich, Switzerland, 2015.

Researchers: Federico Augugliaro, Ammar Mirjan, Fabio Gramazio, Matthias Kohler, and Raffaello D’Andrea.

Related work

[1] F. Augugliaro, S. Lupashin, M. Hamer, C. Male, M. Hehn, M. W. Mueller, J. Willmann, F. Gramazio, M. Kohler, and R. D’Andrea. “The Flight Assembled Architecture Installation”, IEEE Control Systems Magazine, 2014

[2] A. Mirjan, F. Gramazio, and M. Kohler, “Building with Flying Robots”, FABRICATE: Negotiating Design and Making, 2014

[3] F. Augugliaro, E. Zarfati, A. Mirjan, and R. D’Andrea, “Knot-tying with flying machines for aerial construction”, IEEE/RSJ International Conference on Intelligent Robots and Systems, 2015

[4] S. Lupashin, M. Hehn, M. W. Mueller, A. P. Schoellig, M. Sherback, and R. D’Andrea, “A platform for aerial robotics research and demonstration: the flying machine arena”, Mechatronics, 2014

[5] F. Augugliaro and R. D’Andrea, “Admittance control for physical human-quadrocopter interaction”, European Control Conference, 2013

[6] A. Mirjan, F. Augugliaro, R. D’Andrea, F. Gramazio and M. Kohler,” Building a Bridge with Flying Robots”, Robotic Fabrication in Architecture, Art and Design 2016, in press

Article in preparation:

1. Augugliaro, M. Schulz, A. Mirjan, F. Gramazio, M. Kohler, and R. D’Andrea, ”Building a Rope Bridge with Flying Machines”

If you liked this post, you may also enjoy:

• Multiple tethered quadrotors performing high-g, high-speed formation maneuvers 
• Quadrocopter failsafe algorithm: Recovery after propeller loss
• Video: Throwing and catching an inverted pendulum – with quadrocopters
• Quadrocopters learn from prior experience to improve slalom flying

Raffaello D’Andrea’s robots can do amazing things: play soccer and ping-pong, perform complex choreographed acrobatics in the air, build towers brick by brick, and move pallets around warehouses. (See the video at his Web site, and check out his TED Talk.)

He also cofounded Kiva Systems, which was acquired by Amazon.com in May 2012 for $775 million with the hope of using the company’s orange robots to move items around warehouses, get products to consumers faster, and lower costs. Robots have reportedly been deployed in Amazon warehouses around the country.

Massimo Russo, a senior partner and managing director in the Boston office of The Boston Consulting Group and the regional leader of the Technology Advantage practice in North America, recently spoke with D’Andrea about the future of robotics. 

Edited excerpts from their conversation follow.

You’ve developed some amazing robots that play soccer and ping-pong. What novel potential uses for business and society do you see for robots in the future, and where are we liable to see robots next?

In the short term, the biggest impact is going to be from how we make things—how we extract things from the ground (mining, agriculture), how we make things (manufacturing), and how we distribute them. We’re going to see a lot of that innovation find its way into these sorts of realms in the next three to five years.

I still think we’re a ways away from the general-purpose robot in your home that’s going to do your bed, do the dishes, and so on. But I think we will also see in the medium-to-long term an infiltration of automation and robotics in the home.

A lot of your research has been around learning systems and learning robots. Do you see applications of learning algorithms for products such as the driverless car?

One of the biggest advantages of learning and adaptation is that it allows you to create powerful systems that are inexpensive to make. If you try to make a very precise robot, it costs a lot of money. If, however, you relax that requirement to make a machine that right off the bat is high performance, and let it learn and adapt tobecome high performance, it becomes economically viable to deploy such a machine. I think learning and adaptation will have a huge impact on how we make machines.

What has changed over the past few decades, either in terms of the underlying technology or in the economics of robotics, and how do you see robotics changing going forward?

A lot of it is probably best thought of as reaching tipping points. Let’s say you make a robot that could do something and that it costs $1.25 for it to do that, and a person could do the same thing for $1. You’re not going to use that robot. But next year, let’s say it takes 5 cents less to do it, so it becomes $1.20, and then $1.15, and then $1.10. From an outside perspective, you may say there’s no progress being made in this technology. Yet when that tipping point comes, when this robot can do the job for $1 or less, all of a sudden it becomes economically viable and everybody jumps in to adopt it. I think that there has been steady progress in the development of robotics and automation, but I think we’ve just reached various tipping points that will make them, all of a sudden, economically viable to deploy.

What kinds of systems innovations are you seeing? And if you think about this from a company perspective, are there interesting startups or maybe lessons even for the more established robotics players?

One of the neat things that big companies are doing is using robots in a cooperative fashion. A lot of people think that’s been going on for a long time, but actually that’s not true. For example, having multiple robot arms work together to manipulate objects for manufacturing, for installations, and so on—that’s a pretty new development. A lot of the big robotics companies are exploring these ideas—also integrating different types of automation technologies together. I think that’s an exciting development that big companies are mainly undertaking.

With startups, what really gets me excited are the folks who are looking at new ways to think about location and navigation. If we really want robots or autonomous systems to be pervasive, they need to know where they are—not just outdoors but also indoors. A lot of startups are trying to solve this problem by using advances in computer vision—and advances in inexpensive sensors, accelerometers, gyros, magnetometers, you name it—and in the algorithms to fuse all that information together and provide a local awareness for autonomous systems that just doesn’t exist right now.

Your startup, Kiva, was acquired by Amazon. What’s the long-term potential you see for warehouse logistics and management, distribution, and operations going forward?

What’s interesting about Kiva was that in the original business model, the key value that we thought Kiva was bringing to the space was cost. We thought, “Here’s a system that will allow us to reduce the cost of filling orders.” But it turns out that it was many of our secondary value-adds that were really exciting to customers: flexibility, adaptability, and a reduction in human errors. There was the fact that you could set up a warehouse in six weeks instead of two years, and the fact that if you wanted to reconfigure the warehouse, it was easy to do—which you couldn’t do with existing automation. The lesson was that businesses really want flexibility and adaptability.

Do you see applications of Kiva’s technology outside of warehouses, perhaps in more of a manufacturing environment?

Absolutely. We always thought about manufacturing when we were growing the company, but we were really focused on distribution. There was just so much opportunity in distribution that we said, well, manufacturing will come later—maybe we will spin off another company, maybe we will start another division. That still may happen. The underlying ideas can certainly be brought to manufacturing.

There’s been a lot of discussion about big data and companies hiring decision scientists. Are the underlying skill sets very similar, or should companies be thinking about hiring control engineers as well?

The right answer is somewhere in between. You can try to model absolutely everything and never look at data, and try to predict. This is what the Greek philosophers used to do—sit around and try to think of how the world works and forget about observations: “There should be a way to think things through.” That’s one extreme. The other extreme is, “Forget about trying to have any understanding of the physical world, we’re just going to look at data, and the data is going to tell us what causal relationship should take place and how the world works.”

We like to do things in between. We like to have physical models. We like to sit on the shoulders of the giants who have developed all of this understanding about how the world works. But we also like to use real physical data to augment our models, to refine our models, to tell us where our models have deficiencies. The more data you have, the better you can do that. I think there’s a lot to be gained by combining those two approaches. Folks who are only focused on big data—I think that’s limiting. Folks who can only do modeling—I think that’s limiting. You need people who can do modeling and who can work with big data.

Systems are inherently unpredictable and complex when you start connecting them. Do you have any cautionary tales for business leaders about how behaviors could be unpredictable around business systems?

There are many examples of when feedback or interactions with the environments can cause things to go wrong. One example that comes to mind is the Tacoma Narrows Bridge. When this bridge was designed, people didn’t really take into account the interaction of the bridge with a constant flow across the bridge. That caused the bridge to become unstable, and it collapsed. There was Chernobyl, which was basically a system that ran wild and became unstable because of various feedbacks that caused a positive runaway cycle.

Feedback is very powerful, but it’s also very dangerous. You have to be very careful when you start feeding things back without some sort of supervision, because you get unpredictability. It’s a two-edged sword: you want to harness the power, but at the same time you don’t want things to go wrong.

How do you get around that? One, try to understand things before you connect them together and try to predict what could go wrong. You can do that by simply trying to understand the system and trying to reason through it, but also you can make simulation models, agent-based models, you name it. I think that’s the first point. Second, just be cautious. Try things out, and see how it works. For really complicated things, it’s very difficult to prove that something is going to work when interconnected. A lot of times you just have to try it out.

The idea was developed around the following premises:

• The regulatory picture is murky and evolving – note that the FAA will likely miss the 2015 deadline for setting a UAV regulation, and that tethered aircraft are exempt from these rules
• Accountability is important – most people are not happy to be the target of camera that is being flown by an unseen operator

The solution is a tethered, GoPro-fitted quadcopter with onboard algorithms that let it fly instantly and virtually under any conditions, without relying on piloting skills, GPS working or wifi failing.

http://youtu.be/A0ccn9NeZow

Fotokite is designed to be launched and fully operational in one minute. To the user, flying the camera is as simple as holding a kite, and framing the picture is as easy as rotating your hand-held monitor. The tether provides visual accountability for bystanders and property owners.

Fotokite. Credit: Robert Ladig.

The man behind the project, Sergei Lupashin, is a TED Fellow with 10+ years of autonomous systems experience and has assembled an impressive team that includes robotics heavy-weight Raffaello D’Andrea as an advisor. The project is supported by the Swiss National Centre of Competence for Research in Robotics (NCCR), which is also a supporter of the Flying Donkey Challenge.

Already there are several concepts that enable a drone to autonomously follow and film a moving object (AirDog and Hexo+). However, the Fotokite seems particularly well suited for use in media, to provide birds-eye view while allowing those around the device to immediately identify the camera operator. Though its price is still unknown, it seems like a  cool tool that will surely be added to many reporters’ inventory one day soon.

Video and image courtesy of Fotokite.

If you liked this article, you may also be interested in:

• Robohub launches Indiegogo partner page for crowdfunding robot startups
• Quadrocopter failsafe algorithm: Recovery after propeller loss
• Video: Throwing and catching an inverted pendulum – with quadrocopters
• Three new quadrotor videos demonstrate agile control and the power of machine learning
• The thriving drone community of South Africa
• The Distributed Flight Array: Modular robots that self-assemble, coordinate and take flight
• No drone experts were harmed in the making of this video … Or were they?
• Academic and industry collaboration: Interview with Raffaello D’Andrea

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

UPDATE 04/03/2014:

In this video update, we show that a quadrocopter can be safely piloted by hand after a motor fails, without the aid of a motion capture system. This follows our previous video, where we demonstrated how a complete propeller failure can be automatically detected, and that a quadrocopter can still maintain stable flight despite the complete loss of a propeller. 

In the earlier video, we relied on an external motion capture system to measure the quadrocopter’s position and orientation.  By moving more of the algorithm onto the vehicle, the quadrocopter can now be piloted by hand after the failure. The algorithm is executed on the quadrocopter’s onboard micro-controller, and the only sensors required are the quadrocopter’s angular rate gyroscopes. We use blinking LEDs, mounted on the quadrocopter’s arms, to indicate a virtual yaw angle, so that the pilot can control the vehicle with the same remote control commands after the failure. As an alternative to the LED system, an onboard magnetometer could be used to track the vehicle’s yaw angle. Alternatively, by using more sophisticated algorithms, the system could be made to work using only the rate gyroscopes.

ORIGINAL STORY 02/12/2013

The video in this article shows an automatic failsafe algorithm that allows a quadrocopter to gracefully cope with the loss of a propeller. The propeller was mounted without a nut, and thus eventually vibrates itself loose. The failure is detected automatically by the system, after which the vehicle recovers and returns to its original position. The vehicle finally executes a controlled, soft landing, on a user’s command.

The failsafe controller uses only hardware that is readily available on a standard quadrocopter, and could thus be implemented as an algorithmic-only upgrade to existing systems. Until now, the only way a multicopter could survive the loss of a propeller (or motor), is by having redundancy (e.g. hexacopters, octocopters). However, this redundancy comes at the cost of additional structural weight, reducing the vehicle’s useful payload. Using this technology, (more efficient) quadrocopters can be used in safety critical applications, because they still have the ability to gracefully recover from a motor/propeller failure.

The key functionality of the failsafe controller is a novel algorithm that I developed as part of my doctoral research at the Institute for Dynamic Systems and Control at ETH Zurich. This new approach allows such a vehicle to remain in flight despite the loss of one, two, or even three propellers. Having lost one (or more) propellers, the vehicle enters a continuous rotation — we then control the direction of this axis of rotation, and the total thrust that the vehicle produces, allowing us to control the vehicle’s acceleration and thus position.

Even if the vehicle can no longer produce sufficient thrust to support its own weight, this technology would still be useful: one could, for example, try to minimize the multicopter’s velocity when it hits the ground, or steer the multicopter away from dangerous situations such as water, or people on the ground.

This control approach can also be applied to design novel flying vehicles — we will be releasing some related results soon.

This technology is patent pending.

For more information, have a look at the Flying Machine Arena website, the IDSC research page, or just post your question in the comments below.

If you liked this article, you may also be interested in:

• Three new quadrotor videos demonstrate agile control and the power of machine learning
• Live at TED Global 2013: Tweets, photos and video for quadrotor fans
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”
• Quadrocopters learn from prior experience to improve slalom flying

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

TEDGlobal 2013 in Edinburgh, Scotland. June 12-15, 2013. Photo: James Duncan Davidson

In this 4^th interview of our four-part ECHORD series, conducted last June, Sascha Griffiths from TUM talks to Raffaello D’Andrea, Professor of Dynamic Systems and Control at ETH Zurich and technical co-founder of Kiva Systems. The series explores success stories and common obstacles in industry-academia collaborations in the field of robotics, and examines the differences  between these collaborations in the US, Europe and Asia.

[Sascha Griffiths]
Can you describe your background a little? I am especially interested in your transitions between industry and academia. 

[Raffaello D’Andrea]
After getting my PhD at Caltech I became an assistant professor at Cornell University, where I was for ten years. I started the RoboCup program there and we had a RoboCup team that competed there for five years. We won the world championship four times. My area of research is dynamics and control systems, so, for me, robotics was kind of an application area. It’s a good system design problem.

While on sabbatical at MIT in 2003, I met Mick Mountz, an entrepreneur, and he told me about the problem he was trying to solve: distribution. That is when I decided to quit my sabbatical at MIT and join forces with him and Pete Wurman to start Kiva Systems.

In 2008, I officially moved to ETH Zurich but remained on as an advisor at Kiva, and I’ve been here since.

Would you say that Kiva Systems stemmed directly from your research?

No, I wouldn’t say directly. We did fundamental research on controlling robots and controlling dynamical systems. Kiva Systems has a connection to that, but it is not as though my research directly led to Kiva Systems. It was more that the knowledge, experience, and architectures that I learned during RoboCup transitioned into Kiva Systems.

Prof. Asada also mentioned in our interview with him that he thinks the practical character of RoboCup is really important. Do you think that is a good starting point?

Yes, absolutely.  A lot of our first hires at Kiva Systems were former RoboCup students from Cornell University. They learned great skills while they were doing these types of competitions:  how to work in teams and how to work at mechanical, electrical and computer science level issues. There are tight deadlines. You want to get the job done. There are a lot of parallels between doing things in competitions and being successful in the marketplace, and I think that is a great preparation.

Have you been in involved in academia-industry collaborations?

No, in fact I probably have a different view than most people on this topic, especially from most people in Europe. My thinking has been shaped more by the American system, where academia’s focus is on fundamental research. I would like to see more effort to strengthen academia and industry and I think the best way to do it is through entrepreneurs.

What do you think academia can offer to industry in a collaborative project? Do you think there is a reason why collaborations need to take place?

I do. I know a lot of colleagues who work really well with industry. Industry is full of interesting research problems, and it is full of folks who are interested in creating short-term, medium-term, and maybe even long-term gains. So, for them it makes sense to partner with industry. That is one way to do it. I tend to favor the other approach, which is to do fundamental research, and if it is good work, then attract the interest of entrepreneurs who are willing to learn about the new technology. There are also technologists who have an entrepreneurial spirit, who team up with domain experts and figure out how the technology could be used.

Where do you see the hurdles when technology is transferred from academia to the industry? Why do you think the transfer process is so slow?

The reward system in academia is very different to the reward system in industry. In academia, especially if you are doing fundamental research, robustness of what you are doing is not really the primary concern. It is new algorithms, new designs … and you don’t have to worry too much if it doesn’t work all the time. Of course in industry, especially with automation, that is a huge criterion. Systems have to work under different conditions; and they have to work all the time, 24/7. A lot of the time, it is about figuring out what is just a lab demo versus what is ready to be deployed in industry.

How do you think one could speed up the process of getting from the demo to a product that is commercially viable and achieving the robustness you mentioned?

Investment. Having entrepreneurs who understand where the technology is at and what is required to commercialize it, and who seek investment to actually do that process. This investment could come from private money or it could come from public funds, but it’s investment that speeds up the process.  I would favor that mechanism more than just giving money to the industry and academia for collaboration and say “commercialize this technology”. If you have entrepreneurs to lubricate this process, it will speed it up a lot.

You have already mentioned the reward systems. When you think of academic and industrial partners collaborating, how can they measure the success of their collaboration in your opinion?

I think one thing is patents, which are rewarded both in industry and academia. But I also think that a big metric of success is the students that are trained. A lot of the time, if it is a real collaboration between academia and the industry, those folks will transition into industry. I think you have to take a longer-term view than just a few years. You have to understand that if there is a relationship being built, it can form a pipeline of really talented PhDs with skills in that particular area, who then often go to work for that company. That is a huge plus, which is not immediate, but will have its benefits in the medium term.

How would you measure technological progress?

That depends on which way you look at it. If you are viewing it from an academic perspective, again, the reward is if it is getting accepted in top peer-reviewed publications. From the industrial side, the question is whether it is a product. And there is some stuff in between. I think you have to look at it on a case-by-case basis. I don’t think there is a simple metric. You have to look at the technology: how many years will it take to be commercialized? What impact will it have? Will something else replace it?

Do you know of something at the moment, for example, in the research with quadcopters, where you think that it has really improved the technology?

I think the limiting factor is not the technology, it is good business models. That is what is missing. The technology is quite far along … I think that once people figure out what the right business models are to commercialize this technology, things will move very quickly.

Do you see different models of technology transfer in North America and Europe?

I know Switzerland and I know the United States, and I think they are similar in many ways. If you develop technology at ETH, and it is done during research that is being funded, the inventors get a fraction of it, and the university gets a fraction of it. If students want to commercialize a technology themselves, they can license the technology from the university. If you personally don’t, the university can license it on your behalf. They get a certain fraction of the profits, the inventors get a certain fraction… That is also how it is done in the United States. I think that works well. It has the right incentives; it is not too onerous, where the university claims everything, which disincentives the inventors from doing it because they do not get any benefit. On the other hand, it is only fair that the university should get some rewards for developing technology and it is really leaving it up to the entrepreneurs to take these ideas and commercialize them. I think in that way it is similar. I cannot really talk about the rest of Europe.

Do you see any current trends in robotics, where you think real progress is being made towards commercialization?

I think people are looking at consumer robotics. But I think in the short term, it is going to be traditional manufacturing, helping existing businesses improve their operation through automation and robotics. That is where the potential is, simply because it is still a semi-structured environment. If you sell something to a consumer, it has to work in so many different situations. Every home is completely different. Whereas, if you are selling to a business that is trying to improve their manufacturing capabilities, for example, they have leeway on changing the way things are done, rearranging things, instrumenting things to help their operations. So, that is where I think you will see the big in-roads in robotics and automation.

Are we currently in a situation of increasing the market share and selling more, or are we seeing real progress?

No, I don’t think the market size is increasing. Before, it did not make sense to use automation in certain areas, but now the technology is improving in such a way that it does make sense to do so, and to do so in more industries.

Do you think academics lose interest as soon as something reaches market readiness?

Yes, I think that is the case. Academics are interested in doing fundamental research, and there is a different set of values and constraints when you want to commercialize something. For example, robustness – which I think is actually an interesting topic itself – has a lot of interesting research problems associated with it. They tend to be more at the system level. And I think only a few people are really interested in doing those sorts of things. So, I think, for the most part, academics lose interest. Not all, but a fair amount do.

Do you think that is a good or a bad thing?

I do not think it is a bad thing. I think that, again, you need something in between university research and commercialization to facilitate this process. There are many ways to do that. In Germany and in a lot of other European countries, there are strong industry-university collaborations. In the United States, it is more the entrepreneurial folks that are forming the glue. I tend to like the US model best. I think that it enables the quickest transition. Entrepreneurs are the ones who take on the stuff that is not ready for prime time, so to speak, and actually do it. And it takes investment. So, it is not bad that this happens … because without it, you would not get fundamental research done. So, I think that is perfectly fine.

In your TED talk, you talked about science, engineering, and arts coming together. I think you did not mention business and industry as such. Do you see these worlds connecting well or do you think they are all individual communities that tend to not communicate?

I think that they are separate entities, but there are people who are always at the overlap. You need enough of these people to ensure that there is a transfer of knowledge and interests.

Do you think the business, engineering, and science communities communicate enough?

I think they do. They could do more, of course. And I think the right way to do it is to provide the right incentives.

If you liked this article, you may also be interested in:

• The full ECHORD interview series
• Why fostering high-tech innovation and entrepreneurship matters to the economy, and what universities can do about it
• Live at TED Global 2013: Tweets, photos and video for quadrotor fans
• Raffaello D’Andrea on “Do robots kill jobs?”
• Raffaello D’Andrea on “How will robots shape the future of warfare?”
• Raffaello D’Andrea on “What funding scheme is the most conducive to creating a robotics industry?”
• Robots Podcast: Warehouse robots

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

2022 update: A new, one-wheeled version of the Cubli has been developed. The one-wheel Cubli is a three-dimensional pendulum system that can balance on its pivot using a single reaction wheel, as it is shown in this article.

2014 update: My colleagues at the Institute for Dynamic Systems and Control at ETH Zurich have created a small robotic cube that can autonomously jump up and balance on any one of its corners .

This latest version of the Cubli (2014) can jumping up, balance, and even “walk”. This new version is self contained with respect to power and uses three slightly modified bicycle brakes instead of the metal barriers used in the previous version. We are currently developing learning algorithms that allow the Cubli to automatically learn and adjust the necessary parameters if a jump fails due to the deterioration of the brakes and changes in inertia, weight, or slope of the surface.

This robot started with a simple idea:

Can we build a 15cm sided cube that can jump up, balance on its corner, and walk across our desk using off-the-shelf motors, batteries, and electronic components?

There are multiple ways to keep a cube in its balance, but jumping up requires a sudden release of energy. Intuitively momentum wheels seemed like a good idea to store enough energy while still keeping the cube compact and self-contained.

Furthermore, the same momentum wheels can be used to implement a reaction-torque based control algorithm for balancing by exploiting the reaction torques on the cube’s body when the wheels are accelerated or decelerated.

Can this work?

The first step in creating the robot, therefore, was to look at the physics to figure out if a jump-up based on momentum wheels was possible. The image below shows some of the math to figure out the Moment of Inertia (MOI) of the wheel and full cube.

This mathematical analysis allowed a quantitative understanding of the system which allowed to inform design choices, such as the trade offs of using three momentum wheels vs. a design with a momentum wheel mounted on each of the six inner faces of the cube.

Another outcome of this analysis was a good understanding of the required velocities of the momentum wheel to allow the cube to jump up, and the torques required to keep the cube in balance. Both factors were critical for the next steps: Determining the required hardware specs.

Specs and hardware design
Given the required velocities and torques determined above, it was clear that the momentum wheel’s motor and gearbox would be a major challenge for creating the robot. Using the mathematical model allowed to systematically tackle this problem by allowing a quantitative analysis of the trade-offs between higher velocities (i.e., more energy for jump-up) and higher torques (i.e., better stability when balancing).

This mathematics-driven hardware design resulted in detailed specs for the robot’s core hardware components (momentum wheels, motors, gears, and batteries) and allowed a CAD design of the entire system.

Part of this step was the design of a special brake to suddenly stop a momentum wheel to transfer its energy to the entire cube and cause it to jump up.

The photo to the left shows an early design of this brake, consisting of a screw mounted on the momentum wheel, a servo motor (shown in black) to move a metal plate (in blue) into the screw’s path (in light brown), and a mounting bracket (in light brown) to transfer the momentum wheel’s energy to the cube structure. The current design uses a combination of hardened metal parts and rubber to reduce peak forces.

2D prototype
To validate the mechanics and electronics of the jump-up and balancing strategy and prove feasibility of the overall concept a one-dimensional version was built:

The results obtained with this 2D version of the cube were published in an IROS 2012 paper.

The final robot
Following successful tests with the 2D version, a full robot was built. The result is Cubli, a small cube-shaped robot, named after the Swiss-German diminutive for “cube”.

As you can see in the video, Cubli can balance robustly.

However, first jump-up tests showed that the stress resulting from a sudden braking of the momentum wheel led to mechanical deformations of the momentum wheels and aluminium frame. This made repeated jump-ups of the whole Cubli impossible without part replacement. It was therefore decided to tweak the structure and breaking mechanism to reduce the mechanical stress caused by the jump-up.

In addition to balancing, my colleagues are now investigating the use of controlled manoeuvres of jumping up, balancing, and falling over to make the Cubli walk across a surface.

Note 1: This post is part of our Swiss Robots Series. If you’d like to submit a robot to this series, or to a series for another country, please get in touch at info[@]robohub.org.

Note 2: This work was done at the Institute for Dynamic Systems and Control, ETH Zurich, Switzerland and was funded in part by the Swiss National Science Foundation (SNSF), grant number 146717.

Note 3: If you have questions, post them below and we’ll post answers.

Some more photos:

Thanks Gajan!

If you liked this article, you may also be interested in:

• Insect-inspired flying robot handles collisions, goes where other robots can’t
• The Distributed Flight Array: Modular robots that self-assemble, coordinate and take flight
• Designed to mingle: Rezero ballbot’s smooth moves can handle a crowd

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Check out live tweets, amazing photos, TED blog posts and awesome video coverage of the session Those Flying Things at this year’s TED Global, featuring the work of automation and controls expert Raffaello D’Andrea, and drone ecologist Lian Pin Koh. We will be adding new material here as it becomes available, so check back soon. Photo credit: James Duncan Davidson.

This post is part of Robohub’s TED Global 2013 coverage.

Update: TED video | Raffaello D’Andrea: The astounding athletic power of quadcopters

Update: TED video | Timelapse: Building an arena for flying robots
See the full TED Blog post for more about how James Duncan Davidson made the video.

Tweets about “quadrotor OR quadrocopter OR drone OR quadcopter OR quadcoptor OR quadrocoptor OR D’Andrea OR Raffaello OR Lian OR Pin OR Koh OR robot OR Suarez #TEDGlobal”

In his TED Global talk titled “Machine Athleticism” Raffaello D’Andrea shows how he uses control algorithms to push quadrotors to perform at their physical limits. Instead of relying on slides, D’Andrea (who is well-known for his quadrotor demos) demonstrates his flying machines live as they toss a ball, balance a poll, serve a glass of water without spilling a drop, and even fly with their props cut off. In his own words, D’Andrea’s talk is about “the ability of machines to perform dynamic feats that fully exploit their physical capabilities.”

Check out official TED blogger Kate Torgovnick for her coverage of Raffaello D’Andrea’s talk.

See all the posts in Robohub’s TED Global 2013 coverage.

A quadrocopter swoops through the air to serve a glass of water without spilling a drop. Another gets two of its propellers cut off, yet still easily flies across the arena. “It looks like magic!” says ETH Zurich‘s Raffaello D’Andrea, but it took a lot of research, hard work and planning to bring this ‘magic trick’ to TED Global.
This post is part of Robohub’s TED Global 2013 coverage.

D’Andrea’s talk is about “Machine Athleticism” — the ability of machines to perform dynamic feats that fully exploit their physical capabilities. Part  of a session called “Those Flying Things” (which also features the work of drone ecologist Lian Pin Koh, airborne logistics activist Andreas Raptopoulos and sci-fi author Daniel Suarez), D’Andrea delves into what it takes to make a machine athlete:

We use something broadly called Model Based Design: we first build physics-based models that roughly capture the behavior of the machine; these models are then leveraged by a branch of mathematics called Control Theory to synthesize algorithms for controlling them.

He goes on to show a series of pretty amazing quadrotor feats that would be impossible without sophisticated automatic feedback. TED just posted an update on their blog about the demo here, and we will link to the video of the talk as soon as it goes online.

Clearly a lot of work has gone on behind the scenes to get D’Andrea and his team to TED. It’s taken years of research and a whole crew of PhD and Master students to develop the infrastructure and algorithms that make this demonstration possible — and that was just to get to the point where they could do the demos in-house.

D’Andrea and his team usually research and demo their quadrotor tricks at the ETH Flying Machine Arena — a 10x10x10m airspace dedicated to the study of control algorithms, and featuring a high-precision motion capture system, a wireless communication network, and custom software for executing estimation and control. Though the FMA normally lives at its home base at the Swiss Federal Institute of Technology (ETH) in Zurich, it was designed from the outset to be portable, and TED Global is not the first time it has taken to the road:
In 2011 they set up at the FRAC Centre in Orléans, France to create a six metre-high tower – the first architectural installation built by flying machines (see video). Last year the team showed their work at the world’s largest industrial fair, Germany’s Hannover Messe and at Google’s annual flagship conference Google IO in San Francisco.

Still, it takes a lot of time and careful planning to recreate the set up at a new site, and on this occasion they were planning a video shoot as well as a live audience. Says Mark Mueller, one of the team members onsite at TED, “Since April, all of us here have been working full-time, 100% on TED,” referring to all the various administrative hurdles they had to cross, such as planning logistics and agreeing on the space, in addition to rehearsing the demo itself. And then there was the packing to do: “It boils down to fifteen cases, boxes and bags,” says Dario Brescianinini, another member of the crew.

D’Andrea and his team arrived in Edinburgh, Scotland (the host city of TED Global 2013) late Saturday afternoon ahead of most other TED speakers, and started setting up almost right away in a dedicated room where the talk was filmed and where all their demonstrations will take place throughout the week. According to Brescianini and Mueller, it took about three-and-a-half hours to get the first quadrotor hovering. On Sunday they rehearsed the talk — and especially the transitions between each demo — in preparation for the video shoot, which was done in advance of the live talk.

Despite a few hiccups (two quads were damaged during setup and some repairs had to be done on the fly), Monday’s shoot went pretty smoothly. “It took about two-and-a-half hours, but we did one talk without breaks and had most of the things filmed after twenty-five minutes,” says Brescianini. It was a busy set, with fifteen TED crew members, seven people from D’Andrea’s team and one photographer. All in all, it took two fixed cameras (4k HD), one Steadicam, and three GoPros (one in the corner on the floor, one on the ceiling facing downwards, and one flying GoPro attached to a quad) to document D’Andrea’s TED talk. An additional camera shot a “making of” video as well. “It’s pretty busy, and we’re all really relieved after the successful filming.”

Want to learn more?
Check out the pages of ETH Zurich’s Flying Machine Arena team and more Robohub articles on how quadrocopters learn acrobatic maneuvers, cooperate to throw and catch balls, perfect air racing, and throw and catch an inverted pendulum.

Full disclosure: Markus Waibel works with Raff D’Andrea and the Flying Machine Arena team at ETH Zurich’s Institute for Dynamic Systems and Control.

See all the posts in Robohub’s TED Global 2013 coverage.

If you liked this article, you may also be interested in:

• Three new quadrotor videos demonstrate agile control and the power of machine learning
• Live at TED Global 2013: Tweets, photos and video for quadrotor fans
• Video: Throwing and catching an inverted pendulum – with quadrocopters
• Quadrocopters learn from prior experience to improve slalom flying

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

How do you like to go up in a swing,
Up in the air so blue?
Oh, I do think it the pleasantest thing
Ever a child can do!

Child’s Garden of Verses by Robert L. Stephenson

If you’ve ever been a child, chances are that you’ve spent some of your childhood on a swing. And at some point during that time you have probably found out how to move your arms, legs, and body to go higher and higher, without any external help from a friend or parent pushing you – you learned how to pump a swing.

Flavio Fontana, Master student at the ETH Zurich’s Institute for Dynamic Systems and Controls, has taken this concept and used it to control a robot. To make things more challenging, his robot does not just swing, but simultaneously juggles a ball. And, using some neat math to push the idea even further, he and his supervisors Philipp Reist and Raffaello D’Andrea have designed the robot to swing and juggle without cameras – and without ever dropping the ball.

The result of their research is the Swinging Blind Juggler, a robot that juggles a ball from side to side with an actuated paddle attached to a pendulum. The robot is called “blind”, because it has no cameras to “see” the ball. In fact, the Swinging Blind Juggler has no sensors to detect the ball at all.

Here is what Flavio has to say about his video:

In this video, we propose an elegant control strategy for a Swinging Blind Juggler. The key idea is to exploit the dynamic coupling between the motion of the paddle and the pendulum. Children on a swing intuitively use this dynamic coupling to control their amplitude. In a similar way, we control the pendulum with appropriate paddle motions.

So how does this work?

Flavio explains that four conditions must be met for successful juggling:

• First, the paddle used to juggle the ball must have a specific concave shape. This shape will always push the ball back towards the ideal flight path if the ball impact happens to be off-center.
• Second, the paddle has to decelerate as it strikes the ball. This helps to keep the ball bouncing at a consistent height: If the ball flies too high, it will hit the paddle later and hence at a lower speed. Conversely, if the ball flies too low, it will hit the paddle earlier and hence increase the ball’s height on the next bounce.
• Third, the paddle always has to strike the ball at right angles. This ensures that the ball always flies off into the right direction, towards the other side of the swinging motion.
• And fourth, the pendulum must be synchronized to the ball motion. In other words, the pendulum has to reach the peak of its swinging motion at the right time to strike the ball.

To guarantee all of the above, the researchers first derived a mathematical model of the desired motion. Based on these models, they came up with a hardware design that includes a parabolic plate to stabilize the ball’s impact points on the paddle (the first condition), a decelerating paddle motion that stabilizes the ball height (the second condition), and a mechanical mechanism based on four linked bars that automatically tilts the paddle to the right angle for all juggling amplitudes (condition number three).

To meet condition four (to swing back and forth at the right rhythm to hit the ball) the robot can use the same linear motor used to create the decelerating paddle motion for condition two – as long as it still strikes the ball with the right decelerating motion at its peaks. Similar to a child on a playground swing, the robot must raise its body as the swing passes through the lowest point and lower it near the extremes of the motion to pump the swing and swing higher.

The result of the precisely designed hardware setup and control algorithms is a complex motion that’s beautiful to watch – and that blindly juggles a ball from side to side.

To learn more about blind juggling robots, have a look at www.blindjuggler.org.

Note: This post is part of a series that decribes robots developed in Switzerland. If you’d like to contribute a robot to this (or another country’s) series, get in touch at info[@]robohub.org. Like this series? Read more about Swiss Robots!

John Dulchinos feature article: “The great equalizer: How robotics frees manufacturers from consolidating in low-wage nations”

These days it is hard to read an article about the future of robots that does not include a reference to jobs. As a pure roboticist, I object to the constant connection between the two, but as a concerned citizen I think it is a worthwhile discussion …

Read more →

Raffaello D’Andrea on “Do robots kill jobs?”

There is no doubt that robots, and automation in general, replace humans in the work-force: all productivity-enhancing tools, by definition, result in a decrease in the number of man-hours required to perform a given task …

Read more →

Mark Tilden on “Do robots kill jobs?”

Robots do kill jobs but they’re crappy jobs, so good riddance.  If you’ve ever had a job you were desperate for the money, but immediately regretted after you got it, then you know what I mean. …

Read more →

We hope you will join the discussion. Feel free to post your comment below.

See all the posts in Robohub’s Jobs Focus →

This Robotics By Invitation contribution is part of Robohub’s Jobs Focus.

There may be some regional effects that result in an immediate increase in jobs (for example, setting up a new manufacturing plant and hiring workers to maintain the machines), but the global effect is indisputable: overall, robots replace human workers.

What is also true, however, is that robots create jobs as well.  This is simply Economics 101: there is a redistribution of labor from low skilled jobs – what robots can do now, and the foreseeable future – to higher skilled jobs. An analogy from the North Carolina Department of Agriculture: “In 1790, 93% of the population of the United States was rural, most of them farmers. By 1990, only 200 years later, barely 2% of our population are farmers.”  What is also true is that there are many more software engineers now than there were in 1790; or mechanics; or physiotherapists; or professional athletes; or artists.

So the debate about robots replacing human workers is, for the most part, a tired and old one; just replace the word ‘robot’ with any productivity-enhancing tool or development. And as long as the process is gradual, one can reasonably argue that society benefits as a whole.

But the question does have merit, because human workers are at an artificial disadvantage relative to their robot counterparts, and the culprit is artificially low interest rates.  Large companies such as Procter and Gamble can issue 10 year corporate bonds that have astronomically low yields of 2.3%.  With money so cheap, productivity tools – such as robots – that would not be economically viable under normal interest rates and yields are now a bargain.  Why should a company ‘rent’ labor (a human worker) when it can ‘buy’ it (a robot)?  Have we not seen this storyline before?

Read more answers →

See all the posts in Robohub’s Jobs Focus →

[Photo credit: Petr Kratochvil.]

In my answer I have tried (as much as is humanly possible) to put myself in the role of an alien dispassionately analyzing the situation.  And when I do, I keep returning to the following conclusion: the best possible outcome for humanity would be for robots to not play any part (with the possible exception of purely defensive roles such as defusing mines) in warfare whatsoever.

If I were an alien, this is what I would first observe:

1. The ability for the average human to create tools that can harm others is rapidly increasing. This is especially the case for robots and intelligent machines: sensors, actuators, computing platforms, power systems, and other enabling technologies are continually becoming less expensive, more powerful, and more widely available. The result is that increasingly large numbers of people are becoming capable of delivering payloads accurately and over great distances.
2. The knowledge required to create robots, intelligent machines, and the algorithms that bring them to life are widely accessible and impossible to suppress; knowledge wants to be free in the same way that entropy wants to increase.
3. Humans have an incredibly strong sense of fairness, and their own research supports this. One of their greatest triumphs as a species is their ability to channel traits such as tribalism, aggression, and competitiveness – the same traits that lead to warfare – to a benign medium that results in their great joy and entertainment, not to mention a significant economic activity: organized sports. Central to this incredible accomplishment is the establishment of well-defined rules and regulations that strive to ensure a fair competition.

And this is what I would then conclude:

Dominant powers are being seduced by the advantages that robots can bring to the battlefield.  In the short term, this is a perfectly rational strategy. In the long term, however, this leads to an arms race. Even though a dominant power may be able to maintain its lead by continually developing robotic weapons, the capabilities of its adversaries, while inferior, will co-develop and eventually reach levels that will allow them to inflict catastrophic damage. 

Furthermore, asymmetric warfare insidiously erodes the sense of fairness outlined in point 3, with detrimental consequences for both sides.  The losing side is disenfranchised, which coupled with points 1 and 2 above, is extremely destabilizing. The winning side loses its moral compass and the fabric that holds its society together begins to unravel, leading to home-grown disenfranchisement and destabilization there as well. 

The net result of this robotic arms race will be a high-volatility stalemate, with dangerous weapons available to the masses and a lack of social restraint to prevent their indiscriminate use.    

If I were an alien, and thus immune to personal and economic factors that could influence my impartiality (such as having a loved one in combat, or being employed by a weapons dealer or manufacturer), I could only conclude that humanity would greatly benefit from imposing strict and far-reaching bans on the use of robotic technology in warfare.

As a human, not only am I skeptical that this will happen, I admit that my personal views are situation dependent: if my daughter were in combat, I wouldn’t care about asymmetry or fairness, I would want her to be as safe as possible. I don’t think that this makes me a hypocrite, it just makes me human.

Read more answers →

Two of the most challenging problems tackled with quadrocopters so far are balancing an inverted pendulum and juggling balls. My colleagues at ETH Zurich’s Flying Machine Arena have now combined the two.

As part of his Master thesis Dario Brescianini, student at ETH Zurich’s Institute for Dynamic Systems and Control, has developed algorithms that allow quadrocopters to juggle an inverted pendulum . If you are not sure what that means (or how that is even possible), have a look at his video “Quadrocopter Pole Acrobatics”:

(Don’t miss the shock absorber blowing up in smoke at 1:34!)

The Math

A quadrocopter with a plate for balancing the pole. The cross-shaped cut-outs are used for easy attachment to the vehicle and have no influence on the pendulum’s stability.

To achieve this feat, Dario and his supervisors Markus Hehn and Raffaello D’Andrea started with a 2D mathematical model. The goal of the model was to understand what motion a quadrocopter would need to perform to throw the pendulum. In other words, what is required for the pendulum to lift off from the quadrocopter and become airborne?

This first step allowed to determine (theoretical) feasibility. In addition, it showed the ideal trajectory in terms of positions, speeds, and angles the quadrocopter needed to follow to throw a pendulum. And it offered an insight into the throwing process, including identification of its key design parameters.

Reality Checks
The main goal of the next step was to determine how well the theoretic model described reality: How well does the thrown pendulum’s motion match the mathematical prediction? Does the pendulum really leave the quadrocopter at the pre-computed time? How does the pendulum behave while airborne? How well do assumptions for catching the pendulum (e.g., completely inelastic collisions, completely rigid pendulum, infinite friction between quadrocopter and pendulum when balancing) hold?

This second step involved multiple tests with the physical system, including throwing the pendulum by hand to study its aerodynamic properties and precisely timing the quadrocopters’ and pendulum’s motions during the maneuver.

Analyze, Experiment, Repeat

The shock absorber at the end of the pendulum is a balloon filled with flour and attached to a sliding metal cap with zip ties.

Armed with a good theoretical model and knowledge of its strengths and limitations, the researchers set out on a process of engineering the complete system of balancing, throwing, catching, and re-balancing the pendulum. This involved leveraging the theoretic insights on the problem’s key design parameters to adapt the physical system. For example, they equipped both quadrocopters with a 12cm plate that could hold the pendulum while balancing and developed shock absorbers to add at the pendulum’s tips.

This also involved bringing the insights gained from their initial and many subsequent experiments to bear on their overall system design. For example, a learning algorithm was added to account for model inaccuracies.

Dario writes:

This project was very interesting because it combined various areas of current research and many complex questions had to be answered: How can the pole be launched off the quadrocopter? Where should it be caught and – more importantly – when? What happens at impact?

The biggest challenge to get the system running was the catching part. We tried various catching maneuvers, but none of them worked until we introduced a learning algorithm, which adapts parameters of the catching trajectory to eliminate systematic errors.

The long and iterative process of this third step resulted in the final successful architecture to repeatedly throw and catch the pendulum on the real system, including three key components:

First, a state estimator was used to accurately predict the pendulum’s motion while in flight. Unlike the ball used in the group’s earlier demonstration of quadrocopter juggling, the pendulum’s drag properties depend on its orientation. This means, among other things, that a pendulum in free fall will move sideways if oriented at an angle. Since experiments showed that this effect was quite large for the pendulum used, an estimator including a drag model of the pendulum was developed.
This was important to accurately estimate the pendulum’s catching position.

Another task of the estimator was to determine when the pendulum was in free flight and when it was in contact with a quadrocopter. This was important to switch the quadrocopter’s behavior from hovering to balancing the pendulum.

Second, a fast trajectory generator was needed to quickly move the catching quadrocopter to the estimated catching position.

Third, a learning algorithm was implemented to correct for deviations from the theoretical models for two key events: A first correction term was learnt for the desired catching point of the pendulum. This allowed to capture systematic model errors of the throwing quadrocopter’s trajectory and the pendulum’s flight. A second correction term was learnt for the catching quadrocopter’s position. This allowed to capture systematic model errors of the catching quadrocopter’s rapid movement to the catching position.

The Result
As you can see in the video embedded above, at the end of Dario’s thesis two quadrocopters could successfully throw and catch a pendulum.

Many of the key challenge of this work were caused by the highly dynamic nature of the demonstration. For example, the total time between a throw and a catch is a mere 0.65 seconds, which is a very short time to move to, and come to full rest at, a catching position.

Another key challenge was the demonstration’s high cost of failure: a failed catch typically resulted in the pendulum hitting a rotor blade, with very little chance for the catching quadrocopter to recover. A crashed quadrocopter not only entailed repairs (e.g., changing a propeller), but also meant recalibration of the vehicle to re-determine its operating parameters (e.g., actual center of mass, actual thrust produced by propellors) and restarting the learning algorithms.

Says Markus Hehn:

This was a really fun project to work on. We started off with some back-of-the-envelope calculations, wondering whether it would even be physically possible to throw and catch a pendulum. This told us that achieving this maneuver would really push the dynamic capabilities of the system.

As it turned out, it is probably the most challenging task we’ve had our quadrocopters do. With significantly less than one second to measure the pendulum flight and get the catching vehicle in place, it’s the combination of mathematical models with real-time trajectory generation, optimal control, and learning from previous iterations that allowed us to implement this.

Note: The Flying Machine Arena is an experimental lab space equipped with a motion capture system.
Full disclosure: I work with Dario Brescianini, Markus Hehn, and Raffaello D’Andrea at ETH Zurich’s Institute for Dynamic Systems and Control.

More photos:

Credits: Carolina Flores, ETH Zurich 2012

If you liked this article, you may also be interested in:

• Three new quadrotor videos demonstrate agile control and the power of machine learning
• Live at TED Global 2013: Tweets, photos and video for quadrotor fans
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”
• Quadrocopters learn from prior experience to improve slalom flying

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

1. Continue to fund robotics competitions.  Not only is this a great way to educate researchers on how to build real systems, but the approaches adopted by the winning teams are highly correlated to what is feasible with today’s technology. Also, the challenges these teams encounter in a one-year competition cycle are similar to those faced by a young startup in its first year of existence. This funding approach could be expanded to include team projects that create large-scale public installations because, like startups, they too must be reliable and robust to succeed in a public, real-world context.  Finally, one must also make sure that entrepreneurs know about these competitions and team projects, and that they have the right incentives to attend.
2. Provide funding that allows freshly-minted PhDs to transition from fundamental research to applied research with a specific business focus, and at the same time provide support from business mentors and entrepreneurs.  A pre-startup phase, if you will. This not only brings the research closer to application, but it also gives individuals an appreciation of what it takes to commercialize their research, without the immediacy and constraints of being a startup.  A good example of this kind of funding scheme is the Pioneer Fellowships program established at ETH Zurich.

Read more answers →

During the 20 minute presentation, Raffaello D’Andrea revealed some of the key concepts behind his group’s impressive demonstrations of quadrocopters juggling, throwing and catching balls, dancing, and building structures – and illustrated them with live examples with quadrocopters flying on stage.

To watch him hurtle quadrocopters towards his audience, see them juggle balls and balance poles, and to find out what happens when control fails, check out the video:

Other speakers at the Zurich.Minds event included:

• Gerhard Schröder in an interview with host and organizer Rolf Dobelli (in German)
• Tomáš Sedláček on The Economics of Good and Evil
• Kevin Heng on Atmospheric Research of Exoplanets
• Ashkan Nikeghbali on The Fascination of Mathematics
• Lorenz Meier on the Open Hardware Revolution
• Christina Warinner on Evolutionary Diet
• Robert Cialdini on Influence
• Urs Hölzle on Where the Internet Lives, The Hidden Life of Data Centers
• Simone Schürle on Nano Robots for Medicine
• Lars Kolind on Unboss! The new leadership
• John Gray on The Dangers of Faith in Progress, and
• Benjamin Beilmann playing Prokofjew: Sonata for violin solo in D major, Op. 115 on his violin

Full disclosure: I, my colleagues from the Flying Machine Arena, and Raffaello D’Andrea all work in his group at the Institute for Dynamic Systems and Control at ETH Zurich.

Here are some photos of the event:

Photos: Zurich.Minds 2012

Manual programming of robots only gets you so far. And, as you can see in the video, for quadrocopters that’s not very far at all (see the “Without Learning” part starting at 1:30):

On its first try to navigate the obstacle course, the flying robot attempts to navigate based on a pre-computed flight path. The path is derived using a basic mathematical (“first principles”) model. But quadrocopters have complex aerodynamics – the force produced by the propellers changes depending on the vehicle’s velocity and orientation, and thus the actual amount of force produced is quite different from what the simple math describes.

What’s worse, these flying vehicles use soft propellers for safety, which bend differently depending on how much thrust is applied and wear rapidly with use (and even more rapidly when crashing).

Even with continuous feedback on the robot’s position from the motion capture system, manually programming the robots with a control sequence that takes all these imperfections into account is impractical.

My colleague Angela Schoellig and the Flying Machine Arena team here at ETH Zurich have now developed and implemented algorithms that allow their flying robots to race through an obstacle parcours – and learn to improve their performance.

Here is how Angela described the process to me:

The learning algorithm is applied to a quadrocopter that is guided by an underlying trajectory-following controller. The result of the learning is an adapted input trajectory in terms of desired positions. The algorithm has been equipped with several unique features that are particularly important when pushing vehicles to the limits of their dynamic capabilities and when applying the learning algorithm to increasingly complex systems:

1. We designed an input update rule that explicitly takes actuation and sensor limits into account by solving a constrained convex problem.
2. We developed an identification routine that extracts the model data required by the learning algorithm from a numerical simulation of the vehicle dynamics. That is, the algorithm is applicable to systems for which an analytical model is difficult (or impossible) to derive.
3. We combined model data and experimental data, traditional filtering methods and state-of-the- art optimization techniques to achieve an effective and computationally efficient learning strategy that achieves convergence in less than ten iterations.

The result is a robot that learns and improves each time it tries to perform a task.

In this example the robot races through a pylon parcours, calling to mind air races, such as the Red Bull Air Racing Championships or the Reno Air Races – except that there are no human pilots that spent their life learning to fly – here it’s robots doing the learning. And they are efficient, taking less than 10 training sessions to find the optimal steering commands!

Moreover, the learning algorithms are not specific to slalom racing, they can be used to learn other tasks. As Angela points out:

Our goal is to enable autonomous systems – such as the quadrocopter in the video – to ‘learn’ the way humans do: through practice.

The videos below show how learning algorithms can be used for other robotic tasks:

Full disclosure: Angela and the Flying Machine Arena team work in the same lab as I. Also, I’m working on RoboEarth, trying to allow robot learning on a much larger scale.