A Lucie micro drone takes off from a performer’s hand as part of a drone show. Photo: Verity Studios 2017

2017 was the year where indoor drone shows came into their own. Verity Studios’ Lucie drones alone completed more than 20,000 autonomous flights. A Synthetic Swarm of 99 Lucie micro drones started touring with Metallica (the tour is ongoing and was just announced the 5th highest grossing tour worldwide for 2017). Micro drones are now performing at Madison Square Garden as part of each New York Knicks home game — the first resident drone show in a full-scale arena setting. Since early 2017, a drone swarm has been performing weekly on a first cruise ship. And micro drones performed thousands of flights at Changi Airport Singapore as part of its 2017 Christmas show.

Technologically, indoor drone show systems are challenging. They are among the most sophisticated automation systems in existence, with dozens of autonomous robotic aircraft operating in a safety-critical environment. Indoor drone shows require sophisticated, distributed system control and communications architectures to split up and recombine sensing and computation between aircraft and their off-board infrastructure. Core challenges are not unlike those found in modern systems for manned aviation (e.g., combining auto-pilots, GPS, and air traffic control) and in creating tomorrow’s smart cities (e.g., combining semi-autonomous cars with intelligent traffic lights in a city).

These technological challenges are compounded by another: At least for permanent show installations, these systems need to be operated by non-experts. Two years ago, in one of the first major indoor drone shows, a swarm of micro drones flew over the audience at TED 2016. That system was operated by Verity Studios’ expert engineers. Creating a system that is easy enough to use, and reliable enough, to be operated by show staff is a huge technical challenge of its own. All of Verity’s 2017 shows mentioned above were fully client-operated, which speaks to the maturity that Verity’s drone show system has achieved.

Selection of Verity Studios’ indoor drone shows, from the drone swarm at TED 2016 to 20,000 autonomous indoor drone show flights in 2017 alone.

For my colleagues and me, it is these technological challenges, together with the visual impact of indoor drone shows, that makes these systems so much fun and hugely rewarding to work with.

Creative potential

Creatively, the capabilities of today’s indoor drone show systems barely scratch the surface of the technology’s potential. For centuries, show designers were restricted to static scenes. Curtains were required to hide scene changes from the audience, lest stage hands rushing to move set pieces destroy the magic created by a live show. The introduction of automation to seamlessly move backdrops and other stage elements, followed by the debut of automated lighting to smoothly pan and tilt traditional, stationary illumination were revolutionary.

Drones hold the potential for pushing automation further. The Lucies shown in the images above give a first inkling of the creative potential of flying lights that can be freely positioned in 3D space, appearing at will. Larger drones allow to extend that concept to nearly any object, including the creation of flying characters.

Safety

The most critical challenge for indoor drone show systems is safety. Indoor drone shows feature dozens of drones flying simultaneously and in tight formations, close to crowds of people, in a repeated fashion, in the high-pressure environment of a live show. For example, as part of the currently running New York Knicks drone show, 32 drones perform above 16 dancers, live in front of up to 20,000 people in New York’s Madison Square Garden arena, 44 times per season.

There are really only three ways to safely fly drones at live events.

The first way to achieve safety is the same that keeps commercial aviation safe: System redundancy. Using this approach, Verity Studios’ larger, Stage Flyer drones performed safely on Broadway, completing 398 shows and more than 7,000 autonomous flights, flying 8 times a week, in front of up to 2,000 people for a year, without safety nets. The Stage Flyer drones are designed around redundancy. At least two of each components are used (e.g., two batteries, two flight computers, and a duplicate of each sensor) or existing redundancies are exploited. For example, the Stage Flyer drones have only four propellers and motors, like any quadcopter. However, advanced algorithms that exploit the physics of flight allow these multi-rotor vehicles to fly with less than 4 propellers. The overall design allows these drones to continue to fly in spite of any individual component failure. For example, in one of the last Broadway shows, a Stage Flyer experienced a battery failure. The drone switched into its safety flight mode and landed, and the show continued with 7 instead of 8 drones. This approach to drone safety remains highly unusual — all drones available for purchase today have single points of failure.

Verity Studios drone show, 2017 Event Safety Summit, Rock Lititz. Photo: Event Safety Alliance 2017

The second approach to safety is physical separation. This is how safety is usually achieved for outdoor drone shows: Drones perform over a body of water or some roads are temporarily closed to create a large-enough area without people. For example, the Intel drone show at the Super Bowl was recorded far away from the NRG stadium. In fact, for the Super Bowl, safety went even a step further, also adding “temporal separation” to the physical separation (the drone show was actually pre-recorded days ahead of time, and viewers in the stadium and on TV were only shown a video recording). For indoor drone lightshows, physical separation can be achieved using safety nets.

The third approach to safely flying drones at live events is to make the drones so small that they have high inherent safety. Verity Studios’ Lucie micro drones weigh less than 1.8 ounces or 50 grams (including their flexible hull).

As the continuing string of safety incidents involving drones at live events attests, not everyone takes drone safety seriously. This is why my colleagues and I have worked with aviation experts and leading creatives to summarize best practices in an overview paper: Drone shows – Creative potential and best practices.

So, what’s in store for 2018? The appetite for indoor drone shows is huge, which is why Verity Studios is growing its team. And given the 2017 track record, there is a lot to look forward to — your favorite venue’s ceiling is the limit!

Metallica’s European WorldWired tour, which opened to an ecstatic crowd of 15,000 in Copenhagen’s sold-out Royal Arena this Saturday, features a swarm of micro drones flying above the band. Shortly after the band breaks into their hit single “Moth Into Flame”, dozens of micro drones start emerging from the stage, forming a large rotating circle above the stage. As the music builds, more and more drones emerge and join the formation, creating increasingly complex patterns, culminating in a choreography of three interlocking rings that rotate in position.

This show’s debut marks the world’s first autonomous drone swarm performance in a major touring act. Unlike previous drone shows, this performance features indoor drones, flying above performers and right next to throngs of concert viewers in a live event setting. Flying immediately next to audiences creates a more intimate effect than outdoor drone shows. The same closeness also allows the creation of moving, three-dimensional sculptures like the ones seen in the video — an effect further enhanced by Metallica’s 360-degree stage setup, with concert viewers on all sides.

Flying drones close to and around people in such a setting is challenging. Unlike outdoors, indoor drones cannot rely on GPS signals, which are severely degraded in indoor settings and do not offer the required accuracy for autonomous drone navigation on stage. The safety aspects of flying dozens of drones close to crowds in the high-pressure, live-event environment impose further challenges. Robustness to the uncertainties caused by changing show conditions in a touring setting as well as variation in the drone systems’ components and sensors, including the hundreds of motors powering the drones, is another necessary condition for this drone show system.

“It’s all about safety and reliability first”, says Raffaello D’Andrea, founder of the company behind the drones used in the Metallica show, Verity Studios (full disclosure: I’m a co-founder). D’Andrea knows what he is talking about: In work with his previous company, which was snatched up by e-commerce giant Amazon for an eye-watering 775M USD in 2012, D’Andrea and his team created fleets of autonomous warehousing robots, moving inventory through the warehouse around the clock. That company, which has since been renamed Amazon Robotics, now operates up to 10,000 robots — in a single warehouse.

How was this achieved?

In a nutshell: Verity Studios’ drone show system is an advanced show automation system that uses distributed AI, robotics, and sophisticated algorithms to achieve the level of robust performance and safety required by the live entertainment industry. With a track record of >7,000 autonomous flights on Broadway, achieved with its larger Stage Flyer drones during 398 live shows, Verity Studios is no newcomer to this industry.

Many elements are needed to create a touring drone show; the drones themselves are just one aspect. Verity’s drones are autonomous, supervised by a human operator, who does not control drone motions individually. Instead, the operator only issues high-level commands such as “takeoff” or “land”, monitors the motions of multiple drones at a time, and reacts to anomalies. In other words, Verity’s advanced automation system takes over the role of multiple human pilots that would be required with standard, remote-controlled drones. The drones are flying mobile robots that navigate autonomously, piloting themselves, under human supervision. The autonomous drones’ motions and their lighting design are choreographed by Verity’s creative staff.

To navigate autonomously, drones require a reliable method for determining their position in space. As mentioned above, while drones can use GPS for their autonomous navigation in an outdoor setting, GPS is not a viable option indoors: GPS signals degrade close to large structures (e.g., tall buildings) and are usually not available, or severely degraded, in indoor environments. Since degraded GPS may result in unreliable or unsafe conditions for autonomous flight, the Verity drones use proprietary indoor localization technology.

System architecture of Verity Studios’ drone show system used in Metallica’s WorldWired tour, comprising positioning modules part of Verity’s indoor localization technology, autonomous drones, and an operator control station.

It is the combination of a reliable indoor positioning system with intelligent autonomous drones and a suitable operator interface that allows the single operator of the Metallica show to simultaneously control the coordinated movement of many drones. This pilot-less approach is not merely a matter of increasing efficiency and effectiveness (who wants to have dozens of pilots on staff), but also a key safety requirement: Pilot errors have been an important contributing factor in dozens of documented drone accidents at live events. Safety risks rapidly increase as the number of drones increases, resulting in more complex flight plans and higher risks of mid-air collisions. Autonomous control allows safer operation of multiple drones than remote control by human pilots, especially when operating in a reduced airspace envelope.

Verity’s system also had to be engineered for safety in spite of other potential failures, including wireless interference, hardware or software component failures, power outages, or malicious disruption/hacking attacks. In its 398-show run on Broadway, the biggest challenge to safety turned out to be another factor: Human error. While operated by theater staff on Broadway, Verity’s system correctly identified human errors on five occasions and prevented the concerned drones from taking flight (on these occasions, the show continued with six or seven instead of the show’s planned eight drones; only one show proceeded without any drones as a safety precaution, i.e., the drone show’s “uptime” was 99.7%). As my colleagues and I have outlined in a recently published overview document on best practices for drone shows, when using drones at live event safety is a hard requirement.

Another key element for Verity’s show creation process are drone authoring tools. Planning shows like the Metallica performance requires tools for the efficient creation of trajectories for large numbers of drones. The trajectories must account for the drones’ actual flight dynamics, considering actuator limitations, as well as for aerodynamic effects, such as air turbulence or lift. Drone motions generated by these tools need to be collision-free and allow for emergency maneuvers. To create compelling effects, drone authoring tools also need to allow extracting all of the dynamic performance the drones are capable of — another area that D’Andrea’s team has gained considerable experience with prior to founding Verity Studios, in this case as part of research at the Swiss Federal Institute of Technology’s Flying Machine Arena.

Creating a compelling drone show requires more than the drone show system itself. For this tour, Verity Studios partnered with the world’s leading stage automation company TAIT Towers to integrate the drones into the stage floor as well as tackling a series of other technical challenges related to this touring show.

While technology is the key enabler, the starting point and the key driver of Verity’s shows are non-technological. Instead, the show is driven by the show designers’ creative intent. This comprises defining the role of show drones for the performance at hand as well as determining their integration into the visual and musical motifs of the show’s creative concept. For Metallica, the drones’ flight trajectories and lighting were created by Verity’s choreography team, incorporating feedback from Metallica’s production team and the band.

Metallica’s WorldWired tour
Metallica’s WorldWired Tour is their first worldwide tour after the World Magnetic Tour six years ago. The tour’s currently published European leg runs until 11 May 2018, with all general tickets sold out.

Further Robohub reading

• Drone shows: Creative potential and best practices
• Ben Einstein’s great tear-down of D’Andrea’s Amazon robots (although I disagree with Ben’s conclusion that the bots are “cooler than a flying drone” … maybe Ben hasn’t seen Verity’s Lucie drones yet? You be the judge.)
• Verity’s 7,000 autonomous drone flights on Broadway

Some images for your viewing pleasure

James Hetfield with Verity’s drones

Verity’s drones swarming below TAIT’s LED cubes

A glimpse at the 15,000-strong audience of the sold-out concert

Examples of live drone performances (starting top left): Starbright Holiday Drone Show (Intel-Disney), ABB’s 125th Anniversary Celebration (Verity Studios), Meet Your Creator (Saatchi & Saatchi/KMel Robotics) acquired by Qualcomm, Dance with drones on America’s Got Talent (Elevenplay).

Drone shows, such as the ones shown above, are a new and rapidly evolving sector of the entertainment industry.

Preface

At Verity Studios, we’ve published a document outlining this emerging field. For the past 8 years, our group has designed drone systems for shows and live events. With Verity Studios, we are now creating shows and delivering turnkey products for that purpose. Verity Studios is a spinoff company of the Flying Machine Arena, at the Swiss Federal Institute of Technology (ETH Zurich).

Our intent with this paper is to contribute to the emerging ecosystem of drone shows and help accelerate this industry’s safe development. The paper gives a broad overview, including creative, technical, and operational considerations. It also includes a section on best practice, which highlights the main safety concerns related to drone shows.

Further examples of live drone performances (starting top left): Muse Drone Tour, Ars Electronica’s Spaxels, Verity Studios’ drones in Cirque du Soleil’s Paramour on Broadway, Raffaello D’Andrea & Verity Studios’ show at TED 2016.

Abstract

Drone show systems are advanced show automation systems designed for live performances that enable the control of one to potentially thousands of autonomous multirotors or other small flying machines. These novel systems enable the use of drones to extend the traditional palette of light, sound, stage effects, and human performers by choreographing the simultaneous movements of a multitude of flying objects in 3D space. This gives this technology the potential to fundamentally transform the live events experience. Emergent use cases for this technology include animation of flying characters, flying lighting displays, movement of stage props and scenographic elements, and the creation of flying robot actors and synthetic swarms. The key challenges for this new technology are safety and reliability. First solutions for the safe deployment of this technology exist and avant-garde creators are embracing the possibilities offered by autonomous drones.

The full paper can be downloaded here: Drone shows: Creative potential and best practices

Your feedback is welcome at droneshows@veritystudios.com.

Examples of drone costumes (starting top left): Verity Studios, SPARKED; Otto Dieffenbach, Superman Drone; Ctrl.me, Performance Drones; Alan Kwan, Flying Umbrellas.

If you enjoyed this article, you may also want to read:

• New quadrocopter video points to a future for flying machines in entertainment
• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• SPARKED wins at New York City Drone Film Festival
• Flying robots perform 100th show on Broadway, using new localization technology and algorithms
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Photo: Cirque du Soleil – PARAMOUR on Broadway – at the Lyric Theatre. Featuring Flying Machine Design and Choreography by Verity Studios. Pictured: Ruby Lewis as ‘Indigo’ Ryan Vona as ‘Joey’. ©, Cirque du Soleil Theatrical. Photo by: Richard Termine

Since April, a troupe of eight flying machines has been performing in a Cirque du Soleil Broadway show called Paramour. This group of quadcopters has now completed its first 100 shows in front of a live theater audience, without a single incident. Given the string of recent safety incidents with drones (there’s more), this begs the question: How was this accomplished?

The Paramour quadcopters were designed and created by Verity Studios of Zurich and were transformed into flying lampshades in collaboration with Cirque du Soleil. Some readers may recognize them as the direct siblings of the similarly dressed quadcopters in Sparked: A Live Interaction Between Humans and Quadcopters; a short movie shot over the span of two days in the controlled environment of a research lab (see previous Robohub article and Quadrotors vs. CGI Q&A.) What sets the Paramour quadcopters apart, and the key to their ongoing success on Broadway, are a new localization technology and fail-safe algorithms, that keep the fully redundant quadcopters safely on course, performing 8 times a week, in front of a live audience of up to 2,000 people, without nets.

The people behind Verity Studios are no strangers to performing quadcopters. Founder Raffaello D’Andrea is a professor of control systems at the Swiss Federal Institute of Technology (ETH Zurich), from which Verity Studios was spun off. From a lab aptly named the Flying Machine Arena he has been conducting research for over 10 years on complex, autonomous flight, using a high-precision motion capture system to guide the vehicles. The ETH team’s first dancing drones performed in 2009 as part of research by Prof. D’Andrea’s former PhD students, Federico Augugliaro and Angela Schoellig. The group first stepped into the public performance space in 2011 with Flight Assembled Architecture, the world’s first multi-quadcopter performance in front of a large audience. In all, more than 200 live research demonstrations have taken place so far in the Zurich lab with occasional appearances given abroad, such as Google IO 2012.

While the team used motion capture technology extensively over this period, this type of localization technology proved inadequate for a long-running Broadway production when taken out of the research laboratory. The fundamental issue is that motion capture systems rely on a centralized architecture whereby all commands are forwarded to a single computer. A failure in that computer’s hardware or software may cause a sudden loss of control for the complete fleet.

Whereas an occasional failure of this type may be inconsequential in a research environment like the Flying Machine Arena, it is unacceptable for most commercial and industrial applications. The live events industry, in particular, has high safety standards owing in part to the amplification of safety risks when performing close to crowds of people. In addition to the safety risks, the consequences of a mishap entailing personal injury can be severe in terms of negative publicity, brand damage, and litigation.

Nevertheless, motion capture based performances near a live audience have been attempted, such as in this recent America’s Got Talent performance.

https://www.youtube.com/watch?v=1CjX1r-gh8U

To address these shortcomings, Verity Studios moved away from vision-based localization and developed a new system specifically for industrial applications that must meet strict safety and reliability standards. As stated by Raffaello D’Andrea at a recent TED talk, both the new radio-frequency based proprietary localization system and the flying machines are redundant and use fail-safe algorithms such that every vehicle can withstand any single point of failure.

Another key factor in the system’s success is its interoperability within a theatrical environment. Any theatrical system must function within a complex ecosystem of stage automation where stringent industry standards prevail.

“These flying machines have met or exceeded all expectations in terms of safety, reliability, maintenance, and ease of operation,” states Michael Shepp, Automation Carpenter for Paramour.

The technological achievement of creating a safe and robust system for flying a multitude of vehicles before a live audience introduces a rich palette for artistic expression. Creative staff at Cirque du Soleil recognized this potential and have worked side-by-side with Verity Studios to achieve this milestone on Broadway. Cirque du Soleil Chief Creative Officer, Jean-Francois Bouchard, states: “The flying machines are unquestionably one of the most important statements of the Paramour show.”

Cirque du Soleil and Verity Studios are currently exploring the artistic use of flying machines in other upcoming productions.

PARAMOUR runs at the Lyric Theatre (213 West 42nd Street) in New York.

If you enjoyed this article, you may also want to read:

• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• New quadrocopter video points to a future for flying machines in entertainment
• Quadrotors vs CGI: A Q&A with the makers of Sparked
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”
• Video: Throwing and catching an inverted pendulum – with quadrocopters
• Surrounded by his quadrocopter drones on stage, Raffaello D’Andrea explains feedback control and talks about the coming Machine Revolution

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

The first DroneApps Forum in Lausanne, Switzerland, recently assembled 150 drone professionals from Europe, the US, Japan, and Australia for a two-day exchange of ideas. 

European, Japanese, and US drone manufacturers were in attendance, with many reporting on the number of flight hours logged by their customers and their market shares. They pointed out current challenges, including the urgent need for more situational awareness for inspection tasks with multicopters, and the need for higher endurance and higher payload for monitoring with small fixed wing planes. 

Operators, including Robin Murphy (Roboticists Without Borders, Center for Robot-Assisted Search and Rescue), Cyberhawk, COWI, and the SNCF (French national railway company), presented case studies that highlighted the benefits and downsides of using drones to replace existing solutions. 

Regulators and legal professionals discussed the current state of legislation in the UK, France, Switzerland, the US, and Canada. They openly shared their concerns related to privacy, criminal activity, and liability — but also pointed out the importance to stay level-headed in spite of anxieties and fears. 

“Not every new technology has to result in a hectic activity of regulators.” – Peter Müller 

Discussions also covered software providers, including those providing aerial image processing such as Pix4d, Acute3D, and Agisoft. 

A number of the presentations are available here.

The picture emerging from this particular 2-day snapshot of the state of the industry showed what’s working today (e.g., fixed wing drones for surveying in the mining industry) and what isn’t (e.g., multicopters for autonomous inspection). Autonomous flight through airspace without obstacles seems to be well understood and operators have logged thousands of hours. Similarly, autonomous takeoff and landing of small fixed wing planes seems to be working remarkably well. However, autonomous flight near obstacles, such as that required for most inspection tasks, remains a challenge without adequate solution across all platforms. 

One key take-away from the forum was the stark contrast between doers, who constantly fly, and talkers, who stage drone flights for marketing and PR to help sell their vision. (Don’t get me wrong — I think both are valid ways to boot-strap a new business, each with its own risks.)

Some other key take-aways from this particular event: 

• Innovation is led by startups, some of which have already accumulated significant operational experience. Major established players like Lufthansa or Swiss WorldCargo are lagging far behind. 
• Most hardware and electronics seem to be coming out of Europe. Most high-level software, apps, and cloud applications seem to be coming out of the US. 
• Drones are used much more to improve current solutions than to pioneer new application areas. 
• In spite of all the media hype, it’s still very early days for commercial drones. 

We’ve seen robots on stage before: Humanoids have been featured in theater productions in Switzerland, Austria, and Japan, and industrial robots have played parts in dance performances and other staged events. Flying robots have also played a theatrical role, for example in Shakespeare’s A Midsummer Night’s Dream at Texas A&M, a dance set to a Schubert piano trio at MIT, as well as in advertising. Some robots have even gone pro: Robothespian was designed for acting and is rented out for performances.

But the dance between human and machine has never looked quite like this .

The film’s main plot is straight-forward: Lamps in a repair shop magically come alive, rising from their stands to sneak up on an unsuspecting repairman. As the magic unfolds, and the repairman discovers that the everyday objects of his trade have become animated and full of character, viewers become witness to the evolving dance between man and technology.

Everyone on the team, including myself, was impressed by how well the film captures the stunning, at times hypnotic visuals of the flying lampshades. Everything you see is real — no CGI, wires, or speed-ups/slow-downs were used in the film. 

So how did the lampshades come alive?

This short film, titled “Sparked”, was produced in collaboration between researchers at ETH Zurich’s Flying Machine Arena, their spin-off company Verity Studios, and entertainment company Cirque du Soleil; in other words, it’s quadrocopter magic that makes the lampshades fly.

The film marks a step change from previous work by ETH Zurich’s Flying Machine Arena team, catalyzed by their new spin-off company Verity Studios, which also gets credit for designing the quadrocopters’ choreography. Here, in spite of their computer-controlled precision flights, the flying machines become robot actors that are individuals with their own personalities: Pomponette, with her tassels bustling in the propellers’ airflow; the Medusa with its bizarre tentacles waving in the wind; the Runt, sleek and fast but still glowing and twitching from the electric shock. And Lady Purple, absent from most of the film because of her intractable aerodynamics.

Flying machines up, close, and personal

One of the most striking aspects of this film shoot was watching our actor, Nicolas Leresche, interact with the flying machines. Much of this interaction was captured on camera — and you can see the results in the film, in particular the moment of ‘first contact’ — but, for me, it was what happened off-camera that was most revealing.

Naively unjaded by their joys and dangers, this was Nicolas’ first experience with quadrocopters. As he began interacting with them, tentatively at first, and then — in a matter of seconds — more confidently, even playfully, it became crystal clear that I was not merely observing a human using a robotic tool.

As the quadrocopter’s onboard controller pushed back against Nicolas’ hand, immediately reacting to match his actions, it became apparent that the two interacted in a way we don’t usually get to interact with technology. The interaction here was physical and dynamic, and clearly intuitive for Nicolas. There was no noticeable learning curve, no need for manuals or explanations.

But what was most striking about this interaction was the result. Within seconds, Nicolas had established a deep, almost animate connection with the flying robot actors. As their dance developed, driven by the autonomous machines’ choreography and Nicolas’ script, and became ever more fluid and natural, it became clear that what we were witnessing was a relationship in its making; one of mutual respect and, at the same time, confidence and trust. And once there was trust, magic happened.

In his own words:

Actors think that they are the ones who make objects move. I think that, on the contrary, it’s the objects that make us move. In the case of drones, even more so! They are companions (in an etymological sense), confrères, brothers.

Within hours the entire film crew had become comfortable with the flying machines hovering nearby or brushing past them as they swooshed through the air. On one occasion, a mis-timed dolly motion led to a flying machine hovering just centimeters above the camera man, who continued with his work unfazed. On another occasion, when Nicolas missed his cue, he bumped into one of the flying machines. But they just jostled past each other and continued on with their choreography as if nothing exceptional had happened. On yet another occasion, a drone collided with the moving camera without disturbance.

The only damage during the entire two-day shoot was a single broken lamp, knocked over by one of the quadrocopters’ airflow.

Safety

I have no doubt that this level of confidence and comfort with the technology can in no small part be attributed to the system’s reliability. In spite of the packed shooting schedule (a total of 40 shots over two days) and the high complexity of the project (for example, designing lighting conditions using more than 60 light sources including some moving through space), production went smoothly, and importantly for the quadrocopter team, the flying machines did not add any significant delay. Battery swaps and on-the-spot changes to the choreography were efficient and quick, never exceeding a few minutes. Kudos to the excellent film team from who’s mcqueen for brilliantly doing their part to keep the production on schedule.

Collaboration

One aspect that made this project particularly enjoyable was the collaboration with creatives at Cirque du Soleil. From the start there was a lot of excitement about the potential of this technology for entertainment not just at ETH and Verity Studios here in Zurich but also at Cirque in Montreal. Everyone shared the view that the flying machines could be used to amaze audiences and to tell wonderful stories. What ensued was a collaborative, iterative creation process that culminated in a script and a set design that were imaginative to Cirque du Soleil standards, while exhibiting the abilities of the quadrocopters in their full splendor.

As it turned out, figuring out what is easy and what is difficult to achieve with autonomous quadrocopters can be quite counter-intuitive. Having researchers and creatives on both sides collaborate on the key creative elements of the story from the early phases of the project therefore proved invaluable to its success.

Next steps

Why go to all this trouble? We’ve all seen the amazing things CGI can do on film. I love films and movies, how they can condense experience and add life to our most impossible dreams. But nothing compares to the real experience of getting up, close, and personal with robot actors yourself.

What’s next? Our audiences over the past years have had more ideas than we can count. At this point, only one thing’s for sure: They all want to see more flying robot actors.

Some more snapshots from the film shoot below for your viewing pleasure:

All images credited to the “Sparked” team. Markus Waibel is a former member of ETH Zurich, a co-founder of Verity Studios, and one of the contributors to “Sparked”.

UPDATE: New video of a collaborative, cloud-based mapping experiment. Mapping is essential for mobile robots and a cornerstone of many more robotics applications that require a robot to interact with its physical environment. It is widely considered the most difficult perceptual problem in robotics, both from an algorithmic but also from a computational perspective. Mapping essentially requires solving a huge optimization problem over a large amount of images and their extracted features. This requires beefy computers and high-end graphics cards – resulting in power-hungry and expensive robots.

Update

Here is a video of a collaborative mapping experiment using Rapyuta, the open source cloud robotics framework. Instead of just compressing each frame of the stream, we now run a dense visual odometry algorithm to produce keyframes and these keyframes are eventually sent to the cloud for optimization. This technique is a very natural way to compress since keyframes are produced only when the robot has moved a certain distance or angle. The Cloud, in addition to optimization, now also does merging of maps produced by multiple robots. The code is open under the Apache 2.0 license at https://github.com/rapyuta/rapyuta-mapping.

Together with our RoboEarth colleagues Dominique Hunziker, Dhananjay Sathe, Mayank Singh, we have set up an inexpensive, light weight robot so that it can perform full 3D mapping in real-time by offloading heavy computation to the RoboEarth Cloud Engine (nickname: Rapyuta) (see previous Robohub article). In the video below Master student Vlad Usenko shows how robots can sidestep this problem.

How it works
What actually happens here? As you can see from the video and the figure below, the robot is equipped with an RGBD sensor (ASUS Xtion PRO), an ARM-based single board computer (Odriod-U2), and an off-the-shelf wireless dongle. The robot transmits RGB images (QVGA resolution) and depth images (QVGA resolution) at 30 FPS to Rapyuta with a frame by frame compression (JPEG for RGB, and PNG for depth images). The single board computer is only used for compression, communication, and low-level control of the robot – the heavy computation all happens off-board. The optimized map shown at the end of the video is built by manually triggering the optimization process in the cloud after the demo (i.e., not in real time).

The main purpose of this demonstrator is to showcase that today’s wireless technologies allow robots to outsource computationally heavy data processing to the cloud. As shown in the figure below, and as I’ve laid out in an earlier article on Cloud Robotics, bandwidth is a key driver for cloud-based robot services and applications.

Most actual mapping scenarios call for a slightly different split in work-load: You’d want to perform visual odometry locally and only send keyframes to Rapyuta for global optimization. The good news is that we already supports this setup: Rapyuta provides the corresponding software packages to run on ODroid boards along with the necessary cloud computing infrastructure.

What’s next?
This demonstration stops short of actually using the data returned from the cloud: A map is created and available in real time, but not actively used to control the robot. Using the map to control the robot, e.g. to improve exploration, is one logical extension.

Another natural next step would be to use the computational resources in the cloud for localizing in an existing map. Imagine a robot entering a new building and trying to get its bearings. Such localization in an existing map requires performing a search over the entire map – essentially a very large set of images and their features – to find the best match between what the robot is seeing now and a part of the map.

But the cloud can be used to do much more interesting things than improve single robot navigation. Multiple robots can instantaneously share their maps, so applications like multi-robot mapping become much easier. This implies using the cloud as more than a pure computational resource – it becomes a medium to store, share, and collectively improve data. This is interesting for static scenes, but things get even more interesting for dynamic scenes, for example when multiple robots track dynamic objects in the map. With WIRE, some of my RoboEarth colleagues have developed algorithms to do just that. We are currently working on integrating this functionality with Rapyuta and the RoboEarth database and will make it available as a Rapyuta cloud service later this year.

Many of the key advantages of using this Cloud Robotics approach to mapping will only become apparent when maps are brought together with the vast amounts of knowledge required to understand environments. Performing mapping in the cloud not only allows the creation of maps but also the ability to understand them: By bringing computation close to the knowledge required to make sense of all of a robot’s sensor information, Cloud Robotics offers robots a very powerful way to understand the world around them. When working together, cloud computing frameworks like Rapyuta and online knowledge bases like RoboEarth can allow us to make robots that are both smart and inexpensive.

Additional information
For further information, see:

• Technical details and tutorial for this setup (available early July)
• Video of recent live Cloud Robotics Mapping demo at ROSCon 2013
• Cloud Robotics Workshop at IROS 2013 in November, Tokyo
• Homepage of the RoboEarth Cloud Engine (codename “Rapyuta”)
• RoboEarth homepage

Questions?
Feel free to post in the comments below.

2022 update: A new, one-wheeled version of the Cubli has been developed. The one-wheel Cubli is a three-dimensional pendulum system that can balance on its pivot using a single reaction wheel, as it is shown in this article.

2014 update: My colleagues at the Institute for Dynamic Systems and Control at ETH Zurich have created a small robotic cube that can autonomously jump up and balance on any one of its corners .

This latest version of the Cubli (2014) can jumping up, balance, and even “walk”. This new version is self contained with respect to power and uses three slightly modified bicycle brakes instead of the metal barriers used in the previous version. We are currently developing learning algorithms that allow the Cubli to automatically learn and adjust the necessary parameters if a jump fails due to the deterioration of the brakes and changes in inertia, weight, or slope of the surface.

This robot started with a simple idea:

Can we build a 15cm sided cube that can jump up, balance on its corner, and walk across our desk using off-the-shelf motors, batteries, and electronic components?

There are multiple ways to keep a cube in its balance, but jumping up requires a sudden release of energy. Intuitively momentum wheels seemed like a good idea to store enough energy while still keeping the cube compact and self-contained.

Furthermore, the same momentum wheels can be used to implement a reaction-torque based control algorithm for balancing by exploiting the reaction torques on the cube’s body when the wheels are accelerated or decelerated.

Can this work?

The first step in creating the robot, therefore, was to look at the physics to figure out if a jump-up based on momentum wheels was possible. The image below shows some of the math to figure out the Moment of Inertia (MOI) of the wheel and full cube.

This mathematical analysis allowed a quantitative understanding of the system which allowed to inform design choices, such as the trade offs of using three momentum wheels vs. a design with a momentum wheel mounted on each of the six inner faces of the cube.

Another outcome of this analysis was a good understanding of the required velocities of the momentum wheel to allow the cube to jump up, and the torques required to keep the cube in balance. Both factors were critical for the next steps: Determining the required hardware specs.

Specs and hardware design
Given the required velocities and torques determined above, it was clear that the momentum wheel’s motor and gearbox would be a major challenge for creating the robot. Using the mathematical model allowed to systematically tackle this problem by allowing a quantitative analysis of the trade-offs between higher velocities (i.e., more energy for jump-up) and higher torques (i.e., better stability when balancing).

This mathematics-driven hardware design resulted in detailed specs for the robot’s core hardware components (momentum wheels, motors, gears, and batteries) and allowed a CAD design of the entire system.

Part of this step was the design of a special brake to suddenly stop a momentum wheel to transfer its energy to the entire cube and cause it to jump up.

The photo to the left shows an early design of this brake, consisting of a screw mounted on the momentum wheel, a servo motor (shown in black) to move a metal plate (in blue) into the screw’s path (in light brown), and a mounting bracket (in light brown) to transfer the momentum wheel’s energy to the cube structure. The current design uses a combination of hardened metal parts and rubber to reduce peak forces.

2D prototype
To validate the mechanics and electronics of the jump-up and balancing strategy and prove feasibility of the overall concept a one-dimensional version was built:

The results obtained with this 2D version of the cube were published in an IROS 2012 paper.

The final robot
Following successful tests with the 2D version, a full robot was built. The result is Cubli, a small cube-shaped robot, named after the Swiss-German diminutive for “cube”.

As you can see in the video, Cubli can balance robustly.

However, first jump-up tests showed that the stress resulting from a sudden braking of the momentum wheel led to mechanical deformations of the momentum wheels and aluminium frame. This made repeated jump-ups of the whole Cubli impossible without part replacement. It was therefore decided to tweak the structure and breaking mechanism to reduce the mechanical stress caused by the jump-up.

In addition to balancing, my colleagues are now investigating the use of controlled manoeuvres of jumping up, balancing, and falling over to make the Cubli walk across a surface.

Note 1: This post is part of our Swiss Robots Series. If you’d like to submit a robot to this series, or to a series for another country, please get in touch at info[@]robohub.org.

Note 2: This work was done at the Institute for Dynamic Systems and Control, ETH Zurich, Switzerland and was funded in part by the Swiss National Science Foundation (SNSF), grant number 146717.

Note 3: If you have questions, post them below and we’ll post answers.

Some more photos:

Thanks Gajan!

If you liked this article, you may also be interested in:

• Insect-inspired flying robot handles collisions, goes where other robots can’t
• The Distributed Flight Array: Modular robots that self-assemble, coordinate and take flight
• Designed to mingle: Rezero ballbot’s smooth moves can handle a crowd

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

A quadrocopter swoops through the air to serve a glass of water without spilling a drop. Another gets two of its propellers cut off, yet still easily flies across the arena. “It looks like magic!” says ETH Zurich‘s Raffaello D’Andrea, but it took a lot of research, hard work and planning to bring this ‘magic trick’ to TED Global.
This post is part of Robohub’s TED Global 2013 coverage.

D’Andrea’s talk is about “Machine Athleticism” — the ability of machines to perform dynamic feats that fully exploit their physical capabilities. Part  of a session called “Those Flying Things” (which also features the work of drone ecologist Lian Pin Koh, airborne logistics activist Andreas Raptopoulos and sci-fi author Daniel Suarez), D’Andrea delves into what it takes to make a machine athlete:

We use something broadly called Model Based Design: we first build physics-based models that roughly capture the behavior of the machine; these models are then leveraged by a branch of mathematics called Control Theory to synthesize algorithms for controlling them.

He goes on to show a series of pretty amazing quadrotor feats that would be impossible without sophisticated automatic feedback. TED just posted an update on their blog about the demo here, and we will link to the video of the talk as soon as it goes online.

Clearly a lot of work has gone on behind the scenes to get D’Andrea and his team to TED. It’s taken years of research and a whole crew of PhD and Master students to develop the infrastructure and algorithms that make this demonstration possible — and that was just to get to the point where they could do the demos in-house.

D’Andrea and his team usually research and demo their quadrotor tricks at the ETH Flying Machine Arena — a 10x10x10m airspace dedicated to the study of control algorithms, and featuring a high-precision motion capture system, a wireless communication network, and custom software for executing estimation and control. Though the FMA normally lives at its home base at the Swiss Federal Institute of Technology (ETH) in Zurich, it was designed from the outset to be portable, and TED Global is not the first time it has taken to the road:
In 2011 they set up at the FRAC Centre in Orléans, France to create a six metre-high tower – the first architectural installation built by flying machines (see video). Last year the team showed their work at the world’s largest industrial fair, Germany’s Hannover Messe and at Google’s annual flagship conference Google IO in San Francisco.

Still, it takes a lot of time and careful planning to recreate the set up at a new site, and on this occasion they were planning a video shoot as well as a live audience. Says Mark Mueller, one of the team members onsite at TED, “Since April, all of us here have been working full-time, 100% on TED,” referring to all the various administrative hurdles they had to cross, such as planning logistics and agreeing on the space, in addition to rehearsing the demo itself. And then there was the packing to do: “It boils down to fifteen cases, boxes and bags,” says Dario Brescianinini, another member of the crew.

D’Andrea and his team arrived in Edinburgh, Scotland (the host city of TED Global 2013) late Saturday afternoon ahead of most other TED speakers, and started setting up almost right away in a dedicated room where the talk was filmed and where all their demonstrations will take place throughout the week. According to Brescianini and Mueller, it took about three-and-a-half hours to get the first quadrotor hovering. On Sunday they rehearsed the talk — and especially the transitions between each demo — in preparation for the video shoot, which was done in advance of the live talk.

Despite a few hiccups (two quads were damaged during setup and some repairs had to be done on the fly), Monday’s shoot went pretty smoothly. “It took about two-and-a-half hours, but we did one talk without breaks and had most of the things filmed after twenty-five minutes,” says Brescianini. It was a busy set, with fifteen TED crew members, seven people from D’Andrea’s team and one photographer. All in all, it took two fixed cameras (4k HD), one Steadicam, and three GoPros (one in the corner on the floor, one on the ceiling facing downwards, and one flying GoPro attached to a quad) to document D’Andrea’s TED talk. An additional camera shot a “making of” video as well. “It’s pretty busy, and we’re all really relieved after the successful filming.”

Want to learn more?
Check out the pages of ETH Zurich’s Flying Machine Arena team and more Robohub articles on how quadrocopters learn acrobatic maneuvers, cooperate to throw and catch balls, perfect air racing, and throw and catch an inverted pendulum.

Full disclosure: Markus Waibel works with Raff D’Andrea and the Flying Machine Arena team at ETH Zurich’s Institute for Dynamic Systems and Control.

See all the posts in Robohub’s TED Global 2013 coverage.

If you liked this article, you may also be interested in:

• Three new quadrotor videos demonstrate agile control and the power of machine learning
• Live at TED Global 2013: Tweets, photos and video for quadrotor fans
• Video: Throwing and catching an inverted pendulum – with quadrocopters
• Quadrocopters learn from prior experience to improve slalom flying

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

How do you like to go up in a swing,
Up in the air so blue?
Oh, I do think it the pleasantest thing
Ever a child can do!

Child’s Garden of Verses by Robert L. Stephenson

If you’ve ever been a child, chances are that you’ve spent some of your childhood on a swing. And at some point during that time you have probably found out how to move your arms, legs, and body to go higher and higher, without any external help from a friend or parent pushing you – you learned how to pump a swing.

Flavio Fontana, Master student at the ETH Zurich’s Institute for Dynamic Systems and Controls, has taken this concept and used it to control a robot. To make things more challenging, his robot does not just swing, but simultaneously juggles a ball. And, using some neat math to push the idea even further, he and his supervisors Philipp Reist and Raffaello D’Andrea have designed the robot to swing and juggle without cameras – and without ever dropping the ball.

The result of their research is the Swinging Blind Juggler, a robot that juggles a ball from side to side with an actuated paddle attached to a pendulum. The robot is called “blind”, because it has no cameras to “see” the ball. In fact, the Swinging Blind Juggler has no sensors to detect the ball at all.

Here is what Flavio has to say about his video:

In this video, we propose an elegant control strategy for a Swinging Blind Juggler. The key idea is to exploit the dynamic coupling between the motion of the paddle and the pendulum. Children on a swing intuitively use this dynamic coupling to control their amplitude. In a similar way, we control the pendulum with appropriate paddle motions.

So how does this work?

Flavio explains that four conditions must be met for successful juggling:

• First, the paddle used to juggle the ball must have a specific concave shape. This shape will always push the ball back towards the ideal flight path if the ball impact happens to be off-center.
• Second, the paddle has to decelerate as it strikes the ball. This helps to keep the ball bouncing at a consistent height: If the ball flies too high, it will hit the paddle later and hence at a lower speed. Conversely, if the ball flies too low, it will hit the paddle earlier and hence increase the ball’s height on the next bounce.
• Third, the paddle always has to strike the ball at right angles. This ensures that the ball always flies off into the right direction, towards the other side of the swinging motion.
• And fourth, the pendulum must be synchronized to the ball motion. In other words, the pendulum has to reach the peak of its swinging motion at the right time to strike the ball.

To guarantee all of the above, the researchers first derived a mathematical model of the desired motion. Based on these models, they came up with a hardware design that includes a parabolic plate to stabilize the ball’s impact points on the paddle (the first condition), a decelerating paddle motion that stabilizes the ball height (the second condition), and a mechanical mechanism based on four linked bars that automatically tilts the paddle to the right angle for all juggling amplitudes (condition number three).

To meet condition four (to swing back and forth at the right rhythm to hit the ball) the robot can use the same linear motor used to create the decelerating paddle motion for condition two – as long as it still strikes the ball with the right decelerating motion at its peaks. Similar to a child on a playground swing, the robot must raise its body as the swing passes through the lowest point and lower it near the extremes of the motion to pump the swing and swing higher.

The result of the precisely designed hardware setup and control algorithms is a complex motion that’s beautiful to watch – and that blindly juggles a ball from side to side.

To learn more about blind juggling robots, have a look at www.blindjuggler.org.

Note: This post is part of a series that decribes robots developed in Switzerland. If you’d like to contribute a robot to this (or another country’s) series, get in touch at info[@]robohub.org. Like this series? Read more about Swiss Robots!

Two of the most challenging problems tackled with quadrocopters so far are balancing an inverted pendulum and juggling balls. My colleagues at ETH Zurich’s Flying Machine Arena have now combined the two.

As part of his Master thesis Dario Brescianini, student at ETH Zurich’s Institute for Dynamic Systems and Control, has developed algorithms that allow quadrocopters to juggle an inverted pendulum . If you are not sure what that means (or how that is even possible), have a look at his video “Quadrocopter Pole Acrobatics”:

(Don’t miss the shock absorber blowing up in smoke at 1:34!)

The Math

A quadrocopter with a plate for balancing the pole. The cross-shaped cut-outs are used for easy attachment to the vehicle and have no influence on the pendulum’s stability.

To achieve this feat, Dario and his supervisors Markus Hehn and Raffaello D’Andrea started with a 2D mathematical model. The goal of the model was to understand what motion a quadrocopter would need to perform to throw the pendulum. In other words, what is required for the pendulum to lift off from the quadrocopter and become airborne?

This first step allowed to determine (theoretical) feasibility. In addition, it showed the ideal trajectory in terms of positions, speeds, and angles the quadrocopter needed to follow to throw a pendulum. And it offered an insight into the throwing process, including identification of its key design parameters.

Reality Checks
The main goal of the next step was to determine how well the theoretic model described reality: How well does the thrown pendulum’s motion match the mathematical prediction? Does the pendulum really leave the quadrocopter at the pre-computed time? How does the pendulum behave while airborne? How well do assumptions for catching the pendulum (e.g., completely inelastic collisions, completely rigid pendulum, infinite friction between quadrocopter and pendulum when balancing) hold?

This second step involved multiple tests with the physical system, including throwing the pendulum by hand to study its aerodynamic properties and precisely timing the quadrocopters’ and pendulum’s motions during the maneuver.

Analyze, Experiment, Repeat

The shock absorber at the end of the pendulum is a balloon filled with flour and attached to a sliding metal cap with zip ties.

Armed with a good theoretical model and knowledge of its strengths and limitations, the researchers set out on a process of engineering the complete system of balancing, throwing, catching, and re-balancing the pendulum. This involved leveraging the theoretic insights on the problem’s key design parameters to adapt the physical system. For example, they equipped both quadrocopters with a 12cm plate that could hold the pendulum while balancing and developed shock absorbers to add at the pendulum’s tips.

This also involved bringing the insights gained from their initial and many subsequent experiments to bear on their overall system design. For example, a learning algorithm was added to account for model inaccuracies.

Dario writes:

This project was very interesting because it combined various areas of current research and many complex questions had to be answered: How can the pole be launched off the quadrocopter? Where should it be caught and – more importantly – when? What happens at impact?

The biggest challenge to get the system running was the catching part. We tried various catching maneuvers, but none of them worked until we introduced a learning algorithm, which adapts parameters of the catching trajectory to eliminate systematic errors.

The long and iterative process of this third step resulted in the final successful architecture to repeatedly throw and catch the pendulum on the real system, including three key components:

First, a state estimator was used to accurately predict the pendulum’s motion while in flight. Unlike the ball used in the group’s earlier demonstration of quadrocopter juggling, the pendulum’s drag properties depend on its orientation. This means, among other things, that a pendulum in free fall will move sideways if oriented at an angle. Since experiments showed that this effect was quite large for the pendulum used, an estimator including a drag model of the pendulum was developed.
This was important to accurately estimate the pendulum’s catching position.

Another task of the estimator was to determine when the pendulum was in free flight and when it was in contact with a quadrocopter. This was important to switch the quadrocopter’s behavior from hovering to balancing the pendulum.

Second, a fast trajectory generator was needed to quickly move the catching quadrocopter to the estimated catching position.

Third, a learning algorithm was implemented to correct for deviations from the theoretical models for two key events: A first correction term was learnt for the desired catching point of the pendulum. This allowed to capture systematic model errors of the throwing quadrocopter’s trajectory and the pendulum’s flight. A second correction term was learnt for the catching quadrocopter’s position. This allowed to capture systematic model errors of the catching quadrocopter’s rapid movement to the catching position.

The Result
As you can see in the video embedded above, at the end of Dario’s thesis two quadrocopters could successfully throw and catch a pendulum.

Many of the key challenge of this work were caused by the highly dynamic nature of the demonstration. For example, the total time between a throw and a catch is a mere 0.65 seconds, which is a very short time to move to, and come to full rest at, a catching position.

Another key challenge was the demonstration’s high cost of failure: a failed catch typically resulted in the pendulum hitting a rotor blade, with very little chance for the catching quadrocopter to recover. A crashed quadrocopter not only entailed repairs (e.g., changing a propeller), but also meant recalibration of the vehicle to re-determine its operating parameters (e.g., actual center of mass, actual thrust produced by propellors) and restarting the learning algorithms.

Says Markus Hehn:

This was a really fun project to work on. We started off with some back-of-the-envelope calculations, wondering whether it would even be physically possible to throw and catch a pendulum. This told us that achieving this maneuver would really push the dynamic capabilities of the system.

As it turned out, it is probably the most challenging task we’ve had our quadrocopters do. With significantly less than one second to measure the pendulum flight and get the catching vehicle in place, it’s the combination of mathematical models with real-time trajectory generation, optimal control, and learning from previous iterations that allowed us to implement this.

Note: The Flying Machine Arena is an experimental lab space equipped with a motion capture system.
Full disclosure: I work with Dario Brescianini, Markus Hehn, and Raffaello D’Andrea at ETH Zurich’s Institute for Dynamic Systems and Control.

More photos:

Credits: Carolina Flores, ETH Zurich 2012

If you liked this article, you may also be interested in:

• Three new quadrotor videos demonstrate agile control and the power of machine learning
• Live at TED Global 2013: Tweets, photos and video for quadrotor fans
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”
• Quadrocopters learn from prior experience to improve slalom flying

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

So, depending on who you ask, Willow Garage is shutting down, pursuing commercial interests, or changing direction. But things are not as dire as some may think: ROS is fine (the OSRF will support it) and there are many options for Willow Garage to explore. Personally I’m hoping it will be snatched up by Stanford or another academic institution and become a research institute. But given the number of robots they have out (and their price tags!), a commercial path into the future seems even more likely. In any case, I doubt that Willow Garage will disappear just like that.

What caused the changes is that Willow’s founder and funder, Scott Hassan, has decided that it’s time to wean Willow Garage from his private financial support.

I’ve never met Scott, but first and foremost I think we – roboticists and robot enthusiasts around the globe – all owe him a huge THANK YOU! I think it’s fair to say that if it had not been for Scott, we would not have universities around the world sharing a common robot operating system. And, looking at the amazing contributions on the Willow Garage blog, it seems clear that the past five years in robotics would have been a lot less exciting without ROS and Willow Garage’s robots.

Scott brought the model of impact first, monetize later to robotics. Together with Steve Cousins he assembled an amazing team of people and established a network of external collaboration far and beyond anything ever seen in robotics. Did it get us a household robot that can serve us a cup of tea? No. But WOW, did it ever fast-forward us towards that goal.

So thank you Scott, for your vision, incredibly generous financial contributions, and for leading with actions. Willow Garage has already had a huge impact, and I’m sure there’s much more to come. I’m greatly looking forward to the next steps of Suitable Technologies. We all have much to look forward to from Willow Garage’s other already existing spinoffs (convenience links below) – and, who knows, from Willow Garage itself in one form or another.

Change and innovation are usually good friends, at least in my book. In spite of all the negativity out there I, for one, think the future of robotics has never been brighter.

Official info:

• Willow Garage’s official statement: Willow Garage is changing
• The Open Source Robotics Foundation’s statement: ROS @ OSRF

Willow Garage’s spin offs (to date):

• Open Source Robotics Foundation (OSRF)
• Suitable Technologies
• Industrial Perception Inc. (IPI)
• hiDOF Inc.
• Redwood Robotics
• OpenCV
• Open Perception Foundation

The Argo project, which has deployed a network of more than 3’600 robots covering all of the world’s oceans, has returned it’s 1’000’000th measurement.

The yellow underwater robots with their characteristic cylindrical body topped by an antenna have now been successfully used for more than 10 years for precise, long-term measurements of ocean temperatures and salt content. The robots stay submerged for up to 10 days at a time, drifting with ocean currents at a specified depth, or rising and falling in pre-defined paths down to depths of 2000 meters. Upon surfacing, they send their recorded data and current position home via satellite link. The recorded data are then made freely available online in near real-time.

The gigantic network of robot sensors has resulted in a wealth of data, with more than 100 scientific papers published a year. However, it seems scientists on the project agree that Argo’s 10 year data set is not yet long enough to draw reliable conclusions for global change signals such as a global rise in ocean temperatures.

Here are some photos that explain the process:

Photos: Argo project

During the 20 minute presentation, Raffaello D’Andrea revealed some of the key concepts behind his group’s impressive demonstrations of quadrocopters juggling, throwing and catching balls, dancing, and building structures – and illustrated them with live examples with quadrocopters flying on stage.

To watch him hurtle quadrocopters towards his audience, see them juggle balls and balance poles, and to find out what happens when control fails, check out the video:

Other speakers at the Zurich.Minds event included:

• Gerhard Schröder in an interview with host and organizer Rolf Dobelli (in German)
• Tomáš Sedláček on The Economics of Good and Evil
• Kevin Heng on Atmospheric Research of Exoplanets
• Ashkan Nikeghbali on The Fascination of Mathematics
• Lorenz Meier on the Open Hardware Revolution
• Christina Warinner on Evolutionary Diet
• Robert Cialdini on Influence
• Urs Hölzle on Where the Internet Lives, The Hidden Life of Data Centers
• Simone Schürle on Nano Robots for Medicine
• Lars Kolind on Unboss! The new leadership
• John Gray on The Dangers of Faith in Progress, and
• Benjamin Beilmann playing Prokofjew: Sonata for violin solo in D major, Op. 115 on his violin

Full disclosure: I, my colleagues from the Flying Machine Arena, and Raffaello D’Andrea all work in his group at the Institute for Dynamic Systems and Control at ETH Zurich.

Here are some photos of the event:

Photos: Zurich.Minds 2012

Cloud robotics, a shorthand for the idea of leveraging the Internet for robots, offers unprecedented opportunities for robot learning. Apart from using the World Wide Web for faster communication or faster computation, a key opportunity is to allow robots to create and collaboratively update shared knowledge repositories. Hosted in a shared cloud storage infrastructure, such knowledge bases for robots could enable robots to cope with the complexities of human environments and offer a simple, powerful way for life-long robot learning.

As part of RoboEarth my colleagues and I have been working towards this goal by creating something like a “Wikipedia for robots”. By sharing knowledge about objects, maps, and actions in a common framework, RoboEarth allows robots with different hardware and software to exchange – and collectively improve – their knowledge.

The goal of the European-Commission-funded initiative is to develop proof-of-concept demonstrations that show how cloud repositories like RoboEarth’s databases can greatly speed up robot learning and how they may ultimately allow robots to perform well beyond their preprogrammed behaviors.

As one of its first demonstrators, the team has shown that robots with different hardware can use RoboEarth to share maps. Another demonstrator allowed a complex service robot to download hardware-independent object models from the RoboEarth database and use them for perception and manipulation. And a third demonstrator has shown how a service robot can identify, download, and use high-level task descriptions to serve a drink.

Since the start of RoboEarth in late 2009, a number of other groups have begun work on allowing robots to use large, online knowledge repositories. Google announced the formation of a Cloud Robotics team at its Google I/O conference last year. Its main thrust is leveraging existing Google Web services. In a joint talk with Willow Garage Inc., the new Google team also introduced the first pure Java implementation of Willow Garage’s Robot Operating System (ROS). The software, currently in alpha testing, is intended to allow integration of Google’s Android operating system for mobile phones with ROS. Finally, Motoman launched its Web-based remote monitoring service, MyMotoman, in August 2011. It not only enables customers to monitor and update their robots’ status via mobile devices, but also uses historical data to enhance its service offering with predictive maintenance trend data on critical situations. Further initiative are in the works, but, unfortunately, few of them are publicly available (the core RoboEarth software packages are available online).

The above examples mark a wider trend toward harnessing Internet technology for robotics. And this is not surprising: Both the personal computing and mobile phone industries are intimately connected to robotics. Personal computers continue to underpin robot software and act as an important enabler by providing ever-increasing computing power. Mobile phone technology, on the other hand, continues to drive down prices for sensors, including cameras, accelerometers, gyroscopes, GPS sensors, as well as for enabling technologies like wireless networks. In addition, personal computers, mobile phones, and tablets are natural entry points for human-robot interaction for developers as well as B2B clients (think of Telepresence robots) and, increasingly, B2C customers (think of toys like Romotive’s Romo). Both fields have benefitted immensely from Internet connectivity, raising the question if the Internet may, in time, have a similar effect on robotics.

I think there is a likely starting point for such a transformation: The performance of solutions to the field of robotics’ main challenge — robot vision — is tightly coupled with the available resources for computation and data storage. Web services that allow outsourcing computation and data storage to the cloud can offer clear cost benefits for connected robots by replacing expensive local processing capacity and battery power with more powerful cloud services, resulting in cheaper, lighter robots.

With its Goggles mobile phone app, Google has an interesting first version of such an image cloud service. Snap a picture with your smartphone, and you will receive related information from Google product search for objects, translation services for text, or geographic information for landmarks within seconds. Although the current system is aimed at humans and has severe limitations for robotics (e.g., it cannot parse scenes; the returned information is not grounded), it points the way to similarly powerful services for robots.

Vision is not the only robot task that could greatly benefit from web services. Speech-to-text-to-speech services could vastly improve human-robot interaction. Optical character recognition could help robots understand text in all its forms. And any roboticist who has used the turn-by-turn directions offered by a navigation system can appreciate how useful a similar service could be for outdoor or indoor robot navigation.

A transformation like the one the Internet heralded in the personal computing and mobile phone industries, however, is unlikely to result from mere adoption and adaptation of old technology. Just as the killer application for mobile phones was not email, it seems unlikely to me that cloud robotics will be driven by an app store. The app store’s business model, with app sales feeding product sales feeding app sales requires a significant number of users and developers of a common platform for a successful kick-start. This seems difficult to achieve for robotics. However, people are trying and have had some first success.

Another candidate for such a transformation is the use of pooled knowledge to improve robot learning. Kiva Systems, in North Reading, Mass., has pioneered this approach for a homogeneous multirobot system (it is no coincidence that one of Kiva Systems’ co-founders is one of my RoboEarth colleagues) and, as mentioned above, RoboEarth has already made first demonstrations for heterogeneous systems.

As you’d expect, not all knowledge robots can learn is easily exchangeable in a joint knowledge repository. Raw trajectory data or sensor and actuator parameters are often too hardware specific to be exchanged successfully. However, a fair amount of knowledge robots learn can be exchanged: For example, maps, CAD models of objects, and articulation models of doors and drawers have been successfully learnt and shared between different robots. A particularly interesting area for learning are links between shared information, such as where is the fridge (map coordinates), what does it look like (object recognition model) and how do I open it (object articulation model). Another are probabilities, such as given that I see a table, bed, and chair, where would I most likely find a pillow? Robots are well-suited for this type of learning not least because, unlike humans, they are capable of rapid, systematic, and accurate data collection. This capability provides unprecedented opportunities for obtaining consistent and comparable data sets as well as for performing large-scale systematic analysis and data mining.

While the future of Cloud Robotics remains uncertain, my RoboEarth colleagues and I agree that human environments are too nuanced and complicated to be summarized within a limited set of specifications. Any solution to program robots to cope with our daily environments is therefore likely to involve some sort of collective robot learning. Sharing knowledge in a cloud infrastructure like RoboEarth allows robots to perform such learning much faster.

Connecting robots worldwide is a challenging task beyond the scope of any individual initiative. RoboEarth is currently collaborating with related research initiatives in Europe and the United States, as well as with a number of companies through its Industrial Advisory Committee. To join the discussion, feel free to have a look at the RoboEarth website and to get in touch via info@roboearth.org.

Note: A previous version of this article appeared in Robotics Trends.
Credits: Images by Carolina Flores // ETH Zurich // RoboEarth.

My colleagues at the Autonomous Systems Christmas Lab at ETH Zurich have created an awesome Christmas video featuring some of their cool robots:

Mark Hoepfinger writes:

Dear All,

The 2012 Christmas Video from the Autonomous Systems Lab, ETH Zurich. The video illustrates the process of autonomous Christmas present distribution. Starring: Rezero, Firefly, StarlETH and the robotic elevator.

Merry Christmas and a happy New Year!!!
The ASL Team

For more holiday videos, have a look at the ROBOTS Podcast’s Annual Holiday Season Video Special Youtube Playlists. To submit your own, just send us a message at christmas@robotspodcast.com!

Manual programming of robots only gets you so far. And, as you can see in the video, for quadrocopters that’s not very far at all (see the “Without Learning” part starting at 1:30):

On its first try to navigate the obstacle course, the flying robot attempts to navigate based on a pre-computed flight path. The path is derived using a basic mathematical (“first principles”) model. But quadrocopters have complex aerodynamics – the force produced by the propellers changes depending on the vehicle’s velocity and orientation, and thus the actual amount of force produced is quite different from what the simple math describes.

What’s worse, these flying vehicles use soft propellers for safety, which bend differently depending on how much thrust is applied and wear rapidly with use (and even more rapidly when crashing).

Even with continuous feedback on the robot’s position from the motion capture system, manually programming the robots with a control sequence that takes all these imperfections into account is impractical.

My colleague Angela Schoellig and the Flying Machine Arena team here at ETH Zurich have now developed and implemented algorithms that allow their flying robots to race through an obstacle parcours – and learn to improve their performance.

Here is how Angela described the process to me:

The learning algorithm is applied to a quadrocopter that is guided by an underlying trajectory-following controller. The result of the learning is an adapted input trajectory in terms of desired positions. The algorithm has been equipped with several unique features that are particularly important when pushing vehicles to the limits of their dynamic capabilities and when applying the learning algorithm to increasingly complex systems:

1. We designed an input update rule that explicitly takes actuation and sensor limits into account by solving a constrained convex problem.
2. We developed an identification routine that extracts the model data required by the learning algorithm from a numerical simulation of the vehicle dynamics. That is, the algorithm is applicable to systems for which an analytical model is difficult (or impossible) to derive.
3. We combined model data and experimental data, traditional filtering methods and state-of-the- art optimization techniques to achieve an effective and computationally efficient learning strategy that achieves convergence in less than ten iterations.

The result is a robot that learns and improves each time it tries to perform a task.

In this example the robot races through a pylon parcours, calling to mind air races, such as the Red Bull Air Racing Championships or the Reno Air Races – except that there are no human pilots that spent their life learning to fly – here it’s robots doing the learning. And they are efficient, taking less than 10 training sessions to find the optimal steering commands!

Moreover, the learning algorithms are not specific to slalom racing, they can be used to learn other tasks. As Angela points out:

Our goal is to enable autonomous systems – such as the quadrocopter in the video – to ‘learn’ the way humans do: through practice.

The videos below show how learning algorithms can be used for other robotic tasks:

Full disclosure: Angela and the Flying Machine Arena team work in the same lab as I. Also, I’m working on RoboEarth, trying to allow robot learning on a much larger scale.

The Coordination Action Robot Companions for Citizens has submitted its proposal for the European Commission’s FET Flagship Initiative.

The FET Flagship Initiative will provide a colossal EUR 1 Billion funding over ten years, the by far highest amount Europe ever dedicated to purely for academic consortia. 50% of this money will be directly by the European Commission; consortia are expected to raise the other 50% in matching funds from national research agencies, industry or other sources.

Here is how the European Commission describes the initiative:

FET Flagships are science-driven, large-scale, multidisciplinary research initiatives oriented towards a unifying goal, with a transformational impact on science and technology and substantial benefits for European competitiveness and society.

The Robotics Flagship Proposal submitted proposes research focused on developing Robotics Companions. Specifically, the vision calls for:

… an ecology of sentient machines that will help and assist humans in the broadest possible sense to support and sustain our welfare. RCC [Robotics Companions for Citizens] will have soft bodies based on the novel integration of solid articulated structures with flexible properties and display soft behavior based on new levels of perceptual, cognitive and emotive capabilities. RCC will be cognizant and aware of their physical and social world and respond accordingly. RCC will attain these properties because of their grounding in the most advanced sentient machines we know: animals.

The iCub, one of the world's most advanced humanoids.

Robotics Companions is not the only project submitted for evaluation; it is only one of six projects in the race:

• FuturICT – The FuturICT Knowledge Accelerator and Crisis-Relief System: Unleashing the Power of Information for a Sustainable Future
• Graphene – Graphene Science and technology for ICT and beyond
• Guardian Angels – Guardian Angels for a Smarter Planet
• HBP – The Human Brain Project
• ITFoM – IT Future of Medicine: a revolution in healthcare
• RoboCom – Robot Companions for Citizens

By the end of 2012, beginning of 2013 at least two of these projects are expected to be chosen to be launched as full FET Flagship Initiatives in 2013.

Here’s an overview video:

… and some photos that give a flavor of the proposed project’s vision:

 
My colleagues at the ETH Zurich’s Flying Machine Arena have just released a new video. Here is what Robin Ritz, Lead Researcher on the project, has to say about it:
 

This video shows three quadrocopters cooperatively tossing and catching a ball with the aid of an elastic net.
 
To toss the ball, the quadrocopters accelerate rapidly outward to stretch the net tight between them and launch the ball up. Notice in the video that the quadrocopters are then pulled forcefully inward by the tension in the elastic net, and must rapidly stabilize in order to avoid a collision. Once recovered, the quadrotors cooperatively position the net below the ball in order to catch it.
 
Because they are coupled to each other by the net, the quadrocopters experience complex forces that push the vehicles to the limits of their dynamic capabilities. To exploit the full potential of the vehicles under these circumstances requires several novel algorithms, including:
 

1. an optimality-based real-time trajectory generation algorithm for the catching maneuver;
 
2. a time-varying trajectory following control strategy to manage the forces on the individual vehicles that are induced by the net; and
 
3. learning algorithms that compensate for model inaccuracies when aiming the ball.

 
For more information on this and related work, have a look at the Flying Machine Arena’s website or tune in to Prof. Raffaello D’Andrea’s ROBOTS Podcast interview.
 
Some photos of the robots in action:
 

 

 

The Legged Robotics Team at ETH Zurich’s Autonomous Systems Lab is doing research to build fast, versatile, and efficient quadrupedal robots. This movie illustrates some of the work that was done in the past years. Inspired by nature, the team built different types of electrically driven robots that can do various maneuvers from slow and careful climbing to very robust dynamic trotting.

For more information on legged robots, including the Legged Robotics Team’s robots StarlETH, ALoF, and ScarlETH, more information on the use of series elastic actuators for walking robots as well as many more movies, check out the Legged Robotics Team website.

This is the first in a series of Robotics Microlectures produced in cooperation with Switzerland’s NCCR Robotics and Robohub. Watch this space for more!

This is insane – and really cool: This biped robot is made out of paper, folded with origami precision, and driven by an elastic band.

Here’s how it’s made: H/P MPM (Mechanical Paper Model).

An article in today’s edition of the NY Times gives a comprehensive overview of robotic’s influence on assembly lines and warehouses in the manufacturing and distribution industries.

Two quotes to get you thinking:

… the advent of low-cost automation foretells changes on the scale of the revolution in agricultural technology over the last century, when farming employment in the United States fell from 40 percent of the work force to about 2 percent today. The analogy is not only to the industrialization of agriculture but also to the electrification of manufacturing in the past century …

[…]

“We’re on the cusp of completely changing manufacturing and distribution,” said Gary Bradski, a machine-vision scientist who is a founder of Industrial Perception. “I think it’s not as singular an event, but it will ultimately have as big an impact as the Internet.”

Well worth the read, but a pity that it is wishy-washy on the human jobs issue.

See on NY Times for the full article.

ETH Zurich’s summer school on Soft Robotics is over. But as of today, you can have a look at the lecture videos and lecture material online!

A robot that can change colour to either blend in with or stand out from its surroundings has been created by researchers at Harvard University. It was inspired by the camouflage skills found in many sea creatures including octopuses, cuttlefish, and squid.

In a study published in the journal Science the researchers show how simple microfluidic networks allow to change the color, contrast, pattern, apparent shape, luminescence, and surface temperature of soft robots made from polymers and flexible reinforcing sheets. For now, the colored solutions injected into the microfluidic networks were all prepared manually. The color of these networks can be changed simultaneously in the visible and infrared — a capability that biological organisms do not have. These strategies begin to imitate the functions, although not the anatomies, of color-changing animals.

Some photos:

The robot on man-made background. See if you can spot all five robots in panel F.

The robot on natural background:

The micro-fluidic networks developed by the researchers: