Examples of live drone performances (starting top left): Starbright Holiday Drone Show (Intel-Disney), ABB’s 125th Anniversary Celebration (Verity Studios), Meet Your Creator (Saatchi & Saatchi/KMel Robotics) acquired by Qualcomm, Dance with drones on America’s Got Talent (Elevenplay).

Drone shows, such as the ones shown above, are a new and rapidly evolving sector of the entertainment industry.

Preface

At Verity Studios, we’ve published a document outlining this emerging field. For the past 8 years, our group has designed drone systems for shows and live events. With Verity Studios, we are now creating shows and delivering turnkey products for that purpose. Verity Studios is a spinoff company of the Flying Machine Arena, at the Swiss Federal Institute of Technology (ETH Zurich).

Our intent with this paper is to contribute to the emerging ecosystem of drone shows and help accelerate this industry’s safe development. The paper gives a broad overview, including creative, technical, and operational considerations. It also includes a section on best practice, which highlights the main safety concerns related to drone shows.

Further examples of live drone performances (starting top left): Muse Drone Tour, Ars Electronica’s Spaxels, Verity Studios’ drones in Cirque du Soleil’s Paramour on Broadway, Raffaello D’Andrea & Verity Studios’ show at TED 2016.

Abstract

Drone show systems are advanced show automation systems designed for live performances that enable the control of one to potentially thousands of autonomous multirotors or other small flying machines. These novel systems enable the use of drones to extend the traditional palette of light, sound, stage effects, and human performers by choreographing the simultaneous movements of a multitude of flying objects in 3D space. This gives this technology the potential to fundamentally transform the live events experience. Emergent use cases for this technology include animation of flying characters, flying lighting displays, movement of stage props and scenographic elements, and the creation of flying robot actors and synthetic swarms. The key challenges for this new technology are safety and reliability. First solutions for the safe deployment of this technology exist and avant-garde creators are embracing the possibilities offered by autonomous drones.

The full paper can be downloaded here: Drone shows: Creative potential and best practices

Your feedback is welcome at droneshows@veritystudios.com.

Examples of drone costumes (starting top left): Verity Studios, SPARKED; Otto Dieffenbach, Superman Drone; Ctrl.me, Performance Drones; Alan Kwan, Flying Umbrellas.

If you enjoyed this article, you may also want to read:

• New quadrocopter video points to a future for flying machines in entertainment
• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• SPARKED wins at New York City Drone Film Festival
• Flying robots perform 100th show on Broadway, using new localization technology and algorithms
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

National Geographic’s new movie, Robots 3D, proves that robots don’t need to be sci-fi to be compelling.

The team behind RoboThespian, a life-sized humanoid robot designed for human interaction in a public environment, have launched a new YouTube channel: Robot’s World. The robot is very much real and enjoys a bit of profanity in its first episode about the confusion between AI and robots.

Watch here:

Engineered Arts was founded in October 2004 by Will Jackson, to produce mixed media installations for UK science centres and museums, many of which involved simple mechanical figures, animated by standard industrial controllers.

In early in 2005, the Company began work on the Mechanical Theatre for the Eden Project. This involved three figures, with storylines focused on genetic modification. Rather than designing another set of figures for this new commission, Engineered Arts decided to develop a generic programmable figure that would be used for the Mechanical Theatre, and the succession of similar commissions that would hopefully follow. The result was RoboThespian Mark 1 (RT1).

RoboThespian being assembled in the Engineered Arts’ workshop. Source: Engineered Arts

“After 10 years designing and building full sized humanoids, similar questions kept coming up again and again. So, in part, this is a way to address those,” said Will. “The first episode is to try and dispel some myths about AI, for example, a talking robot tells the audience it is stupid — talking does not automatically equivolate intelligence. Also, just because a machine looks like a person does not mean it has any human qualities.”

Each episode takes about 2-3 weeks to create. The team programmes the movement in their virtual simulation software, and films in their workshop.

“I wanted to focus a little more on creative uses for our robots, rather than always leaving the fun part to someone else,” said Will. “We are now making episode 2, which looks at robots as domestic servants. Humanoids are not at all likely to be useful for utility tasks around the home in the foreseeable future.”

And the comical profanity?

“Well … Robot’s ranting personality is somewhat a proxy for myself,” said Will. “The second episode is a lot less profanity-laden — I think I just needed to get that rant out!”

“Most of all, we are not afraid to make fun of ourselves. We do some extremely difficult engineering. The 4th iteration of our dynamic biped, Byrun, will be completed soon — at the same time, we know primarily that using humanoid robots is for entertainment value and communication. We are looking to push the boundaries in that space.”

You can subscribe to Robot’s World on their YouTube channel.

If you liked this post, you may also like reading:

• Robots Podcast #173: RoboThespian, with Will Jackson
• Robots at #ERW2015: From imagination to market 
• RoboThespian co-hosts new robot documentary from CBC – “Roboticize Me”

Copyright: Wanda Tuerlinckx

Amsterdam-based photographer Wanda Tuerlinckx feels inspired by the slow style of classic photography. So influenced, in fact, you can find her observing all her subjects through an old-fashioned lens, an authentic 19^th century ‘camera obscura’ (Latin for “dark chamber”).

Initially, she began by taking portraits of people. Now she concentrates her artistic endeavours photographing robots, cleverly juxtaposing these unlikely high/low tech companions, to create visually stunning black and white images.

“Robots through the eye of a very old camera can be an extreme combination. It gives the robots a type of personality, or soul. Robots are not typically defined as being ‘romantic,’ but looking at these images the robots feel more alive,” said Tuerlinckx.

“I find it fascinating how people in the tech world react; they find the robots beautiful. The lens doesn’t have a hard effect, like with digital cameras, the focus is softer. The camera I use is not a cold camera, it’s a warm camera made out of wood.”

Wanda hopes to document robots in their evolution over the next 10, 20+ years using her portrait method. “With a camera this old you need to have patience — not always easy to find in our fast-paced world.”

Check out a collection of her photographs below. All images copyright Wanda Tuerlinckx.

Interact Centaur Rover. European Space Agency. Copyright: Wanda Tuerlinckx

SERGIO. Second Edition Robot for Generic Indoor Operations. Tech United/ University of Technology Eindhoven. Copyright: Wanda Tuerlinckx

Dual-hemi Continuously Variable Transmission. Project: ROSE, University of Twente, RaM. Copyright: Wanda Tuerlinckx

KASPAR. Therapeutic robot for children with autism. EIZT/Zuyd Hogeschool. Copyright: Wanda Tuerlinckx

Robird Jerrie. Clear Flight Solutions. Copyright: Wanda Tuerlinckx

Entry Descent and Landing (EDL) Demonstrator Module. European Space Agency. Copyright: Wanda Tuerlinckx

Nao. Robots.nu. Copyright: Wanda Tuerlinckx

ReflexLeg. Project: ReflexLeg, University of Twent, RaM. Copyright: Wanda Tuerlinckx

TUlip: Tech United/ University of Technology Eindhoven. Copyright: Wanda Tuerlinckx

Underactuated hand prosthesis with compliant control. Project: MyoPro, University of Twente, RaM. Copyright: Wanda Tuerlinckx

PARO. Seal robot for people with psycho geriatric disorders. EIZT/ Zuyd Hogeschool. Copyright: Wanda Tuerlinckx

TURTLE. Tech United RoboCup Team Limited Edition. Tech United/ University of Technology Eindhoven. Copyright: Wanda Tuerlinckx

Cheetah, Fast Running Energy Efficient Quadruped Project: FREEQ. University of Twente, RaM. Copyright: Wanda Tuerlinckx

AMIGO. Autonomous Mate for IntelliGent Operations. Tech United/University of Technology Eindhoven. Copyright: Wanda Tuerlinckx

Aerialtronics Altura. Clear Flight Solutions. Copyright: Wanda Tuerlinckx

Welby Altidor, Executive Creative Director of Creation, Cirque du Soleil (L) and Benjamin Dupont (R). Source: 2016 NYCDFF

SPARKED: A Live Interaction Between Humans and Quadcopters won the 2016 New York City Drone Film Festival (NYCDFF) Featuring Drones category.

NYCDFF is the world’s first event exclusively dedicated to celebrating the art of drone cinematography. The festival offers an international platform for filmmakers from every corner of the globe to exhibit their work in front of industry professionals and the drone cinema fan community. 

Watch their award winning film here:

Cirque du Soleil, ETH Zurich, and ETH spinoff Verity Studios partnered to develop the short film that features 10 quadcopters in a flying dance performance. The collaboration resulted in a unique, interactive choreography where humans and drones move in sync. Precise computer control allows for a large performance and movement vocabulary of the quadcopters and opens the door to many more applications in the future.

You can also watch a short film about the making of SPARKED here:

If you like this post, you also may also enjoy reading:

• New quadrocopter video points to a future for flying machines in entertainment
• Quadrotors vs CGI: A Q&A with the makers of Sparked

Robots have featured in the movies since as early as 1897, with the first celluloid robot turning up in a French movie called Gugusse and the Automaton. Since then, the history of movie robots has been filled with mechanical maids, cruel cyborgs and angry androids.

Most fictional robots are not realistic. Back in February on the Robotiq blog, we investigated how realistic the robots in Star Wars really are. However, even some of the earliest movie robots are still far beyond what our modern technology can produce. Perhaps we’ll never be able to create robots that live up to our fictional ideal. Neill Blomkamp, director of the recent movie, Chappie, thinks not. Recently he said that his research for the movie brought him to the conclusion that humans are incapable of producing an AI as advanced as science-fiction portrays. It might not be such a bad thing if he’s right.

Even though AI might never catch up, some of the other associated robotic technologies have developed quite significantly; enough that we are now able to create some robotic technologies which were first dreamed up for the silver screen.

Lets take an informed eye to four classic movie robots, to separate fanciful robot fiction from realistic robot fact …

1. Power Loader – Aliens (1986)

Exoskeletons are a perfect example where technology is getting pretty close to the movies. The reason is partly that they don’t require much AI – all of the high-level processing is done by the human brain.

This year, the Center for New American Security released a report saying that the US military is aiming to provide soldiers with the TALOS exoskeleton by 2018. There are also other contenders, such as Lockheed Martin’s HULC.

Some have compared the XOS 2 exoskeleton to Tony Stark’s suit in the recent Iron Man movies. However, given the bulk of modern exoskeletons, they’re really more like the Power Loaders from Aliens. Panasonic also claims that they are developing a power loader, which you can see an early design of in this video:

https://youtu.be/nTRUoanbYjU

Whichever movie robot you choose to compare them to, exoskeletons have well and truly arrived.

2. ED-209 – Robocop (1987)

The ED 209 was a giant, machine-gun wielding robot that malfunctions at the start of Robocop, killing a room full of suited executives — possibly the most catastrophic case of Vorführeffekt ever (a German word meaning “demo effect” that refers to the fact that something will always go wrong in a live demo).

Military robots with guns are making their way onto the battlefield whether we like it or not. The HDT Protector might not have legs like the ED 209, but it does have a machine gun. It’s not automated, but new military robots are capable of automated targeting. The physical design imagined for ED 209 was an imposing two-legged affair, and looked worryingly similar to the real-life US mech-suit Megabot 2.

3. KITT – Knight Rider (1982 TV series and 1991 movie)

If the rumors of a new Knight Rider movie coming out in 2016 were true (it seems they’re not, but it’s hard to tell with internet rumors) it would be perfect timing to correspond with the birth of a real life Knight Rider. Next March, Chinese startup Youxia Motors is set to release its KITT inspired electric car. Modeled on the famous robotic automobile, it comes complete with an Android-based operating system called KITT OS.

The Youxia X might not be a complete KITT in itself, but autonomous cars have really come far in the last few years and several of the technologies featured on-screen are now possible. Self driving and collision avoidance are looking to be standard features on many new cars. The so-called “anamorphic-equalizer” — a flashing line of red lights that acted as the cars “eyes” — were infra-red on TV but nowadays they’d probably be LIDAR (though hopefully not so easily hacked as today’s LIDAR). Also, KITT’s engine was a true hybrid, running on various fuels including hydrogen and electric, like a growing number of modern cars.

4. R2D2 – Star Wars Trilogy (1977 to 1983)

Yes, we have covered Star Wars robots already. Despite this, R2D2 really deserves a mention. This droid has earned its place in the robot hall of fame because it’s actually one of the most realistic robots on this list. Largely, this is because the physical design of the robot is pretty basic. However, many of its more advanced, futuristic technologies have actually become possible since they were first dreamed up in the 70s.

R2D2’s impressive list of equipment included a fusion welder, buzz saw and all-terrain treads; all well-proven technologies. In fact, modern technology has surpassed some of the droid’s more futuristic capabilities. Instead of the universal connectivity it got through a plug-in “scomp link”, we’d now probably use bluetooth. These days, its “power recharge coupler” could be upgraded to wireless charging. Its flight capabilities were similar to that of modern autopilots.

But what about the famous holographic projector? This August we all learned how to make holographic projectors out of our smartphones, using the same technique that full sized holographic projectors have been using for a few years.

There is even a real-life R2D2 Builders Club, whose members often meet to show off their hundreds of R2D2 replicas. Most of these replicas probably don’t have the droid’s more advanced features, but it would only take one enterprising robot-builder to turn R2D2 into a reality.

***
As more technologies catch up with fiction, we might continue to see some of the classic movie robots become realistic possibilities. Right now, it looks like AI will always be the limiting factor from making truly realistic replicas, but you never know.

If you liked this article, you may also be interested in:

• The real soft robots that inspired Baymax, with Chris Atkeson
• The robots in Ex Machina: VFX Q&A with Andrew Whitehurst
• Why do people love robots? Find out in this Botscar-winning Robot Film Festival short
• From sideshow to main event, drones are making their mark in entertainment

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

The video features two drones zig-zagging in the forest, bumping into trees and rolling around leafy branches. Each drone is equipped with 6 meters (20 feet) of neon wire, two dozen of RGB LEDs and fluorescent material. The drones’ trajectories are remote-controlled by operators, while the lights execute a pre-programmed sequence and use data from onboard accelerometers to react in sync to contacts and collisions. The drones’ cages are freely rotating, which adds to the magic of the unique choreographies that are possible with this system.

The movie was shot in one summer night, and required several takes in order for the pilots and camera operators to synchronize their motions. The movie reminds us of fireflies at night, or perhaps of birds-of-paradise courtships that involve dance, color and amazing choreographies.

One of the goals of the video was to demonstrate how drones can be used for art and entertainment; the GimBall platform allows drones to safely and smoothly interacti with objects or humans, which really contributes to bringing them alive.

Atelier D. Schlaepfer’s undertakings are at the crossroads of art and design. They take place in variable settings such as public buildings, private homes, and urban or natural settings. Atelier D. Schlaepfer regularly collaborates with architects and construction specialists to offer a scenography of space in the form of lighting, coloring, or a formal concept.

The drones are manufactured by Flyability SA, a swiss company specializing in collision-resistant drones for industrial inspection. Flyability’s patent-pending technology is based on a freely rotating spherical protective cage, which allows drones to remain perfectly stable after contacts. They can thus access hard-to-reach places while being remotely operated. Equipped with a full HD camera, they can efficiently carry out close-up visual inspection of various structures or confined places, such as bridges, factories, boiler rooms, tanks, etc.

The two drones are each equipped with 6 meters (20 feet) of neon wire and 24 RGB LEDs.

If you liked this article, you may also be interested in:

• Flyability wins $1M Drones for Good competition
• Insect-inspired flying robot handles collisions, goes where other robots can’t
• New quadrocopter video points to a future for flying machines in entertainment

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Beautiful short documentary by The Atlantic about amputee drummer Jason Barnes and Georgia Tech professor of music technology Gil Weinberg that explores “how robotics and artificial intelligence are fundamentally changing the way that music is created and performed.” 

Via Daniel Lombroso. Thank you!

Callan McAuliffe in Robot Overlords

In the low-budget British sci-fi adventure, Robot Overlords, alien automatons rule the streets and ordinary citizens are locked under curfew in their homes. Just by stepping outside, you risk being vaporised by a hulking Sentry, or picked off by a lethal Sniper.

Amid the ruins of civilisation, Sean Flynn (Callan McAuliffe) leads his group of young friends on a quest to join the resistance forces that are standing against the robot invaders. Hot on their heels, however is their old teacher – now treacherous robot collaborator – Robin Smythe (Ben Kingsley) and his captive Kate (Gillian Anderson). Who will prevail? Man or machine?

Directed by Jon Wright, Robot Overlords features some 265 visual effects shots delivered by Soho-based Nvizible, supervised by the company’s founder and co-owner, Paddy Eason. The film’s complex tracking requirements were fulfilled by Peanut FX, with additional compositing support being provided by Boundary VFX, under the supervision of Nick Lambert.

Watch a wide selection of Nvizible’s VFX shots in the official video for Robots Never Lie, written by Matt Zo for the Robot Overlords end credits

Origins of the Overlords

The original story concept for Robot Overlords came to director and co-writer, Jon Wright, in a dream. “I dreamed that I was playing cowboys and Indians with a little boy,” Wright recalled. “A huge two-storey robot came marching up the street and swung its laser cannon arm towards him, and a voice boomed out, ‘Citizen, drop your weapon immediately!’ I assumed I was just recycling a movie that I’d already seen, but eventually, I came to the conclusion that maybe it was an original idea.”

After expanding the dream concept into a two-page treatment, Wright began developing a script with co-writer Mark Stay. “Jon and I are both big fans of those 1980s Amblin movies like E.T.: The Extra-Terrestrial and The Goonies,” said Stay. “But those films were always set in American suburbia – we wondered why nobody had ever set a film like that in the UK.”

“It came together very quickly, especially for a British film,” Stay continued. “Our elevator pitch was, ‘It’s like The Goonies, but with robots blowing stuff up!’ We took Jon’s treatment to Piers Tempest – one of the producers from Jon’s previous film, Grabbers. The BFI put us under their wing, and before we knew it we had some development money.”

VFX supervisor Paddy Eason was also consulted at this early stage. “I knew Jon because we did the effects for his first two films – Tormented, and an Irish monster movie called Grabbers.” Eason commented. “The first thing I received was the treatment, and the next throw of the dice for me was getting invited to be part of a group that critiqued the script – Jon calls it the ‘Tiny Think Tank’.”

The Tiny Think Tank

The Tiny Think Tank is an initiative conceived by Wright as a means by which a group of filmmakers can gather together periodically, in order to review scripts in development. “It’s a gang of writers, actors, directors, visual effects people,” Wright explained. “There aren’t any producers or financiers involved – it’s purely creatives. We come together, read a script completely cold, then have a round-table discussion. It was inspired by Pixar. They have their team of creatives – who are also shareholders – who come together and critique each other’s films. The Tiny Think Tank is our low-budget UK version of that, and I think it’s one of the reasons Robot Overlords got made. Our script accelerated in terms of quality much more quickly than it would have done had we been writing it in isolation.”

Nvizible’s VFX breakdown for a key sequence in “Robot Overlords”

As part of the Tiny Think Tank, Paddy Eason was able to offer a level of opinion and criticism not commonly afforded people in the visual effects profession. “I had some suggestions from my own visual effects point of view,” Eason remarked. “But I also put some story ideas out to Jon and Mark. As a visual effects person, that’s not normally part of your remit. In fact, on many shows, to put forward story ideas would be viewed as a breach of etiquette. So it was delightful to be invited to give that kind of feedback.”

Nvizible’s association with Robot Overlords developed further when the company became a production partner on the project. Nevertheless, they were still required to bid for the work. “Robot Overlords was a regular production with a bond company and other financiers, so it had to be seen not only that our bid was value for money, but also that our costs weren’t unrealistically low,” Eason pointed out. “Also that, if the unthinkable happened and we dropped the ball, there would be somewhere else to go to get the work done. Obviously it didn’t come to that!”

How to Design a Robot

Key to the success of Robot Overlords was the convincing creation of the alien machines that have descended from outer space to occupy planet Earth. Design concepts for the mechanical marauders developed naturally alongside the story.

“The Robot Empire is stretched across the galaxy,” explained Stay. “If you compare it to the Roman Empire, the Earth is like Caledonia – it’s the last place any of them want to be. Resources are stretched thin. So the Sentries, for example, are quite sorry figures. They walk with a kind of stoop, and they’ve got chips and dents and dinks in their armour, because they’ve been here for three years and survived a war.”

According to the storyline, the Robot Empire studies the inhabitants of each planet they plan to subjugate, and designs an automated invasion force specifically to tap into the psychological profiles of that planet’s inhabitants. This bespoke engineering approach is evident in the design of the Sentries: two-storey, bipedal robots with small heads and enormous, hulking arms.

According to co-writer Mark Stay, the robot Sentries are designed to tap into our instinctive fear of “people with tiny brains and big muscles”.

“In studying us, the robots have discovered that we’re frightened of people with tiny brains and big muscles,” explained Stay. “That informed the design of the Sentry. Our other robots – like the Snipers and Air Drones – are very functional, each with a specific purpose. ”

A different psychological insight led to the design of the Mediator – a kind of robot diplomat that acts as an interface between human beings and their automated oppressors. “The Mediator is made to resemble a child, because they’ve observed that, on the whole, adults don’t behave aggressively towards children,” Jon Wright remarked. “But it’s been designed by an alien species who aren’t familiar with the minutiae of how we react, how our culture is organised. So they’ve got it a bit wrong. They’ve attempted to make something likeable, but they’ve actually made something that’s extremely creepy.”

Early conceptual designs for the robots were developed by storyboard and concept artist, Gabriel Schucan. “Jon and I sat with Gabriel,” Stay recalled. “We scribbled ideas, and he would develop them and send drawings back to me and Jon. That was still very early on in the writing of the script.”

“Robot Compendium” concept document – Sentry illustration by Jack Dudman

Also at this early stage, the team created the “Robot Compendium”, a manual containing a piece of key art for each robot, drawn by Jack Dudman, along with a description of the machine’s capabilities. “The Robot Compendium, helped us to sell in the script to investors,” remarked Stay. “So it was a really important document for us.”

During the story development process, visual effects constraints influenced what might be achievable, both practically and financially. “While we were still talking about the script, we started doing visual effects breakdowns in a very loose, back-of-an-envelope way,” said Eason. “Some scenes would turn out to be looking very expensive, and that was fed back into the writing process. For example, there was a scene in the woods with our heroes being chased by robots called Octobots. That was quite heavy-duty in terms of visual effects, and wasn’t particularly important in the story. So it was elbowed fairly early on.”

“Robot Overlords” concept art by Paul Catling.

The Overlords Become Real

Once Robot Overlords was greenlit, the initial robot concepts were handed over to designer and illustrator Paul Catling, known for his work on films including Guardians of the Galaxy, Captain America: The First Avenger and the Harry Potter series.

“Paul is incredibly productive,” Eason remarked. “He did lots and lots of pen-and-ink designs, just exploring how the robots might look. Early on, some of them were much more organic-looking, and skeletal, and frightening in a slightly Satanic kind of way. Ultimately, we decided the robots should be slightly crude-looking. We wanted to avoid comparisons with Transformers – Jon wanted our robots to look much more industrial. We also wanted them to be slightly reminiscent of WWII Nazi technology.

Paul Catling’s concept designs for the Sentry.

Although the design of the Sentry was inspired directly by Wright’s inspirational dream, it nevertheless went through a lengthy development process. “One of the original tests we did for the Sentry was a lot more like Robby the Robot,” Wright revealed. “Then he developed into more of an Iron Giant character – we just pushed that design to make it slightly more peculiar. He’s slightly lop-sided, and made up of different geometric shapes. That was partly driven by the plot device of having them fold up into cubes, which was our way of showing that they’ve been deactivated.”

The Mediator robot was played by Craig Garner, whose appearance was enhanced in post-production to give the actor’s skin an unsettling plastic sheen. For some scenes, frames were removed to create staccato movements.

The Mediator, in contrast, was conceived from the beginning as a machine that looked like a human – but not too much like a human. “Right from the beginning, we wanted deliberately to put the Mediator in the uncanny valley, rather than try to avoid it,” Wright noted. “We gave the actor, Craig Garner, perfect false teeth, trimmed his hairline to a perfectly straight line, gelled his hair rock-solid, and tried to remove all of his imperfections and blemishes with make-up. Then that was enhanced in post-production with clean-up, skin glows, making his eyes uniform. Sometimes we’d take out every other frame to give him a kind of clockwork, staccato movement.”

“The bulk of the Mediator work was done in the grade by the colourist, Gareth Spensley, on his Baselight,” Eason elaborated. “However, the software we use at Nvizible for skin work is a suite of NUKE tools that we use as part of our ‘Nhance’ service. These allow us to selectively filter out features of a certain size – for example, wrinkles or spots – while retaining everything else, like image grain, shadows and regular skin texture.”

Robots at Nvizible

With the exception of the Mediator, Nvizible further developed the robot designs in their CG workspace. This process was facilitated by initial 3D work done by Paul Catling. “As Paul progresses his designs, he starts working in 3D,” explained Eason. “He’ll do a render, then go on top of it in Photoshop and paint lighting effects and so on to turn it into an illustration. But the underlying geometry is there. He even rigs it a little so he can move the arms and legs around to come up with certain key poses. So he ended up with some pretty well-realised CG models, which he then gave to us. We had to rebuild them, but they gave us a head-start.”

Early 3D concept for “Robot Overlords”

Nvizible’s robot models were built, rigged and animated primarily using Maya, with additional sculpting work done using ZBrush. “The Sentry ended up being a very complicated rig, with hundreds of moving parts,” Eason noted. “But it does play the biggest part in the film.”

Texturing was applied in MARI, using photographic reference chosen to support the story point that the robots have been operating in a hostile environment for many years, and are therefore both weathered and battle-worn. “There was no decoration on the robots – no painted symbols or information; it was all slabby, raw, metal, moving parts,” Eason noted. “We took photographs of similar unadorned, mechanical things, like old steam engines, big industrial lifts, big heavy things made out of cast iron. Part of the fun was making them nice and weathered – chipped and broken – which also helped to give them scale.”

Ncam and the Air Drone

Many key scenes featuring the robots were shot using a real-time camera-tracking system provided by Nvizible’s sister company, Ncam. The Ncam system allows a camera operator to view pre-visualised CG assets through the viewfinder, adjusted to conform to the parameters of the lens and superimposed in the correct position over the live-action.

“Ncam works without any tracking markers. It lets you see a previs version of your visual effects assets through the camera, or at video village,” Eason explained. “Ncam combines the live video feed from the camera with a render from a workstation, all in real-time, showing your VFX asset in situ. It could be anything from a set extension to a creature.”

Ncam was used during the film’s opening sequence, in which a crazed man runs down a suburban street shouting defiance at his robot oppressors. An Air Drone flies down, warning the man to return to his home; when the man ignores the robot, it vaporises him.

“My view with visual effects is to say, ‘Go ahead and do everything practically that you can, and we’ll deal with it in post.’ Rain? No problem!” – Paddy Eason, Nvizible.

The sequence was shot on location in Bangor, on the outskirts of Belfast, Northern Ireland, using primarily hand-held cameras. Following Wright and Schucan’s storyboards, Nvizible created previs for the sequence, using survey data of the location gathered by members of their Belfast office.

During the night shoot, interactive lighting was coordinated by Fraser Taggart, director of photography, to simulate the glow from the Air Drone’s blue hover-jets, while practical rain simulated a torrential downpour. “Rain gives you a huge amount of visual texture, so it’s a lovely thing to have,” observed Eason. “Generally speaking, my view with visual effects is to say, ‘Go ahead and do everything practically that you can, and we’ll deal with it in post.’ Hand-held cameras? No problem! Rain? No problem! We’ll add CG rain, and we’ll make sure it’s composited behind your rain. But there were a couple of shots where the camera was looking up into the sky, where it would have been too much of a nightmare to have rain falling into the lens, so I did ask them to shoot dry versions to give us the option.”

Practical pyrotechnics were used for the moment when the man is blown up by the robot, somewhat in the manner of a human firework. “We shot the plate with the actor, a clean plate with no actor, and a third plate with the pyro going off,” Eason revealed. “We used a similar camera move each time, but there was no motion control – which is fine as long as everyone’s on the same page. The practical pyro gave us real interactive lighting, and a good clue of how the rain is lit up. We enhanced it with additionally-shot bluescreen pyro elements afterwards.”

For the shots featuring the Air Drone, Ncam was used extensively to aid the shooting of the live-action plates. “It was exciting to see the reaction of the camera operator,” Eason recalled. “Like a lot of people, he was a bit sceptical about Ncam – sometimes people view it as a bit of an encumbrance. But when he realised he was actually able to see the hovering Drone through his viewfinder, and walk around it and choose angles, he was really excited and impressed.”

In one close-up shot, the camera executes a 180° move around the muzzle of the Air Drone’s gun, with the camera racking focus on the barrel. “I don’t think you’d have shot that plate without Ncam,” Eason asserted.

While the live-action plates were used for most scenes featuring the flying robot, Nvizible created some backgrounds using 360° HDRI panoramas shot on location by VFX assistant Erran Lake, using a DSLR camera. “Sometimes the plate done with the movie camera might be nice, but technically very difficult,” Eason explained. “So we would create our own version of that using my stitched stills, and put the rain in ourselves.”

Ncam was used for a scene in which a giant robot Sentry pursues Connor (Milo Parker) down a suburban street.

As director, Jon Wright appreciated the freedom afforded by Ncam, notably during a sequence in which a gigantic Sentry chases a boy down a suburban street. “You’re able to shoot something as if it was actually there, so it unlocks your shooting style,” he commented. “We were able to imbue those shots with a hand-held quality, and be quite cavalier about the way we shot. Normally when you shoot a visual effect like that, and there’s nothing there, you’re worried about not framing it up properly.”

“It was quite difficult to set up, positioning the robot and getting good tracks on the camera,” Wright continued, “but it meant we were able to do a running hand-held shot backwards with the boy, framing up the robot above him. The Sentries are massive, so they tend to go out of frame. If there’s nothing there, it makes the operator nervous and they tend to err on the side of caution and give you a slightly peculiar, lop-sided frame. If they can see the thing, they frame it much more naturally, and you get a much more interesting shot.”

NCam workflow – final composite (before grading)

“I imagine in years to come this will be fairly standard practice,” Wright concluded. “The days of tennis balls on sticks are numbered.”

Watch a video of the live NCam feed as seen by the camera operator:

By the time post-production began, Robot Overlords already existed in rough-cut form, since editor Matt Platts-Mills had been present on location throughout production, cutting each day’s rushes together into a rough assembly. “It was very helpful to have Matt with us, cutting as we went,” Eason observed. “By the time the shoot was done, we were in a good position to hit the ground running on certain key visual effects scenes.”

Since much of the film had been shot hand-held using anamorphic lenses, one of the first challenges for the visual effects team was tracking. Peanut FX handled all the film’s tracking requirements, delivering 3D layouts for roughly 115 shots, mostly with 3D camera and/or object tracks for set extensions and robot integration.

“We mainly used 3DEqualizer,” commented Amelie Guyot, matchmove supervisor at Peanut. “We find it great for solving anamorphic camera tracks when lens distortion grids and camera info are available. We were provided with a lidar scan model for scenes involving a castle, which we used to perfectly match the plates – 3D scans are always very helpful, especially with anamorphic shots. We also used Autodesk’s Matchmover to calibrate sets; it’s still one of our favourite tools for image-based 3D reconstruction.”

“Tracking was a concern early on,” Eason admitted, “but the guys at Peanut absolutely rocked it and did a brilliant job.”

Tracking for “Robot Overlords” was handled by Peanut FX, who worked meticulously to deal with director Wright’s extensive use of hand-held cameras, and some challenging sequences in which the robots interact directly with humans and their technology.

During post-production, Jon Wright made himself available to the Nvizible team on a regular basis. “I made a point of remaining on the film throughout the time that Nvizible were working on the visual effects, so I could go in and out of their offices at my leisure,” the director stated. “It saves a lot of work if you’re seeing updated shots regularly – you tend to have less waste. I don’t know how to run Nuke or anything, but I think I’m quite geeky, and have a good understanding of how effects work. I can tell if something’s going to be quick to do, or time-consuming. I think maybe some directors don’t have that.”

With regard to the animation of the robots, Wright paid particular attention to the way the movement of the metal invaders expressed their character – or rather, their lack of it. “There’s a tendency to anthropomorphise robots,” Wright observed. “Our robots are quite implacable, so it was important to strip out emotion.”

Even given this direction, nevertheless Wright found that individual animators inevitably had their own take on how the robots should move. “After a bit I could guess quite accurately who’d animated what, because they bring their own personality to it,” he commented. “It got to a point where I would say, ‘I know this shot would be great for this animator,’ on the basis of what they’d done before.”

Lighting the CG robots brought its own set of challenges, due to the difficulty of making metallic objects look both highly reflective and convincingly three-dimensional. “It’s a bit of a double-edged sword,” Eason remarked. “If you’ve got good HDRIs for everything, you get good reflections, but because most of what you see with metal is based on reflections, it’s hard to make stuff that’s physically juicy, with a nice keylight and a nice fill light. You’re always fighting that chrome robot thing, where it all looks a bit floaty and unlit.”

Skyship concept art by Paul Catling.

Riding the Skyship

One of the most difficult visual effects sequences occurs at the film’s climax, when Sean finds himself riding through the skies on the front of the enormous robot Skyship.

“They were really hard shots, because they were extremely ambitious,” Wright admitted. “I’d imagine that even the likes of ILM working for J.J. Abrams would have found those shots tough to get right.”

Once again, NCam was used to combine a previs version of the Skyship with the live-action as it was being shot on a small studio set. “The set had a slight bounce, which made things interesting,” Eason recalled. “We ended up replacing all the practical set with CG, although it was useful to have it there for reference, shadow catching and so on.”

The film’s ambitious climactic scenes, during which Sean (Callan McAuliffe) rides on the exterior of the massive Skyship, proved particularly challenging.

In post-production, the low-resolution Skyship model was replaced by its high-resolution counterpart. “The main challenges with the Skyship came in look-dev, working out how to make a big shiny collection of metal blocks look big and scary and cool,” Eason commented. “In reality, large steel things tend to look like concrete, so we had to cheat the physics of the renders quite a bit. We had a lot of fun with the Skyship engines – big jets of dirty orange and blue fire. Our FX lead, Wayde Duncan Smith, did a great job of simulating all that stuff in Houdini.”

Because the sequence featured multiple camera angles, with moves that were very specific to the animation of the various characters and craft, it was deemed unfeasible to shoot live-action background plates. Instead, generic aerial plates of landscapes were shot using an array of DSLR cameras.

“We flew the route in a chopper several times, shooting 160° panoramas using three Canon 5Ds, shooting stills at 3fps,” Eason revealed. “We fed these image sequences through a pipeline of undistortion, stitching, stabilising, optical flow and re-projection, to make beautiful background plates of moving land for all the flying Skyship shots. It was hard work, but our Nuke artist, Antoine Jannic, absolutely nailed it.”

Watch a video of a completed shot from the Skyship sequence:

Enter the Spitfire

Also featured during the film’s finale is that most British of icons – a Supermarine Spitfire. The old-school technology of the WWII fighter plane proves pivotal in the battle against the robots, whose jamming devices have grounded more sophisticated jet aircraft.

The production shot live-action of the historic plane on the ground using a mock-up provided by Gateguards UK. An authentic Spitfire was also made available for one day of the shoot by The Aircraft Restoration Company, Duxford. In addition, Nvizible built a CG Spitfire which they used for additional aerial shots.

Jon Wright (right) directs the action around the museum-piece Supermarine Spitfire, on location in the Isle of Man.

“We used the mock-up Spitfire in a remote, disused quarry location on the Isle of Man,” explained Eason. “Then we also had the real Spitfire fly out to us for the day. We positioned it on the airfield and dressed it similar to how the mock-up had been in the quarry. It was a huge privilege to have a real museum-piece there, and a dream for us in terms of creating our CG Spitfire for the flying shots, because we were able to get perfect reference, taking thousands of photographs, right down to every rivet.”

Eason was doubly delighted when he found himself booked in for a flight in a two-seat Spitfire. “The producer, Piers Tempest, knew I was a big fan of Spitfires, and he made a casual comment: ‘We’ll get you a flight in it.’ I didn’t believe it would ever happen, but a couple of months later it was arranged for me to go to the Imperial War Museum at Duxford. I got to fly around, and control the Spitfire, and everything. Ostensibly it was to record audio inside the plane – I was wired for sound – but also it was just a marvellous day out.”

Robot Reflections

Looking back at his experience of working on Robot Overlords, Mark Stay noted the benefits brought by the collaborative process: “I’ve been involved in the whole procedure in a way that many other writers aren’t usually. I was writing the novelisation at the same time, so I would email Paddy to ask questions like, ‘What colour are the engines?’ He would invite me over to Nvizible so I could sit in on VFX test screenings, and I would be there furiously scribbling all this stuff down for the novel. I’ve been really blessed with the access that I’ve had.”

“It was a delightful experience, because of our creative involvement from beginning to end,” Paddy Eason added. “I love the fact that it’s quite an unusual British film – very charming and quirky, starting at a very small personal scale and ending up quite epic and extraordinary.”

The film’s ambition is something that Jon Wright is proud of. However, he remains sanguine about the challenges involved in bringing such a film to the screen.

“It’s a bit of a shame that we aren’t really making these kinds of movies in Britain and Ireland, and that British cinemagoers almost prefer to go and see American films over British films. And we’ve got all our best filmmakers going across the water to do remakes,” Wright commented. “But the truth is, when you make a movie like this, you go up against massive Hollywood blockbusters, and it’s difficult to deliver in the way that they deliver. So we tried to focus on character, and give the film a different tone to a typical Hollywood movie at the moment – not dark and moody, but kind of light-hearted and optimistic. Our kids swear a lot and are playful, rather than being the earnest, soul-searching types you get in typical American YA movies.”

Echoing Wright’s thoughts, Eason recalled, “I remember one meeting we had early on, where we pitched the project to quite a well-known film producer. This producer said, ‘If you pull it off, I’ll eat my hat.’ So now we’re going to get a nice hat-shaped cake made for him!”

Having enjoyed limited theatrical release across the UK through Signature, Robot Overlords may also be gaining momentum as a TV series. “We have strong interest from two big broadcasters, in the US and the UK,” Wright revealed. “In 2015, television is probably the way to go with this sort of project. We could really explore into the human politics of the situation – life under enemy occupation. Obviously, the occupation is going on across the entire planet, so there’s a million different directions you could expand the story into. And ten hours of television is probably more exciting than another 90-minute film. It could be very good – watch this space.”

Robot Overlords is now available on DVD. 

• Robot Overlords on Facebook
• Nvizible
• Peanut FX
• Boundary VFX
• Jon Wright at IMDb
• Paddy Eason at IMDb
• Mark Stay – Unusually Tall Stories
• Amelie Guyot at IMDb
• The Foundry – NUKE, MARI
• Autodesk – Maya, Mudbox, Matchmover
• Pixologic – ZBrush
• Side Effects – Houdini

Special thanks to Marek Steven, Alex Coxon, Piers Tempest and Phil Guest. “Robot Overlords” photographs and video copyright © 2015 and courtesy of Tempo Productions and Nvizible.

• The robots in Ex Machina: VFX Q&A with Andrew Whitehurst
• True AI, or clever simulation? Transcendence movie has Johnny Depp crossing the Singularity
• 12 favorite robot movies – new and old
• The real soft robots that inspired Baymax, with Chris Atkeson

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

The latest addition to this recent crop of AI movies – which includes Caradog James’s The Machine and Spike Jonze’s Her – is Ex Machina. Written and directed by Alex Garland, the film introduces young computer coder Caleb (Domnhall Gleeson) into an experiment designed to establish whether sexy and cerebral android Ava (Alicia Vikander) is truly self-aware.

Vikander’s on-set performance as Ava was meticulously preserved during the post-production process. While much of her body was replaced by a digital counterpart, Vikander’s face and hands were retained throughout. The result is a seamless blend of live-action and CG animation that remains convincing in a film characterised by long takes and intricate dialogue.

The production visual effects supervisor for Ex Machina was Double Negative’s Andrew Whitehurst, who was assigned to the project for around 16 months. Under Whitehurst’s supervision, Double Negative delivered over 300 robot shots, with an additional 250 VFX shots being provided by Milk VFX, Utopia and Web FX.

Watch the trailer for Ex Machina:

https://youtu.be/XYGzRB4Pnq8

How did you get involved with Ex Machina?

Double Negative were approached by DNA films and the writer/director, Alex Garland. We quickly worked out that we saw things very similarly, and that we would be able to work together.

How did you divide up the visual effects work among the various vendors?

The work on Ex Machina was divided amongst four facilities. Double Negative created the android, Ava. Milk VFX created Ava’s brain, “Ava vision” and did a number of monitor inserts and clean-up effects. Utopia created a CG mobile phone for Caleb and added extra CG buildings to the Norwegian location plates. Web FX provided a number of clean-up and monitor composites.

This is Alex Garland’s feature debut as director. How was he to work with?

An absolute pleasure. Alex is visually very driven, and has a great appreciation of art, comics, film, and games – all of which he brings to bear on his work. He’s a very collaborative director, and we had a great many design meetings in which the two of us did quick sketches with a pad of copier paper and a fistful of Sharpies, to work out design issues. It’s a very fast way of working; it also allowed us make better decisions, because we could rule many things out before working up more finished designs.

Did you previs the film?

No, we didn’t do any previs. It was always very character-led, which meant that the blocking was something that only the actors – working with Alex and the director of photography, Rob Hardy – could do. We had to make sure we could work with anything they decided to shoot.

Alicia Vikander stars as Ava in Alex Garland’s “Ex Machina”, with visual effects by Double Negative, Milk VFX, Utopia and Web FX.

How did you go about designing and developing the character of Ava?

When Alex came to Double Negative, he’d already had some concept paintings done by Jock, who’s an amazing artist. As the film had to come in at a certain budget, we needed to design a character that could be realised for that amount of money.

Once we knew our practical constraints we began to work up painted concepts. We included O-rings and connection studs in the design. They gave us something to track and provided clean edges to roto around, so we could layer in CG behind the in-camera costume. We worked very hard to give Ava a plausible mechanical quality that also had a lot of feminine beauty: the audience had to believe that a human could fall in love with her.

What other sources did you draw visual inspiration from?

The only rule I set for anyone who was working on Ava’s design – including myself – was that we weren’t allowed to look at other robots, especially androids. Instead, I put together a collection of images of Formula One car suspensions, high-end road bicycles, lightweight aircraft airframes, human anatomy and procedural sculpture. These gave us inspiration for both form and materials.

I think that – possibly subconsciously – the design of Ava owes a lot to the French comic artist, Moebius. Both Alex and I are big fans of his work, and even though we didn’t explicitly refer to his images during production, when I look at Ava now I see a lot of Moebius’s influence there.

Ava concept art by Karl Simon Gustafsson

How did you translate the concept art into three dimensions?

We soon reached a point where it was clear that we’d pushed painted concepts as far as they could go, and that we had to start modelling in 3D. We began by building an arm, starting with simplifying human bone shapes and changing pivot points to more mechanical forms. Then we started adding musculature and cabling. Richard Durant and Alexis Lemonis did the majority of the modelling of Ava.

We worked hard to make sure that we designed something which could work practically, which looked like it had the right weight distribution, and which still had “form follows function” beauty. We continually removed pieces that seemed superfluous. The great industrial designer, Dieter Rams, has a motto: “Less, but better.” We constantly kept that in mind. In fact, when the design was 3D-printed for the laboratory set, it did all fit together beautifully. That was a proud moment!

Was it a complex model to rig for animation?

We started rigging Ava as soon as we began modelling, and the rig developed throughout the show – it was still being tweaked right up to the end. Mark Ardington and Fernanda Moreno rigged Ava, with Mark carefully building versions of the rig that could be used by the body track artists to duplicate Alicia’s movement. We also had a higher res version of the rig that could be swapped out at render time; that one had cables, muscles bulging, and all the secondary animation that gives Ava weight.

How did you capture Alicia Vikander’s on-set performance as Ava?

Almost every scene involving Ava features lengthy dialogue shots. So, from the beginning, it was clear that Alicia would need to be on set, and that her physicality would have to drive the performance and give the other actors something to respond to. That immediately ruled out performance capture in post as an approach, so we opted for body tracking.

The majority of the shots in the film are around 200 frames long, with one clocking in at 1600 frames. It was a huge challenge to track them as the precision needed to be maintained throughout. It really was a herculean effort by all involved.

Additionally, the film was shot using Xtal Express anamorphic lenses which, whilst beautiful, provided a number of challenges to the tracking artists. The lens distortion was often not even, and changes in focus could radically change the lens geometry. Alex Maciera led the tracking team and I think it’s no exaggeration to say that body tracking Ava was the hardest task on the show.

Describe the workflow for a typical shot in which Ava appears.

Our approach to most Ava shots was to allow Alex, Rob and the actors to shoot what they wanted on set. When a set-up was completed, we would step in and shoot HDR lighting reference with bracketed stills, and shoot clean plates for the shot with the main unit camera.

In post, this allowed us to rotoscope the parts of Alicia we wanted to keep: always the face and hands, often the shoulders and feet too, and sometimes the shorts. Then we used the clean plates to prep out the rest of Ava, giving us a plate to composite with.

Meanwhile, camera- and body-tracking gave us a mesh that could be rendered with the HDR lighting captured on set. We used PRMan for rendering.

Finally the CG, the roto, and the clean plate were handed over to the compositing artists to final the shots. The 2D team – under the supervision of Paul Norris – not only graded the CG to perfectly match the plate, but often also did tracking tweaks and stabilisations. They also added the incredibly complex lens distortions, flares, and other optical effects.

It sounds like a lot of work for the roto department. Did you use greenscreens to help with the keying?

Well, the shoot was scheduled to last eight weeks, and we knew that we would be shooting between 15 and 25 set-ups every day. This pushed us to design a character that wouldn’t require heavy amounts of bluescreen or greenscreen on set; at the pace we were shooting at, there just would never have been time to light them properly. In fact, Ex Machina is the first show I’ve done where we didn’t use a single greenscreen.

It’s also worth pointing out that, because the sets had a lot of glass in them, we would often be tracking, prepping, rendering, and compositing several Avas at once. Some shots included as many as three reflections of Ava, in addition to the Ava in front of the camera.

What’s your favourite Ava shot?

I’m especially proud of the work that was done on the scene where Ava puts on clothes. On set, Alicia dressed herself over the top of the Ava costume. We not only had to prep in the background where Ava’s arms, legs and torso were, but we also had to paint in other details – the back side of the stockings, for example. Greg Shimp did great paint work on those shots.

The lighting in the scene was very subtle, and we ended up adding a lot of additional CG lights to give the look we wanted. Michael Ranaletta composited the shots, dealing with the shifting, shallow focus, and subtle edge treatments around the garments to get everything to sit together perfectly. It’s a such an intimate, delicate scene. It’s a treat when VFX is allowed to make something beautiful.

How do you feel about the film, now that the work is done?

Ex Machina is a really beautiful film. The imagery is – and I realise I’m biased – fantastic. I really believe that this is the result of every department working closely together, helping each other out, and making good practical decisions every step of the way. The collaboration with Alex, the art department, camera, make-up, costume and editorial was exceptionally deep on this show, and I think it shows in the finished result.

If you’re lucky enough to have great crews – as I did at Dneg, Milk, Utopia and Web – then you can make some very beautiful pictures. I’m very proud of all of them.

Links:

• Ex Machina – official website
• Double Negative
• Milk VFX
• Utopia

Special thanks to Sarah Harries and Karl Simon Gustafsson.

In this video lecture, Massimiliano Zecca from the Healthcare Technology and Head of the Healthcare Technology group at Loughborough University discusses emotional robotics, musical robotics and wearable bio-instrumentation. He centers his research on robotic systems and technologies that assist those in need due to advanced age or illness.

Zecca is interested in robotic systems and human interactions on an emotional level and in designing a mental model for the humanoid robot. This lead to the development of the WE-4RII robot. His second interest and research focus lies in musical robotics and making robots capable of interacting with other musicians, as if the robots were band members. This is how the Waseda flutist robot N.4 refined VI was born, followed by the Waseda saxophonist. His work also touches on how people perceive the interaction with these robots, rather than concentrating solely on advancing the technology. This brought his attention to wearable sensors for humans, where the data collected is directly fed to robots for interpretation.

Zecca explores issues such as how people feel about interacting with the robot on an emotional level, how the robots interact within groups, and how can they “sense” what humans feel during the interaction.

Download the PDF presentation here. 

Massimiliano Zecca holds a Ph.D. in Biomedical Robotics from the Scuola Superiore Sant’Anna, Pisa, (Italy), and currently teaches Healthcare Technology at Loughborough University (UK), where he also leads the Wearable BioRobotics research group.

He is a member of the National Centre for Sport and Exercise Medicine in the East Midlands, and of the NIHR Leicester-Loughborough Diet, Lifestyle and Physical Activity Biomedical Research Unit.

Zecca M. IJARS Video Series: Human-Robot Emotional and Musical Interactions [online video]. International Journal of Advanced Robotic Systems, 2015, 12:V3. DOI: 10.5772/60801

Baymax, Chris Atkeson and Hiro from Disney’s Big Hero 6. Source: Chris Atkeson

Baymax, the lovable, inflatable robot star of Disney’s recent hit, Big Hero 6, is far from a movie fantasy. With their soft cushiony bodies, robots like Baymax have very real prospects as future care-givers, space-travellers and more. Robohub’s Helmut Hauser spoke to the man who inspired Baymax – Chris Atkeson, Professor of Robotics at Carnegie Mellon University – about the hard science behind soft robotics.

[HH] Professor Atkeson, can you give us the background – how Disney approached you and how they decided to use a soft robot for the role of Baymax?

[CA] When Disney makes one of these animation movies, they do research. For example, for the movie Frozen, they had a bunch of animators who were from Southern California and had not really experienced snow. They sent a lot of them to Norway to play in the snow there and film it. That led to the movie Frozen.

For Big Hero 6, they knew they were going make a movie about a robot. They had the outline for the story from a Marvel comic, but they really wanted a robot that was new and different, they didn’t want a Terminator or a Transformer. So they contacted Disney Research, which tours robot laboratories around the world. One of the laboratories they visited was mine. They saw our work on inflatable robots to help dress, people, feed them and comb their hair, and the Disney folks were very happy because they’d found their robot: an inflatable robot that was very, very different from anything that had been in the movies before. And they also found a character for Baymax – a health or personal care, companion robot.

You are working in the field of soft robotics. Could you please explain what that is?

We’d like to have robots help take care of people and trying to do that with a very heavy robot is difficult, because sometimes computers break down or crash and the robot moves in unexpected ways, and sometimes people aren’t very co-operative. When I give a tour of my robotics lab to children, they often try to grab the robot by the legs and [it] could fall down. If a robot is very heavy, let’s say 100-150 kilograms, it can hurt the children or a pet. So I decided the best way to make a robot safe is to make it as light as possible, because mass is what causes gravitational force. Your momentum and your energy are proportional to mass. So, if we can reduce mass, we can reduce force and momentum; we can reduce energy and make it a much safer robot. So I was actually interested in light-weight robots – and maybe that’s what the field should be called – but we ended up deciding that the best way is to make inflatable robots and they’re inherently soft, so we got classified as “soft robotics.”

Can you also talk a little bit about the technological challenges that you have to face when you build inflatable robots?

Many people worry that an inflatable robot is too weak to do anything useful. So, one of the things we tested was: can they lift heavy loads? And yes, they can lift 10 or 15 kilograms and that’s enough for a lot of the care tasks that we want to do. If you don’t use a robot in the configuration of an arm of a humanoid and, instead, put the inflatable part underneath something, you can lift a larger mass. Inflatable devices are used to lift cars and buildings, so they can be very, very strong.

The second fear with inflatable robots is, if you puncture it, it will deflate and be useless. It turns out that the suits used for space walks, on the International Space Station, are also inflatable robots. They’re designed to self-seal if they’re damaged, so they don’t deflate immediately; there are techniques to help inflatable robots survive when they’re punctured, they can have compartments. So, puncturing an inflatable robot is not a big problem. In Big Hero 6, the robot repairs itself after it has several holes in it.

Yes, exactly – with scotch tape.

Yes.

Can you say a little bit more about energy efficiency? Is there a lot of energy needed in order to keep the pressure up in the robot or is that quite easy to do?

I don’t think it’s a lot of energy. It all depends on how much leakage you have. If you have a system that’s continually leaking and inflating, it may run up your energy bill. But if you have a situation where the structure doesn’t leak very much, and you only have to add a little air once in a while, that’s not a big problem.

Lightweight carbon fiber skeleton for an arm. Source: Chris Atkeson

You’ve already talked a little bit about possible applications for such types of robots. Do you have any other applications, outside of care-taking, in mind?

Robots that are toys and interact with children need to be very, very safe so I think soft, or lightweight, robots have an application there. Basically, [in] any kind of situation where you’re interacting with people, we need a high amount of safety.

Do you use any special sensors for your inflatable robots? Maybe a stretchable, bendable sensor, or do you use off-the-shelf sensors?

I think when you have a robot that’s not rigid, you tend to rely more on vision and external sensing. A rigid model can rely on sensors in the joints to achieve accuracy, but with the soft robots, you tend to have to look at where your hand is and where the target is in order to figure out: are you reaching the target accurately?

Closely connected to that, the control approach might be different as well, right? Or do you employ the same techniques that are used for a rigid robot?

I think the control approach may well be different. You can think of it as a much more flexible or complicated dynamic, essentially everything bends. And that means you have to be a little more cautious and use lower “control gains” or “stiffnesses” in order to move the robot around. So I think there need to be slightly different techniques, both in sensing and in control.

Your name is well-known in the field of machine learning too. I was wondering if you have any plans to link your expertise there to soft robots. Do you see the soft body as an advantage there?

We have already used learning for the “kinematics” of the robot; in other words, how to reach accurately, even though the parts of the robot are soft and bendy. The big surprise is that, when you start building complicated metal robots, hard robots, they also bend if they’re complicated enough – like a humanoid robot – because they have so many joints and there is play in them. So some of the techniques that we originally designed for soft robots are actually turning out to be useful for learning with the metal robots.

Baymax concept skeletons. Source: Walt Disney Animation Studios

In the movie, Hero builds rigid body parts for Baymax to enable it to fly around and to fight. Do you think that would make sense in real-life? And, if so, what applications could this have?

I think it’s a very good idea to have robots that can change modes. In this case, Baymax put on a hard outer shell in order to become stiff, and I think that makes a lot of sense. If you have a service or house robot, and it needs to be strong, it could use tools or put on a suit in the same way that people use tools. I think it’s a very cost effective way to get robots to do a wide range of things.

Do you also see a use for such types of robots outside the house, for disaster relief or for military use, for example? Because I would assume that a soft robot doesn’t have to model its environment in order to move around in it. Would that be a good application?

It depends on what kind of disaster you’re imagining. If you have an earthquake, for example, and a lot of rubble, you need small, agile robots to find victims, and very strong robots to dig them out. So maybe we’d have more specialized robots, say, in Japan preparing for an earthquake. If it’s a disaster that involves a lot of people getting sick, like the Ebola situation, then the soft robots may make more sense. They can be made very cheaply because we are using techniques similar to the ones used to make clothes, so we could even have disposable robots rather than trying to clean them. So, in the United States, when the Ebola problem was much greater a few months ago, there was some discussion about what kind of robots we should develop and soft robots was a possible answer.

What about using such types of robots for exo-planetary exploration, since, in space travel, weight and mass are a big issues?

Sure. Did you know we use inflatable landing systems to land probes on Mars? So, that’s already an example. If you’re going to build a habitat that people live in, it would probably be inflatable, if you want to deploy very large structures in space. Currently, we use unfolding wing-like things that are quite complicated. It’s possible that inflating them would be a better way to go – to make very large solar panels or antennas.

Baymax concept. Source: Walt Disney Animation Studios

You have a project called “Build Baymax”. Can you tell us a little bit more about that?

Yes. Of course, it’s inspired by the movie. There are two parts to the project. One is building the body: Can we build something with the capabilities of the robots in the movie? The second part is building the brain: Can we build robots that are capable of having the artificial intelligence that Baymax showed in the movie? We have groups to pursue those two targets, building the body and building the brain.

The movie is – I think – really inspiring, for adults as well as for kids. Because it talks about being a good scientist and being curious about things. So, if you were to give advice to kids who may be interested in soft robotics, what good sources are there for inspiration? What kind of skills does someone need to join the next generation of soft roboticists?

I would, of course, emphasize that studying mathematics and science is very important. But, I also think it’s possible for children to build soft robots, because it’s like making clothes and children can do it at home. If you want to build a metal robot, you typically need access to parts. With soft robots, it’s like building a kite and you can do that in your home. You can have some fun and get some experience. In the United States, we have a term, “home economics,” which is about teaching people to sew and things like that. Soft robotics could make home economics popular again.

Further reading on soft robotics:

Chris Atkeson’s webpage
Build Baymax website
Robosoft – An EU research community around soft robotics

• While most robots are either geek or jock, CHAPPiE is a lesson in sexy
  
• Robot love: Spike Jonze’s new sci-fi film ‘Her’ may be closer to reality than you think
  
• Automata: The new “sci-fi” blockbuster set to put robot ethics under a spotlight
  
• Ability to self-repair isn’t the same as consciousness: Analyzing Automata from a tech standpoint
  

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

YouTube starlet and CBBC actress Kate Mandeville sports a Swarovski-crystal-adorned prosthetic at London’s Wearable Tech Show. Designed and 3D printed by Open Bionics, Mandeville’s bionic arm is a statement in fashion and personality.

The personalisation of healthcare devices has been a growing trend in the maker-sphere. From gold-plated hearing aids, neon walking sticks, and sparkling blade prosthetics to 3D printed arm casts, people with disabilities are no longer waiting for health services to catch up – they are dragging their medical devices into the future on their own.

These medical aids are getting a long-awaited makeover and today it’s the turn of the bionic hand with Open Bionics unveiling of their latest 3Dprinted prosthetic at London’s Wearable Tech Show.

Open Bionics, a startup of four based inside the Bristol Robotic’s Laboratory’s Technology Incubator, has 3D printed a custom-fitted bionic hand with enough sparkle to rival a disco ball for a woman born without a hand.

Grace Mandeville is a YouTube starlet and CBBC actress who applauds the growing popularity for diverse prosthetics.

Grace has taken to YouTube on multiple occasions to discuss diversity and her love of inventive prosthetics that can show off a bit of her vibrant personality.

Grace said: “This is my favourite thing about this whole topic. I really love fashion, and therefor dress to illustrate my personality, so being able to wear a creative prosthetic that shows who I am seems awesome – it’s like a one off accessory that nobody else can wear, basically like vintage Chanel.”

“You should be proud of what makes you different, and I think being able to wear a fun looking prosthetic is something to be proud of! You’re basically saying to the public “my arms cool and I know”.”

Open Bionics’ COO Samantha Payne said that the idea behind the Swarovski hand was to show off the possibilities for prosthetics within 3D printing.

Samantha said: “We printed Grace a socket and robotic hand in three days, and because 3D printing is so affordable we can add Swarovksi crystals and create something really eye-catching that will not break the bank. We also added four fibre optic wires to the socket so that whenever Grace closes her hand, a blue light would shoot up her 3D printed arm.”

“Prosthetics are entering the realms of fashion and we wanted to show how bionic prosthetics can be functional and fun.”

“We’ve been very experimental with Grace’s hand. This is a completely new socket design and this is the first time we’ve experimented with placing the EMG sensors above the elbow. Grace is actually controlling her hand by the muscle signals from her back.”

“The idea is to give hand amputees more option and a choice to have something they’d get some enjoyment out of wearing.”

“We’ve been told a lot by amputees that they want something that will get a compliment not a strange stare, something far away from a ‘flesh’ coloured prosthetic.”

Grace’s sister, Amelia Mandeville, said that having an attractive prosthetic could help turn something that is seen as ‘negative into a positive’. Amelia echoed her sister’s stance for having the option to stand out, asking “Who wants to be the same?”

As Grace eloquently puts it, “Why try to blend in, when you can have a piece of art as an arm instead?”

Grace was given a traditional cosmetic prosthetic when she was little and has one now but says, “I never wear it, I don’t like wearing it, it gets in the way.”

Grace said: “I love what Open Bionics is doing. So many people at the ‘Wearable Tech Show’ thought I had a hand and that I was wearing a fashionable sleeve, making some kind of fashion statement. I had to keep pulling my arm out and showing people that I wasn’t wearing some kind of glove but an actual bionic arm.”

“I found the hand really easy to operate, I tried it on for the first time Monday and I could control the hand straight away. I thought it was going to be really heavy but it wasn’t. I obviously still feel the difference, I was born with a foreshortened forearm so wearing anything is going to feel different and will always be an added weight.”

“I don’t ever wear prosthetics because I don’t feel like I need to. I would however absolutely love a bionic hand like this for events and evening’s out. I love fashion and this looks incredible.”

Open Bionics is still developing their robotic prosthetics and hope to be selling 3D printed hands within a year. They have won multiple awards for their open source 3D printed robotic hands and was recently named as one of the Top 50 international robotics companies to watch along with Google.

• This robotic prosthetic hand can be made for just $1000
• CES 2014 – Robotics on the Runway
• Robots: the “ables” of the next technology trend
• Designing Jibo: Subtracting the obvious and adding the meaningful
•  Rehab robots, smart prostheses and exoskeletons to reach $1.8 billion by 2020

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

This fall, Fjuze – an entertainment company from the Netherlands – hopes to launch a massive drone entertainment show at the Amsterdam Arena. Aptly called AIR, the event is slated to feature an as yet unspecified (but presumably large) number of drones performing “ballet and battles,” “races and lasers,” and “circus and illusions.” That such a large-scale entertainment event featuring drones is even being planned at all speaks volumes about the mainstreaming of drone culture.

The Amsterdam Arena is the largest stadium in the Netherlands and is typically used to host football matches and concerts. “The Amsterdam Arena is suitable for a show like this because it’s a big indoor football stadium,” explained Pim Jansen of Fjuze, the entertainment/event company behind the AIR project. With the stadium’s capacity pegged at 50K for this show, AIR organizers will be banking on the surge in drone popularity to fill seats.

This is ambitious for a drone-focused event.

To give some perspective: AUVSI’s Brett Davis is expecting their upcoming Unmanned Systems Show (one of the largest drone-focused events in terms of attendance and square footage) to attract 8K attendees; likewise UAVSA’s Nicco Bugarin reports that they are the fastest growing commercial UAV association in the US, with their 2014 Drone Expo attracting over 3K attendees and their 2015 show expecting ~5K. Events such as these have been increasing their efforts to feature more commercial drones and broaden their appeal base. “There’s a lot of excitement and enthusiasm for the civil and commercial uses of UAS,” said AUVSI President and CEO Brian Wynne. Many different industries, including agriculture, oil and gas and even the movie industry, are looking to use UAS once the regulatory framework is in place.” 

Yet while events such as these are in essence trade shows and thus tend to attract a niche audience, AIR is seeking to entertain the masses. And this puts it into a whole other category.

If you are looking to gauge drones’ potential for mass appeal, Andra Keay of Silicon Valley Robotics suggests that it may be best to look beyond trade events to the growing number of major entertainment and events companies that are using or plan to use drones to attract attention:

• Spaxels and the ARS Electronica Future Lab have been partnering to light up the night sky with drones at festivals and events since 2012;
• KMel (recently acquired by Qualcomm) likewise got their claim to fame doing a drone light show at Cannes in 2012 and have gone on to develop many notable drone promos, including this Lexus ad, for example;
• Disney applied for three drone-related patents in 2014 that include possible use of “aerial display systems”;
• Paramount partnered with Ascending Technologies to promote the Star Trek movie “Into the Darkness”;
• Maker Faire has been hosting Game of Drones, a drone flying competition featuring aerial combat, and obstacle courses;
• Cirque du Soleil has partnered with Verity Studios and ETHZ to produce a short film, and according to Cirque’s Science and Technology Advisor Bill Keays “the project demonstrates a whole new avenue for channeling creativity” … suggesting that more drone projects may be in Cirque’s future;
• and though the swarm of drones in this Subaru ad are CGI and not real, the automaker clearly wished to associate itself with the trend.

“We’re just starting to see arenas turned into 3D entertainment units, where every drone is effectively a pixel in a giant array,” says Keay.

If one broadens the scope to look at robotics in entertainment more generally, there is every reason to believe that drones could one day soon move from side show to the main stage. Sarcos (which was recently bought back from Raytheon) has seen its robots put to use in attractions at Disney, Universal Studios and the robotic fountains at the Bellagio casino. In May of last year, Royal Carribean announced a partnership with Robotic Arts to bring “high tech entertainment” to its cruise guests aboard the Quantum of the Seas, and just last fall announced a partnership with the developers of the bartending MakrShakr robot. Perhaps most spectacular, however, is South Korea’s $666 million USD public-private investment in Robot Land – a theme park dedicated to creating large-scale consumer demand for robotics in that country.

Safety considerations will always have to be taken into account in any public event that includes drones; no event organizer would want a repeat of the Australian triathlete who was struck by a drone during a marathon, or the Virginia man who was hit in the face by a drone while watching Virginia’s bull run. But the drones that caused accidents at both these events were interlopers and outside the control of the organizers. At controlled events such as AIR, or this TED demonstration by Raffaello D’Andrea, drones are piloted by experienced operators and/or autonomous systems, and spectator safety may not be any more of a concern than it would be at other events featuring airborne projectiles (think baseball).

“Of course we have to guarantee visitors’ safety,” says Jansen, who explained that AIR will have safety nets to protect spectators.

At the moment, AIR’s only official partner is The Dutch Royal Air Force, which according to Jansen, is looking for new ways to communicate with their target audience and to recruit the “pilots and techs of the future”. With AIR scheduled to launch at an as-yet undetermined date this fall, organizers will need to get cracking! Ticket sales are slated to start soon.

Other drone events to watch out for this year:

• AUVSI’s Unmanned Systems Europe – March 3-4 Brussels, Belgium
• The New York City Drone Film Festival – March 7, New York City, NY, USA
• Cabourg Drones Festival – May 2-3, Cabourg, France
• AUVSI Unmanned Systems Conference & Trade Show – May 4-7, Atlanta, GA, USA
• Drone Apps Conference – September 14-15, Lausanne, Switzerland
• UK Drone Show – December 5, Birmingham, UK

In this episode, Audrow Nash speaks with Hunter Lloyd, who is a Professor of Robotics at Montana State University and a comedian. Hunter performs a comedy act for all ages with partner Looney, a NAO Humanoid Robot from Aldebaran Robotics. Lloyd discusses making people laugh with his robot partner, why he does it, and how what he’s learned as a comedian relates to robotics.

Hunter Lloyd

Hunter Lloyd is an award winning Professor, business owner, inventor, author, comedian, and family man. While Hunter was an undergraduate student studying accounting, he postponed his college career to start stand-up comedy. For six years Hunter toured the country working with acts such as Jerry Seinfeld and Tim Allen. Hunter headlined his first comedy show at Laffs in Tucson, Arizona; at the age of 23, he appeared on television networks such as MTV, VH1 and The Comedy Channel. After touring for several years, Hunter went back to complete his undergraduate degree and graduate school where he studied engineering and computer science, which eventually led him to the field of robotics. Hunter is now a Professor of Robotics at Montana State University in Bozeman, Montana. Hunter and his students have excelled in a number of competitions including the NASA Robotics Championship, the Robot Olympic Games, and Robogames.

Hunter is also a robotics entrepreneur: founding a robotics company in 2005 that helped develop the RangeWatcher and WatchKeeper technologies for the Predator Unmanned Aircraft Group. Today, Hunter combines his skills as a robotics engineer and comedian to perform to audiences of students, inspiring them to follow their dreams in the fields of math and science.

Links:

• Download mp3 (16.3 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Looney the Robot website
• Hunter Lloyd’s website

NEW: Transcript below.
In this episode, Ron Vanderkley speaks with Mythbusters‘ Grant Imahara, and Richard McKenna from The Creature Technology Company about robotics in the film, television and theatre industries.

Grant Imahara
Grant Imahara graduated from the University of Southern California with a degree in electrical engineering. It was shortly after that Imahara was hired as an engineer at LucasFilms and Industrial Light & Magic, building and operating a number of visual effects, models and robots for popular films/film series (such as Star Wars, Galaxy Quest, Jurassic Park, Terminator, The Matrix and AI: Artificial Intelligence). Imahara also built the Energizer Bunny, for the battery company’s commercials, Deadblow robot on BattleBots and Jeff Peterson from The Late Late Show with Craig Ferguson. Imahara is perhaps best known as a presenter on Discovery Channel’s MythBusters, and is often seen making robots or robotic rigs needed to aid in the testing of various myths. Imahara appeared at Supanova 2014 promoting his casting as Mr. Sulu in the popular professional web-series Star Trek: Continues.

http://youtu.be/iJv6LfI1yxk

Richard McKenna
Richard McKenna is Chief Engineer at The Creature Technology Company. He joined CTC in 2010 and has worked on all of the major projects since that time, including How to Train Your Dragon; King Kong and the Sochi Olympic Mascots.  He has a Bachelor of Engineering (Hons) in Mechatronics, Robotics and Automation Engineering from Swinburne University and is certified as a “Chartered Professional Engineer” by Engineers Australia, registered on the National Professional Engineers Register (NPER). Prior to joining CTC, the majority of Richard’s time was spent in the defence industry, and he has also worked in special effects for film and television.

Links:

• Download mp3 (15MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Myth Busters’ Website
• The Creature Technology Company’s website

References: 

• Stanley “Stan” Winston was an American television and film special make-up effects creator. He was best known for his work in The Thing, the Terminator series, the first three Jurassic Park films.

• Gerry Ryan of Jayco Caravans the largest recreational vehicles manufacturer in Australia, owner of Creature Technology, Global Creatures, (Gerry Ryan is listed in BRW Rich 200 list 2014: 148)

• Dreamworks “HOW TO TRAIN YOUR DRAGON LIVE SPECTACULAR”, where dragons with 40 foot wingspans fly and breathe fire. This must-see arena spectacle breaks all the rules of traditional entertainment, immersing audiences into a magical and mythical world of Vikings and Dragons.

A camera man gets up, close, and personal with the human and machine actors on the “Sparked” set.

If you thought “SPARKED” – the new short film by Cirque du Soleil, ETH Zurich, and Verity Studios – is too real-looking to be CGI, you were right. But why go to all the trouble of using quadrotors to get those lampshades dancing in the air for real? We asked Bill Keays, Science and Technology Advisor at Cirque du Soleil, to give us an insider’s perspective about Cirque’s motivation for the film and how it came together.

Bill Keays on the SPARKED film set.

[Robohub] What was the inspiration for the film?
[Keays] The combination of autonomous flying machines with Cirque du Soleil magic was so evidently bursting with potential we committed ourselves to make something original together not knowing what it would be at first. An iterative creative process ensued involving all participants, and what emerged was a wonderful script that carried the talent from both sides of the formula to a whole new level.

Who are the main partners on the project? 

It was a highly democratic and highly collaborative process. Raffaello D’Andrea of ETH Zurich and Verity Studios, and Welby Altidor of Cirque du Soleil were the project leaders, but the true collaborative nature of this project is reflected in the fact that no participant roles are specified in the credits.

Who initiated the project? 

The very first discussions were held between Prof. Raffaello D’Andrea at ETH Zurich and myself as Science and Technology Advisor at Cirque du Soleil in March 2012. After ongoing discussions and considering various options, the two organizations eventually committed to a creation project in January 2014.

The “Sparked” film set. The shoot took place in the Flying Machine Arena at ETH Zurich, a portable space devoted to autonomous flight.

Why was the film made?

For Cirque the project demonstrates a whole new avenue for channeling creativity. It allows Cirque to reach a new audience externally and provides inspiration internally to push the creative envelope in new directions.

As such is a highly effective, multifaceted communications vehicle, and it will certainly be submitted for competition.

What were the production costs? 

The large bulk of the cost came in terms of human resources split between three organizations. The exact figures cannot be known.

How long did the project take from first concept to finished film? What were the major steps/milestones?

The whole project, from go-ahead to YouTube release, took under 8 months. It was a back and forth process whereby content was driven by discussions between Cirque, Verity, and the film director. Through this method we progressed through initial concept design, script development, set and prop design, and finally the shoot, with intense hardware and software development taking place throughout. The product was then handed off to post production and then to distribution.

Setting up before a flight scene.

Who designed the lampshades? The photos show researchers from ETH and Verity fitting the quads with them. Did you provide dimensions to props experts and iterate? How did that process work?

It was indeed an iterative process. The lampshades were designed and produced by the creative team at Cirque in Montreal who used their own quadrocopters for flight tests to ensure that they would be compatible with the flight environment at ETH Zurich. Guidance from the ETH and Verity teams was indispensable of course.

Was the actor controlling the quads with hand motions (via Kinect)? Or was this pre-programmed and choreographed? At times it looks like he is ‘conducting’ their movements …

The Flying Machine Arena fully possesses the ability to control the flying machines with a Kinect or other similar devices, or even hand gestures. But given that this is a movie production, the movements of the actor and the flying machines were choreographed as is the convention in film practice.

Was there any CGI at all in the film? How much is real?

None whatsoever, it’s all real; and even though it looks terrific in the movie, this pales in comparison to seeing it in the round. It was mindboggling to watch live.

Has Cirque worked with robots before?

Yes, Cirque has experimented with robots in the past, but never flying ones.

What were the biggest challenges of putting live actors together with the quads on stage?

Given that this is such an unusual circumstance for making a movie, we didn’t know how our lead actor would react to the flight environment. But the high precision of the flying machines and terrific expertise of ETH Zurich’s Flying Machine Arena staff quickly inspired confidence. By the second day, remarkably, our actors, both human and mechanical, were moving about the set with terrific ease, paying little attention to each other.

Was safety ever an issue?

The design of the lampshades was convenient in that it shielded the blades of the quadrocopters, thus making them far less likely to strike anyone or anything. There were a couple of crashes and a few low passes, but at no point was there any real apparent danger.

What about lighting?

Believe it or not, all the scenes with the flying lampshades are completely lit by the lampshades themselves! Another fortuitous design feature.

Changing batteries.

Were there any other challenges?

The rapid power consumption of the flying machines meant that battery changes would be frequent. With time, the switch-overs began looking like Formula 1 pit stops, performed with great speed and precision.

What was it like for the researchers from the lab to have to ‘perform’ for the camera? How long did they have to rehearse before you were ready?

The lampshades arrived in the weeks prior to the shoot so there was time for Verity Studios and ETH Zurich to fully test their aerodynamic abilities, and to run through the choreography. In the days preceding the shoot we performed flight tests according to the story board and had one practice day with the lead actor. However, due to the extreme time constraints imposed by the film crew, the control software had to be designed such that flight plans could be altered at a moment’s notice to meet specific requests from the director. This proved to be an invaluable tool and was instrumental in completing the shoot in the allotted time.

Had your previous work (e.g. demos, presentations) prepared you for working with quadrotors on a film set?

Absolutely, the Flying Machine Arena team had performed extensively with flying machines in public spaces and conferences (including TED), so much of this experience transferred well onto the movie set.

An early flight test in the Flying Machine Arena.

The quadrotors peek through in the film at times, as do the retro reflective markers for the motion capture system – was this intentional?

It is true that the quadrotors are sometimes visible, but the visual quality of the movie is so overwhelming that it is difficult to remain focused on technical bits. We actually liked the fact that they were occasionally visible; as sort of a wink to those of us who are knowledgeable about them.

With previous demos, the quadrotors played a leading role – but in this film, they are literally hidden behind lampshades. How does it feel to see them step back into a supportive role for this film?

But the flying machines are not hidden. They are in dress like all actors in this movie; they have character, motivation and certainly no lack of presence. As the devil’s advocate, I would be inclined to challenge the assumption that they are playing a supporting role.

Why not just do it with CGI?

There are at least two good reasons:

First of all, because the Flying Machine Arena is not about CGI. It’s about control systems, aerodynamics, and sophisticated algorithms that yield a physical manifestation of uncommon beauty.

The second reason is because you can’t do CGI in front of a live audience and Cirque du Soleil is in the business of live entertainment. Even though this extraordinary little movie is a gem in its own right, it is primarily the proof of concept for something far more powerful; something that must be seen live.

All images credited to the “Sparked” team. For more insight on the film see also New quadrocopter video points to a future for flying machines in entertainment by Markus Waibel of Verity Studios. 

We’ve seen robots on stage before: Humanoids have been featured in theater productions in Switzerland, Austria, and Japan, and industrial robots have played parts in dance performances and other staged events. Flying robots have also played a theatrical role, for example in Shakespeare’s A Midsummer Night’s Dream at Texas A&M, a dance set to a Schubert piano trio at MIT, as well as in advertising. Some robots have even gone pro: Robothespian was designed for acting and is rented out for performances.

But the dance between human and machine has never looked quite like this .

The film’s main plot is straight-forward: Lamps in a repair shop magically come alive, rising from their stands to sneak up on an unsuspecting repairman. As the magic unfolds, and the repairman discovers that the everyday objects of his trade have become animated and full of character, viewers become witness to the evolving dance between man and technology.

Everyone on the team, including myself, was impressed by how well the film captures the stunning, at times hypnotic visuals of the flying lampshades. Everything you see is real — no CGI, wires, or speed-ups/slow-downs were used in the film. 

So how did the lampshades come alive?

This short film, titled “Sparked”, was produced in collaboration between researchers at ETH Zurich’s Flying Machine Arena, their spin-off company Verity Studios, and entertainment company Cirque du Soleil; in other words, it’s quadrocopter magic that makes the lampshades fly.

The film marks a step change from previous work by ETH Zurich’s Flying Machine Arena team, catalyzed by their new spin-off company Verity Studios, which also gets credit for designing the quadrocopters’ choreography. Here, in spite of their computer-controlled precision flights, the flying machines become robot actors that are individuals with their own personalities: Pomponette, with her tassels bustling in the propellers’ airflow; the Medusa with its bizarre tentacles waving in the wind; the Runt, sleek and fast but still glowing and twitching from the electric shock. And Lady Purple, absent from most of the film because of her intractable aerodynamics.

Flying machines up, close, and personal

One of the most striking aspects of this film shoot was watching our actor, Nicolas Leresche, interact with the flying machines. Much of this interaction was captured on camera — and you can see the results in the film, in particular the moment of ‘first contact’ — but, for me, it was what happened off-camera that was most revealing.

Naively unjaded by their joys and dangers, this was Nicolas’ first experience with quadrocopters. As he began interacting with them, tentatively at first, and then — in a matter of seconds — more confidently, even playfully, it became crystal clear that I was not merely observing a human using a robotic tool.

As the quadrocopter’s onboard controller pushed back against Nicolas’ hand, immediately reacting to match his actions, it became apparent that the two interacted in a way we don’t usually get to interact with technology. The interaction here was physical and dynamic, and clearly intuitive for Nicolas. There was no noticeable learning curve, no need for manuals or explanations.

But what was most striking about this interaction was the result. Within seconds, Nicolas had established a deep, almost animate connection with the flying robot actors. As their dance developed, driven by the autonomous machines’ choreography and Nicolas’ script, and became ever more fluid and natural, it became clear that what we were witnessing was a relationship in its making; one of mutual respect and, at the same time, confidence and trust. And once there was trust, magic happened.

In his own words:

Actors think that they are the ones who make objects move. I think that, on the contrary, it’s the objects that make us move. In the case of drones, even more so! They are companions (in an etymological sense), confrères, brothers.

Within hours the entire film crew had become comfortable with the flying machines hovering nearby or brushing past them as they swooshed through the air. On one occasion, a mis-timed dolly motion led to a flying machine hovering just centimeters above the camera man, who continued with his work unfazed. On another occasion, when Nicolas missed his cue, he bumped into one of the flying machines. But they just jostled past each other and continued on with their choreography as if nothing exceptional had happened. On yet another occasion, a drone collided with the moving camera without disturbance.

The only damage during the entire two-day shoot was a single broken lamp, knocked over by one of the quadrocopters’ airflow.

Safety

I have no doubt that this level of confidence and comfort with the technology can in no small part be attributed to the system’s reliability. In spite of the packed shooting schedule (a total of 40 shots over two days) and the high complexity of the project (for example, designing lighting conditions using more than 60 light sources including some moving through space), production went smoothly, and importantly for the quadrocopter team, the flying machines did not add any significant delay. Battery swaps and on-the-spot changes to the choreography were efficient and quick, never exceeding a few minutes. Kudos to the excellent film team from who’s mcqueen for brilliantly doing their part to keep the production on schedule.

Collaboration

One aspect that made this project particularly enjoyable was the collaboration with creatives at Cirque du Soleil. From the start there was a lot of excitement about the potential of this technology for entertainment not just at ETH and Verity Studios here in Zurich but also at Cirque in Montreal. Everyone shared the view that the flying machines could be used to amaze audiences and to tell wonderful stories. What ensued was a collaborative, iterative creation process that culminated in a script and a set design that were imaginative to Cirque du Soleil standards, while exhibiting the abilities of the quadrocopters in their full splendor.

As it turned out, figuring out what is easy and what is difficult to achieve with autonomous quadrocopters can be quite counter-intuitive. Having researchers and creatives on both sides collaborate on the key creative elements of the story from the early phases of the project therefore proved invaluable to its success.

Next steps

Why go to all this trouble? We’ve all seen the amazing things CGI can do on film. I love films and movies, how they can condense experience and add life to our most impossible dreams. But nothing compares to the real experience of getting up, close, and personal with robot actors yourself.

What’s next? Our audiences over the past years have had more ideas than we can count. At this point, only one thing’s for sure: They all want to see more flying robot actors.

Some more snapshots from the film shoot below for your viewing pleasure:

All images credited to the “Sparked” team. Markus Waibel is a former member of ETH Zurich, a co-founder of Verity Studios, and one of the contributors to “Sparked”.

Hollywood has already been quick off the mark to explore issues thrown up by robotics, in movies such as Simulation and Robot & Frank. Automata, by Spanish director Gabe Ibanez, promises to provide not a little food for thought and throw a more immediate moral debate into the mix.

The film stars Antonio Banderas as an insurance agent living in the year 2044, where robots are now a common sight. To keep these metallic slaves under our control, there’s a law that expressly forbids them from modifying themselves – but nothing, it seems, can stop the rise of artificial intelligence.

The ability to self repair and optimise, it’s argued, is what distinguishes robots from mere machines as it implies some degree of consciousness.

Self-repair

In fact, robots with the ability to self-repair are not actually that far off and there are systems that are already able to do this – albeit in a basic form.

Bristol Robotics Laboratory researcher Alan Broun has spent the past couple of years working on artificial intelligence that will enable a robot to recognize itself, specifically to enable it to identify its own hand. He likens the process to that of a newborn baby building a mental model of itself over time and adapting that model as its body changes and its capabilities improve. It’s an area of research that has been termed “active vision.”

“It’s the view that a robot can’t be passive, it needs to interact to learn about itself and its world,” says Broun. “An adaptive system can work out its own dimensions, for example. Currently, if you change a robot’s arm, someone has to reprogram the whole system.”

Giving a robot the ability to learn about itself is, Broun believes, the first step towards being able to self repair.

“The foundations for it are there,” he says. “As a system becomes more capable, we’ll see the technology integrated, and it will be able to operate for longer without human interference.”

Broun believes that it’s only the commercial value that’s presently holding this in check, but research is moving fast.

If this is true, and self repairing robots are literally around the corner, debate is needed – and now – about the values, status and allowances we should accord these new systems.

Addressing preconceptions

The general public’s ideas about robots and their capabilities also need to be urgently addressed. A number of major studies have recently found that despite optimism about the technology’s potential, public perceptions of robots are often influenced by misconceptions and fears. In order to improve the image of robots and to increase public acceptance, we need to better understand public opinion and even take robots out onto streets, into schools and hospitals, to show people what the technology is capable of.

The creative arena is good one within which to explore the ethical challenges robotics will throw up. A Hollywood movie, such as Automata, has the potential to reach millions and provoke much needed debate. It may provide a springboard – the first step towards developing a regulatory framework, which must be in place before advanced robotic technologies can really take off.

The Japanese have for a while been way ahead of the rest of the world in their interest and acceptance of the issues robotics raises. Much of this has been due to the all pervading nature of the technology in Japan. Automation is de rigueur and the issues it raises have long been of interest to artists in the form of anime and other popular art forms.

In the rest of the world, grass roots awareness of the massive changes that robotics will throw up is evolving slowly, but workshops and seminars are introducing robots to a public that may have, until recently, given them a purely fictional status.

At Brighton’s Digital Festival, taking place throughout this month, artists Lorenza Ippolito, Anna Dumitriu and Alex May are investigating the philosophical and practical questions behind robotics.

Their event, My Robot Companion, is aimed at older audiences wanting to look at the technology and consider what the future might hold for us.

By introducing their audience to HARR1 (Humanoid Art Research Robot 1, made in collaboration with the University of Hertfordshire’s Adaptive Systems Research Group), the artists aim was to stimulate questions about the uses and functions of robots, and to explore the ethical implications of how they could be used in the future in a range of social settings.

Prone to malfunction

Such social robots hold a great deal of promise for society, but it’s also crucial to manage public expectations. The first systems are likely to be basic and could be prone to malfunction. But it’s what are perceived as failed interactions that can actually provide valuable data for the designers of future systems.

These themes are explored in a workshop that is part of next month’s International Conference on Social Robotics (in Sydney, Australia). Misbehaving Machines hopes to promote interaction and allow confrontation in robotics but, just as importantly, it seeks to bring engineers and scientists together with artists to look at how unexpected behaviour impacts on user perceptions and how this can be explored in the medium of art.

As is becoming clear, this is a field ripe with opportunity, not just for engineers and computer scientists, but also creatives – whether they work in the movie industry, the media or design. Robotics has got to be the most multidisciplinary technology known to man. Previously, largely the domain of engineers and computer scientists, now it’s pulling in psychologists to draw up the interaction boundaries that humans and robots must traverse, legislators and the creative industry – as a whole – to alert the public to the fact that major change is most definitely afoot. And that the sooner we make way for it and equip ourselves to face the physical and emotional obstacles robotics will throw up, the more successful that transition will be.

Automata is premieres in the US on 23 September 2014 and is released ion 10 October. There is no UK release date yet.

The Brighton Digital Festival runs to 28 September 2014: http://www.brightondigitalfestival.co.uk

The Misbehaving Machines workshop takes place on 27 October 2014 in Sydney Australia, part of the International Conference on Social Robotics.

This article was originally published on RAS SIG – the Robotics and Autonomous Systems Special Interest Group – part of Innovate UK’s Knowledge Transfer Network.

It seems like our most popular visions of the future always end with environmental collapse, authoritarian oppression … and the ever-popular zombie apocalypse. Science fiction, what Clive Thompson called “the last great literature of ideas,” seems to function today primarily as a literature of warning, of cautionary tales and parables of technology run amok.

What if we could also use science fiction to inspire hopeful thinking about the future, to imagine how technology could be a tool to help us solve our most pressing challenges, instead of being the purveyor of our destruction or a symbol of our hubris? Project Hieroglyph, founded by science fiction visionary Neal Stephenson in 2012 and headquartered at Arizona State University’s Center for Science and the Imagination, seeks to use science fiction to restore the creative, practically ambitious thinking about technology that made Golden Age science fiction so powerful and compelling.

The project’s first anthology, Hieroglyph: Stories and Visions for a Better Future (William Morrow/HarperCollins) features hopeful visions of the near future, grounded in real science and technology, created by top science fiction authors like Stephenson, Cory Doctorow, Elizabeth Bear, and Bruce Sterling, and scientists, technologists, and other experts in fields ranging from robotics and space exploration to education, sustainability, and neuroscience. (Disclaimer: I work at the Center for Science and the Imagination, and am one of the project managers for Hieroglyph.) We sometimes call this “science fiction of the present” or “science fiction of five minutes from now,” because the stories feature technologies – from a 20 kilometer steel tower to crowdfunded 3D printing robots creating a lunar base from moon dust to a global network of drone pilots protecting wildlife from poachers – that we could start building today, if we had the right story to tell about how and why.

The fundamental idea driving Project Hieroglyph is this: if we want a better future, we need to start by crafting better stories , better dreams. Major challenges like climate change, world hunger, and many of our most deadly infectious diseases are well understood. What holds us back from implementing solutions on a global scale is a lack of social and political will – fundamentally, a lack of imagination. We need stories that engage our collective imagination and inspire us to reallocate resources, change our priorities, and do something different, ambitious, and transformative.

When Stephenson talks about a “Hieroglyph,” he means an icon, like Asimov’s robots, Heinlein’s rocket ships, and William Gibson’s cyberspace, that “supplies a plausible, fully thought-out picture of an alternate reality in which some sort of compelling innovation has taken place” (read more in his article “Innovation Starvation“). Hieroglyphs become deeply etched in our cultural consciousness and shape generations of actual research. For example, it’s impossible to work on robotics without grappling with Asimov’s Three Laws.

Project Hieroglyph also challenges science fiction writers to collaborate deeply with researchers working at the cutting edge of knowledge, not just as fact checkers, but as equal partners. Stephenson’s story about a 20 kilometer tall steel tower, for example, was created in collaboration with Keith Hjelmstad, a professor of structural engineering at Arizona State University, and the idea is filtering back into Keith’s teaching, research, and work with students who are exploring how super-tall structures can cope with punishing jet stream winds and exotic upper-atmosphere weather conditions.

A great science fiction story can act as a first prototype (an idea developed by Intel’s futurist Brian David Johnson, one of our collaborators), a napkin sketch of a possible technological solution, to explore how the technology might affect communities, economies, and individuals. The best stories are also invitations, bringing people from a variety of perspectives together to have a conversation about whether we should pursue certain innovations, how we should use our vast technological capabilities, and what limits and regulations need to govern emerging technologies. At Project Hieroglyph we believe that weaving a human story around a whiz-bang technological idea is an important step in beginning to understand its social, cultural, and ethical implications, and to give a broad range of people an opportunity to respond, with enthusiasm, curiosity, caution, or consternation. In this spirit, each story in our anthology is coupled with a “story page” on our web platform that features expert responses, links to additional information, and most importantly, discussion forums that enable readers, experts, and skeptics to weigh in and contribute their own ideas.

We’re excited to welcome in Robohub community members to Project Hieroglyph, especially because several of the stories in our first anthology engage with the incredible possibilities of emerging robotics technologies. Cory Doctorow’s novella “The Man Who Sold the Moon” imagines how crowdfunding, the DIY culture, and 3D fabrication can open up new possibilities for democratized robotic forays into space. Gregory Benford’s “The Man Who Sold the Stars” (a lot of Heinlein readers contributed to this one!) imagines a future where robotic mining of asteroids and comets enables humans to extend our reach throughout our solar system and beyond. Lee Konstantinou’s “Johnny Appledrone vs. the FAA” and Brenda Cooper’s “Elephant Angels” consider how drones, currently a major bogeyman haunting our collective imagination, might be used for good. Vandana Singh’s “Entanglement” imagines self-organizing networks of robots scouring the floor of the Arctic Ocean and plugging methane leaks that threaten to exponentially accelerate climate change.

Project Hieroglyph has a network of amazing writers, artists, scientists, and engineers engaged in creating stories that make transformative change feel possible. But we know that we can’t accomplish anything significant without you. These stories are arenas for conversation, collaboration, and creativity, and they are designed to spawn real-world research projects, inventions, and solutions. We need your help transforming these Big Ideas into new collective realities. Won’t you join us?

To learn more about Project Hieroglyph and join the community, visit http://hieroglyph.asu.edu. Check out a video trailer for the project here:

http://vimeo.com/105817777

Read free excerpts from the book here, here, and here.

Bruno Lefevre-Brauer (known as +Brauer), is a graphic designer living in Paris. Over the past 20 years he has designed numerous album covers for French and international artists and pursued his personal artistic expression through painting and sculpture. An artist who regularly exhibits in Paris, these illuminated metal upcycled robots are from his series “Viva la Roboluciòn!”

+Brauer carefully chooses vintage objects that have an industrial past, that are marked by time and whose patina has been moulded by years of manual use. He admires the beauty of these discarded industrial parts, alters their appearance, sculpts them, and incorporates light sources into their structure before assembling the parts together to create a unique and poetic artwork.

The materials and the venerable patinas express their beauty in the light of day, while at night, evocative light fittings to reveal their magic. Right from conception, the element of light is an integral part of the artwork: each robot is designed to interact with it’s environment in a different way, whether it is turned on or off.

Abandoned or forgotten in workshops and garages, the industrial parts are reborn in unique works of art that embrace us with their kind presence, imposing personality, and amazing humanity. Each piece is a statement of poetic resistance to mass-consumption.

You can view the full image gallery here +Brauer | sculptures robots

This is an excerpt from The Drone Primer: A Compendium of the Key Issues, a free, one-stop, handbook addressing the basic and fundamental questions around drones in all their contexts, from foreign theaters of war to domestic civilian use. This basic report covers technology, history, strategy, law, and culture, and includes a portfolio of drone art. It addresses what the authors identified as as a serious problem: the lack of any single, well-written, comprehensive introduction to drones that was not driven by a political, moral or commercial agenda. Written for a general audience, The Drone Primer explains how drones work, their history, how they are used, the concerns that they have raised, their practical benefits, the debate that they have sparked, and the art trend that they have inspired.

Click here to read the full Primer for free.

Chapter 8 – DRONE ART

“The policy discussion is a highly specialized system with its own language and its own elite— almost a priesthood—who understand it and can explain it. By bringing back that analysis, by demystifying it, by looking at how we can explain it better and have clearer discussions around it, I think art has the potential to perform a very important role in policy.” – James Bridle

Over the past few years, artists have responded to the drone in various mediums. Art that treats the drone as its subject, or an art-making tool, has served to interrogate, protest, and promote this technology. This art has been well received by the public and has earned institutional recognition, to the extent we could confidently claim that it is an art trend. The art that makes up this movement is most commonly referred to as drone art.

Because of its cultural resonance, the drone has embedded itself in the popular visual vocabulary. Visions of the drone—usually some kind of variation on the U.S. Predator or Reaper—pop up in graffiti murals, collages, animations, and conceptual art with increasing frequency. Its distinctly sharp, cockpitless form has become a visual motif representing technological advancement, state power, and the spectre of autonomous technology.

In established art practices, we have seen drone art range from straight-served activism— as in the case of Heather Layton and Brian Bailey’s exhibition Home Drone, which asked its American audience to imagine what it would be like to live under drones—to more subtle forms, as Mahwish Chishty’s drones are painted in the aesthetic of Pakistani truck art—to the oblique, like Himanshu Suri’s song “Soup Boys”:

That drone cool, but I hate that drone Chocolate chip cookie dough in a sugar cone Drones in the morning, drones in the night I’m trying to find a pretty drone to take home tonight.

A key feature of the drone art movement is that the drone has served a dual role as both a subject of the artwork and a tool for creating it. In the case of artist and vandal KATSU’s drone graffiti, a drone is the graffiti-making tool. One of the earliest examples of drone art was the Natalie Jeremijenko and the Bureau of Inverse Technology’s BIT Plane project (1995). The team flew a camera-equipped, remote-control plane over Silicon Valley, capturing footage of large tech campuses that, on the ground, were closed to the public. In an interview with the Center for the Study of the Drone, Jeremijenko described the project as an “an exploration into this new territory called ‘information space.” The artistic product was a grainy, black-and-white video containing very little useful information about Silicon Valley, but it was a powerful statement about aerial technology and its surveillant potential.

Tomas van Houtryve’s drone-made aerial photographs of American domestic scenes are similarly bi-layered. On one hand, he is, like Layton and Bailey, experimenting with the idea of hypothetical drone strikes in the U.S. and protesting what he believes is a violent and unjust U.S. policy. On the other hand, his compositions flex the power of the drone, and in doing so point at what he considers to be the worrying implications of unmanned aerial technology. Van Houtryve protests the targeted killing campaign by partially demonstrating what the killing machine can do.

This project diverged from aerial drone art that celebrates the machine for its photographing ability without raising objections or concerns. Drone aerial photography, much of it created by amateurs, is a burgeoning field. While aerial photography has been practiced in various forms since the 19th century, its accessibility and popularity has exploded in recent years. Drone aerial photography was popularized by Raphael “Trappy” Pirker, who in 2010 grabbed U.S. headlines for flying a camera-equipped drone over the Statue of Liberty, the Brooklyn Bridge, and parts of Lower Manhattan.

While Trappy contends that his interest is in flying rather than art, his videos have inspired an aesthetic that disabuses itself of the political discussion and focuses instead on the sheer beauty of the aerial perspective.

Chapter 9 – PORTFOLIO

The Drone Art movement is important because it has contributed a vibrant visual vocabulary to the drone debate. It is also a test case of how different sectors of the art world—from fine gallery art to conceptual virtual art—respond to (and sometimes incorporate) a new technology.

Pakistani artist Mahwish Chishty relays the image of the drone through a classic Pakistani truck art aesthetic; in doing so, she raises questions about how the continuous presence of drones over parts of Pakistan have earned the drone a place in the country’s culture.

The BIT Plane project made use of the drone as a flying tripod, but in doing so also made a statement about the technology itself. Fernando Brizuela’s watercolors create a space of intersection between an old- fashioned, high art practice and a commentary on modern warfare.

James Bridle and Trevor Paglen, two seminal drone artists, meditate on the invisibility of the drone. Paglen takes super-long-range photographs of U.S. military drones flying at high altitude. In the resulting images, the drones often appear as nothing more than dark specks in the wide desert sky.

Bridle, on the other hand, traces 1:1 scale outlines of military drones in public spaces–he calls them Drone Shadows. “The Shadows are not really about what the drone looks like,” Bridle told the Center for the Study of the drone. “They’re about the absence of the drone in the contemporary discourse.”

Fernando Brizuela. “drone,” 2013. Watercolor on Paper.

Trevor Paglen, “Untitled (Predators; Indian Springs, NV),” 2010 [Detail]. Credit: Altman Siegel

James Bridle, “Drone Shadow 007: The Lavender Hill Drone” 2014

Mahwish Chishty, “MQ9/1,” 2011. Gauche, tea stain and gold leaf on paper. 8” x 28.5”

When I was in high school I was a big fan of graphic novels. In art class I used to splash out paintings – portraits mostly – and would write on them, giving the portraits word balloons, like a comic book. I wanted to portray personalities more than portraits. Of course my art teacher hated this and she’d box my ears, telling me that painters should not write, and that writers should not paint. She even called me an artistic criminal.

But I went on (as you did, dear reader) to make things like Twitter accounts and LinkedIn profiles and selfies that I’d then tag with some text and did just that – put words with the picture.

Today we’re all  portrait artists and writers. But with cameras in our phones and publishing tools that allow us to be both exhibitors and viewers in the often perverse universe of social media, it’s not so much about self-expression as it is about identity. We all write blog posts and put an image in them, and rarely does a selfie not include some tagline that’s supposed to be clever or witty, since, like all self-effacing artists, we want to be clever and witty and get into the mosh pit of show-n-tell.

Selfie = Profile = Account = Avatar = Self-portrait = ID

Whether you call it a profile, an account, or an avatar, you’re making a self-portrait. It’s your ID. It might a 2D profile, or it might be a 3D avatar for a game, but when you upload a photo of yourself and then you add some text, you’ve made a self-portrait ID that’s got something to say.

And you make IDs for yourself more often than you might think: if you’ve got a Twitter account, then you’ve done this; if you’ve got a Facebook or LinkedIn account, then you’ve done this; even when you pick some dumb photograph like the flower or the horse to use as your login image for OSX or Windows, you portray yourself with that image, and then you write your name next to it. We all have dozens (if not hundreds) of these weird little text-garnished self-portraits scattered all around the Internet.

Your avatar is your self-portrait, and it has more value than you think.

Regardless of your ability to draw or write a verse, these self-portraits you’ve made are very, very valuable.

Portraits have always been valuable, and this is because our identities are valuable. The royalty of Europe used to have portraits painted as a means of immortalizing their ID onto the wall of a castle, and the money required to hire an artist to paint their face for all the castle-folk to see was a declaration of power, influence, and wealth. Even when the camera was invented, portraits were still valuable, and sitting for a portrait was a sign of status. Today a face on the cover of a magazine is the same. It’s a social value.

Not all of us can be Portuguese Kings, but those days are past us anyway. Today we wallow in the attention of social media by building pages that proclaim how great our life is (or how miserable), or draw attention to things that interest us, and we have little numbers in social media sites that measure our self-portrait avatar on some social scale. Our avatar self-portraits, therefor, not only have text, but they also have numbers so that we can see how they’re performing – and that number ranks the value of our avatar self-portraits.

These avatar self-portraits generate various kinds of value, in terms of privacy, relationships, organization, access, and even money:

• Edward Snowden demonstrated that value in terms of privacy with the NSA revelations; why would the NSA spend all that money, time and effort collecting the information if it wasn’t valuable?
• Every time you talk with family, friends and work colleagues via Skype, you demonstrate that value in terms of relationships; why would you spend all that time on Skpye if the relationships were not valuable?
• Your user account on Apple’s iCloud is something that is of value to you to help you organize your information, schedule your day, and purchase the virtual commodities that help you navigate the city, purchases, and social network. That’s value expressed in terms of quality of life, quality of organization, and quality of access.
• And, most obviously, companies like Google, LinkedIn, Twitter, and Facebook mine our avatars’ data and build a cultural value system that is so valuable it is measurable in terms of billions of dollars per year.

Whether it’s privacy, money, identity, relationships, or money, the value of our self-portrait avatars is important to the very structure of society, and this has been the case for a very long time.

It used to be that only the very wealthy were able to afford portraits that lasted after their death and secured their position in history: a portrait hanging on a castle wall is evidence of the individual’s importance (both historically and financially) to the building itself; people used to pray to the portrait of the Pope; swear allegiance to the image of the king; and even today it is common practice in the US and many other countries to hang the portrait of the President or head of state in government offices.

But pop culture has a way of transforming things that were untouchable into things that can be handled and manipulated by the average person. It takes things that were ‘out of touch’ down off their pedestals and out of their glass cases, and makes them into something smaller that can be changed, edited and owned.

The avatar is a pop culture version of the self-portraits of classical history, and they allow us to change and distribute the portraits of ourselves as the royalty of the Renaissance used to do with their portraits. We use our avatars today to buy, sell, negotiate, advertise, rate, share, trade, learn, teach, train, talk, read, listen, write, and have fun with family and total strangers. Our avatars can do far more than a classic portrait can do because they are interactive, we can handle them, and they belong to us in many small pieces.

What if your avatar could talk for you? 

What if you had an autonomous avatar – a version of you that can talk for you when you’re not there –  that was your doppelgänger? What if it was able to protect your privacy, support your relationships, improve your quality of life, and make you money while you slept?

At Geppetto Avatars, we wonder about this next step. What does it mean if an avatar is able to talk without a person behind it pulling the strings? If you were to take all of the content you’ve written, and then provide your avatar with the ability to speak all that knowledge that bangs around in your skull, what would it say? If you had a self-portrait that could talk to you, what would you discuss with each other? If you could walk away from a mirror, and your reflection stuck to the glass, what would it say to the people it encountered after you left?

Now consider your job and the things that you do to make money in this world. Specifically, imagine the things that most bore you, and that you repeat daily. Imagine taking those words and gluing them, like a word-balloon, to that reflection of yourself in the mirror. Imagine that your avatar could discuss those things, share your knowledge, and contribute to a larger ecosystem in which you yourself no longer had to spend as much time doing these tasks.

Would it provide you with more privacy? Would you use it to interact with work colleagues? Or family members? Would it be a virtual assistant? What if it could answer your phone for you, or perform simple business tasks like ordering a shirt, calling a taxi, booking a reservation, letting someone know you’re running a bit late, buying a book from Amazon, calling to find out why your phone bill increased, looking for a good sandwich shop, and finding out what hours they are open?

Would we want these things? Would we learn about ourselves, or become more ignorant of what we like? Would we protect our privacy, or would we become vents of deprivatization?

Of course there is a nightmare that hides in these notions, a dark thing called polyopic heutoscopy – a psychotic state in which schizophrenics find themselves surrounded by their various alter egos. Could that nightmare, instead, be our future?

In the coming years these questions will be answered.

One day we will be able to build avatar self-portraits and set them loose in a larger ecosystem where they can speak with other people and avatars, to complete tasks for us and, much as a company might today, deliver messages, answer questions, listen, observe, collect information, organize information, barter, buy, sell, and serve as knowledge brokers and doppelgängers in an increasingly virtual world.

Today Geppetto Avatars builds healthcare avatars and instructional avatars that can discuss, listen, observe, and share information. They’re doppelgängers that liberate busy individuals from performing rote tasks, so that they can spend their time on higher value work instead.

Already there are a bunch of talking avatars running wild out there, and they’re turning out to be pretty valuable:

• Craig Mundie, Microsoft’s Chief Research and Strategy Officer showed how Microsoft’s Xbox Kinect and an avatar could facilitate group therapy sessions; people generally feel less intimidated by an avatar than by a human group leader.
• Research suggests that the experience of watching your digital look-alike uneventfully reach the front of the simulated line and order a pretend drink is real enough to help you learn to cope with similar situations in the actual world (thus reducing social anxiety).
• Virtual coaches can help people with PTSD.  The director of this project, Skip Rizzo, says: “It leverages latent inhibition. If you expose somebody to something in a controlled environment in advance, when they face that challenge in the real world, there’s less chance of fear conditioning — which is at the base of PTSD.”
• According to Prof Jeremy Bailenson, avatars are becoming more popular. Kids in the US spend about an hour a day inside an avatar. To put that in perspective, they read print for about 38 minutes a day.
• And this week Watson acquired Cognea (formerly My Cyber Twin),which means that we’re about to see a lot more of this kind of thing.

The avatar is becoming the medium of the future.  If your avatar could talk, what would it say?

All images courtesy Dito Von Tease / ditology Follow @DitoVonTease. If you liked this article, you may also be interested in:

• Facebook – Our ffriendly automated identity bender
• “A Paro walks into a club…” (or, Notes on the design of automated body language)
• The uncanny valet (or, Notes on the design of robot psychology)

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Audrow Nash interviews Max Rheiner from Zurich University of the Arts (ZHDK) about his  project Birdly. Birdly explores the experience of a bird in flight with several methods. Unlike a common flight simulator, the user embodies a bird, the Red Kite. To evoke this embodiment, Birdly mainly relies on the sensory-motor coupling. The participant can control the simulator with their hands and arms, which directly correlates to the wings and the primary feathers of the bird. Those inputs are reflected in the flight model of the bird and displayed physically by the simulator through nick, roll and heave movements.

Visualized through a head-mounted display (Oculus Rift), the whole scenery is perceived in the first person perspective of a bird. To intensify the embodiment, Birdly has additional sound, olfactory and wind feedback. The participant hears the roaring of the wind and the flaps of the wings. The olfactory feedback is based on the changing scenery and ranges from the scent of a forest, or soil, to several other odors of the wilderness. According to the speed of the bird, the simulator regulates the headwind with a fan.

Max Rheiner
Max Rheiner is a senior lecturer at Zurich University of the Arts (ZHDK) where he teaches bachelors and masters programs for the Department of Interaction Design. He also developed the Physical Computing Laboratory there. He received his Diploma from Zurich University of the Arts in the field of New Media Arts in 2003.

Rheiner’s research and artistic interests center on interactive experiences which utilize methods from Virtual/Augmented Reality and Immersive Telepresence. His artistic work has been recognized and exhibited in a number of international and well-renowned venues such as Biennale Venice, Italy, Ars Electronic Linz, Austria, and Yamaguchi Center for Arts and Media, Japan.

Links:

• Download mp3 (12.1MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Birdly’s website

“Why We Love Robots” is a 5-minute short documentary film co-directed by UC Berkeley Prof. Ken Goldberg and his wife, award-winning filmmaker Tiffany Shlain. The film was nominated for an Emmy Award and won a “Botscar” (Robot filmOscar) at the 2014 Robot Film Festival, and explores the human fascination with robots and new research trends in the field.

The film combines cultural references and found footage to explore our human fascination with robots and emerging research in Cloud Robotics and Belief Space planning.

Thematically, I think the natural world provides a wonderful contrast when held up against humanity and the technology we create. Describing a synthesis of high technology and the dirty, complicated natural world forces the reader to consider what makes something natural versus unnatural. The comparison also serves as a metaphor for what it might mean to have a relationship with a machine – another combination of natural and unnatural. In the end, I believe all technology is “natural” because it is an outcome of what human beings are naturally designed to do – build tools, and use those tools to build better tools.

Hope you enjoy the excerpt!

— Daniel H. Wilson

Part 3. Chapter 4: MIND OF LIFE

Neuronal ID: Takeo Nomura

What is a mind, but a pattern? My mind or yours. Man or machine. Simply an arrangement of atoms. Each of us, a unique expression of the mind of the universe.

Thoughts are precise bullets of electricity, fired through our neurons in timed pulses. Our bodies are layers of folding skin and muscle laced with fractal lightning. Natural, like veins on a rain-soaked leaf. Cracks in a tumbling stone. Or the sigh of this wave, lapping my shins until they are cold and numb. The clear liquid flows in, suspends the fine black hairs on my legs, and then retreats, laying the hairs down in new configurations. The sky leaks raindrops over my bony shoulders like Morse code.

We are patterns. Trapped inside other patterns.

After six months of listening, the sea sends an emissary. The towering bulk of a bizarre machine approaches our outermost seawall. A glistening spire, afloat, the size of a skyscraper. It forges ahead, slow and steady, through azure stripes of rain. From the salt encrusted on it, I would say it came from the open ocean. The dreamer only knows what this derelict has been doing out there on those endless blue plains.

I am staggered by its layered complexity.

Half submerged in Tokyo harbor, the spire is saturated with living things. It sprouts so far into the heavens that its upper reaches are shrouded in the rain haze. Some kind of muscular fiber makes up the main trunk, braided thick as the Tokyo Skytree, resembling bark but clearly with much higher tensile strength. It bears further study, as does the stability mechanism. The island-machine flutters delicately on flat fins the size of baseball fields, rising and falling, surfaces curling with wet sea grass on top and studded with barnacles below. Each pulse of the surf surges over the lip of the rear fin and washes straight through the marshy ecosystem.

“It is beautiful,” I say to Junshi-88.

The humanoid robot stands in the surf with me, trinocular lenses protruding at maximum zoom. It wears no human clothing, only a camouflaged green and black armored outer casing. Spurning human adornments is a mark of autonomy.

I hear the raucous squawking of seagulls from here. Hundreds of them circle the treelike structure, hunting the fish that swarm below in its safe harbor. The birds are nesting in the upper branches and have been for many generations, it seems, above clouds of insects. The fiber base is shit-stained and covered in seaweed, riddled with dens and nests and burrows. The voice of the sea has manifested itself to me in the most enormous and ancient form of life possible.

“Do you think it is shinboku?” I ask.

“Hai,” says the 88, its voice a grinding whine. Unlike the generations of overly polite robots that populated Japan before the New War, my freeborn ally is barely willing to speak in human audible frequencies, much less exercise impeccable manners. He is his own, not a servant, and there is no risk of me forgetting it.

I nod vacantly, staring. It must be a form of shinboku, a divine tree, honored by the monks and called upon to protect Shinto shrines. This mightiest of shinboku is beached here, as if lost from some other dimension. It forms a pattern so intricate that it places a gentle flame of awe into the pit of my belly. I am glad that I can simply coexist with a thing of such beauty and complexity. The shinboku has come from the unknowable flat wasteland of the open ocean, through tides of war, crafted by the voice of the sea and now sent into our harbor.

My equipment detects communications being relayed from hidden antennae located in the upper reaches of the tree. I slide a pair of modified binoculars over my eyes. Flip on a radio overlay of my own design. Scan the patterns until I find an alcove, nestled in the top. Some kind of control center is perched in the crook of two large branches, its entrance covered by sweeping vines. The binoculars reveal radio communications floating from the tower. Greenish wisps of communication meander over infinite glittering waves, to the horizon.

The voice of the sea is speaking.

I tap the 88 on its shoulder and we return to our little boat. Continue puttering up the Sumida River toward Nomura Castle. Behind us, the swaying island-tower watches balefully from where it rises out of the bay, the size of a movie monster kaiju.

Reaching the top of the shinboku will not be easy. I will need to retrieve my best tools if I wish to climb the tree of life. And returning to my workshop is, unfortunately, complicated. Mikiko will be there and she will disapprove of this mission. My place is on the throne, she says. I made my people a promise to protect them. I haven’t been back to the castle in weeks, staying on the streets with the Junshi-88.

The darkness settled over my people in the last months like silent falling strips of black silk. In the field of optics, they call the phenomenon a “just noticeable difference threshold.” A slight darkening of things. Each tiny gradation impossible to perceive.

Until the suicides began.

One month ago, I returned from a night expedition to the bay. In the frigid predawn, I had just docked my little wooden motorboat on the river. Junshi-88 was walking behind me in quiet pneumatic steps. It stopped. Ground out a verbal warning in Robspeak. My eyes lifted from the roadway and thoughts of the roiling sea evaporated.

A curious sight.

Nomura castle lies on a small hill, giving it a view of the surrounding Adachi ward. Scarred and leaning, its curtained walls of flash-welded steel and iron surround a star-shaped central keep. The roof of the castle is a curved square, the roofline bowed, edges thrusting out angrily like the horns of a kabutomushi beetle.

A little fellow was standing on the sweeping arch of the keep tower, his body sheathed in layers of hazy morning light, face empty, taking deep, slow breaths. Fish- scale flakes of armored roofing winked around him and I remember the roosting pigeons were giving him polite space.

“Not good, Junshi,” I murmured.

The crack of the young man’s bones on the castle steps was like the report of a pistol. We hurried to his body and tried our best to move him into a respectful position. With the fighting over, I could not imagine why the man would step away from this life. The akuma offer no more threat. But the war must have torn holes in our hearts. When it ended, no hope arrived to fill them.

“You must be very sad and lonely,” I whispered to the corpse. “But your friends will come for you soon and they will help put you to rest.” Junshi-88 blinked at me, processing my words. I do not know if it understood. By treating the dead as if they are living, we give them respect. We make life easier for those who remain.

Junshi-88 helped me arrange the body and it did not complain. I do not fear to touch a dismantled machine. On that morning I learned that this bravery can go the other way, too.

The young man was not the first to leap. Nor was he the last. Many of my people are falling. They are drowning. Hanging and suffocating and burning. I cannot say why my people are leaving us. I have never been good with emotions. But I can feel the wrongness of the empty act. The despair and meaninglessness that have settled over us like still, glassy water.

Without an enemy, we are falling forward. Nothing to push against. Flailing into empty space. We do not know how to start over. There is no route back to the beginning. The pattern of the world is torn. Living is the ruins of our memories is painful, and many would rather die.

My people may despair, but I do not.

For many years I have lived in a bare room with a woman and a workbench. A lamp and a chair and well-oiled tools spread out on a reed mat. Warm fingers on my shoulders. Hot tea. The bright smell of washed hair and the warm lingering scent of the soldering iron. It is a world of hope. From in here I can see the tools of rebirth everywhere. Each mangled wire or melted scrap is another piece of the puzzle.

I say a prayer as I cross the square where that nameless young man stepped out of the world. Beside me, 88 marches dutifully. The awakened military humanoid is a chilly friend. Never a recipient of my services. But Mikiko asked it to protect me, and as a freeborn, 88 takes her command very seriously. It required two nights before I became used to having it watch me sleep.

“There he is,” calls someone.

“Nomura!”

“What is in the harbor!?”
“Are we under attack!?”

My people are gathered. Each measured shout sends my head ducking lower and puts an extra scurry in my steps. Junshi-88 clears the way as we trot up the sweeping promenade of steel steps. They have been destroyed and repaired in a cycle these last three years. Burrowed under, demolished, heat-blasted, and soaked in the blood and oil of our defenders. The enemy akuma never made it inside. Not after that first time.

No one tries to stop my passage. I glance up, just once, and see that there are hundreds of my people outside the closed doors to the throne room. They are milling around and talking to each other in concerned whispers. A hush shudders through them like a wake as 88 pushes a path through.

I am not good at talking to them. Head down, as usual, I climb the steps. My workshop is still located in the main hall, on the same spot where I first knelt and began to work on a nearby senshi robot arm. Back when this place was an abandoned factory and not a shining fortress.

The 88 and I enter through an arched front door made of crosshatched steel beams. The fortified door gleams like the armored scales of some giant prehistoric fish. It was built by the great crane-arm senshi that rests now, coiled and deadly, high against the ceiling of what is now the throne room.

Slipping inside, I pad across the vast space. A neat corridor of senshi honor guard flanks the path to the throne. Each robot arm is folded in a salute, coated in glistening, nail polish–red paint. My terra-cotta army, always capable of animating, but not called on to defend the central keep in more than a year. Are they unhappy to be without purpose now? I wonder. Or will the time soon come again when they must build?

The scrap-metal throne is empty.

I leave the 88 behind and trot around the throne. My table and lamp are shoved against the stone foundation of the dais. Polished steel flooring whispers under my paper sandals. My amorphous reflection spreads below like a dark puddle on the metal. Quiet now. A little farther and I can make it out without alerting Mikiko.

Hastily, I stuff the tools I will need into a brown canvas backpack. Ransacking my work desk, I pull out all manner of tools and trinkets. Soon, the bag is bulging. Last thing, I grab my trusty toolbox and tuck it under my other arm.

“Mr. Nomura,” calls Mikiko.

Her voice stills my feet. My queen steps out from behind the dais that supports our thrones. I cannot remember the last time I climbed those steps to sit on that ostentatious chair. At some point, it has been decorated with a fan spray of sharp, twisting scrap metal, collected from our destroyed city.

“Mikiko,” I whisper.

“Did you notice your people outside?” she asks.

“Oh, uh. No,” I reply. “Too busy, in any case.”

Mikiko does not react to my obvious lie.

“Listen to me now. I cannot stop this darkness. The survivors need a human being to lead them. Someone who understands the despair they feel. An emperor.”

“No time,” I say.

“The rate of suicide is increasing,” she says. “I do not know how to help them. They need a purpose. You gave them that, once.”

“I’ve got to get back to the harbor,” I say.

“What are you afraid of, Takeo? Really?”

The question lingers, her synthetic voice echoing.

“Very busy,” I whisper, taking a step.

As I turn to go, she speaks: “My darling, you will never find what you have lost. The answers you seek are not in the sea. They are in here.”

I stop moving. My skin has gone cold. I am thinking of those wide, round little eyes. The crushing press of the wave against my back.

“It’s . . . a shinboku,” I say, voice shaking. “In the harbor. The voice of the sea has sent it here to me. I must find out why.”

“The platform that washed up? It is an artifact of war. Broken and derelict.”

“I am curious,” I whisper.

So difficult to explain, these patterns in my mind. The razed remains of Tokyo and the phantom images of buildings that are gone. The wailing ghosts of millions of dead, their bodies churned under the ground and burned to smoke on the wind and swept out to the bottom of the sea.

I need to understand it. I need to find the meaning in it.

“Trust me, my love,” I say. “The voice of the sea—”

“Is in your imagination!” she shouts. Her voice echoes from the thick rib supports that hold up the vast arched ceiling. “I am losing you to the past. To the same despair that is taking your people. Stop this madness. Come back, Emperor Nomura. Do your duty.”

Now there is exquisite emotion on her face. Anger and sadness. I know she places it there for me alone, an affectation. Each careless wrinkle on her face, every strand of gray in her hair puts a thumping into my heart. Passion and dread. I try to imagine returning to her side and ignoring the voice calling to me from the sea.

She is her own and you know it, old man. You are going to lose her.

The thought makes my fingertips numb, puts a thickness in my throat and a warm waiting tide behind my eyes. I cannot face it.

“This is my duty to myself,” I whisper, and scurry away.

Daniel Wilson will be at the Waucoma Bookstore in Hood River, OR on Friday, August 1 at 5pm as part of his Robogenesis book tour:

Waucoma Bookstore, 5PM
212 Oak St.
Hood River, OR, 97031 

Robohub is an online platform that brings together leading communicators in robotics research, start-ups, business, and education from around the world. Learn more about us here. If you liked this article, you may also be interested in:

• How will robots shape the future of warfare?
• If you could enhance yourself with some robotic accessory or implant, what would it be?
• The truth is no stranger than fiction when it comes to robots
• Robot tourism coming soon to Korea: Masan Robot Land project finally breaks ground
• Robot love: Spike Jonze’s new sci-fi film ‘Her’ may be closer to reality than you think

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.