Image: ESA/ATG medialab

In Darmstadt, Germany, European Space Agency (ESA) teams are scrambling to confirm contact with the Entry, Descent and Landing Demonstrator Module (EDM), Schiaparelli—the spectre of Philae still haunting the European Space Operations Centre (ESOC). Whether or not ESA ever speak to Schiaparelli again, the risky business of space robotics is once more laid bare.

Editors note: Since this article was written, ESA are attempting to decode a partial signal from the EDM

After Schiaparelli’s release from its mothership—The Trace Gas Orbiter (TGO)—the 577 kg EDM was programmed to autonomously perform an automated landing sequence, with parachute deployment and front heat shield release between 11 and 7 km above the Martian surface, all the time collecting data on the pressure, surface temperature and heat flux on its back cover as it passed through the atmosphere.  Then, 1100 m from the ground, retrorocket braking was supposed to slow the EDM before a final fall from a height of 2 m, protected by a crushable structure. But sometime during this process the signal was lost. ESA_Shiaparelli’s twitter feed soon fell silent also.

If Schiaparelli reached the surface safely, its batteries should be able to support operations for three to ten days, offering multiple opportunities to re-establish a communication link and collect data from its suite of sensors designed to measure wind speed and direction, humidity, pressure, atmospheric temperature, transparency of the atmosphere, and atmospheric electrification. But if contact is never restored, it’ll be a hefty blow to ExoMars and ESA’s aim of testing key technologies in preparation for their contribution to subsequent missions to Mars, including a planned 2020 rover that will carry a drill and a suite of instruments dedicated to exobiology and geochemistry research.

Image: ESA/ATG medialab

Such robotic technology has the potential to stretch our cosmological horizon, and perhaps even detect traces of extra-terrestrial life. But there’s a cost.  The ExoMars program runs up a bill totaling €1.3 billion, and such failures (if a failure it be) naturally raise questions of risk versus reward, particularly in tight economic times. Michael Johnson of the Pervasive Media Studio, Bristol, and founder of PocketSpacecraft.com, once told me during an interview: “About 50% of space missions fail after launch.” He shrugged. “That’s life… This is a high risk, high reward activity.”

In no other field of robotics is the potential of catastrophic failure so present and accepted. But how long can this attitude be maintained? The consensual public funding of—and indeed existence of—future space missions is parasitic upon the success of space robots and automated technology, and vice versa. Can you hear us Schiaparelli?

…

Come in, Schiaparelli.

…?

Image: ESA/ATG medialab

BUT ALL IS NOT LOST. The difficulties surrounding the mission’s EDM might potentially overshadow the successes of its TGO, which successfully performed the long 139-minute burn required to be captured by Mars, and has entered an elliptical orbit around the Red Planet. TGO’s Mars orbit Insertion burn lasted from 13:05 to 15:24 GMT on 19 October, reducing the spacecraft’s speed and direction by more than 1.5 km/s. The TGO is now on its planned orbit around Mars, primed with a suit of scientific equipment to measure the Martian atmosphere from on high. You win some, you lose some. This 50:50 success rate exemplifies the apparent nature of space robotics.

Click here to keep up to date with the latest ExoMars developments.

If you enjoyed this article, you might also be interested in:

• Drone learns to see in zero-gravity
• Robots Podcast #217: LunaRoo, with Jürgen Leitner
• The four coolest NASA robots
• Exoskeletons: From helping people walk to controlling robots in space
• NASA Space Robotics Challenge prepares robots for the journey to Mars

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

The successful landing of Philae on comet 67P/Churyumov-Gerasimenko this November was an extraordinary achievement and of course demonstrates – despite the immense challenges – that it is possible. The Philae mission was, in a sense, a proof of concept for cometary landing and this, for me, answers the question ‘what does it mean‘.

Of course there is a very large distance between proof of concept and commercial application, so it would be quite wrong to assume that Philae means that space mining (of planets, asteroids or comets) is just around the corner. Undoubtedly the opportunities are immense and – as pressure on Earth’s limited and diminishing resources mounts – there is an inevitability about humankind’s eventual exploitation of off-world resources. But the costs of space mining are literally astronomical, so unthinkable for all but the wealthiest companies or, indeed, nations.

Perhaps multi-national collaborative ventures are a more realistic proposition and – for me – more desirable; the exploitation of the solar system is something I believe should benefit all of humankind, not just a wealthy elite.

But politics aside, there are profoundly difficult technical challenges. You cannot remotely control this kind of operation from Earth, so a very high level of autonomy is required and, as Philae dramatically demonstrated, we need autonomous systems that are able to deal with unknown and unpredictable situations then re-plan and if necessary adapt – in real-time – to deal with these exigencies. The development of highly adaptive, resilient, self-repairing – even self-evolving – autonomous systems is still in its infancy. These remain fundamental challenges for robotics and AI research. But even if and when they are solved there will be huge engineering challenges, not least of which is how to return the mined materials to Earth.

Bearing in mind that to date only a few hundred kg of moon rock have been successfully returned* and Mars sample-return missions are still at the planning stage, we have a very long way to go before we can contemplate returning sufficient quantities to justify the costs of mining them.

*and possibly a few grains of dust from Japanese asteroid probe Hayabusa.

For other views on the Philae touchdown and the future of space mining and exploration, see:

• Touchdown! Rosetta’s Philae makes first ever landing on a comet
• One giant leap: Rosetta and the space mining frontier
• Space mining: Robots in the final frontier

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Still image from animation of Philae separating from Rosetta and descending to the surface of comet 67P/Churyumov-Gerasimenko in November 2014 (photo:ESA)

UPDATE: ESA’s Rosetta mission has soft-landed its Philae probe on to the surface of Comet 67P/Churyumov-Gerasimenko. The signal confirming the successful touchdown arrived on Earth at 16:03 GMT (17:03 CET), it’s the first time in history that such an extraordinary feat has been achieved and it’s a great milestone for space exploration and Europe. Read here the full press release from ESA.

LIVE VIDEO FEED | In a few hours a major milestone in space exploration will take place when the Rosetta orbiter deploys a small robotic lander named Philae towards comet 67P/Churyumov–Gerasimenko. This is the first time a spacecraft will reach the surface of a comet and, if successful, will greatly increase our knowledge in this field. Rosetta was launched over 10 years ago and followed an extremely complex trajectory with deep space maneuvers that used the gravity of Earth and Mars to propel it alongside its target and successfully orbit around it. Now the next step, Philae’s landing, will be transmitted live. See the video feed below.

Landing on the comet will be quite tricky: its weak gravity could allow the lander to bounce and escape into deep space. The shape of the surface was unknown until Rosetta came close enough so the landing spot was only recently selected. Philae is equipped with shock absorbers, spring loaded drills and even a harpoon that should be enough to keep it attached upon landing.

You can watch the live transmission from ESA below:

Tweets by @ESA_Rosetta

Overview of Philae lander instruments and their function, from DLR

Reaching 67P wasn’t easy. A rare Ariane 5 failure delayed its launch and the mission was repurposed to its current destination. The trajectory was extremely complicated with multiple gravity assist maneuvers, and required the spacecraft to enter hibernation mode, going into standby to save energy. Rosetta is powered by two huge solar panels that demand constant solar exposure instead. Other spacecraft (like Voyager 2) that went this far from the sun, used radioisotope generators.

The video below show the full trajectory of the mission:

An excellent overview of Rosetta/Philae mission is provided by the following Royal aeronautical society podcast. Paolo Ferri (Head of Mission Operations Department, European Space Operations Centre, ESA) introduces the Rosetta mission and its scientific objectives, describing also the spacecraft, its payload and its lander.

Additional resources:

ESA – Rosetta mission
ESA – Rosetta blog
DLR – Rosetta mission

#CometLanding (official hashtag)
@ESA_Rosetta
@Philae2014

Philae separating from Rosetta and descending to the surface of comet 67P/Churyumov-Gerasimenko in November 2014. (Credit:ESA/ATG medialab)

If you liked this article, you may also be interested in:

• One giant leap: Rosetta and the space mining frontier
• Space mining: Robots in the final frontier
• Astronaut on International Space Station successfully controls K10 rover on Earth, supporting use of telerobotics in future deep space missions

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

As researchers at the European Space Agency’s Advanced Concepts Team, we wanted to study how visual cues could be used by robotic spacecraft to help them navigate unknown, extraterrestrial environments. One of our main research goals was to explore how robots can share knowledge about their environments and behaviors to speed up this visual learning process.

© European Space Agency, Anneke Le Floc’h

Similar to the RoboEarth project, the central idea is that a group of robots sharing visual information such as raw camera images or abstracted mathematical image features would have a much broader visual experience to learn from than a single robot operating on its own.

We had already studied the use of motion-based cues in navigation, such as optic flow and the size changes of visually salient features like SURF, but we we wanted to verify our findings in this area and at the same time investigate if the appearance of visual features could be used to aid navigation as well. To this end, we needed a very large robotic data set of visually salient image features paired with corresponding robotic state estimates, and it was simply infeasible to gather all this data by ourselves.

That’s where the Parrot AR.Drone entered into play. The AR stands for Augmented Reality, and the drone is indeed a toy quad rotor meant for playing games. We thought we could develop a game for the Parrot that would serve as a means for crowd-sourcing the data we needed. And because the Parrot was designed to be used in augmented reality games and can be controlled by an iOS device, data-gathering could be done in a gaming context, making it fun for people to participate in the experiment.

To this end, we designed a mission-based game where players simulate docking the Parrot with the International Space Station as quickly as possible while maintaining good control. Bonus points are given for correct orientation and low speed on the final approach.

At the end of the mission, players can log their high scores on a score board, and at the same time contribute to the experiment by anonymously sharing abstract mathematical image features and velocity readings.

The Parrot AR.Drone is uniquely suitable for this purpose because:

• It is widely available to the public, with over a half-million units sold world-wide.
• It is a real robot with many useful sensors, including two cameras (at the front and on the bottom), sonar, accelerometers, gyrometers, and (on the Parrot AR drone 2.0) a pressuremeter and magnetometer. Further, it already uses these sensors to achieve accurate onboard state estimation and (consequently) stable hovering.
• It can be controlled by smart devices such as the iPhone or Android devices, which can be used send data over the internet.
• The code for communicating with Parrot is open source.

Methodology
The question we faced experimentally was how to couple the real-world object to the virtual space in which the drone would be flying. Having a reference object of known size was a goal since it would provide some form of ground-truth measurement. As it happens, the Parrot AR.Drone is delivered with a marker that is recognized in the images by the onboard firmware. The recognition gives an image position (x,y) and a distance to the marker. Note, however, that this information is not enough to disambiguate both the position and orientation of the drone with respect to the marker. Figure 1 shows two configurations that give exactly the same marker detection data. In the ‘red’ case, the marker is in the left part of the image, because the drone is located on the right of the marker. In the ‘blue’ case, the marker is in the left part of the image, because the drone has turned to the right. Note that the distances to the marker are the same.

Figure 1: Ambiguity of retrieved marker data. The angle β and distance d are the same for the red and the blue case, but their position / heading are different.

In order to render the virtual space environment, we perform state estimation on the basis of the drone’s estimates of its height, speeds, angles, and its detection of the marker. The docking port of the ISS (the marker) is assumed to be at location (0,0,0). The state estimation uses an Extended Kalman Filter (EKF), and in the light of the discussion of the marker’s ambiguity, we decided to only update the position on the basis of the marker’s readings, and not the heading. The heading is estimated quite well on board the drone, where the Parrot AR.Drone 2 also uses a magnetometer.

Figure 2 shows what computational processes are happening on which device during the game. The marker detection and the estimation of the drone’s attitude, height, and speeds are performed by the firmware on board the drone. The app is running the drone’s control interface, sending and receiving data from the drone, performing state estimation (X,Y,Z) with respect to the marker and rendering the 3D world.

Figure 2: Distribution of processes over the devices involved in the game.

Of course, for the experiment we also need to process the drone’s onboard images. Because of the iPhone’s computational constraints, we decided to save 5 subsequent images while the player is trying to dock and to process the images after the flight. This processing now happens when the player visits the highscore table and agrees to join the experiment. After extracting the vision data and concatenating this with the state estimates, the data is encrypted for transmission. Finally, since some players may use 3G for sending their data, we considered that the data should be as compact as possible. Therefore, the data is compressed on board the iPhone before being sent. The data from 5 textured images on average takes around 77 kb.

© European Space Agency, Anneke Le Floc’h

Testing and Launch
Our original plan was to work on the game from February to April 2012, launch it in May or June, and analyze the data from August to September. In reality, we didn’t converge on the final code until November 2012.

Of course, when you are developing the app, you know the program inside out and by habit you may avoid playing the game in such a way that would make the program malfunction. So in November we invited other people try out and rate the app. The ratings and comments were used to perform some final adjustments, and we sent the app to the Apple store for review in January 2013. At the same time, we began to execute our plans for the launch and related PR-activities. We saw this as an essential part of the project, as the “crowd” first has to know about a project in order to contribute to it.

The app was launched on the 15^th of March 2013, accompanied by a press release about AstroDrone from the European Space Agency and the video shown below. Parrot also promoted our app on their site. The news was soon picked up by BBC Technology, Wired, and the Verge, as well as news papers, and television shows such as ARD’s NachtMagazin. This (ongoing) media attention has helped us to reach the public: just one week after the launch, some 4,000 persons downloaded the AstroDrone app, and 458 of these people had already contributed to our experiment by sending their data. This is a nice start, but of course we hope there is more to come.

Next Steps
We have not arrived at the end of this project, but rather at the beginning. We are first solving some small issues by means of app updates. For example, if an iPad or iPod is connected to the drone’s WiFi, it cannot send its data. There was no message that showed the absence of an internet connection (solved in version 1.2). At the same time, we have started analyzing all the data that is coming in.

In the future, we may also improve our crowd-sourcing game in various ways, in order to make it as fun and easy as possible for players, and as useful as possible for our research. In any case, it’s an adventure to work with this exciting new way of gathering robotic data!

[Ed. Hallie Siegel]

Key dates:
Application deadline :15 March
Development phase : 4–12 April
Finalist down-selection phase : Beginning May
Competition event : Mid-October

As mentioned on the ESA website:

Robots will receive their instructions wirelessly. To make the competition more challenging, the robot operators will not have a direct view of the playing field but have to rely on video from four cameras in the playing area and from sensors on their robot, if they choose to equip their robots with them.

The playground is a scaled-down model of the International Space Station, starting at ESA’s Automated Transfer Vehicle (ATV) and finishing at the European laboratory, Columbus.

In under five minutes, the robots must unload cargo of different sizes from ATV, carry it to Columbus and stow it safely. Like in any game, there will be bonus points  and penalties.

The route to Columbus includes scaling a ramp and all robots must end the game in one piece to qualify. ‘Astronauts’ floating around the game area must be avoided and teams must cope with losing video signal for up to 15 seconds.

This is a sketch of the competition area layout:

The competition rules are available on pdf (Volare space robotics competition details – PDF ) and you can see a presentation of the concept by ESA astronaut Luca Parmitano:

Volare robotics challenge link – esa.int