John McCormac discusses his takeaways from the RSS 2016 Workshop ‘Are the Sceptics Right? Limits and Potentials of Deep Learning in Robotics’ and highlights interesting themes and topics from the discussion.

As a PhD student in the Dyson Robotics Lab at Imperial, I was delighted to be given the chance by my supervisors Professor Andy Davison and Dr Stefan Leutenegger to attend RSS 2016.  The conference began with an extremely interesting workshop on the role of Deep Learning in robotics.  This post is by no means an exhaustive overview of that workshop, but I hope to highlight some of the themes and topics of the discussion that I personally found most interesting.

Larry Jackel, President of North C. preparing for his talk on the history of deep learning, and NVIDIA’s autonomous driving system. With organizer Niko Sünderhauf at the podium. Source: John McCormac

Where are the sceptics?

Overall it was an extremely interesting workshop, but I (and others) started asking ‘Where are the sceptics?’ Of the 150-200 audience members and panellists in the workshop only about 5-10 raised their hands when Pieter Abbeel directly asked the audience how many were sceptics at the end of the day. Many of those in the sceptics camp, when questioned, appeared still to endorse deep learning as being an important tool in the roboticist’s toolbox. One area of scepticism appeared to come from the lack of guaranteed behaviour in particular situations (discussed at more length below in the ‘Problems’ section), and closely tied to that notion is also the lack of interpretable uncertainty measures in many areas of DL (some solutions to this issue have already been worked on).

The dividing line between camps was obviously a fuzzy one, and some attempts were made to clarify the meaning of the term ‘sceptic.’ Larry Jackel had an interesting objective measure of a sceptic based on a famous bet (there were actually two bets, and he lost the other one) he won against Valdimir Vapnik, which was whether in ten years anybody in their right minds would be using neural nets. If anybody fell within this ‘sceptic’ camp in the audience, they did not openly voice that opinion.

A more expansive interpretation of the ‘non-sceptic’ camp came from Pieter when pressed by Oliver Brock. Pieter’s stance appeared to be that Deep Learning provided a new toolkit that had the potential to be a stepping stone beyond simply the ‘next task’ to a more general form of robotic intelligence. As a measure of success of Deep Learning, he in particular pointed towards the sheer number of breakthroughs that had occurred in the field of artificial intelligence within the past five years. There were some who worried this phase of optimism was potentially another boom before a bust (think AI winter) but Pieter argued the bust was much less likely to occur now given large companies already use these tools directly to generate revenue (Google’s Ad targeting system was specifically mentioned by Pieter).

The prevailing mood was a pragmatic approach to DL; that the roboticist’s task is to provide priors and model constraints to structure problems which can then be tackled with deep learning. Oliver Brock drew a comparison between the current state of deep learning in robotics and alchemy (he even described himself as a ‘proud alchemist’) – everybody has their black-magic methods of designing architectures. He argued the field as a whole should be aware of this, and should strive towards chemistry, or the ‘periodic table of robotics.’ He argued that researchers should keep in mind and highlight in papers the general applicability of their approach to other problems areas. Ashutosh Saxena’s vision of what this combination of networks and priors may look like in a robotic setting was principled factor graph approach where each factor can be a learned RNN. He believes forming this canonical factor graph structure also allows people to share components of different subtasks, both through time for a single robot and between different robots.

Dieter Fox was very enthusiastically in support of closely coupling deep architectures with prior models. He claims he was converted from sceptic to enthusiast by the robustness of techniques for hand pose estimation. He claims that although purely model-based approaches have the promise of confidence bounds, often in practical problems it is very difficult to achieve this, and the resulting algorithms themselves are very brittle. He highlighted ICP’s requirement of a good initialisation as an example of this brittleness for hand tracking, and said that single frame detection with a CNN appeared much more robust in this domain at least. He did not weigh in heavily on a question by Ben Upcroft on whether tracking was even relevant anymore and could be replaced entirely by detection, but instead focused again on the benefits of tighter coupling between model-based approaches and DL.

John Leonard began the introduction by saying that he was very much a sceptic until five months ago when he saw the work Larry Jackel has been doing with NVIDIA on autonomous driving (raw pixels to control commands) and stated that one ‘should ignore deep learning at their peril’. He did also highlight the importance of the work that has already went into traditional probabilistic inference in robotics, and that it could continue to form an important role in research to come.

Where is the training data?

Many talks touched upon the theme of where people get their training signal for these networks. There was a plethora of synthetic datasets in use for various experiments and validations. For RL tasks Pieter Abbeel highlighted the OpenAI gym for providing a platform for synthetic RL robotic challenges.

Experimentation on synthetic toy datasets before moving into more complex simulations and real-world data was the favoured approach by many. The work on Progressive Neural Networks by Raia Hadsell from Google Deepmind was a perfect example of this progression.  The work tackles continual learning by stacking and sharing columns of features learned in previous tasks and makes them available for future tasks (with locked weights to ensure the network does not catastrophically forget), but with additional columns of capacity to learn attributes specific to the current task. This work began by training on Atari games such as pong to explore and experiment with the architecture.  It then progressed to a simulated 3D robotic arm control problem, because the ability to generate orders of magnitude more training data as well as large hyperparameter searches, is far easier in the synthetic setting that for a real arm. The features from this synthetically trained network was then forced to bridge the ‘reality gap’ (the term used by Raia) to a real robotic arm task, where it learned faster and performed better on than a comparable system without the pretraining and PNN architecture.

Work presented by Theophane Weber Attend, Infer, Repeat: Fast Scene Understanding with Generative Models, also from DeepMind, aimed towards modelling 2D and 3D scenes in an object-centric latent space with Variational Auto-Encoder style generative models, also started with multi-MNIST toy datasets before moving to 3D synthetic scenes, and it too has very recently been tested on real images with a corresponding 3D model.

The projects from Dieters Fox’s lab followed a similar approach. Work on modelling fluids within a scene by Connor Schenck began by training on images from a blender fluid simulation before progressing to the real world. To move to the real world setting a very cool and clever method was used for collecting ground truth labellings. The fluid was heated and the video was monitored with a thermal camera to gain a robust pixel-wise labelling of the fluid vs. everything else in the scene.

Dieter also showed some very interesting work using dynamic fusion as a source of dynamic correspondences. He used this sort of training data to build siamese networks which labelled close points on a 3D dynamic model of a person in an embedded feature space and then used the networks labelling of this space on a new video to fit the old 3D dynamic fusion model to the new scene. He also stressed that in the robotic domain temporal information can provide an excellent supervisory signal for ‘free,’ his example for this was the SE3 physics prediction work by Arunkumar Byravan which is trained on information in the future. This is the same approach taken in the work presented by Ingmar Posner on End-to-End Tracking and Semantic Segmentation Using RNNs where it was the future that was the training signal for tracking objects through occlusions. This paper went on to win the best paper award.

For large-scale data collection in the domain of autonomous cars, John Leonard was also excited about the announcement by Tesla that it had collected hundreds of millions of miles of driving training data by recording both camera and sensor inputs (as well as I believe the driver’s actions). It was this sort of data that was used by NVIDIA and Larry Jackel to produce their autonomous ‘pixel-to-torque’ deep learned driving system.

Open problems for DL in robotics

One problem that was often touched upon was the issue of incorporating deep learned systems into real-world robotic products. Whether in domestic robotics, autonomous driving, or any other mass produced robotic application. On this there was some heated debate, the majority of people (all of the panel – Deiter Fox, Pieter Abbeel, Raia Hadsell, Ashutosh Saxena, Walter Scheirer, Larry Jackel, and Oliver Brock) felt that as enumerating every single possible scenario in testing was impossible, some set of standard failure tolerances would have to be set (e.g. number of crashes per million miles). Scheirer made the analogy to fault tolerances in a factory where you rely on statistics in testing. Nicholas Roy (who called DL ‘hyperparameterised function approximators’ everytime he spoke which became something of a running joke in the workshop) was very outspoken on this matter and seemed to be in favour of a more model-based approach as taken in airplane engineering and safety. Although many argued that in the domain of automobiles where the environment has the potential to become much more complicated a similar approach would not appear feasible, and given the huge number of avoidable human fatalities it would possibly be of great benefit even without the same level of rigid testing.

Walter Scheirer’s talk was an important datapoint for the question of robustness, it was all about the fragility of CNNs. I often found this in my research, for example, I give a picture with a cat and a dog to be classified and the CNN returns that it’s most likely a hair dryer. The network was obviously trained for single image classification, but the lack of robustness can sometimes be quite disappointing. He borrowed ideas from psychophysics to evaluate neural networks, testing how robust they were to image blurring and occlusion injection. It resulted in many state-of-the-art networks producing terrible results without hugely adverse degradations in the image from a human perspective and he, therefore said much of the hype about ‘super-human’ performance is really about bad metrics which fail to evaluate of the robustness of the algorithms. Pieter highlighted this issue as an important next step in the practical usability of DL and he argued it was likely to come about in the next five years, a robust switch that can highlight when a network is no longer equipped to deal with the current environmental ‘distribution’.

Pieter highlighted among other open problems, an interesting problem in reinforcement learning, namely the hierarchical nature of learning, and sparse rewards. If your agent spends a long time searching around the action space and finally figures out how to walk, only to walk into something dangerous or off a cliff, it would negatively impact all of the hierarchy of actions below it– do you give up on walking entirely?

Some other problems that were highlighted in the discussion panel were dealt with by quite funny one-word retorts. One person questioned the ability of these systems to surpass human level performance if we primarily had human guided training data, to which Oliver Brock replied ‘AlphaGo.’ Another question was whether online training was an important area of research, and whether we care that a network’s weights are trained offline and may never learn more as the move on. Raia responded with ‘Yes.’ She then did elaborate that this is the core challenge of much of the RL research that they are working on at Google Deepmind.

The optimistic future for DL in robotics

Pieter ended with a ‘crazy’ thought that outlines a very exciting picture for the future of deep learning in robotics. He argued that our brain is not significantly more evolved than when the primary tasks demanded of it were to simply find food, make shelter etc. and a lot of the amazing advancements of intelligence possibly just ‘fell out’ of this task set. This thought underlined a sense of optimism that deep learning still has a lot of mileage to be had in robotics and that the solutions found there could have much greater implications for artificial intelligence in general.

Machines have been beating humans for years in games and contests: chess, Scrabble, checkers, Jeopardy! and more recently, the Chinese board game Go. Google’s AI machine won 4 out of 5 games in a $1.3 million contest against South Korean Lee Sedol, rated as the world’s best Go player. Sedol comments to ABC News at a post-game press conference:

I didn’t expect to lose. I didn’t think AlphaGo would play the game in such a perfect manner. Personally, I am regretful about the result, but would like to express my gratitude to everyone who supported and encouraged me throughout the match.

The 2,500-year-old board game is purported to be more complex than chess. It is played with black and white stones and the aim is to surround more territory than the opponent. Go appears to be a simple game and begins with an empty board. Two players (one using black stones, the other white), alternate placing stones in squares, trying to grab territory without getting their pieces captured.

According to Alan Levinovitz, writing in Wired a few years ago, “there are 400 possible board positions after the first round of moves in Chess and 129,960 in Go. There are 35 possible moves on any turn in a Chess game, and 250 for Go.” DeepMind’s David Silver and Demis Hassabis noted that the number of possible board configurations in Go is larger than the number of atoms in the universe.

Drake Baer, in a Tech Insider piece, describes DeepMind’s history and role in the development of AlphaGo:

DeepMind didn’t “program” AlphaGo with evaluations of  “good” and “bad” moves. Instead, AlphaGo’s algorithms studied a database of online Go matches, giving it the equivalent experience of doing nothing but playing Go for 80 years straight.

“This deep neural net is able to train and train and run forever on these thousands or millions of moves, to extract these patterns that leads to selection of good actions,” says Carnegie Mellon computer scientist Manuela Veloso, who studies agency in artificial intelligence systems.

Google acquired DeepMind in 2014. Founded in 2010 by chess prodigy-turned-artificial intelligence researcher Demis Hassabis, the company’s mission is to “solve intelligence,” and it claims that “the algorithms we build are capable of learning for themselves directly from raw experience or data.”

In February 2015, DeepMind revealed in Nature that the program learned to play vintage arcade games like Pong or Space Invaders as well as human players. Now it’s about to master a game that once seemed unmasterable for artificial intelligence [Go].

A Google DeepMind paper describes the technical details. You can also watch the Nature video below with DeepMind’s Demis Hassabis, who then describes the process required by the AI.

What does it all mean?

Go’s appeal lies in depth through simplicity, which is why its difficult for computers to master up — until now. There’s limited data available looking at the board. Choosing a good move demands a great deal of intuition. Until AlphaGo, no one had built an effective evaluation function. But AlphaGo’s use of deep learning and neural networks to teach itself enables it to process millions of Go positions and moves from human-played games, then deciding upon moves predicted on that experience.

Sam Byford writes in The Verge:

The twist is that DeepMind continually reinforces and improves the system’s ability by making it play millions of games against tweaked versions of itself. This trains a “policy” network to help AlphaGo predict the next moves, which in turn trains a “value” network to ascertain and evaluate those positions. AlphaGo looks ahead at possible moves and permutations, going through various eventualities before selecting the one it deems most likely to succeed. The combined neural nets save AlphaGo from doing excess work: the policy network helps reduce the breadth of moves to search, while the value network saves it from having to internally play out the entirety of each match to come to a conclusion.

Thus, AlphaGo gets better by playing itself. DeepMind believes that the principles it uses in AlphaGo have broader application and implications. Hassabis makes a distinction between “narrow” AIs like Deep Blue and artificial “general” intelligence (AGI), the latter being more flexible and adaptive. He thinks AlphaGo’s machine learning techniques will be useful in robotics, smartphone assistant systems, and healthcare.

Intuition has been the sole domain of humans. But now AlphaGo, Google, and DeepMind have shown they can build systems with an effective evaluation function to make choices based on a growing universe of experiential data. Reading medical scans and upgrading Siri (and other online resources and social assistants) are just the tip of the iceberg. This technology has implications for self-driving vehicles of all types (air, land and sea), medical diagnostics, and predictive stock picking to name just a few.

If you liked this article, you may also enjoy:

• Google DeepMind challenge match: Lee Sedol vs AlphaGo
• Talking Machines: Machine learning in healthcare and the AlphaGo matches

AlphaGo playing challenger. Source: Google DeepMind

Update: The final match was played on Tuesday, 15 March, with AlphaGo winning over Lee Sedol 4-1! Watch all the livestream matches below.

In October 2015, AlphaGo became the first computer program ever to beat a professional Go player by winning 5-0 against the reigning 3-times European Champion Fan Hui (2-dan pro). That work was featured in a front cover article in the science journal Nature in January 2016.

If you like this article, you may also enjoy:

• To err is human: raising the bar on AI
• Talking Machines: Machine learning in healthcare and the AlphaGo matches
• Talking Machines: Gaussian Processes and OpenAI, with IIya Sutskever

Google DeepMind Challenge Match: Lee Sedol vs AlphaGo. Credit: Deepmind/YouTube

In episode five of season two, Ryan walks us through variational inference. We put some listener questions about Go and how to play it to Andy Okun, president of the American Go Association (who is in Seoul, South Korea, watching the Lee Sedol/AlphaGo games). Plus, we hear from Suchi Saria of Johns Hopkins about applying machine learning to understanding health care data.

If you like this podcast, you may also be interested in:

• Talking Machines: AI safety and the legacy of Bletchley Park, with Nick Patterson
• Talking Machines: Robotics and machine learning music videos, with Michael Littman
• Talking Machines: Women in Machine Learning, with Hanna Wallach
• Talking Machines: Gaussian processes and OpenAI, with IIya Sutskever
• Talking Machines: Real human actions and Women in Machine Learning (WiML), with Jenn Wortman Vaughan

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

AlphaGo playing challenger Fan Hui. Source: Google DeepMind

Last week, media outlets reported about the latest ‘sure sign’ of an imminent robopocalypse. Google-backed DeepMind created an AI sophisticated enough to beat a human champion in the Chinese board game Go. AlphaGo defeated European champion Fan Hiu, surpassing IBM’s Deep Blue chess-playing victory over Gary Kasparov in 1997, marking another milestone of development and progress in AI research. Go is a more difficult game for humans to master – and it was thought also computers –  because of its complexity and need for players to recognise intricate patterns.

When I set off to write a note about this achievement I expected to find the usual sources in popular press with their characteristically subtle declarations, heralding the End of the Human Race is Nigh! However, thankfully, responses seem to be more sanguine and muted. The British tabloids even avoided using that picture of Terminator that invariably accompanies their reports on new developments in AI and robotics.

Perhaps this is a sign things are changing, and that the popular press are becoming more sensible and responsible in their technology reporting. Well, maybe lets see how many weeks – or days – we can go without this sort of thing before claiming victory, or remarking we’ve turned a significant corner.

From a cultural perspective, there is a lot of interest about DeepMind’s success, even if it hasn’t stirred the usual panic about the robopocalypse. It made me recall a conversation from an EURobotics event in Bristol in November. We humans like to think that we’re special. And maybe the possibility that robots or AI are a threat to that special status is another reason why we are so afraid of them. Maybe we fear another blow to our narcissism, like when that crazy astronomer Copernicus spoiled things by showing that Earth wasn’t the centre of the Universe. Or when Victorian poo-pooer Darwin demonstrated we evolved on Earth and weren’t placed here at the behest of some Divine Creator. Perhaps we don’t really fear that robots and AI will destroy all of humanity – well maybe we fear that, too – but perhaps part of the fear is that robots and AI will destroy another of those special things we reserve for ourselves as unique beings amidst creation.

Yet, our scientists aren’t going to let us sit wrapped in the warmth of our unique being. They keep pushing and developing more and more sophisticated AI that threatens our… specialness. How do we as a culture respond to this persistent challenge? Like any good politician, it seems we have decided to confront the inevitability of our failure by constantly changing the rules. Choose your sporting metaphor such as: we ‘move the goalposts‘ or we ‘raise the bar’.

Once upon a time it was enough to think of ourselves as the rational animal, the sole species on Earth bequeathed with the capacity for reason. Then, as evidence we could reason as the basis for unique status for humanity crumbled – both thanks to proof that animals were capable of sophisticated thought and the lack of proof that humans were rational – we shifted those goalposts. We transformed ourselves into the symbolic animal, the sole species on Earth endowed with the capacity to manipulate signs and represent.

We learned that whales, dolphins and other animals were communicating with each other all the time even if we weren’t listening. And that’s before we taught chimps how to use sign language (for which Charleton Heston will never thank us).

Then computers arrived to make things even worse. After some early experiments with hulking machines that struggled to add 2 + 2, computers eventually progressed to leave us in their wake. Computers can think more accurately and quickly than any human. They can solve complex mathematical equations therfore demonstrating they are pretty adept with symbols.

Ah, BUT…

Supercomputer Watson and human challengers on Jeopardy. Source: youtube

Humans could find solace in the comforting thought that computers were good at some things, yes, but they weren’t so smart. A computer could never beat a human at chess, for example. That is until May 1997 when chess champion Gary Kasparov lost to IBM’s Deep Blue.  But, that was always going to happen. A computer could never, we consoled ourselves, win at a game that required linguistic dexterity. Until 2011, when IBM’s Watson beat Ken Jennings and Brad Rutter at Jeopardy! the hit US game show. And now, Google’s DeepMind has conquered all, winning the most difficult strategy game we can imagine.

What is interesting about DeepMind’s victory is how human beings have responded again to the challenges of our self-conception posed by robots and AI. If we were under any illusion that we were special and alone among Gods’ creations as a thinking animal, a symbolising animal, or a playing animal, that status as been usurped by our own progeny again and again, in that all too familiar Greek-Frankenstein-Freudian way.

Animal rationabile had to give way to animal symbolicum, that in turn gave way to animal ludens. What’s left now for poor, biologically-limited humanity?

Lieutenant Commander Data in Star Trek. Source: youtube

A glimpse to this latest provocation can be seen in Star Trek: The Next Generation. Lieutenant Commander Data is a self-aware android with cognitive abilities far beyond any human being. Despite these tremendous capabilities, Data is always regarded – by himself and all the humans around him – as tragically, inevitably inferior, less than human. Despite lessons in Shakespeare and sermons in human romantic ideals from his mentor Captain Jean-Luc Picard, Data is doomed to be forever inferior to humans.

It seems now that AI can think and solve problems as well as humans, we’ve raised the bar again, changing the definition of ‘human’ to preserve our unique, privileged status.

We might now be animal permotionem – the emotional animal – except while that would be fine for distinguishing between us and robots at least until we upload the elusive ‘consciousness.dat’ file (as in Neill Blomkamp’s recent film, Chappie)  this new moniker won’t help us remain distinct from the rest of the animals. Because to be an emotional animal and a creature ruled by impulse and feeling is.. to just be an animal, according to all of our previous definitions.

We might find some refuge following Gene Roddenberry’s example in the notion of humans as unique animal artis: the animals that creates and engages in artistic work. The clever among you will have realised some time ago I’m not a classical scholar and my attempts to feign Latin fell apart some time ago. Artis seems to imply something more akin to ‘skill’, which robots arguably have already achieved; ars simply means ‘technique’ or ‘science’. Neither really capture what I’m trying to get at, suggestions are welcome.

The idea that human beings are defined by a particular creative impulse is not terribly new. Attempts to redefine ‘the human’ along these lines have been evident since the latter half of the twentieth century. For example if we flip back one hundred years, we might see Freud defining human beings (civilised human beings to clarify) as uniquely able to follow rules. But by the late 1960s, Freud’s descendants, like British psychoanalyst D. W. Winnicott, are arguing the exact opposite – what makes us human is creativity and the ability to fully participate in our being in an engaged, productive way.

What’s a poor AI to do? It was once enough for artificial intelligence to be sufficiently impressive maybe even deemed ‘human’ if it could prove capable of reason or symbolic representations, win at chess, Jeopardy!, or Go. Now we expect nothing less than Laurence Olivier, Lord Byron, and Jackson Pollack all in one.

https://www.youtube.com/watch?v=FcZ2NcCEC0o

This reminds me of Chris Columbus’s 1999 film Bicentennial Man (based on a story by Isaac Asimov). Robin Williams’s Andrew Martin begins his life, for lack of a better word, as a simple robot. Over the decades becomes more like a human and sentient, demonstrating artistic skill and feeling genuine emotion. At each stage he hopes to be recognised as a being on par with humans. No, he’s told, you’re not sentient. When he’s sentient he’s told he cannot feel. He’s told he cannot love. No achievement is enough.

Even once he has achieved everything and become like a human in every respect, perhaps even superhuman, he is told it is too much and has to be less. In a reversal of the Aristotelian notion of the thinking, superior animal Andrew is told to make mistakes. He is too perfect. He cannot be homo sapien, he needs to be homo errat, the man that screws up. It is not until Andrew is on his deathbed that the Speaker of the World Congress declares world will recognise Andrew as human.

Perhaps this will be the final line and definition of human that will endure every challenge posed by robots and artificial intelligence. No matter the level of technological progress, regardless how far artificial life leaves human beings behind: we will be homo mortuum. The rational animal that can die.

If Singularity enthusiasts and doomsayers are to be believed, this inevitable self-conception is not long off. Perhaps humans’ strength in our ability to adapt and the talent to re-invent ourselves, could mean there’s life in the old species yet. Regardless, it will serve us very well to create a conception of both ourselves and of artificial life forms that try to demarcate the boundaries and decide when these boundaries might be crossed. And what the implications will be for crossing that line.

This post originally appeared in Dreaming Robots.