Embodied communication: Looking for systems that mean what they say

The lay notion of how human communication works is, basically, that we exchange packages of information that are encoded and then decoded in our heads using language. But it just takes a little observation to see that this “computer metaphor” just doesn’t apply to the human way of communicating. We can’t reduce communication to a transfer of abstract information, and the same happens to be true of human-machine communication if we expect it to be of any use.

Think of turn-taking during a conversation. Speakers and listeners switch roles very quickly, often faster than typical reaction times. Silences between turns last less than 300 milliseconds, leading us to believe that a lot more is going on than simply waiting for the information to arrive in order to decide the next step. In fact, a lot of anticipatory planning is happening in both the speaker and listener. There’s a continuous monitoring of the other in search of subtle cues as to when the turn will be yielded. Such cues include changes of prosody, gestures, gaze shifts toward the listener, implicit verbal cues such as filler utterances…. Turn-taking is just one example of embodiment, a critical dimension of communication that most classical models ignore.

In fact it was the the study of human language that led the way toward the revolution of embodiment in the cognitive sciences. Cognition, in this view, unfolds in time while we perceive and act. Cognitive processes are continuously linked to sensory information and continuously coupled to motor systems that impact on the sensed world.

Communication is embodied in the obvious sense that natural communication occurs between people with physical and physiological bodies, and their cognitive abilities originate from processes in their nervous systems. But the embodiment stance to human communication is deeper than that. It extends the scientific approach to human language by adding a new emphasis on the grounding of language in the ongoing perception of the scene.

We begin to understand the emergence of language during development as a consequence of the immersion of the child in an environment shared with speakers. Development helps us explain how gesture and other non-linguistic cues are tightly coupled into language processing in the here and now of human communication. Embodied communication is also essential for researchers building artificial cognitive systems that autonomously generate language to communicate with a human and, conversely, make sense of human language and gesture by tying it to the system’s own scene perception.

Researchers exploring these different approaches to the study of embodied communication will share their ideas and insights during the Fifth EUCogIII Members Conference in Bochum 19-20 March 2014. Peter Dominey from the INSERM Stem Cell and Brain Research Institute in Lyon, France will show, for instance, how the iCub robot acquires “meaning” by establishing a link between physical action and language, and Yiannis Aloimonos from the University of Maryland in the USA will demonstrate how an “action grammar” is able to parse human action into meaningful sequences. Psychologists like Larissa Samuelson and Susan Wagner Cook from the University of Iowa will present the developmental perspective on embodied communication, focusing on how children learn words and how gesture and language co-develop. Srini Narayana will show in theoretical models how meaning and grammar are intimately tied together in the learning process.

Our aim is to advance the interdisciplinary interchanges between artificial and human cognition research fields. Both technical and natural aspects of communication will therefore be represented at the conference. To develop cognitive systems we need to understand the variety of ways that speech and gesture are coordinated, and develop technical aspects like creating automatic semantic descriptions of complex human activity from perceptual data. Knowing more about how humans really communicate and make sense of human communication is important if we want to create machines that are smart like us.

EUCog is the European Network for the Advancement of Artificial Cognitive Systems, Interaction and Robotics.

Gregor Schöner is the Director of the Institut für Neuroinformatik at the Ruhr-Universität Bochum in Germany.

Gregor Schöner is the Director of the Institut für Neuroinformatik at the Ruhr-Universität Bochum in Germany.