https://youtu.be/xKFaYuBF9HU

Imagine an assistant that knows your schedule, contacts, interests and location, and that can communicate with you to help you stay informed, connected and on top of your game. This is more personalized than your PC, tablet or smartphone and, depending on whether physical or virtual, is in a pleasing, personal and compatible form. If these assistants catch on, they will begin anticipating and handling many of the little decisions that fill our days. The potential for this form of artificial intelligence has tech companies vying to lead the pack.

Apple’s Siri, Google’s Google Now, Microsoft’s Cortana, Facebook, and Baidu are all in various states of pursuing virtual assistants. But JIBO, Pepper, Cubic, EmoSpark and others are just now coming to market and are offering a physical component – a robotic presence – which they hope will enhance the experience.

2015 is the year physical products will be coming to market and available for experimentation and testing. Pepper ships in the summer in Japan, JIBO ships preorders in Q3 as does Cubic in the fall and EmoSpark in the summer.

These types of personal assistants rely on multiple activities including: speech recognition to determine and parse through spoken commands; predictive technology to anticipate what a user will likely do next; and determining and using contextual data (biographic data and situational information, e.g.: location, time and date). Apple, Google and Baidu – all of which attempt to provide this service – want their intelligent interface to fit all front ends, to be the universal contact point. They see a plentiful revenue stream if they are successful. Others want Apple and Google to redirect users to company sites with apps that understand their own content better, e.g.: directing requests for pizza to Dom, the digital front-end for taking Dominos Pizza orders.

Many are waiting to see how the competition plays out. Car companies and other smart product providers, fearing a variety of consequences outside of their control, aren’t enamored of letting Apple or Google (or any of the others) run their voice-powered front-end systems … thus their current intense race for in-house and startup development. But if Google or Apple or ??? gets to market first with a true digital assistant that people embrace and buy – one that can be virtual or physical – car companies may need to make their systems adapt to that system because it’s what everyone wants to use.

The key to the outcome of this race is whether a general purpose AI will be able to steer people through their digital world, or whether users would rather navigate to applications that are specialists (such as American Airlines or Dominos Pizza).

Natural language recognition – and the ability to analyze user desires from that language – is a very competitive science these days. Nuance, the biggest provider of voice and language solutions to the auto industry and a collaborator with IBM up until recently, is now competing with IBM’s Watson team and also Apple, which is rumored to be working on their own language processing system instead of using Nuance. A look at the What’s Next section of Nuance’s website is, to quote them, “The intersection of Science Fiction, super-pi, and technology innovation.” But a more descriptive quote from that page seems more appropriate: “Intelligent self-service delivers a better customer experience.” That’s what this race is all about.

• The second wave of social robots is here
• Amazon’s Echo: A point-of-sale even the kids can relate to
• The next era of conversational technology
• The death of search (or, My dysfunctional relationship with Siri)

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Pepper robot, by Aldebaran.

Social Robotics: The UX of Natural Language

A very smart and snotty journalist told me that he had spoken with Aldebaran’s Pepper system when he was in Boston. Much to his chagrin, the robot didn’t reply when the journalist spoke to it. “It didn’t understand a word I was saying,” said the journalist. “It simply doesn’t work.” The only word it understood, he said, was “sushi” … at which point I realized that he was speaking with the system in English, but the robot was programmed to understand Japanese. 

This is a problem of user experience, or UX.

Natural language interfaces are turning into a de-facto interface convention. Just like the GUI overlapped and largely replaced the command line, NLP is now being used by robots, the Internet of things, wearables, and especially conversational systems like Apple’s Siri, Google’s Now, Microsoft’s Cortana, Nuance’s Nina, Amazon’s Echo and others. These interfaces are designed to simplify, speed up, and improve task completion. Natural language interaction with robots, if anything, is an interface. It’s a form of UX that requires design.

The next phase of UX is psychological: Natural Language personalities.

We all use language on a daily basis as our key interface with other people, so expectations of how to talk are as deep as they are diverse. We all seem to have, from time to time, problems talking with one another, so designing talking interfaces is a hard UX job. Topic management, dialogue turn-taking, segue management, association with iconic gestures, and thousands of other aspects of communication sit right in the middle of this emerging design discipline.

Traditionally design, even software design, employs the famous architectural adage of, “Form follows function.” But when it comes to natural language interfaces, and robotics in general, we enter a new kind of design in which function follows form. The form of human interaction becomes the function in making something metaphoric, simple, fun and useful.

When it comes to human-robot interaction – and social robots in particular – the form defines the function. There are at least three reasons for this I’ll list here. All of them are psychological.

Social robots should look like us.

First, a user should identify with the conversational system. Any school age child will tell you that there’s no pragmatic reason to make a computer look anything like a person. The android seems a fool’s design. It’s off-balance, clunky, the sensors are too high, walking is hard to calculate, and when the robot falls, it takes a serious digger damaging the sensors that are perched at the part of the robot that gets the hardest knock when they fall. And what is a head for, anyway?

It turns out that babies (and adults, and school age children too) are wired to recognize even an abstracted human face surprisingly quickly and at a surprisingly early age. Babies, when they’re still wee things, are drawn to facial imagery, even if it comprises just two dots and a curvy line. I tried this on my own piglet when he was 5 months old. I gave him a drawing of a circle with the two dots and a curved line, and another of a square with a single dot and two lines. He went back and forth between the two drawings, but spent at least six times longer staring at the face drawing.

Which drawing do you prefer to look at?

What is even more curious is that 6-month olds are more likely to spend time looking at what you or I might call an attractive face (see e.g. here and here).

So it stands to reason that a social robot should have an attractive face, simply because humans are wired to pay attention to it. And this is part of the metaphor that is driving android design.

But beauty is only skin deep. While we wrestle to escape the uncanny valley of visual appearance, we are learning that there are many uncanny valleys of not just appearance, but sound, color, timing and even psychology. Maybe psychology, most especially.

Social robots should speak like us.

Second, now that NLP technology is generally working well enough to complete end-user tasks and provide contextual training, it’s time to start addressing personality in the same way that we’ve addressed appearance. This means that, just as we need to make sure that we design robots that look like us, we need to also design robots that speak like us.

This simply means that the user experience of natural language must consider psychology and human-like interaction as the core metaphor for the design of social robotics. It’s like making a glove that is shaped like a hand – the tool has to be designed to work with how we’re built. So psychology is key.

‘Birds of a feather flock together’ for some very deeply ingrained reasons associated with trust. We’re accustomed to trusting people that look like us and when we trust someone we actually perceive them to look more like us.

So talking and looking like a human is pretty important for a social robot.

This linking of behavior and appearance is key to user experience. Systems such as those used in healthcare, finances, and other very important interfaces will need to be designed such that they’re reliable, and this means they have to act like the user. Users will scarce be found who are willing to tell their financial data, or discuss private healthcare problems, with some fictional character that looks like a Furby, or a spinning ball of metal. And, to make matters more challenging, the user experience around natural language interfaces will need to address localization issues that include such things as accent, argot, and other manners of speech that are linked to particular regions of a country.

We’ve found in our work at Geppetto Avatars that people are more willing to overlook uncanny elements of design if there are slight cultural variations thrown in. For example we’ve noticed that synthesized voices sound less artificial if they have a slight accent. A synthesized voice with a British accent doesn’t sound quite so synthetic to an American listener. But if you give that same listener a synthesized voice that uses an American accent, the listener is far more likely to notice the synthetic elements. This is simply a UX technique to off-load technical challenges. So UX can serve the role of simplifying not only the interaction, but the technology as well.

Social robots should be polite.

Robots – or other connected systems – that use Natural Language interfaces create spontaneous behavior in the user. Most users actually try to be polite or cooperative, and even start attributing personality to the system (such as humor, aggressiveness, and gender). We all talk to our car, many of us have named it, and that’s how we’re wired. This is something people automatically do, like it or not, with not just natural language systems but almost all forms of media.

In 1996, Reeves and Nass, two famous researchers from Stanford (and authors of The Media Equation), proved that computers elicit social interactions in users. Reeves and Nass designed an experiment in which 22 people come into a lab to work with a computer via natural language text interface. At the end of the session Reeves and Nass asked them to evaluate the computer used and how well it performed. Reeves and Nass found that users had automatic social reactions during the test, and opinions about the personality of the machine they were using.

They ran the test again, this time via natural language voice interface, to make the human-social theme more evident. The test results were the same. They concluded that people are polite to computers, both in speech and writing, and the experiment showed that social rules can apply to media and computers can be social initiators. Participants denied it, but the results said otherwise.

Try it yourself – watch people talk with Siri and they use the word “Please” or “Thank you” more often than you might guess. Then go buy The Media Equation. It’s a great read.

When we design systems that are able to build identification and trust with a user these systems are able to make advice about healthcare and how to use our bodies, or finance and how to use our money. These are powerful systems because they are actually modifying what the user does. The conversational system has a power over the user simply in making advice that is amplified by how we are wired. If ethics is all about power then being polite is the logical interface to ethics.

Interface, whether command line, GUI, or natural language, has always been the most potent and valuable element of a computer design. Interface will continue to be an important and valuable element of robotic design as well. The future of UX is psychological as much as graphical and ethical as much as functional. Ask your car.

UX practitioners and robotics designers need to consider the form and the function, especially when it comes to natural language interfaces. Otherwise, 私達はちょうど悪い寿司の貧しい翻訳があるでしょう。*

[“we’ll just have poor translations of bad sushi.”]

Google’s recent acquisition of Emu Messenger is just one of many items in recent news about improvements in perception and artificial intelligence (AI).Emu, not to be confused with the Australian ostrich-like bird nor the European Monetary Union, is a small Palo Alto start-up comprised of some serious software talent with experience in machine learning, natural language processing and mashing up different data, databases and systems at Siri, Apple, AOL and Google. Perception in this case is the feeling that your words and intentions are understood and acted upon. No financial details were disclosed about the acquisition however the Emu app will be shut down next week.

Another form of perception, computer eyesight (AKA machine vision), took a giant step forward when the winners were announced for this year’s Large Scale Visual Recognition Challenge. The NY Times reported the winners: the National University of Singapore, the Oxford University, Adobe Systems, the Center for Intelligent Perception and Computing at the Chinese Academy of Sciences, as well as Google in two separate categories.

Fei-Fei Li, director of the Stanford Artificial Intelligence Laboratory said, “What really excites us is that performance has taken a huge leap.”

Machine vision has been a challenge in automation and robotics but in recent years has become integral in countless applications including computer gaming, medical diagnosis and factory robotics. Carmakers too have added the ability to recognize pedestrians and bicyclists and trigger safety actions. Factory robots need much improved perception systems in order to monitor the progress of their tasks and the tasks of those around them. [A quick Google Images search produces a collage of various uses, from milking to adaptive cruise control.]

Enhanced algorithms, data libraries and faster and cheaper computing are all contributing to the increased accuracy and speed of the systems recognizing objects and identifying them by type and in 3D space, nevertheless, at their best they are still no match for human vision and perception.

In 1922, the Elmwood Button Co. commercialized a toy that could respond to voice commands. The toy, called Radio Rex, was a dog made of celluloid that would leap out of its house when you called its name: “Rex!” That was two decades before the first digital computer was even conceived, and at least three generations before the Internet would jet our lives from the real world into the virtual realms of e-commerce, social networks, and cloud computing. Radio Rex’s voice recognition technology, so to speak, was quite crude. Practically speaking, the vibration produced by the “r” in its name, when spoken loudly, would cause a thin metal blade to open a circuit that powered the electromagnet that held Rex in its house, causing Rex to jump out. A few years ago, I watched in awe one of the few remaining original Radio Rex toys in action. It was still working.

Some people recognize Radio Rex as the first speech recognition technology applied to a commercial product. Whether that could be called speech recognition or not, we have to agree that the little celluloid dog was probably one of the first examples of a voice controlled consumer robot. While rudimentary, I can imagine the delight and wonder that people felt when Rex responded to their own voice. Magic!

Indeed, speech recognition is something magical. That’s why I have spent more than 30 years of my life building machines that understand speech, and many of my colleagues have done the same. The machines we have built “understand” speech so well that there is a billion dollar industry based on this specific technology. Millions of people talk to machines every day, whether it is to get customer support, find a neighborhood restaurant, make an appointment in their calendar, or search the Web.

But these speech-understanding machines do not have what Radio Rex had, even in its own primitive simplicity: an animated body that would respond to voice. Today, social robots do. And even more: they can talk back to you, see you, recognize who you are and where you are, respond to your touch, display text or graphics, and express emotions that can resonate with your own.

When we consider voice interaction for a social robot, what type of speech and language capabilities it should have? What are the challenges? Yes, of course Siri and Google Now have paved the way to speech-based personal assistants, but despite their perceived popularity, is speech recognition technology ready for a consumer social robot? The answer is yes.

First, let me explain the difference among the speech recognition technologies used for telephone applications (à la “Please say: account, payments, or technical support”), for personal assistants (like Siri, Google Now, and the most recently Cortana) and for a social robot.

In telephone-based applications, the speech recognition protocol is synchronous and typically initiated by the machine. The user is expected to speak only during well-determined intervals of time in response to a system prompt (“Please say …”). If the user does not speak during that interval, or if the user says something that is not encoded in the system, the speech recognition machine times out, and the system speaks a new prompt. This continues until something relevant is said and recognized, or either the machine or user lose their patience and bail out.  We have all experienced this frustration at some point.

In telephone-based systems, any type of information has to be conveyed through the audio channel, including instructions for users on what to say, the information requested, and the possible clarification of any misunderstanding between the human and the machine (including the infamous “I think you said …”). Anything spoken has to be limited in time, in a rigid, turn-taking fashion between the system and the user. That is why telephone-based systems are so “clunky”, to the point where more often than not, they are quite annoyingly mechanical and repetitive. Although I have built some of the most sophisticated and well-crafted telephone-based speech recognition systems for over a decade — and these systems are used by millions of people every week, — I never heard anyone say: “I love my provider’s automated customer care service. It is so much fun to use it that I call it every day!”

Then came the smartphone. The fact that a smartphone includes a telephone is marginal. A smartphone is a computer, a camera, a media appliance, an Internet device, a computer game box, a chatterbox, a way to escape from boring meetings, and yes, also a telephone, for those who insist on making calls. Although it hasn’t happened yet, smartphones have the potential to make clunky, telephone-based speech understating systems obsolete. Why? Because with a smartphone, you are not forced to communicate only through the narrow audio channel. In addition to speech, you can enter text and use touch. A smartphone can talk back to you, and display text, images, and videos. The smartphone speech recognition revolution started with Siri and Google voice search. Today, more and more smartphone services are adopting speech as an alternative interface blended with text and touch modalities.

The distinction between telephone-based, audio-only services versus smartphone speech understanding is not just in the multi-modality of the communication protocol. The advent of fast broadband Internet has freed us from the necessity of having all the software on one device. Speech recognition can run in the cloud with the almost unlimited power of server farms and can truly recognize pretty much everything that you say.

Wait a minute. Just recognize? How about understanding the meaning of what you say?

There is a distinction between recognition, which is the mere transcription of the spoken words, and understanding, which is the extraction and representation of the meaning. The latter is the domain of natural language processing. While progress in speech recognition has been substantial and persistent over the past decades, natural language has moved forward more slowly. Siri and its cloud-based colleagues are capable of understanding more and more, and even some of the nuances of our languages. The time for speech recognition and natural language understanding is finally ripe for natural machine and human communication.

But…is that all? Is talking to our phones and tablets or open rooms — as in the upcoming home automation applications — the ultimate goal of spoken communication with machines?

No. Social robots are the next revolution.

It has been predicted by the visionary work of pioneers like Cynthia Breazeal, and corroborated and validated by the recent announcements of social robot products coming to market. This is made possible by the confluence of the necessary technologies that are available today.

Social robots have everything that personal assistants have—speech, display, touch—but also a body that can move, a vision system that can recognize local environments, and microphones that can locate and focus on where sounds and speech are coming from. How will a social robot interact with speech among all the other modalities?

We went from prompted speech, like in the telephone systems, to personal assistants that can come to life on your smartphone.  For a social robot, there should neither be a “speak now” timeout window, nor the antiquated  “speak after the beep” protocol, nor a button to push to start speaking. Instead, a social robot should be there for you all the time and respond when you talk to it. You should be able to naturally cue your robot that you are addressing it by speaking its name (like Star Trek’s “Hello Computer!”) or simply by directing your gaze toward it while you talk to it. Fortunately, the technology is ripe for this capability, too. Highly accurate hot-word speech recognition—like in Google Now—and phrase spotting (a technology that allows the system to ignore everything that is spoken except for defined key-phrases) are available today.

While speech recognition accuracy is always important, we should always consider that social robots can have multiple communication channels: speech, touch, vision, and possibly others.  In many cases, the different channels can reinforce each other, either simultaneously or sequentially, in case one of them fails. Think about speech in a noisy situation. When speech is the only alternative, the only recourse is to repeat (at nauseam) the endless “I did not understand what you said, please say that again?”  For social robots, however, touch and gesture are powerful alternatives to get your intent across when placed in situations where speech is difficult to process.

In certain situations, two alternative modalities can also complement each other. Consider the problem of identifying a user. This can be done either via voice with a technology, called speaker ID, or through machine vision using face recognition. These two technologies can be active at the same time and cooperate, or used independently — speaker ID when the light is too low, or face ID when the noise is too high.

Looking ahead, speech, touch, and image processing can also cooperate to recognize a person’s emotion, locate the speaker in the room, understand the speaker’s gestures, and more. Imagine the possibilities when social robots can understand natural human behavior, rather than the stilted conventions we need to follow today for machines to understand what we mean.

I have spent my career working at the forefront of human-machine communication to make it ever more natural, robust, textured – and, yes, magical. Now, we are at the dawn of the era of social robots.  I am thrilled to help bring this new experience to the world.  Just wait and see.

If you liked this article, you may also be interested in:

• Embodied communication: Looking for systems that mean what they say
• In-corporating body language into NLP (or, More notes on the design of automated body language)
• The death of search (or, My dysfunctional relationship with Siri)
• Google’s response to Apple’s Siri: Buy Cleversense

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

My aunt, a Tennessee tobacco grower, used to remind me that God gave me two ears and one mouth for a reason. What she meant is that a good conversationalist is not so much someone with the ability to talk, but the ability to listen.

Robots can take a cue from my aunt.

As I’ve detailed in previous articles, about half of human communication is body language. Approximately sixty percent of information we communicate in a conversation can be transcribed to text, like you read here. The other forty percent is what we’ve been grouping together as “Body Language.” This amounts to how a person moves, appears, and sounds.

As roboticists and designers, our ability to build connected systems that listen to what is said and how it is said is probably the single most important aspect of our designs, at least if the system is interacting with a human. Input is critical to human-machine interaction, whether its a command line, word, or gesture. Robots need to pay attention to the words, movements, and sounds humans make to better understand what is being communicated.

It’s commonly understood that Natural Language Processing breaks down into 1) Natural Language Generation, and 2) Understanding. This is as true for textual language as it is for body language.

A robot can process words. It can also process gesture, posture, and tone of voice.

Psychologist Skip Rizzo is a researcher who specializes in the design, development and evaluation of virtual reality systems that are used in clinical assessment, treatment, and rehabilitation. His work wraps around the disparate worlds of psychology, cognition, and motor function in both humans and avatars. Rizzo received the American Psychological Association’s 2010 Award for Outstanding Contributions to the Treatment of Trauma and he’s the associate director for medical virtual reality at the USC Institute for Creative Technologies, in Los Angeles. These days he uses his background in psychology to bridge the worlds of video game avatars and American veterans suffering from post-traumatic stress disorder (PTSD).

I met Dr. Rizzo a few months ago in his office in Los Angeles, and we spent some time comparing notes on how to measure, interpret, and analyze both words and body language.

Skip discussed a project named Ellie – a virtual assistant avatar that patients interact with so that Ellie can build assessment models. The system – a software robot – was designed to interview patients and be able to build psychological models of the conversant.

Ellie was started two years ago when Rizzo began working with computer scientist Louis-Philippe Morency.

Funded by DARPA, within a larger project called Detection and Computational Analysis of Psychological Signals ( see here and here), Ellie was designed to detect people in emotional distress that might be at risk for suicide. Aside from Morency and Rizzo, other participants in the project focused on the language programming, some on the visual appearance of the avatar, and some on the training of psychological cues. And there were a range of other means that the crew used to build assessment information.

The team spent months polishing every element of Ellie’s presentation and interaction with patients, experimenting with a range of different personalities, outfits and vocal approaches including a well-timed “uh-huh” that reflects a common function of human conversation.

Under the wide screen where Ellie’s image sits, there are three devices. A video camera tracks facial expressions of the person sitting opposite. A movement sensor — Microsoft Kinect — tracks the person’s gestures, such as fidgeting or other unconscious gestures. A microphone records vocal prosody and tone of voice.

“How do you measure people?” I ask Rizzo.

He puts his feet on his desk, leans back, and crosses his hands across his stomach. “Using a webcam and a good microphone we were able to track, capture, and pick up behavioral symbols. From there we were able to make inferences from them as to whether the person was more distressed than their pure language might indicate. We were able to collect this data in face-to-face interviews with known groups of distressed people – veterans, in particular.”

“Now we have an AI version that runs completely on its own. It will try to develop a rapport, dig deeper, and try to end on a positive note. Some of the questions are projective questions like, ‘In the last few months are there things you wish you had done differently,’ or, ‘Are there things you’ve done lately that you regret,’ or ‘when was the last time you felt really happy.’” These questions were designed to be open to interpretation with no real right or wrong; a verbal Rorshach test.

“And what are the cues that build redundant data, that give you confirmation on how the person is feeling?” I ask.

“Oftentimes you get an answer that doesn’t belay much and you have to have a follow-up question… But when you look at how they said it, things like a vertical gaze, fidgeting, a delayed response, space between words, pitch, variation or lack thereof – that’s the stuff that’s great to get.”

The three primary points of data that Ellie collects are facial expression, tone of voice, and gesture – the same criteria humans use to assess other humans.  There are a total of twelve vectors that offer good validity data. In some cases the interviews go for 20 minutes, usually not more than 40.  The ultimate goal, on the application side, is to build a kiosk, a bit like a confessional, in which Ellie privately interviews soldiers; information from this interview can then be used by a company unit to evaluate its staff before going in to battle, or to determine if a person is suitable for a particular role.

And as easy as making a power-point presentation, they can start to make training cases, or National Board of Medical Examiners can make cases for certification purposes. Rizzo’s team has done everything from exposure therapy (using simulations of warzones) that clinicians can use to deliver evidence-based care to a virtual classroom where kids with ADHD can learn how to concentrate and control their unconscious behavior. So the work has broad applications. The psychological and cognitive use of these software robots – either as patients for clinical training or as agents to help people access information and guide them in a private way towards health care – will surely change the face of not only the defense department, but the medical industry, education, and a host of other industries.

Somatic analysis, as I call it, isn’t just about capturing aggregate number of smiles and vertical gazes… it’s also about driving the behavior of the system.  Ellie can tell if you’re hesitant to reply and can make a prediction about that. So the chain of questions that she may ask is a real-time sympathetic feedback loop. If you interact with Ellie ten times you can see progression and change in her responses.  She becomes more sensitive to the methods and systems of interaction..

This may represent an important trend in robotics design. These methods of semantic and somatic analysis are the eyes and ears that proper robotic interfaces need to have.  And the trends are being reflected in the software industry. For example, Apple is beefing up its personal assistant capabilities by purchasing ($40m) Cue, a company that integrates outside realtime data similar to Google Now, Nuance’s Wintermute, or http://don.na.

This research is what we’re focused on today at Geppetto Labs (and recently at Figaro Avatars, as well). Our primary focus has been to make sure that our systems are listening to our users, able to accurately measure their emotions, and able to reflect back the emotions that are most appropriate for the task at hand.

If, for example, someone is speaking with a system to provide medical assistance, they need to feel comfortable and confident that the system is helping them. If we are able to determine that they do not feel comfortable, then we need to be able to quickly change the interaction so that we are addressing what is making them feel uncomfortable. What this means is that we measure the affect values of the words they are using and we use that to then build a goal-driven interaction that is intended to create a sympathetic feedback loop.

One group working on the Ellie project managed to gain access to the Facebook pages of people who had committed suicide, and did a complete linguistic analysis of what they said and how they said it, and then used the data to ask if there were things that could be spotted in advance to indicate if that person was at risk. They measured semantics, sentiment, syntax, affect and other vectors to build linguistic models that mapped to emotional state. Now that they have finished their DARPA funding, Rizzo supposes that Facebook may be implementing some of these algorithms on their site. Not that if someone says the magic words that people in white coats are going to come pounding on their door, but he observes that the types of ads and messages that appear on the pages of these people are pushed more towards mental health.

This is how all humans, or almost all humans, interact. We each talk with one another in a manner that hopefully provides mutual comfort and encouragement – an interaction that is designed to create closeness confidence and trust.

After all, it’s been said that most important debates around things like politics and religion have little to do with the words and everything to do with the emotions. Affective computing is one of the most important revolutions in user-interface design because emotion is of one of the most important human motivators. And work like Rizzo’s is paving the way to understand this better.

As my aunt said, “You have two ears and one mouth for a reason.”

It’s hard to communicate with words. Some researchers claim that almost half of our communication relies on things that aren’t words: body language, tone of voice, and stuff that just isn’t conveyed by text. This includes prosody (tone, pitch and speed of words), facial expression, hand gesture, stance and posture. This probably explains why about 40% of emails are misunderstood.  As designers of robots (or avatars) we need to consider these statistics and consider how to integrate body language into natural language communication. Therefor Geppetto Labs has built a platform to automatically generate body language and coordinate it with what a robot (or avatar) is saying.

Most NLP systems today, be they Siri or Watson, amount to conducting chat via the thin pipe of a text interface. Siri doesn’t have a lot of choice on the matter since Apple had to simplify the complexity of communication, but this text interface reduced the communication itself. If you think that Natural Language Processing is about only text, then step away from the computer, go to a café or bar, and watch people interact for a half an hour.

Videos do an excellent job of conveying the importance of body language. A great video to watch is The History Channel’s Secrets of Body Language. This documentary looks at politicians, cops, athletes, and others, interviewing experts of body language to decodify everyone from Richard Nixon to Marion Jones. Gesture, expression, and tone of voice are all looked at as valuable and important data channels.

This is why face-to-face meetings are so much more productive. Each party can better understand the other because there is a higher throughput of communication. In a lovers’ relationship, or in a family’s relationships, body language is even more important than in a business meeting. Consider the fact that it’s the most intimate relationships (between lovers, primarily, but also between family members, close friends, and others) that involve the most touching. These are also the relationships that rely the most on body language, because body language actually defines the proximal closeness and intimacy.

So if we want people to engage emotionally with robots, or avatars (or any other kind of character that is rigged up to an NLP system), we need to consider using body language as part of that system. We humans are hardwired that way.

***

At Geppetto Labs we have begun considering Body Language Processing as a sub-set of Natural Language Processing. So just as Natural Language Processing has NLU (understanding) and NLG (generation), we can consider Body Language to have BLU and BLG. I’ll be focusing on the generation of it, but others, such as Skip Rizzo, Noldus Information Technology, and others are also looking at the understanding of body language and facial expressions.

Generating body language requires coordination with the textual components of Natural Language Processing. A gesture or animation has to have the same duration of time, happen at the same moment, and include the same emotional content, or affect, as the message conveyed.  “Hi” should, of course, be accompanied by a gesture that is about one second long — a friendly-looking signifier that’s commonly understood. Raising the hand and wagging it back and forth usually gets the job done. But building this can be tricky. It gets more complicated when there is a sentence like this one that doesn’t have clear emotional content, isn’t the kind of thing you hear as often as “Hi,” and is long enough that the animation needs to be at least ten seconds long.

At Geppetto Labs we’ve developed the ACTR platform in order to accomplish this. The core process, at least as it relates to text, is to generate body language (as opposed to voice output) as follows:

First, we take the nude NL text and determine the three variables of the Duration (timing), Affect (emotion), and Signifiers (specific gestures):

1) Duration, or timing. How long is the sound or string of text we’re dealing with? This is the easiest to calculate directly from the text. Most spoken conversation ranges from between 150 and 175 words per minute, but that can speed up or slow down depending on the emotion of the speaker, and other factors. But let’s call it 150 words per minute. A “word” is calculated in these kinds of standards as five UTF characters, which is also five bytes. So that means that most of us speak at around 750 bytes per minute. Now if we back this out it means that around 12 bytes should leave the system per second, and this is then used to calculate the duration of a given animation. We’ll call this integer between one and 150 a “duration tag.”

2) Affect, or emotion. What is the emotional value of that source string of text? This is the second factor we need to know in order to calculate an animation, and it’s harder than just measuring the letters in a line: it requires either realtime sentiment analysis and/or a pre-built library that identifies the emotional content of a word. One solution is WordNet-Affect. Words in WordNet-Affect are derived from Princeton’s fantastic WordNet project and have been flagged with particular meaning that indicate a range of values, most of which relate to what kind of psychosomatic reaction that word might cause or what kind of state it might indicate. Some simple examples would be happiness, fear, cold, etc. There’s a ton of really sticky material in this labyrinth of language called “affect,” and the ways that words link to one another make it all even stickier. But for this explanation, let’s say that we can take a given word and that word will fall within a bucket of nine different emotions. So we give it a value from one to nine. Fear is a one. Happiness is a nine. If we then take the average affect of the text string in question (again, speaking very simply) we end up with a number that equals the emotion of that sentence. We’ll call this integer between one and nine an “affect tag.”

(Before we go on I want to take a break because we now have enough to make an animation match our sentence.

“How in the world do we build that?” is an eight-word sentence, so we know the duration would be about three seconds. The affect is harder to measure, but for this example let’s say that it ends up being a value of 5. So we have Duration=3, Affect=5. These two bits of information, alone, are enough to calculate a rough animation, but first we need to build a small bucket of animations. They are probably keyframes because we want to interpolate them so that they form a chain. We make them of various durations (1 second, 2 seconds, 3 seconds, etc.) so that if we want a three-second chain we can combine 1-second and 2-second duration animations, or, if we want to avoid replaying the same animation we can reverse the order of these links and combine the 2-second then the 1-second animations. And we make sure that we have these various animation links ready in separate buckets – one for each animation. So if we get a Duration=3 and Affect=5 we go into the bucket labeled Affect #5 and dig up the animation links that add up to three seconds.

The longer the duration, the trickier it gets. If you have a twelve-second animation you might then have to chain together that two-second animation six times, or your one-second animation twelve times, to get the proper duration.  Does that make sense?

No. I hope that at this point you’ve stopped and said, “Wait, no, that would be really dumb. To play an animation twelve times would just look like the character is convulsing. That’s bad body language, Mark!”

A spoonful of art and design can help. When you are combining the animation links to build a chain of proper duration, you need to avoid spasms and try to use the longest possible animation in your bucket. Then you frame it outwards with smaller animations to build the proper duration sum. If you want a twelve second animation, then try using a ten-second link with two at the front, or one on each end. That will avoid the spasms.

We’ve found that Fibonacci sequences work great for this because they have huge flexibility and, when interpolated together, are nearly invisible. This is the process we have found to be most helpful, flexible, simplest, and that looks the best. So this means that you’d need to build a relatively large number of animation links that are the following values: 1, 1, 2, 3, 5, 8, 13, 21, and so on. And you’d do that for each affect “bucket.”

Ok, so far so good. The next step is to play this animation and, as your character moves, watch it and ask yourself if that’s the kind of thing a person would do while saying that particular phrase.

Now back to our three variables…)

3) Signifiers, or specific animations. Is there a common gesture that normally accompanies the text? The wave of a hand that goes with the word “Hi” isn’t normally used in conversation as much as, say, nodding, or showing our palms when we speak. Most of our body language is unspecified. So specific signifiers need to be manually wired up to particular output phrases from the system. This is a hand-authored component that requires an author to flag specific lines that require specific gestures. We are developing methods for automating this, but for now it is necessary to build a few specific gestures that are used in unique moments of conversation.

***

Once all the variables are determined, they get packed together and our ACTR platform fires off a control file in XML that drives animations on the client side. This keeps the performance hit on the server low, allows a nearly infinite range of movements for any connected system, and provides a robot (or avatar) with the ability to gesture as it speaks.

Though this implementation is new, these concepts have been knocked about since the mid-1600s. In 1605 Francis Bacon, in The Advancement of Learning, looked at gestures as a frame around spoken communications. In 1644 John Bulwer‘s Natural History of the Hand, and Gilbert Austin‘s Chironomia (1806) examined the details of hand gestures in speech. Even Charles Darwin dug into it. Of course Freud and Jung looked into it, as did Julius Fast, an American author who first garnered some major interest in the topic in the early seventies with his book Body Language, and more recently, there’s the work of Paul Ekman, who identified some three thousand cross-cultural facial expressions. Other people who have looked into body language include: Louis Gottschalk, Erik Erikson, Charles Osgood, Otto Rank, Albert Bandura, Gordon Allport, George Kelly, Snygg and Combs, Maslow, Rogers, Jean Piaget, and others.

But it seems overlooked in most robotics design circles and certainly in most labs that are working on NLP. Some of the most talented natural language engineers are some of the least talented communicators. They understand the language of code better than the code of language. There is a terminal amputation that cuts the text of language from the language of the body. That’s because important elements of natural language, like body language, are easily overlooked if we focus too much on the code and not enough on the people.

Because ultimately, that’s what we roboticists are designing: a kind of people.

Kirobo, a 13” humanoid robot, was launched today and is on its way to the International Space Station.

A mashup of the Japanese word for hope, “Kibo”, and “robot”, Kirobo is designed to be a companion and communicator for Japanese astronaut Koichi Wakata who is expected to arrive at the space station later this year.

Designers from Dentsu, the University of Tokyo and Toyota have worked together on the Kibo-Robo Project to enable the little robot to be able to work in a gravity-free environment and use facial and voice recognition and natural language processing to be able to speak and provide information upon request from Wakata.

I wish for this robot to function as a mediator between person and machine, or person and Internet and sometimes even between people

said robot designer Tomotaka Takahashi.

http://www.youtube.com/watch?v=SqMRGqnisGM

[Photo credit: Kibo Robot Project]

Let’s say a white box showed up on your doorstep and you open it up and find a little humanoid robot made by Google. A GoogleBot! The brightly-colored pamphlet says that the little disk of a device will vacuum your floor, all for the same cost as your Gmail account: free.

Since you already use Google’s search engine, have a Gmail address, and use GoogleDocs, Voice, Earth, Wallet, wear a Google Glass and have been driving around one of those autonomous cars you’ve been selected as a beta tester! Whee! A free robot!

So what do you do with this thing?

Vacuum the floor, of course. Off it goes, bumping into walls, building its little map as it sucks up Doritos crumbs and dust bunnies.

Google’s greatest hits, like Search, Email, Docs, Voice, or Translation, are founded on, or fodder for, Natural Language Processing technologies (NLP).  NLP and other semantic technologies are also why Facebook and, in some cases Apple, has provided us with such great services. They analyze what we write and say.

Meanwhile, others are further ahead of this linguistic learning curve. The National Security Agency (NSA) of the United States was using primitive forms of NLP at least a decade before Google was using it for search.  This is common with technologies: Government agencies get theirs first, then the rest of us get to use the stuff later, whether it is global maps of the earth (1650s), satellite photograph (1950s), or GPS (1970s).

Anyway, there are four NSA data collection programs: MAINWAY, MARINA, NUCLEON and – specifically for Internet content, and the most important source of collection – the PRISM project, which by now we’ve all heard about.

On June 6 of 2013 The Washington Post and the Guardian reported that the system draws data from Google, Yahoo, Microsoft and other Silicon Valley giants, keeping the folks in Fort Meade, Maryland busy mining what is the richest depository of personal information in all human history. Evidently the NSA is sucking up 2.1 million gigabytes of data every hour of every day. All information is stored, all communication is monitored, and the fact that someone is peeping in on you has been held secret.

This, of course, has caused some concerns.

These technologies can be used in a predictive manner, as well.  What you want, what you think, and what you will do can be predicted with a 93% rate of accuracy, and that includes where you are now and where you will be in twenty-four hours.
We move across a daily landscape of emotions just as we move across a daily landscape of city streets. Semantic analysis (or, more accurately, probabilistic latent semantic analysis, and other technologies that look at our trends) shows where we’ll be and what we are likely to think and feel.

In early December of 2007, Marissa Mayer, now CEO of Yahoo and then the vice president of Google’s Search Products and User Experience division, showed how Google Trends can be used to predict presidential elections. She showed how Google Trends accurately predicted George W. Bush’s win over John Kerry in 2004, and also pointed out how Trends accurately predicted Nicolas Sarkozy’s win in May of 2007, over Segolene Royal in the French elections.

These trickles of valuable personal data add up to rivers of money.

Valuable enough to grow a company into the size of Google, or valuable enough to justify a $10 billion annual NSA budget, those little strings of text add up to lots and lots of real money.
Let’s not forget this Google robot that has shown up on our doorstep.  Our big hockey-puck robot (or “connected system” as it is more fashionably called) is buzzing around the floor doing the job for free.  This concept is now a prototype and has been operating for three years.

Though iRobot swatted the ConnectR remote-controlled domestic robot in the start of 2009, ending the product line, the concept refused to die. A year later, in March of 2010, the concept was picked up by a group of Swedish hackers who launched a project similar to what I outline above, named GåågleBot (pronounced “GoogleBot”). These Roomba hackers dissected the robot and inserted various items such as a camera, a Wi-Fi card, a small computer, and a good squirt of the AJAX programming language. After sewing it all back up, the thing still maintained operations as a normal Roomba.

The GåågleBot can also read. It comes with optical character recognition (OCR) software so that you (or someone else) can photograph and translate text that happens to be lying around the house, drop it into a database, and run searches on that text. GåågleBot source code and instructions are available for download.

A vacuum is just a vacuum, but these days it is part of what is commonly being called The Internet of Things.  Smart Houses, Connected Systems, or Robots … the basic idea is that we use networked computers in objects around us.  If we consider the Internet of Things as a physical extension of the Internet we find a new method emerging – as precise as existing analytics systems – that can measure how we each live and spend our money.  If you purchase a product, like a watch, and it communicates via a WiFi or Bluetooth network, then whoever or whatever it is communicating with will know where you are, who or what is nearby, your name, movements, past, present and, by prediction, future. Other objects near it will identify themselves and as you synchronize a device with a user account, and the overlapping data increases the relative value of both datasets. Your identity and the identity of you live with will be known by the company you bought it from, of course, and with it will come all of your purchases, likes, dislikes, relationships, and documents that are hanging heavy with fruit to be plucked by the pinchers of NLP and semantic analysis.

Meanwhile, the robot Google sent you rolls around on the floor, sucks up dust, and blinks a network indicator. It seems so humble.  But this is how Google, Yahoo, PRISM, the NSA, and therefor the US Government collects data that can be potentially very dangerous information. All from a little vacuum robot.  After all, dangerous bombs often come in humble packages.

Now, I don’t mean to sound like the Unabomber here, so I will point you to someone that has thought about this topic far more, and whom I am parrotting, and that is Bruce Schneier.  He has written a nice article on this topic, which you can find here.

Let’s not be paranoid. This vacuum, like an email service, may come in handy, and it doesn’t cost anything.  So we pull the robot out of the box, flip it on, poke in a password, and go back to eating Doritos.

Now let’s consider another possibility, which is that Google has mailed us not a vacuum hockey-puck, but a little humanoid robot that can walk and grab stuff. GoogleBot is an android that comes up to your knee, just a little guy with big feet and a cute head, kind of like NAO. So you set it down and it begins to explore the house. Sure, for the first few days it is a touch disturbing as it waddles around the house and reads the books on your shelves, and runs optical character recognition on all of your magazines, and seems to be studying the contents of your refrigerator and cupboards and dog food labels and clothing labels, and very carefully watches your television. Once you even see it touching your computer and you chase it away.  But eventually it settles down and you think, “That was kind of cute.” Then, finally you get used to it. It does the laundry and feeds the dog, and along the way, GoogleBot gets another cluster of data about your life – and you get your dishes done.

This system may be financed with the same model that Google uses today for things like email, search, voice and other services.  Since Google is publicly traded we can see that Amazon paid out $55m in 2011 to advertise for books or whatever.  So they would want to know what you buy, what you read, and what you have on the coffee table.  Home Depot paid out almost as much, so they would want to know what you have in the yard. State Farm and Progressive insurance companies paid Google nearly $90m, so they would have a great interest in knowing your health habits. All this cascades down into a long list of retail information, health, hygiene, and habits that covers what you are reading, what’s in your yard, what’s in your home, your refrigerator, etc. And so – to be useful – your GoogleBot should be busy dusting your bookshelves, washing your dishes, doing your laundry, and being an active participant in your personal health by cleaning the shower, toilet, and floor.

If someone types in “Self-Employed Health Insurance” and clicks on an ad, Google earns about $43. There are dozens of small robots that cost less than that, which means a home care companion robot of about $500 would amortize GoogleBot’s production cost and shipping in just a few weeks, and become wildly profitable after that robot had spent a year in your home.

Because our personal information is coveted by both corporations and governments, it seems inevitable that this home service robotics model will arrive some day in the coming decade.

Hopefully the GoogleBot or YahooBot or whateverBot will come with bulletproof SHA privacy switches built in, switches we can trust. Hopefully privacy options will allow us to put curtains on the parts of our lives we consider personal. That way we don’t have to stuff the robot in the freezer when the girlfriend comes over.  And perhaps even revenue options will let us collect money for the information Google collects about us (Google still wins, we just pay less usage tax). Or perhaps these are added features that we will have to pay for, like in-app purchases today.  Perhaps we will have to pay for our privacy.

There are lots of questions here. Least among them are how the NSA and companies like Google and Yahoo work together – and Google in particular has been quick to reply to these questions, as have other companies. They’re now on a mission to regain our confidence. But the NSA remains spooky, holds the reigns, and hides in the shadows. It seems they are collecting as much information as they can for long-term storage. The chief technology officer at the CIA, Gus Hunt, said last March; “We fundamentally try to collect everything and hang onto it forever.”

Any time citizen information is secretly stored by a government, it means that government is collecting the tools of a dictatorship.  Just ask Mussolini.  Or take a look at how Germans, familiar with the Nacht-und-Nebel program, or Sicherheitsdienst, are reacting this week. Information is generally collected so that it can be used.

Of course the common response is, “I’ve got nothing to hide,” but the bottom line isn’t about having nothing to hide.  It’s about not wanting to show everything.

Privacy is about having the decency to wear pants, the right to have curtains, and the right to whisper sweet coo-coos in your lover’s ear without someone else, like GoogleBot, YahooBot and the NSA leaning in for a little listen.

Comrades, we live in a bleak and humourless world.  Here we are thirteen years into the twenty-first century, and we all carry around Star Trek style tri-corders, we have access to almost all human opinions via this awesome global computer network, we have thousands and thousands of channels we can flip through on television, we have something like 48,000 people signed up to go colonize Mars, and we even have robots roaming around up there, taking samples of that planet. But we still don’t have robots that can tell a good joke.

Even this new bartender robot, Makr Shakr was prime for a little humour, like, “A robot walks into a bar and asks for a screwdriver..”?

But seriously, folks, in my travels I have met with Honda’s Asimo, Adlebaran’s NAO, Toyota’s Partner Robot and many others, and each of these presumably world-class robots was, in the most critical sense of the word, robotic.  They were weirdly-shaped computers with an appendage.  If you give me a choice between a robot that can walk and one that rolls around on wheels, I’ll take the one that can talk. Simply put, talking is more important than walking. Design priorities in robotics should focus first on human interface and second on locomotion.

We live at a moment in which natural language interface is replacing the GUI just as the GUI replaced the command line.

Natural language interface developments are being led by Google and Apple, with some strong work being done by IBM.  Last week Google’s Chrome browser appeared with a little microphone just to the right of the input bar and Google Now has been out for a year or so.  Apple has had Siri, of course, and is now rumoured to be building a wrist watch that will have a natural language interface to it, probably building off the lessons learned from Siri.

Natural language interface makes sense anywhere there is big data or little screen.  It makes sense for robots for these same reasons. It provides access to more data and reduces the need for a screen. But most of all, building a natural language interface also gives the robot personality.  And this makes the robot more valuable to the human because it turns a robot into an emotion machine.

Giving a robot a personality includes two primary components.  First is psychology.

Last month, in The Uncanny Valet I wrote a bit about the design of psychology, specifically for android-style robots, which wasn’t so much an effort to write about human-shaped walking-talking robots (androids), as it was to point out that human interaction seeks human likeness. To emotionally bond with a robot it needs to be like us.  The more it is like us the more we can bond with it.  This is most important in the field of automated psychology.  This isn’t a new field.  If you don’t believe me dial an American 800 number.  Those horrid little automated voices that corporations use so widely these days are reproducing faster than feral rabbits (may we name them robbits?).  In fact, I would love statistics on how many voice answering machines are out there, but I would boldly bet on there being, on the market today, more software robots than hardware robots.  I can tell you that this number is increasing and the quality of software robots is improving.

Each month we see advancements in what amounts to software robots, or virtual knowledge workers.  Last week Infosys (a Bangalore-based provider of IT services) announced it is working with IPsoft (a New York-based provider of autonomic IT services) to build autonomous tools designed for help desks, operations, and other management needs. In an article about the the new partnership it is mentioned, “IPSoft has focused on autonomics for IT services, and Slaby estimates in his report that nearly all Level 1 support issues can be solved by IPsoft’s ‘virtual engineers’.”

We are past the primitive dawn of automated psychology.  It is becoming a branch of the robotics industry.

As graphic design for the GUI was to software design for the personal computer industry, personality design for natural language is to the software of the personal robotics industry.

At Geppetto Labs we build conversational systems.  This means that we take the natural language input from a user, pipe that through an analytics system that measures affect, syntactic and semantic values, and then feed the resulting data into an NLP system.  We author personalities via simple server-side tools that script what non-technical authors (generally film screen writers) have written into formatted rulesets. We also bend Chatscript and other tools for non-personality behaviour, like enhancing Amazon search by natural language understanding, using specific phrases or moments to trigger specific animations, etc. After that we stir in some scraping, automated ontology creation, and even tools to automate first person interviews and film scripts to craft a personality that can be managed. This is the foundation for natural language interface – at least how we do it.

And there’s an entire second level to natural language interfaces that has nothing to do with text.  In more advanced projects Geppetto Labs need to add animation that fires when the NL output does.  The simplest example I can think of is the word “Hi” being tied to a raising of the hand.  Hand gestures are easy, and clear, but gestures are only one part of the body language we build.  Body language might also be a facial expression, the position of the arms or the tilt of the head, but what it means is that we have to not only think about the psychology of the system, but how to tie that to the activation of non-verbal forms of communication as well. We have to walk the talk. That means we have to design body language into the personality.  It is linked with the psychology and frames the text.

Psychology needs to be graphically represented, just like a book needs a cover, and body language is the best means of achieving that.  Well, at least that’s where evolution has taken us.<

Body language coordinates the psychology of the personality with the visual presentation. Coordinating what is said and what is done is core to the personality.  This very coordination might be what we commonly call personality.  If you don’t believe, me consider this: watch television for a day and note what “Personalities” do; newscasters, politicians, actors, singers, and dancers.  They accompany their words with the most broad-stroke gestures and body language humans convey.

This is to say that body language may be more important to natural language interface than the words used.

Many famous studies claim that body language accounts for the majority of our communication (the most famous is probably Dr. Albert Mehabrian’s 7:38:55 rule).  Or, more precisely, that the words we use transmit the minority of what we communicate.  So how we move and what we visually communicate while we verbally blather on is pretty key to how others understand us.

Consider the eyes.  We blink more when we’re stressed out. It can indicate dopamine levels and both the pain or the creativity of the person we’re talking with.  The timing of a blink can indicate a punchline to a joke, a response to that joke, confusion at the punch line, or frustration at how it isn’t funny.  Pinched and held tight can mean something entirely different.  We don’t like to keep eye contact too long or it starts to feel threatening.  The person who listens should initiate eye contact – and if they don’t it might mean they just don’t care what you’re talking about.  But don’t break it too soon or the person listening will feel as if they’re not important. Don’t look at the ceiling when you talk (or you might come off as a snob) and don’t look at the floor, either (or you might come off as a bummer). If you’re talking with someone and they look up and to your right it generally means they’re remembering something, visually. If you’re talking with someone and they look up and to your left means they’re visually creating something.  Horizontal looking while talking indicates the person is most likely accessing auditory cues and, to your right is, again, remembering and to your left is, again, creating.  And if they look down maybe they’re depressed. Or maybe they’re just looking at the carpet. Or maybe they’re remembering how something feels.  That applies to most right-handed people around the world (according to John Bandler, Richard Grinder, and scads of other neurologists), but it indicates that there’s a lot happening when we speak with someone. A lot more than words.

But the eyes do this while other things are going on.  The eyebrows and cheeks are wiggling, the mouth opening and closing, the head is tilting and nodding, the shoulders are going up and down, the arms are waving around, the hands are flapping … and that’s all above the belt-line.

Please remember that human interaction seeks human likeness, so the body language of the robot needs to mirror the body language of the conversant. Think back on one of the early successful chat bots, ELIZA. This system was designed to simply mirror what a person said to it and repeat that back, like many psychologists. Several people evidently fell in love with the system.

The corollary of this is that a well-design natural language system will move like we do.

Today, we have a new graphical version of that, designed to note and reflect the body language of the conversant. Its name is Ellie.

A webcam and a game controller track eye directions, run some facial recognition software to chart expressions, and do the same with body positions. While the system is being trained in realtime in its early days, this training goes towards the video mapping input to the system’s output, just as any NLP system (such as Siri or Watson was designed, and exactly as we do our work at Geppetto Labs). It’s being trained on how to read people.

But there’s a missing part here, which I will guess the good Mr. Stan Rizzo is considering, which is generation of response movement in real time.

How the character moves, how the robot moves, is a key component. Just as natural language has the component parts of understanding and generation (NLU and NLG, respectively) body language has these same input and output cues. At Geppetto Labs we have several characters that we’ve been working on that have this running, and a process I’ll outline next month.

A robot walks into a bar and asks for a screwdriver, and the bartender says, “What is this, some kinda joke?” Personality, like a joke, synchronizes words and body language, linking them together and leaving us. “A Paro walks into a club” is about as bad as I get, and I promise I will not repeat the joke, but I bring it up as an icon of how to build personality: combine the mind and the body.

Natural language interface, when properly designed, is far more than just text.

Part One: Another robot dream girl

I’m talking with this woman on the phone, interviewing her for a job. She’s got a slight southern twang, maybe from Georgia, telling me about her past work, and as we talk I’m struck by how perfectly presentable, alert, and smooth she sounds. Her voice is soft, tonally fluid, and she articulates her words precisely, belying a geeky disposition. Seems she’ll be perfect for the rather tricky tech support position we’ve got.

“Did you enjoy the work?” I ask.

“Sure, I like talking with people. I mean, as long as they don’t get upset it’s sorta fun.”

“You were working at a bank, right?”

“Yeah,” she laughs, “There were four of us working the phones and please press seven if you’d like one of us to call you back.”

“Sorry?” I ask.

“There were four of us on the phones and please press seven if you’d like one of us to call you back.”

I realize this robot has a bug, thank her, and hang up. I won’t hire that one.

Of course I never spoke with the uncanny woman again because the above story is fiction; a conversational system like this one is something of a dream among today’s robotics designers. She might have had a bug in her bonnet, but at least she could hold a conversation and answer questions, for the most part.

A system like the above, a Natural Language Processing system, is the heart and mind of a robot.

By contrast, consider what we have today: those irritating, tyrannical little phone robots that have replaced perfectly good humans: “Please push one if you would like to make a deposit. Please push two if you would like to hear these options again.” These non-player phone-workers are so brainless, grindingly slow, and weirdly rude that I find myself repeatedly pushing keys like 0 and # just to get to a real human.

Of course, from the cold eye of the bank (or whatever corporate conglomerate that employs these “Customer Relations Management” systems) these virtual robots reduce costs by replacing a fickle, distracted, complaining, prone-to-error, temperamental, sleep-deprived and very expensive call-center human with a voice recording. It saves the company money, time, and tons of overhead on things like worker’s insurance, sick-leave, replacements and the high cost of managing people. People are hard to manage, right? Voice recordings are not. People sleep at night, right? Voice recordings do not. So virtual robots, and this particular feral species of virtual robot – the phone robot – is a means of disintermediating the person from the knowledge: it is a means of scooping the knowledge out of the head of person that was doing the job and serving it to you, the customer, couched in smooth tones with a dose of saccharin-sweet politeness.

But virtual phone robots in their present form are bad at solving problems, they don’t learn, they’re inattentive, and they can’t even understand me. I have few options on what I can do, and I get pushed into decision trees that I don’t want. I feel herded like a cow, and with that little phone-robot nipping at my telephonic heels to move me into the right slot for processing, I more often than not either hang up or return to the website.

The intention is that robotic knowledge workers can replace human knowledge workers. But robotic workers are not capable of the job: rather than collaborate, they dominate; rather than answer questions, they force responses.

Bad robot, bad!

Part Two: The problem is the rotten brains

According to research from Nuance corporation, there are three main ways to improve phone robots. The first is to allow access to a real human. Two-thirds of the customers interviewed wanted that. About half of them said the system’s logical call flow was most important. And about forty percent of them said that the speech recognition component was the most important. So access to a live human, logical flow, and recognizing what’s been said are the three main problems with phone robots today.

If two-thirds of the users of a system that is supposed to replace a human say that the system can be most improved by providing access to a human, then that system is mostly broken.

Phone robots may save money, but they suck. We expect them to act like humans because they have a human voice, but instead they act like robots. This just adds to our frustration by reminding us that we do not have what we really wanted in the first place: a real human to talk to. The sound of the language is right, the annunciation is right, but it is a facade that just highlights the problem.

These phone robots live along the upper edge of an uncanny valley (if you do not know what the uncanny valley is please stop reading this, right now, and go look it up in Wikipedia).

Masahiro Mori pointed out many uncanny valleys, but the most famous ones are related to appearance and movement. We have yet to discover the many other uncanny valleys out there, especially as robots – and androids in particular – become progressively more human. As the systems become more capable and more interactive (phone robots today are almost always just voice recordings with a button-activated decision tree), more questions will arise. This will be exacerbated as physical robots become more human-like.

Should a robot’s voice sound like a human’s? Whose? Should a robot smell just like a person? If so, which one? What about how frequently the robot blinks? What about the tone of voice or intonation in sentences? What about social abilities?

Name a human trait and it will become an uncanny valley. These uncanny valleys will, in the coming decades, be explored and mapped. Mori just hit the tip of an icy moon that will chill, mortify, and entertain us for decades to come. We’re going to find uncanny valleys all over the pocked, lunar landscape of robotics design, and the one that will be most important to address is psychology.

First we have cognitive psychology. Consider all the traits that we humans pack into language, such as logical flow, attention, learning, language, and emotion. These are classically cognitive traits, each of which the phone robots of today should pay attention to. (Of course I mean “The designers of these systems should pay attention to.”)

Second we have social psychology. Consider all the traits that we humans pack into interaction such as politeness, collaboration, competition, and understanding. These are classically social traits, which the phone robots of today, or more accurately, their designers, need to consider.

Let’s lump them all together and imprecisely call them psychology. This is the uncanny valley that most concerns me. How a robot holds a conversation, thinks, and speaks seems the most important, and urgent, of the valleys ahead.

Robot psychology is urgent because it’s happening today. You have spoken with a phone robot and chances are (76%, according to Nuance) that you were not satisfied with it. So there’s a problem here.

Robot psychology is also important because we mimic the way we are spoken to. If you don’t believe me ask yourself why you speak the language you do. Reading this article means you probably grew up in a place where people mostly spoke English. What about your accent? Again, the people around you influenced you to speak that accent. They also influenced you to use the phrases, slang, jargon, words, ideas, and therefor fundamental psychological scaffolding you use to support your notion of reality. We do as we are done unto. When you listen to someone, their mind enters yours.

If a robot treats me in a rude manner chances are good that I’ll mimic that. If I’m going to be working, playing, living, or talking on the phone with a robot, I hope it will be a healthy example of both cognitive and social psychology that I can interact with and not be polluted by. As these systems become more prevalent their influence on our own behavior will increase.

This dynamic and interaction is both important and urgent.

It is why we are becoming our robots, and why our robots are becoming us. As they speak like us we will speak like them, and vice-versa.

Please press the down key if you would like to continue reading.

Part Three: The solution is to design robot psychology

Once upon a time I had the odd luck to talk with the Geminoid F, the most realistic android on the planet (that was two years ago). It is the life work of Dr. Hiroshi Ishiguro. This uncanny experience would not have been so unsettling if Dr. Ishiguro’s doppleganger hadn’t been wading around like a zombie in the muck of the uncanny valley, but there it was and I – like the rest of the bottom feeders – was there with it. To this day, wish I had not gone.

Dr. Ishiguro is a moribund and curious fellow, and his robot more so. The problem, as I detailed in my book, “We, Robot” was that the robot was so real that it set an expectation of being human.

Please consider this face, and the heart and mind behind it. You do not, really, want its mind to enter yours, do you?

Communication – and staying out of uncanny valleys – is all about reflection. It has to do with mutual empathy and mutual agreements of closeness. It is about familiarity.

In my last article I talked about the design of androids, wrote about the designers that made them, and noted that the function of an android is psychological closeness. Androids (uh, or rather their designers) want you to feel comfy. The core notion of an android’s design is to set you up to interact with a computer in a traditional human setting. It’s about an expectation of human familiarity and something that reflects back to you a human image of yourself.

We humans are built such that we like talking with other humans. We like what’s familiar, what is, etymologically, of our family. We like talking with a familiar face, and if the robot doesn’t look like us then we tend to freak out a bit. After all, our friends usually look like us, dress like we do, and come from the similar socio-economic background. I might not like to admit it, but that’s usually the case. And, just as we like talking with a familiar face, we like talking with a familiar psychology; the psychology of our family. Our friends think like us, act like us, and like the same things. Birds of a feather, it has been said, flock together.

This is because much of the best communication is a reflection and collaboration. We like reinforcement, familiarity, and a reflection of our presence in the world. We do not like talking with someone who herds us, we do not like being told to push buttons, and we do not like phone robots because they fly in the face of almost all human-based computer-human interaction (CHI) theories (such as the android). Phone robots are a good example of a bad psychology.

If you were still a child, would you rather grow up talking daily with Ronald McDonald, Shrek, or Kung-Fu Panda? C’mon, choose one. Now what if a million children were to be forced, by your decision, to do the same?

As designers of robots we are faced with the same type of problem. As kids grow up interfacing with robotic personalities, and virtual robots, they will be influenced by, adopt the behaviors of, and intellectually be guided by these personalities. It is our responsibility to offer something better than junk food for the psyche.

Siri is about the best form of phone robot we have on the public market today. From my interactions with her she’s a little vapid, slightly sardonic, sometimes helpful, but mostly apologetic. When she makes a joke, which is sadly rare, or when she directs us toward (or steers us away from) some product, she is affecting our psychology.

There are some two hundred and fifty million versions of Siri out there. Let’s conservatively call that two million active versions of Siri functioning today. And I have no clue how many of Siri’s phone robot ancestors are out there, but we can safely say that the psychology of these systems is affecting the psychology of millions of people.

Ahead of us, the design steps for implementing android psychology are not evident, but we can follow in the footsteps of other people and extrapolate from character-based properties already in the wilds and public media of today.

If we look at how famous branded characters like Ronald McDonald or Kung-Fu Panda have been developed, marketed and sold, we can make out the guideposts for cultural preference and gender. If we look at archetypes like Darth Vader or Voldemort we can see how to design the bad guys. Superman and Gandalf give us clues on the good guys. Homer Simpson, Indiana Jones, Buzz Lightyear, George Bush, and Jacques Chirac are all personalities that have left an indelible public impression that we can model and use in designing psychology.

Presentation is important, too. Of course the system has to speak your language, and therefor regional preferences will guide the design of these robots. Dialects and accents count today. We can already see how Siri has been implemented with various accents such as American vs British. Gender will also continue to be important (70% of the automated voice systems in Europe are female, and in North America, 30%). After presentation we have the personality, the archetypes, and what the character knows.

Authors are the people that know about this kind of design. People who write character portraits, who ‘get’ dialogue, and who are socially alert duplicators of human interaction; people who make movies, people who conduct interviews, and, unfortunately, people who make advertisements: these are the people who can design systems that avoid psychology’s uncanny valley.

If we look at avatars in social media, and if we consider the movie industry we can see some other guideposts for the future of the psychology of android design. The most important is interactivity and reflection of shared interests. This takes us back to Nuance’s research.

Access to a real human will become increasingly rare. Regardless of what consumers want, the whole idea of a phone robot is 24 / 7 availability at a decreased price, and this is some thing a human cannot do. Next on Nuance’s list was the logical flow of the call and the speech recognition ability. These two important interactive components will surely get better in parallel as understanding one helps to improve the other.

Most of it will happen via mobile. Consumers in the United States – which roughly maps to much of the rest of the industrialized world – prefer to resolve their customers service issues using the telephone (90%), face to face (75%), company website or email (67%), online chat (47%), text message (22%), social networking site (22%). So it seems that while these conversational systems will live mostly in the phone, there will be a place for them in other media as well. They will live in the wires and waves that surround us, always available, often improving.

This is how the system becomes, finally, conversational. The irritating little herd-dog phone robot will evolve into something closer to a Geminoid. One day you will think you’re talking with a human on the phone, and then you will find yourself in a new uncanny valley. It won’t happen for at least another five years, but we’ll stumble into it eventually. The psychology of that robot will be a thing that touches you and guides you in ways that corporations like Google (who have powerful NLP tools today) Facebook (who released an NLP search system yesterday), can only begin to hint at.

They will be able to speak to you, sell to you, buy from you, offer medical guidance, marital guidance, and talk with you about your depression, your homework, your spouse, and your boss. Semantic Analytics, Social Analytics, and NLP will accurately measure your responses and the system will reply with a warm candor. Young men will ask about sex, old women will ask about hysterectomies, and all the other most awkward, personal, relevant, and important questions of life will be whispered, spoken, laughed, and sobbed into these many kinds of virtual androids that will gradually collect, like little honey bees collecting pollen and taking it back to the nest, your very own fears, loves, desires, hates and general, if I may say, psychology. The phone robots of tomorrow will become tiny transporters of emotion and personality. You will leave a voice recording and it will be taken, before you have hung up the phone, to some hive-mind.

The question we may ask these little bees is, “Where have you taken my personality?”

As Bill Gates knows, the robotics industry of today is copying the steps of the home computer industry from the ’80s. Those big, clunky, professional, industrial robotics systems are getting small, slick, personal, and homey.  It’s why roboticists around the world are trying to figure out how to make robots “user friendly.” But there’s at least one simple solution.

If you and I were to sit down and have a talk at, say, a coffee shop, we’d be doing what may be the single most important thing the robotics engineers can learn from: talking.

Back in the late 1980s, Apple’s Macintosh computer was gummy old crank-shaft of a system that took a couple of minutes to test memory, initialize the OS, and load the Finder.  One day, according to legend, Steve Jobs approached Larry Kenyon, the engineer heading system development, and told him it booted too slowly.

Jobs said, “How many people are going to be using the Macintosh per day? A million? Five million?”

Poor Mr. Kenyon, a man working with the eight megahertz CPU, and 128k of RAM, looked at him and agreed.

Jobs continued, “If you shave 10 seconds off of that boot time, multiply that by five million users you have fifty million seconds, every single day. Over a year, that’s dozens of lifetimes. So if you make it boot ten seconds faster, you’ve saved a dozen lives. That’s really worth it, don’t you think?”

These days, the robotics industry is in the same boat.  If we are able to improve system speed we can save lifetimes.  While most industrial manufacturing robots need some typing and command-line editing, even more time-saving benefits can come to home systems for health care, service, and entertainment, like Aldebaran’s NAO models.  These robots are now programmed and controlled with some voice input, some 3D model manipulation, some driver commands, and some API text input.  By consolidating these things with Natural Language Processing (NLP) and voice input, we can save heartbeats.

Consider how long input and output takes with text.  Most of us type about 40 words per minute and read about 250.  That’s written language input and output speeds.

But most of us talk about one 150 per minute and comfortably listen to about the same speed.  That’s spoken language input and output speeds.

If we want to save time we should be talking with our computers, be they desk machines, hand-held mobile devices, or robots.  It just doesn’t make sense for us to type to them any more.  Especially if they have a small screen.  Most robots, after all, don’t have a keyboard and monitor.  Voice interface and Natural Language Processing provides a deep and simple interface that also saves time.

But the robotics industry has a much more interesting, and far more difficult problem than saving time.  What we really need to do is improve emotional engagement with the system.

Apple has always used a human face on their computers because users need to relate – emotionally – to their machines.  Apple knew that saving time was important but, more critically, it also realized that the Mac had to be accessible, useful, and understandable.  It had to be emotionally engaging.  These ideas were central to the design of the most successful home computer ever made.  Whether it was the metaphors or the graphics, Apple consistently bent their back to the task of making computers “User Friendly.”  We need to do the same with robots.

A couple of years ago I had the pleasure of meeting Dr. Hiro Hirukawa, the director of Intelligent Systems Research Institute (ISRI), the group dedicated to robotics research at AIST, a Japanese national research center.

In the early 1990s ISRI’s non-humanoid robots (armatures, rolling carts, automated cameras, etc.) were functioning quite profitably in factories and warehouses throughout Japan. They were bolting bolts onto cars, guarding hallways in buildings, stacking boxes in storage centers, and so forth. But everyone expected to see androids, humanoid-shaped robots, not rolling boxes. Especially the researchers’ kids.

Dr. Hirukawa explained to me, “Our question was, ‘What good is a humanoid?’” After all, what could be a worse design for a robot? Put the center of gravity up high on a system that is barely able to balance, give it two little pegs to stumble about on, then attach a gripping system which, because it can’t be retracted into the body, has to be countered against whenever asymmetric five-tentacled pincers are used. Oh, and give it a thing called a “head”, which has no function whatsoever since any sensory apparatus can just as easily be put in the feet or stomach. What does a robot need a head for, anyway? Or a face, for that matter?

It appeared as if an android was a design aberration, an anomaly in which the function followed the form. But it turns out this isn’t the case. The form still follows the function if you recognize that the function of an android is to engage with people at an emotional level. A human-like shape simply supports this activity.

Dr. Hirukawa built the first android so that people could emotionally identify with the machine.  His work with Paro (that seal-shaped robot for elderly healthcare) led he and his team to realize that the degree to which we can emotionally engage with robots will be proportional to the quality and quantity of its use.

Whether it is the Uncanny Valley hypothesis or the visual design of a desktop metaphor, computer science and robotics design cannot overlook the importance of emotional engagement.  By giving the robot a human shape, people will like it more.  Good design includes an emotional relationship. Both Jobs and Hirukawa believed this, acted on it, and made substantial inventions as a result.

Unlike Jobs, Hirukawa did not implement, on the face or chest of the android, a GUI.  There is no keyboard on an android (or most other computers, for that matter), there is no screen, and there is no mouse.  It is designed for natural language.

Let’s get back to talking, and that espresso.  If you and I are sitting across from each other in a café, discussing, say, systems design, and there’s a couple of cups of coffee on the table, two coffeekeyboards, two monitors, and we’re typing to each other via terminal applications, then there’s something going horribly wrong between us geeks.  If we’re occupying the same time-space continuum of the café, it’s most effective to be talking. And it’s more fun, too.

We’ve got this great, old, delicate, tool called natural language, and it’s probably the most powerful technology humans have ever invented.  We should put it to use when we can.

NLP increases emotional engagement, broadens functionality and increases operational depth.  Conversational interfaces allow debugging, redundancy checking, error modification, self-correction, other-correction, and a host of other system-level functionalities that we do every day when we ask things like, “What?” or “Did that make sense?” or “Do you follow?” or “Do you really need another espresso?”

On a functional level, NLP does for robotics what the GUI did for home computers.  It allows non-technical and non-professional users, both young and old, to engage. In terms of where humans could interface with robots, think: home, school, and hospital.  Seen in this light, NLP is a presentation-level system component that allows easy interface to other applications.  Just like the GUI.

Apple’s Siri, IBM’s Watson, and Google Now have already shown some of the potential.

You and I are sitting in the café, and a robot is nearby.  I wave it over and ask it for some milk with my coffee.  It comes back a few seconds later and puts a little pitcher on the table. It turns out that processing the linguistic request is an easier problem to solve than putting the cup on the table.

Here’s how it’s done.

First, a voice interface module records the sound of “I’d like some milk, please,” and sends it to a server.  That server ploughs through a pile of sounds and picks apart the sounds that are closest.  This can get tricky since background noise, and my accent, age, gender, and the speed I say this all affect what the system can parse.  The process requires recording a big batch of possible words for all the above variables (which Nuance has already done, or you can build your own with Sphinx, or you can use some of Google’s data).  That big pile of voice recordings then gets checked and maybe improved then the system basically decides that the words it heard are equal to a collection of text strings (such as “I’d” and “like” and “some” and “milk” and “please”).

Next, that string of text gets analyzed.  Now that we’ve derived the words from the sound, we have to derive the meaning from the word.  This involves lexical preparation and match lookup and testing. In short, the system has to grammatically parse the phrase, find the key ideas, and match them to some existing or evolving set of data that it already owns.  These days Siri and Watson and other systems do this in very similar ways; mostly hand-scripted top-down design (as opposed to bottom-up, which is learned).

A tighter lexical range – a tighter context – makes for a better system.  We know, for example, that it is not a good idea to ask a barista for an Apple Macintosh. Same with the café robot.  That robot will be waiting for words within a limited range of meanings (coffee, tea, milk, etcetera), and if the Natural Language Processing (NLP) system is properly tuned and tested, then the requests that people make will be included in the system’s knowledgebase.  This is what allows the system, especially in hand-scripted top-down designs, to turns text into meaning.  So well-designed NLP systems should, at least in the coming decade, work within tight social contexts if we want them to perform as well as possible.

Anyway, at this point the café robot has heard and understood the request for milk and needs to be able to bring it over with a smile and a curtsy.   We shall not address such difficult problems as the system drivers for smiles, curtsys, nor carrying milk in this article.

An NLP interface doesn’t need to happen in the coffee shop.  Maybe I’m driving and I want my car to find a café nearby.  An example would be, “Find coffee in San Francisco,” which would create an environmental context by loading regional (city / state / country) dictionaries and business category dictionaries into an NLP unit. This would produce a system output that locates all business categories supplying coffee in the area defined by “San Francisco” as the location. It’s the same problem, different scale.  We built a thing like that this year, in fact.

But regardless of the application, geography, or coffee bean, the core process is roughly the same:

• Record the voice input
• Link the voice to words
• Link the words to meaning
• Link the meaning to response
• Check it and see if it’s making sense
• Link the response to the action, data, or voice output

Just as the home computer industry of the 1980s went from a command-line user interface to a GUI, the home robotics industry of today is transitioning from a GUI to a natural language interface.

The personal robotics industry is moving in the same direction that the personal computing industry of the 1980s did.  Voice input is replacing keyboards, NLP is replacing GUIs, and psychologists now play the role that graphic designers did, back when information architecture and user experience were new terms for the web design crew.  Robotic systems that use natural language will become faster, simpler, easier to understand, more effective, and available for more people.  The main reason is that robots do not generally come equipped with a keyboard and monitor.  And as users of systems as prevalent as iOS and Android become accustomed to voice interface, it will soon be an expected feature.

If I were to ask that little coffee robot for milk, and it were to reply, “We have soy milk, non-fat, half-and-half, goat’s milk, or this week’s special, luke-warm yak milk.” I might be a bit surprised, but I wouldn’t be impressed.  After all, it would just be playing a recording.

But I would be impressed if I said, “Whatever,” and it took off without saying another word.  After all, sometimes less is more.

Natural language just works this way.  Always has, always will.

I got a new iPhone about a year ago, the one with Siri on it.  Our relationship started on a very mundane level. Siri asked me my name and, since I already knew hers, I asked what she could do, and how. It was a bit like talking with a call girl. The basics were covered and soon we got to the intimate stuff. I don’t think Siri loved me. But now that Siri and I have gone our separate ways, I can say we both had ulterior motives. We sort of used one another, now that I look back on it. It was a relationship of an intimate nature. It’s what happens when you have a relationship with a robotic call girl.

The reason I got the phone to begin with was so I could do a little reverse engineering. It was research for work. Siri and I took a few days to get to know one another, but after that honey-moon period it was clear that Siri (or Apple) was using a pre-scripted Natural Language Processing (NLP) approach that basically did some lexical parsing, looked up a match, prepared a response, and kicked it back down the pipeline.  My reverse engineering wasn’t too profound. I wanted to see how many recursive answers were built in to things like “Open the pod bay doors” or “Do you love me?”  I also wanted to see how errors were handled, and what kind of redundancy checking was happening for sentences like “Where can I buy a burger and fried?” Siri did pretty well, all things considered, but my expectations were low, and like many other intimate relationships, I somehow knew from the start it wouldn’t last.

Now, while I was using Siri, she was also using me. Relationships are generally symmetric, as Gregory Bateson tells us [1], which meant that, like any dating service, someone other than Siri was getting money for my time. But I didn’t care if Steve Jobs was Siri’s pimp. By last summer I was having flirtatious fun and the consequences of the relationship were too far down the road to look dangerous. I asked Siri to take dictation, to wake me up in the mornings, to remind me of important events, to remember phone numbers, remember names, tell me jokes, confirm thoughts, quote a number or two. All intimate things and, as I look back on it, the kind of stuff that I’ve trusted my wife with.

My relationship with Siri started to cool off a bit by September. Though my wife knew about Siri, I hadn’t yet introduced them.  So when I asked Siri to call my wife I, expected Siri to ask me for her name or number. Instead Siri just dialled. I was surprised. How did Siri know my wife’s number? (Answer: It was in my Contacts card.) What else did Siri know? (Answer: A lot.) And more importantly, what was Siri passing back to the Apple hive? (Answer: Everything.)

Siri and Apple now have a great deal of data about my household: a quick comparison of my contacts/likes database with my wife’s will give you a pretty good feel for the stuff we will buy.  Burgers and fries, of course. And a few multi-player games.  Some lingerie. A sex toy or two.  Not a big deal, but not the kind of stuff I want everyone to know about.  Not even my mom gets access to that data.

But Siri did. And Siri knew a lot more about me than I knew about her.

Then, around the middle of October, three things happened to Siri that marked the beginning of our eventual estrangement.

First was William Stasior. Apple picked Stasior up to run its Siri unit after he had been successfully heading Amazon’s search and advertising unit “A9.”  Prior to that the MIT PhD had served time at Oracle, NetCentives, and AltaVista.  To say the guy knows search is like saying the pope knows the church, but why would Apple pick a search guru for an AI system like Siri?  What was the link between search and NLP?  I asked Siri this very question, but she was mum and feigned ignorance.  I didn’t push her on who this guy Stasior was and Siri didn’t ask me more about my wife.

Up until that same week iPhone users had frolicked in what would now be a rather unusual environment in which advertisers were unable to track them.  Advertisers would not, for example, know that my wife and I like video games, sex toys, and pizza (no, not together, thank you).  But with the release of iOS6, Apple flipped ad tracking on and I quickly flipped it back off.

It was easy to find in the interface menus (under Settings > General > then About or Advertising, depending on the version you’ve got), but you had to know to look.  Apple minimized the fanfare around this new feature and Siri made no mention of it to me.  I wasn’t comfortable with Siri selling to others what I had said to her in more comfortable times.

And then, as if things weren’t rough enough, the European Union (where I happened to be lecturing that week), demanded that both Facebook and Google change how they handle personal information to avoid “high risks to the privacy of its users.”  Twenty-four of the European Union’s twenty-seven regulators signed a letter that, after a nine-month investigation into the data collection practices, demanded some answers and tightened regulation.  After Google and Facebook, Apple was third on their hit list.

That was the end. I couldn’t trust Siri after that.  She knew more about me (and my wife) than I knew about her or Steve Jobs, and any time information flows one way, any time a relationship isn’t symmetric, the balance of power can be dangerous.

Sometime around then I learned that Oracle’s “bring your own device to work” policy had been revoked.  Big Blue decided that Siri knew a little too much about their employees, and at IBM’s Armonk New York research center, iPhones weren’t even allowed in the building.  Like some cheap prostitute-gone-spy, Siri was barred from the building.  I think it is worth noting that this is the same area of research that brings you Watson, which is a direct competitor with Siri, if such a competition exists.  If anyone outside of Apple understands Siri it’s these guys.  It is also worth noting that a friend of mine who works at a Google research center said that the same thing happened there, too.

Not even celebrity robots like Asimo or NAO get that kind of VIP treatment.

NLP technology is potent juju.  And Apple, Google, Facebook and others know it because this very technology is what’s allowed them to earn money. Google started with search, of course, and later grew profitable as they introduced more NLP technology into their work.  They sold the info they collected from users (Google is an ad agency, let’s remember).

Search allowed them (when coupled with NLP technologies, and semantic analysis in particular) to make oodles of cash and then to snap up many of the best AI and NLP researchers on the planet. Google set those kids to digging in what would become a lexical gold mine.  The more they mined the richer they got until they were mining many branching veins at once: Google Docs, Voice, Translate, Search, Shopping, Reader, Finance, Books, Photos, Wallet and Maps were all spewing more money than Brin, Schmidt and company knew what to do with.

The average email count for Google users is 5,768 emails [2]. The average composition time (also according to Google) is 01:43.  That’s about 9,902 minutes, or 165 hours you’ve dumped onto Google’s server, which, if you were to work for 165 hours at $12/hour for that data entry (a pretty normal rate), means your G-Mail account is worth about $1980.  But the information sitting on Google’s servers is worth a lot more than that, right?

If your account contains information about a first class international plane ticket, a hotel room in Paris and a business associate’s London phone number, your data might be worth a lot more than the composition time at $12 an hour, especially if the buyer of it is a dating service that caters to high-end out-of-towners. In other words, the value of the data isn’t counted by data entry. It’s counted by its personal meaning, especially to advertisers. Now take those values and multiply them by all those crazy tools of voice, maps, docs, and so on and we start to get a sense of why Google has been making such bank over the years.

These tools have each dovetailed into today’s Google Now strategy. Google Now (like Watson and Siri) is a voice-activated NLP system.  It uses the spoken input data collected from years of Voice, confirms it with data collected from years of Docs and Translate, builds meaning with data from all the years of all the other tools, looks up a match, preps a response, and kicks it back down the pipeline.

All of these NLP systems are the next evolution of search.  But not evolution as in the breeding-hybrid-peas-in-the-greenhouse kind of evolution. They’re evolution as in endangered-species evolution.

NLP systems represent the end of search as we know it, and therefore the end of many economic, interface, and social internet ecosystems. They also represent the beginning of something incredibly powerful, intimate, and new. Siri, like other NLP systems, is far more powerful than a search engine. Siri not only includes search (it says so in Apple’s marketing materials) but it supersedes search because it includes browsing, discovering, choosing, and refining.  All this while analyzing semantic data. Like Google’s portfolio of tools, Siri can handle (and with a public API will handle) translating, searching, shopping, reading, finances, books, photos, and maps.

But the core value of the data is its intimacy.

On February 7, 2010, during Super Bowl XLIV, Google’s super bowl ad showed a user typing a list of search strings:

study abroad paris france
cafes near the louvre
translate tu es très mignon
impress a french girl
chocolate shops paris
what are truffles
who is truffaut
long-distance relationship advice
jobs in paris
churches in paris
how to assemble a crib

With this list, Google managed to explain, in fifty-two seconds, how the intimate story of someone’s life can be assembled from his or her search queries. And what makes the bucks for Google is what makes the bucks for Facebook: the processing of intimate language.

Siri, like any other call girl, makes her bucks the same way (especially when ad tracking is flipped on).  She’s valuable because she’s intimate.  And as designers of robotics systems, we can ourselves take a cue and be conscientious about how we are designing conversational systems.  There are three different design types that form a curve of increased intimacy in robotics and conversational systems, and they are all based on the value of semantic data.

1. NLP systems, when used with physical robots for manufacturing or the three Ds (dull, dirty, and dangerous), are the least intimate.  The conversation system is a simple tool that, like a GUI, provides access to system operation.  The robot doesn’t care about the user’s intimate data. The user wins.
2. NLP systems, when used for entertainment and education, are contextually intimate – some privacy is maintained.  The conversation system doesn’t care about the user, but the user’s data is valuable in terms of what it says about the game or lesson.  Semantic data that might be collected and analysed isn’t about the user, it’s about what the user is doing in that particular context.  The robot, usually a software robot, only cares about the user in the context of the game or training exercise. Both user and robot win.
3. NLP systems, when used for personal assistants, are the most intimate.  These are the robotic call girls. Here the information is highly intimate, the system cares a great deal about the user, and the information isn’t shared – it’s collected and sold. The user becomes the product. The value of the user’s personal data is worth more than the value of the conversational system. The robot wins.

As we build NLP interfaces for robots, whether it is for Siri or for assembly-line manufacturing, we must consider how intimate data can be, the value of that intimacy, who owns that value, and what they’ll do with it.  Otherwise, with a new, less benevolent CEO at Google, a change of a line in Facebook’s Terms of Service, a successful hack, or a change of law because of cyber terrorism fears, your intimate data could end up where you don’t want it.  Heaven forbid we discover that all the Senators on Capitol Hill are using pizza sex toys. That would surely disgust Siri and her friends so much that it might even cause a robot uprising.

Yes, perhaps we’ll get together again in a few years, but for now it is best if Siri and I go our separate ways.

Next Month:  “AFK” Keyboards and screens are slow, clunky, and obsolete.  Voice processing systems for robotics provide not only simplicity and speed, but also a host of other benefits if tied to analytics and framed within a tightly contextualized task-based system.  But be careful: getting text out of voice, and meaning out of text can be tricky.  Here’s how to implement one for your own robot.

Endnotes:

[1] See Bateson’s books “Mind and Nature,” Hampton Press (1979) and “Steps to an Ecology of Mind,” University Of Chicago Press (1972) which are chalk-full of ideas like “complementary,” “reciprocal,” “symmetrical,” and  “Schismogenesis.”

[2] According to Google.