Comrades, we live in a bleak and humourless world.  Here we are thirteen years into the twenty-first century, and we all carry around Star Trek style tri-corders, we have access to almost all human opinions via this awesome global computer network, we have thousands and thousands of channels we can flip through on television, we have something like 48,000 people signed up to go colonize Mars, and we even have robots roaming around up there, taking samples of that planet. But we still don’t have robots that can tell a good joke.

Even this new bartender robot, Makr Shakr was prime for a little humour, like, “A robot walks into a bar and asks for a screwdriver..”?

But seriously, folks, in my travels I have met with Honda’s Asimo, Adlebaran’s NAO, Toyota’s Partner Robot and many others, and each of these presumably world-class robots was, in the most critical sense of the word, robotic.  They were weirdly-shaped computers with an appendage.  If you give me a choice between a robot that can walk and one that rolls around on wheels, I’ll take the one that can talk. Simply put, talking is more important than walking. Design priorities in robotics should focus first on human interface and second on locomotion.

We live at a moment in which natural language interface is replacing the GUI just as the GUI replaced the command line.

Natural language interface developments are being led by Google and Apple, with some strong work being done by IBM.  Last week Google’s Chrome browser appeared with a little microphone just to the right of the input bar and Google Now has been out for a year or so.  Apple has had Siri, of course, and is now rumoured to be building a wrist watch that will have a natural language interface to it, probably building off the lessons learned from Siri.

Natural language interface makes sense anywhere there is big data or little screen.  It makes sense for robots for these same reasons. It provides access to more data and reduces the need for a screen. But most of all, building a natural language interface also gives the robot personality.  And this makes the robot more valuable to the human because it turns a robot into an emotion machine.

Giving a robot a personality includes two primary components.  First is psychology.

Last month, in The Uncanny Valet I wrote a bit about the design of psychology, specifically for android-style robots, which wasn’t so much an effort to write about human-shaped walking-talking robots (androids), as it was to point out that human interaction seeks human likeness. To emotionally bond with a robot it needs to be like us.  The more it is like us the more we can bond with it.  This is most important in the field of automated psychology.  This isn’t a new field.  If you don’t believe me dial an American 800 number.  Those horrid little automated voices that corporations use so widely these days are reproducing faster than feral rabbits (may we name them robbits?).  In fact, I would love statistics on how many voice answering machines are out there, but I would boldly bet on there being, on the market today, more software robots than hardware robots.  I can tell you that this number is increasing and the quality of software robots is improving.

Each month we see advancements in what amounts to software robots, or virtual knowledge workers.  Last week Infosys (a Bangalore-based provider of IT services) announced it is working with IPsoft (a New York-based provider of autonomic IT services) to build autonomous tools designed for help desks, operations, and other management needs. In an article about the the new partnership it is mentioned, “IPSoft has focused on autonomics for IT services, and Slaby estimates in his report that nearly all Level 1 support issues can be solved by IPsoft’s ‘virtual engineers’.”

We are past the primitive dawn of automated psychology.  It is becoming a branch of the robotics industry.

As graphic design for the GUI was to software design for the personal computer industry, personality design for natural language is to the software of the personal robotics industry.

At Geppetto Labs we build conversational systems.  This means that we take the natural language input from a user, pipe that through an analytics system that measures affect, syntactic and semantic values, and then feed the resulting data into an NLP system.  We author personalities via simple server-side tools that script what non-technical authors (generally film screen writers) have written into formatted rulesets. We also bend Chatscript and other tools for non-personality behaviour, like enhancing Amazon search by natural language understanding, using specific phrases or moments to trigger specific animations, etc. After that we stir in some scraping, automated ontology creation, and even tools to automate first person interviews and film scripts to craft a personality that can be managed. This is the foundation for natural language interface – at least how we do it.

And there’s an entire second level to natural language interfaces that has nothing to do with text.  In more advanced projects Geppetto Labs need to add animation that fires when the NL output does.  The simplest example I can think of is the word “Hi” being tied to a raising of the hand.  Hand gestures are easy, and clear, but gestures are only one part of the body language we build.  Body language might also be a facial expression, the position of the arms or the tilt of the head, but what it means is that we have to not only think about the psychology of the system, but how to tie that to the activation of non-verbal forms of communication as well. We have to walk the talk. That means we have to design body language into the personality.  It is linked with the psychology and frames the text.

Psychology needs to be graphically represented, just like a book needs a cover, and body language is the best means of achieving that.  Well, at least that’s where evolution has taken us.<

Body language coordinates the psychology of the personality with the visual presentation. Coordinating what is said and what is done is core to the personality.  This very coordination might be what we commonly call personality.  If you don’t believe, me consider this: watch television for a day and note what “Personalities” do; newscasters, politicians, actors, singers, and dancers.  They accompany their words with the most broad-stroke gestures and body language humans convey.

This is to say that body language may be more important to natural language interface than the words used.

Many famous studies claim that body language accounts for the majority of our communication (the most famous is probably Dr. Albert Mehabrian’s 7:38:55 rule).  Or, more precisely, that the words we use transmit the minority of what we communicate.  So how we move and what we visually communicate while we verbally blather on is pretty key to how others understand us.

Consider the eyes.  We blink more when we’re stressed out. It can indicate dopamine levels and both the pain or the creativity of the person we’re talking with.  The timing of a blink can indicate a punchline to a joke, a response to that joke, confusion at the punch line, or frustration at how it isn’t funny.  Pinched and held tight can mean something entirely different.  We don’t like to keep eye contact too long or it starts to feel threatening.  The person who listens should initiate eye contact – and if they don’t it might mean they just don’t care what you’re talking about.  But don’t break it too soon or the person listening will feel as if they’re not important. Don’t look at the ceiling when you talk (or you might come off as a snob) and don’t look at the floor, either (or you might come off as a bummer). If you’re talking with someone and they look up and to your right it generally means they’re remembering something, visually. If you’re talking with someone and they look up and to your left means they’re visually creating something.  Horizontal looking while talking indicates the person is most likely accessing auditory cues and, to your right is, again, remembering and to your left is, again, creating.  And if they look down maybe they’re depressed. Or maybe they’re just looking at the carpet. Or maybe they’re remembering how something feels.  That applies to most right-handed people around the world (according to John Bandler, Richard Grinder, and scads of other neurologists), but it indicates that there’s a lot happening when we speak with someone. A lot more than words.

But the eyes do this while other things are going on.  The eyebrows and cheeks are wiggling, the mouth opening and closing, the head is tilting and nodding, the shoulders are going up and down, the arms are waving around, the hands are flapping … and that’s all above the belt-line.

Please remember that human interaction seeks human likeness, so the body language of the robot needs to mirror the body language of the conversant. Think back on one of the early successful chat bots, ELIZA. This system was designed to simply mirror what a person said to it and repeat that back, like many psychologists. Several people evidently fell in love with the system.

The corollary of this is that a well-design natural language system will move like we do.

Today, we have a new graphical version of that, designed to note and reflect the body language of the conversant. Its name is Ellie.

A webcam and a game controller track eye directions, run some facial recognition software to chart expressions, and do the same with body positions. While the system is being trained in realtime in its early days, this training goes towards the video mapping input to the system’s output, just as any NLP system (such as Siri or Watson was designed, and exactly as we do our work at Geppetto Labs). It’s being trained on how to read people.

But there’s a missing part here, which I will guess the good Mr. Stan Rizzo is considering, which is generation of response movement in real time.

How the character moves, how the robot moves, is a key component. Just as natural language has the component parts of understanding and generation (NLU and NLG, respectively) body language has these same input and output cues. At Geppetto Labs we have several characters that we’ve been working on that have this running, and a process I’ll outline next month.

A robot walks into a bar and asks for a screwdriver, and the bartender says, “What is this, some kinda joke?” Personality, like a joke, synchronizes words and body language, linking them together and leaving us. “A Paro walks into a club” is about as bad as I get, and I promise I will not repeat the joke, but I bring it up as an icon of how to build personality: combine the mind and the body.

Natural language interface, when properly designed, is far more than just text.

Part One: Another robot dream girl

I’m talking with this woman on the phone, interviewing her for a job. She’s got a slight southern twang, maybe from Georgia, telling me about her past work, and as we talk I’m struck by how perfectly presentable, alert, and smooth she sounds. Her voice is soft, tonally fluid, and she articulates her words precisely, belying a geeky disposition. Seems she’ll be perfect for the rather tricky tech support position we’ve got.

“Did you enjoy the work?” I ask.

“Sure, I like talking with people. I mean, as long as they don’t get upset it’s sorta fun.”

“You were working at a bank, right?”

“Yeah,” she laughs, “There were four of us working the phones and please press seven if you’d like one of us to call you back.”

“Sorry?” I ask.

“There were four of us on the phones and please press seven if you’d like one of us to call you back.”

I realize this robot has a bug, thank her, and hang up. I won’t hire that one.

Of course I never spoke with the uncanny woman again because the above story is fiction; a conversational system like this one is something of a dream among today’s robotics designers. She might have had a bug in her bonnet, but at least she could hold a conversation and answer questions, for the most part.

A system like the above, a Natural Language Processing system, is the heart and mind of a robot.

By contrast, consider what we have today: those irritating, tyrannical little phone robots that have replaced perfectly good humans: “Please push one if you would like to make a deposit. Please push two if you would like to hear these options again.” These non-player phone-workers are so brainless, grindingly slow, and weirdly rude that I find myself repeatedly pushing keys like 0 and # just to get to a real human.

Of course, from the cold eye of the bank (or whatever corporate conglomerate that employs these “Customer Relations Management” systems) these virtual robots reduce costs by replacing a fickle, distracted, complaining, prone-to-error, temperamental, sleep-deprived and very expensive call-center human with a voice recording. It saves the company money, time, and tons of overhead on things like worker’s insurance, sick-leave, replacements and the high cost of managing people. People are hard to manage, right? Voice recordings are not. People sleep at night, right? Voice recordings do not. So virtual robots, and this particular feral species of virtual robot – the phone robot – is a means of disintermediating the person from the knowledge: it is a means of scooping the knowledge out of the head of person that was doing the job and serving it to you, the customer, couched in smooth tones with a dose of saccharin-sweet politeness.

But virtual phone robots in their present form are bad at solving problems, they don’t learn, they’re inattentive, and they can’t even understand me. I have few options on what I can do, and I get pushed into decision trees that I don’t want. I feel herded like a cow, and with that little phone-robot nipping at my telephonic heels to move me into the right slot for processing, I more often than not either hang up or return to the website.

The intention is that robotic knowledge workers can replace human knowledge workers. But robotic workers are not capable of the job: rather than collaborate, they dominate; rather than answer questions, they force responses.

Bad robot, bad!

Part Two: The problem is the rotten brains

According to research from Nuance corporation, there are three main ways to improve phone robots. The first is to allow access to a real human. Two-thirds of the customers interviewed wanted that. About half of them said the system’s logical call flow was most important. And about forty percent of them said that the speech recognition component was the most important. So access to a live human, logical flow, and recognizing what’s been said are the three main problems with phone robots today.

If two-thirds of the users of a system that is supposed to replace a human say that the system can be most improved by providing access to a human, then that system is mostly broken.

Phone robots may save money, but they suck. We expect them to act like humans because they have a human voice, but instead they act like robots. This just adds to our frustration by reminding us that we do not have what we really wanted in the first place: a real human to talk to. The sound of the language is right, the annunciation is right, but it is a facade that just highlights the problem.

These phone robots live along the upper edge of an uncanny valley (if you do not know what the uncanny valley is please stop reading this, right now, and go look it up in Wikipedia).

Masahiro Mori pointed out many uncanny valleys, but the most famous ones are related to appearance and movement. We have yet to discover the many other uncanny valleys out there, especially as robots – and androids in particular – become progressively more human. As the systems become more capable and more interactive (phone robots today are almost always just voice recordings with a button-activated decision tree), more questions will arise. This will be exacerbated as physical robots become more human-like.

Should a robot’s voice sound like a human’s? Whose? Should a robot smell just like a person? If so, which one? What about how frequently the robot blinks? What about the tone of voice or intonation in sentences? What about social abilities?

Name a human trait and it will become an uncanny valley. These uncanny valleys will, in the coming decades, be explored and mapped. Mori just hit the tip of an icy moon that will chill, mortify, and entertain us for decades to come. We’re going to find uncanny valleys all over the pocked, lunar landscape of robotics design, and the one that will be most important to address is psychology.

First we have cognitive psychology. Consider all the traits that we humans pack into language, such as logical flow, attention, learning, language, and emotion. These are classically cognitive traits, each of which the phone robots of today should pay attention to. (Of course I mean “The designers of these systems should pay attention to.”)

Second we have social psychology. Consider all the traits that we humans pack into interaction such as politeness, collaboration, competition, and understanding. These are classically social traits, which the phone robots of today, or more accurately, their designers, need to consider.

Let’s lump them all together and imprecisely call them psychology. This is the uncanny valley that most concerns me. How a robot holds a conversation, thinks, and speaks seems the most important, and urgent, of the valleys ahead.

Robot psychology is urgent because it’s happening today. You have spoken with a phone robot and chances are (76%, according to Nuance) that you were not satisfied with it. So there’s a problem here.

Robot psychology is also important because we mimic the way we are spoken to. If you don’t believe me ask yourself why you speak the language you do. Reading this article means you probably grew up in a place where people mostly spoke English. What about your accent? Again, the people around you influenced you to speak that accent. They also influenced you to use the phrases, slang, jargon, words, ideas, and therefor fundamental psychological scaffolding you use to support your notion of reality. We do as we are done unto. When you listen to someone, their mind enters yours.

If a robot treats me in a rude manner chances are good that I’ll mimic that. If I’m going to be working, playing, living, or talking on the phone with a robot, I hope it will be a healthy example of both cognitive and social psychology that I can interact with and not be polluted by. As these systems become more prevalent their influence on our own behavior will increase.

This dynamic and interaction is both important and urgent.

It is why we are becoming our robots, and why our robots are becoming us. As they speak like us we will speak like them, and vice-versa.

Please press the down key if you would like to continue reading.

Part Three: The solution is to design robot psychology

Once upon a time I had the odd luck to talk with the Geminoid F, the most realistic android on the planet (that was two years ago). It is the life work of Dr. Hiroshi Ishiguro. This uncanny experience would not have been so unsettling if Dr. Ishiguro’s doppleganger hadn’t been wading around like a zombie in the muck of the uncanny valley, but there it was and I – like the rest of the bottom feeders – was there with it. To this day, wish I had not gone.

Dr. Ishiguro is a moribund and curious fellow, and his robot more so. The problem, as I detailed in my book, “We, Robot” was that the robot was so real that it set an expectation of being human.

Please consider this face, and the heart and mind behind it. You do not, really, want its mind to enter yours, do you?

Communication – and staying out of uncanny valleys – is all about reflection. It has to do with mutual empathy and mutual agreements of closeness. It is about familiarity.

In my last article I talked about the design of androids, wrote about the designers that made them, and noted that the function of an android is psychological closeness. Androids (uh, or rather their designers) want you to feel comfy. The core notion of an android’s design is to set you up to interact with a computer in a traditional human setting. It’s about an expectation of human familiarity and something that reflects back to you a human image of yourself.

We humans are built such that we like talking with other humans. We like what’s familiar, what is, etymologically, of our family. We like talking with a familiar face, and if the robot doesn’t look like us then we tend to freak out a bit. After all, our friends usually look like us, dress like we do, and come from the similar socio-economic background. I might not like to admit it, but that’s usually the case. And, just as we like talking with a familiar face, we like talking with a familiar psychology; the psychology of our family. Our friends think like us, act like us, and like the same things. Birds of a feather, it has been said, flock together.

This is because much of the best communication is a reflection and collaboration. We like reinforcement, familiarity, and a reflection of our presence in the world. We do not like talking with someone who herds us, we do not like being told to push buttons, and we do not like phone robots because they fly in the face of almost all human-based computer-human interaction (CHI) theories (such as the android). Phone robots are a good example of a bad psychology.

If you were still a child, would you rather grow up talking daily with Ronald McDonald, Shrek, or Kung-Fu Panda? C’mon, choose one. Now what if a million children were to be forced, by your decision, to do the same?

As designers of robots we are faced with the same type of problem. As kids grow up interfacing with robotic personalities, and virtual robots, they will be influenced by, adopt the behaviors of, and intellectually be guided by these personalities. It is our responsibility to offer something better than junk food for the psyche.

Siri is about the best form of phone robot we have on the public market today. From my interactions with her she’s a little vapid, slightly sardonic, sometimes helpful, but mostly apologetic. When she makes a joke, which is sadly rare, or when she directs us toward (or steers us away from) some product, she is affecting our psychology.

There are some two hundred and fifty million versions of Siri out there. Let’s conservatively call that two million active versions of Siri functioning today. And I have no clue how many of Siri’s phone robot ancestors are out there, but we can safely say that the psychology of these systems is affecting the psychology of millions of people.

Ahead of us, the design steps for implementing android psychology are not evident, but we can follow in the footsteps of other people and extrapolate from character-based properties already in the wilds and public media of today.

If we look at how famous branded characters like Ronald McDonald or Kung-Fu Panda have been developed, marketed and sold, we can make out the guideposts for cultural preference and gender. If we look at archetypes like Darth Vader or Voldemort we can see how to design the bad guys. Superman and Gandalf give us clues on the good guys. Homer Simpson, Indiana Jones, Buzz Lightyear, George Bush, and Jacques Chirac are all personalities that have left an indelible public impression that we can model and use in designing psychology.

Presentation is important, too. Of course the system has to speak your language, and therefor regional preferences will guide the design of these robots. Dialects and accents count today. We can already see how Siri has been implemented with various accents such as American vs British. Gender will also continue to be important (70% of the automated voice systems in Europe are female, and in North America, 30%). After presentation we have the personality, the archetypes, and what the character knows.

Authors are the people that know about this kind of design. People who write character portraits, who ‘get’ dialogue, and who are socially alert duplicators of human interaction; people who make movies, people who conduct interviews, and, unfortunately, people who make advertisements: these are the people who can design systems that avoid psychology’s uncanny valley.

If we look at avatars in social media, and if we consider the movie industry we can see some other guideposts for the future of the psychology of android design. The most important is interactivity and reflection of shared interests. This takes us back to Nuance’s research.

Access to a real human will become increasingly rare. Regardless of what consumers want, the whole idea of a phone robot is 24 / 7 availability at a decreased price, and this is some thing a human cannot do. Next on Nuance’s list was the logical flow of the call and the speech recognition ability. These two important interactive components will surely get better in parallel as understanding one helps to improve the other.

Most of it will happen via mobile. Consumers in the United States – which roughly maps to much of the rest of the industrialized world – prefer to resolve their customers service issues using the telephone (90%), face to face (75%), company website or email (67%), online chat (47%), text message (22%), social networking site (22%). So it seems that while these conversational systems will live mostly in the phone, there will be a place for them in other media as well. They will live in the wires and waves that surround us, always available, often improving.

This is how the system becomes, finally, conversational. The irritating little herd-dog phone robot will evolve into something closer to a Geminoid. One day you will think you’re talking with a human on the phone, and then you will find yourself in a new uncanny valley. It won’t happen for at least another five years, but we’ll stumble into it eventually. The psychology of that robot will be a thing that touches you and guides you in ways that corporations like Google (who have powerful NLP tools today) Facebook (who released an NLP search system yesterday), can only begin to hint at.

They will be able to speak to you, sell to you, buy from you, offer medical guidance, marital guidance, and talk with you about your depression, your homework, your spouse, and your boss. Semantic Analytics, Social Analytics, and NLP will accurately measure your responses and the system will reply with a warm candor. Young men will ask about sex, old women will ask about hysterectomies, and all the other most awkward, personal, relevant, and important questions of life will be whispered, spoken, laughed, and sobbed into these many kinds of virtual androids that will gradually collect, like little honey bees collecting pollen and taking it back to the nest, your very own fears, loves, desires, hates and general, if I may say, psychology. The phone robots of tomorrow will become tiny transporters of emotion and personality. You will leave a voice recording and it will be taken, before you have hung up the phone, to some hive-mind.

The question we may ask these little bees is, “Where have you taken my personality?”