Virtual assistants are setting the bar for the kind of user experience people will expect from the social robots of tomorrow.

At the Apple party earlier this month, at the Bill Graham Civic Auditorium in San Francisco, CEO Tim Cook and other key execs took to the stage to announce their next-generation product offerings. Apple’s been setting trends in user interface (UI) and technical design for over two decades now and its latest products didn’t disappoint: Siri already inhabits our iPhones and responds to voice commands, but now she will soon be embedded in Apple’s TV remotes too, so that users can change channels, find weather forecasts and get sports scores simply asking for them.

Siri demonstrates just how far we’ve come from the early days of interfacing with computers, databases and other information systems. Whereas systems used to be able only to respond to constrained queries, such as a Google search, Siri is now intelligent enough to respond appropriately to fairly detailed, human-like, requests, such as “Show me the Modern Family episode with Edward Norton” or “Skip ahead seven minutes.” She can even respond to questions such as: “What did he say?” Siri will simply skip back 15 seconds to the previous line the actor said. It’s an innovative new interface for controlling TV and a host of other media, and it’s likely to change how we interface with computing even more significantly than the introduction of the GUI (which, if we remember, Apple also had a hand in rolling out).

In addition to Siri, there’s a swarm of other software robots descending from the clouds: Google’s Google Now, Microsoft’s Cortana, Nuance’s Nina, and the Amazon Echo, which plays music, reads books and can buy stuff listed on their site. All of them keep track of what you like, when, and are able to draw massive amounts of data from both your voice (gender, age, region and other vectors) as well as the words you use. Baidu announced Duer in the first week of September with the intention of providing a voice interface for the home and internet of things (IoT) services and healthcare support. The company also  plans to integrate it into self-driving cars.

And recently Facebook’s M was touted as a concierge service available through its messaging app. This is a smart play, as Facebook is manning the assistant with a crowd of  employees (called M trainers) to simultaneously answer requests and train the system to improve responses. It seems a short step for Facebook to place ads in M (much as they do in site content today), but the real revenue stream will be in collecting user data. For the time being, as with Baidu’s Duer, Facebook’s M is not voice-driven, but uses text as a feature within the Messenger app.

Our mobile devices are becoming natural language interface hubs for life management and, as a result, having a gravitational pull on an increasingly complex buzz of connected services and APIs.

There are many others: Samsung’s S Voice, LG’s VoiceMate, BlackBerry’s Assistant, Sirius, HTC’s Hidi, Silvia and Braina, to name a few.

We can also foretell the future by looking at less advanced natural language systems. Bots – essentially natural language oriented scripts – are a good indicator of where the robotics industry is at because bots are pervasive, useful, and simple to author. TwitterBots and FacebookBots crawl through these systems like bees in a hive, industriously providing retweets, reposts, summaries, aggregations, starting fights and flocking to followers. They can be bought, auctioned, sold, and deleted; you can buy 30,000 Twitter followers on eBay for as little as for $20, provided they’re all bots.

Several years ago Facebook estimated that around five percent of all accounts are bogus – this would put the number of Facebook fakes at around 50 million – while other, now-antique estimates range as high as 27 percent (that’s about 200 million trash-trawling bots). They’re relevant because emergent technology arrives from the fringe.

According to Gartner’s 2015 Hype Cycle chart, Intelligent Virtual Assistants still have a ways to go before they get good enough to go mainstream (likely, 5-10 years). But also according Gartner, roughly 38% of American consumers have recently used virtual assistant services. They predict that, by the end of 2016, around two-thirds of consumers will be using them daily. Other sources forecast that the global Virtual Assistant market will grow at a CAGR of 39.3% between 2015-2018, and the total market is projected to climb to more than $2.1bn by 2019. These numbers cluster around the trends in natural language interfaces.

These trends show no sign of altering their flight path. Our mobile devices are becoming natural language interface hubs for life management and, as a result, having a gravitational pull on an increasingly complex buzz of connected services and APIs. This means that things like search will change: we will no longer have to speak Googlese; paper and page metaphors will be supplanted by the more dynamic (and cognitively more addictive) character metaphor. And if trends in virtual assistants and intelligent helpers – software robots – continue, then knowledge-bases (such as Wolfram Alpha or IBM Watson) will continue to come peppered with a patina of natural language, allowing us to move through data faster, with less training, and in a more human manner.

This is the trend in software robots, and it is headed towards hardware, too.

Hardware is the new software

Software robots show us what hardware robots will be expected to do in the near future. First, natural language is the de-facto interface for a range of functions from social robotics to customer service and personal healthcare companions – not only because it increases the quantity of data collected, but also because it decreases the cost of collecting it. Not only because it replaces people, but because it amplifies them.

That said, the natural language interface, or voice UI, is only one of many that robots will be expected to provide. Virtual assistants can now perform a range of functions based on the online services that are commonly integrated. Since these services can now be integrated into any connected system, we can put together a laundry list of what hardware robots will be expected to do:

• Understand voice input (respond to commands, answer questions, etc.)
• Offer personal assistance (schedule appointments, set reminders, check stocks, etc.)
• Provide mobility assistance (give directions, predict traffic, find parking, etc.)
• Interface with connected devices (TVs, garage doors, lights, surveillance cameras, etc.)
• Allow you to buy stuff (and keep track of what you’ve already bought)
• Interface with social media (send photos, read tweets, etc.)
• Detect mood (read emotions, body language, save state, etc.)
• Build predictive models (track user behavior, save user state, etc.)

Robotics will increasingly adopt a voice UI, akin to what we see in today’s personal assistants, because task completion is simpler, faster, and more effective, menus go away and personality (the UX of NLP) is a lot of fun. These software robots are testbeds and proving grounds for what users expect from tomorrow’s hardware robots; they set a bar for user experience. Personal assistants, like their distant cousins from a hardware lineage, will soon be smart enough to reply to other robots that call us on the phone, and we will want to equip our Jibo, Nest or Alitalia system with it’s own resident assistant that will answer the phone when the robot from the cable company calls.

Whether you happen to be Siri or the gal working for customer service at the cable company, your job, your life, your car and your family is being invaded by the swarms of personal assistants changing tomorrow’s robotics industry today. We have a choice: either we design hardware robots as assistants and companions (to help us accomplish tasks and keep us company), or we design them like bots (to mull the garbage, aggregate the trash, and harass one another).

The future, it seems, contains both.

Now, I don’t know about you, but a patent on personality seems like a patent on DNA, as if Google had announced, “All your souls are belong to us.”  It made me a little jumpy because my company writes personalities for voice-control systems, be they robot, IoT appliances or avatars. Maybe, I thought, this will stunt the growth of the robotics industry and we will finally be faced with Evil.

What about all the people developing robots? Maybe they would save us. Maybe, on the other hand, all the heroic robots from 1970s would save us. A robotic personality isn’t a new concept, afterall.

I wasn’t the only one to sit up and raise an eyebrow. To get a sense of perspective I asked Alexander Reben (robotics artist and engineer) what he thought. He replied, via email, that “what Google has laid out in this patent appears to almost define the field of social robotics. It seems to be quite broad claiming to cover aspects which maybe obvious to those in the field.” Kate Darling (researcher of robot ethics and intellectual property) also thought it was quite broad.

Diana Cooper, who specializes in the legalities and and ethics of robotics (she’s Head of UAS and Robotics Practice Group at LaBarge Weinstein), was more optimistic. She wrote, “I can see this technology being very beneficial, particularly when robots are tasked with assisting in providing treatment to vulnerable populations such as children with disabilities. On a broader scale, I am optimistic that the personalization of robots will counteract society’s anxiety about robots and lead to increased consumer adoption.”

Andra Keay, of Silicon Valley Robotics, pointed out, “I think that Google are a little confused about what a robot is. To quote from one section of the patent: ‘something with processing capabilities and sensors‘ … If there weren’t pictures, I’d have assumed the patent was describing a phone or tablet.”

Personality, like the GUI, is a user interface.

However you want to think about it, the patent was issued. So there it is.

Personality is core to user interface. Personality is – like the graphic design of a GUI – the element that not only allows the sorting and categorization of information but also allows a person to emotionally bond with a robot (or avatar or talking IoT appliance or car or whatever). Personality in a video game interface is different from personality in a business SAAS. Personality also squats at the very foundation of affective computing, human-machine interaction, and even human-human interaction.

When it comes to robots, personality might actually BE interface.  If I walk into a hotel room and there’s a robot waiting to greet me, its personality will frame what we do together. If it says “Good afternoon. Would you care to order something from room service?” that’s a very different system than if it says, “Yo! Wanna throw back a few vodkas and play Call of Duty!?” Personality interface, like graphical interface, frames user interaction.

Why This Invention Is Not Patenting Personality

Titled, “Methods and systems for robot personality development” the patent (#US8996429B1 issued by the USPTO on March 31st, 2015) was authored by two employees at Google, one a researcher (Anthony G. Francis, Jr.), the other an R&D UX Creative Lead (Thor Lewis). Knowing their past work is important for understanding this invention – and what place the patent has in our emerging robotics ecosystem.

Google’s Robot Personality Patent

First, according to the work of Francis (a PhD in AI from Georgia Institute of Tech), context influences how we act. Context is something that allows us to remember, reason and then expend energy appropriately. It allows us to make the right decision at the right time. The context of a task guides its action. Francis has also spent a good deal of time looking into affective computing. And it just so happens that he’s a science fiction author.Second, Lewis (artist, 3D modeler, designer, and science fiction buff, himself), has authored other patents based around machine vision, and came to the world of AI vision from the inside out. He went from making visual things with computers to making computers visual things.

So these two get together and we may imagine their conversation in the Google Coffee shop one afternoon:

“Hey, if a robot can see a thing, like a person, and build context, then doesn’t that inform what the robot does?”

“Yeah, and maybe the robot can get a sense of who that person is based on the stored personal information via their online accounts.”

“Sure, so if the person is at a wedding or a funeral and that would change the way the robot would interact with them, right?”

“Right, and it could remember its past personalities and interactions!”

“Totally! And it would live in the Cloud!”

“Awesome!”

“Hey, you got a pen?”

As I read the patent I realized that this isn’t an invention for personality. This is an invention for how to modify a personality in a context. The way that the robot looks, moves, sounds and feels (and touch is mentioned, too), all impact the personality, which can be transferred via the cloud, so personalities can be uploaded, downloaded, shared, and modified. But the patent isn’t as broad as it might first appear. This isn’t inventing personality, per se. It’s inventing how a personality builds context and interacts. The last word of the patent’s title explains that this is not about what personality is, but how personality is developed.

How To Develop A Personality

There are a few pithy lines in the invention. “Personality may be thought of as a personification in the sense of human characteristics or qualities attributed to a non-human thing.” Then the authors set out how those characteristics develop in a context.

Multiple devices are cross-referenced. This allows the robot to collect information on a user and cross-reference data to build context. This means that GoogleBot can now tell that I like to eat cake at a wedding, but not at a funeral, and then modify its personality to align with that context. It will know to offer me cake when people are getting married, not buried.

The patent reviewer, in the “Reasons for Allowance” notes that: “processing to obtain ‘data’ as opposed to information theoretic ‘information’ occurs in response to ‘obtaining’ the information.” This is specifically cited as a part of cloud services as they relate to collecting visual information. Thus the patent differs itself from prior art.

But why would a robot personality development patent be useful to Google?  This is core to Google’s advertising business model. GoogleBot will know whether I prefer to eat cereal in the morning or evening, what kind of cereal, what kind of milk, what kind of bowl, what kind of spoon, table, and the timing it takes me to eat that cereal. And just like the business model of web searches from the late 1990s, Google will then be able to sell that data to the many advertisers that support Google’s highly valuable business. Not just cereal of course, but shoes, shirts, shit, shinola and all the other things we consume, produce, and interact with in our daily lives. This context means that the GoogleBot will become one of the most powerful marketing, advertising, and search mechanisms ever built.

A personality that can consolidate our data is one that truly knows us.

GoogleBot’s adaptive personality can consolidate our data and that consolidation knows us. The GoogleBot, with its emphasis on vision, will be able to tell what other cereals I like, what other bowls I use, what other shirts I wear and when I leave for work. By simply observing it will (like search, email, docs, voice, Google+, and the host of other services that Google offers) know me. What is key to context – and to the value of Google’s USD$50 billion-a-year advertising system – is the consolidation of channels of information.

Channel aggregation is a proven and valuable business model. Apple did something similar. When Apple released the iPad (leashed to iTunes and other services), they consolidated the movie channel, the video rental store, the movie theater, and the television into a single device. Apple allowed us to consume more by aggregating media delivery channels.

Apple aggregates output channels to facilitate media sales and Google aggregates input channels to facilitate advertising sales.

Remember that your data is Google’s product, and they sell it to their customers. Imagine a Google Search engine housed in a little teddy bear that modified what it presented based on your past searches. Now imagine that the little teddy bear understood how you felt about those searches, and why. And it told you so, moved, sounded, and even felt a bit different as a result. Now imagine that the context of your search, via this sensitive, sweet robot that is a cuddly animal with an immense, encyclopedic brain, is now completely understanding you and adapting to your needs. Add the Google AdSense, flavor with Google Analytics, stir in Nest, DoubleClick, Uber, 23andMe, residential fiber, and the other Google goodies, and you’ve got something like the Google Panda Robot that was announced the same day the patent was issued (and no, this wasn’t an April fool’s joke). The Amazon Echo appears to be following a similar model, as I mentioned in a previous article.

Internet penetration growth rate is set to peak in 2016.

Google has to solve some important problems in the coming year, so this robotics approach is also timely. Internet penetration growth rate is set to peak in 2016. Since Google’s P&L is proportional to Internet usage, they need to invent some new models. Their three existing models – consumer devices, ad-supported consumer services, and software-as-a-service – are approaching maximum impact, and we can expect that they will consolidate all three into this robotic personality that will develop so as to offer each aggregated input channel when it is most contextually appropriate. Google can now consolidate the contexts of buyer-seller to complete the holy circuit of advertise-search. And the context of personality is the interface.

To Know Me is To Own Me

If we thought that value was inherent in goods themselves, then we were wrong. As advertisers have long known, the personal information of what cereal I like is far more valuable than the cereal itself. It will be the opposite of what we thought, especially if what is valuable is what the robot collects. The personal information collected to determine if we like cereal or not will be far more valuable than the cereal itself. So like Shroedinger’s cat, the robot will look for our personality in the box, and our personalities will shift as a result. The context will change and this sensitive interface is will adapt because it was invented to do so. Just like search on Google’s webpage, the robot’s personality is more about collection than display.

Google collects our personalities by cross-referencing our context; this new patent is about cross-referencing context to reflect personalities back to us. And this allows for an improved collection method.

So while Google’s recent patent at first appears to be a method for displaying robot personalities, it is actually a means of collecting our personalities, and therefor our purchasing decisions. It’s not that Google will own our souls, but they will own our personalities, and therefor our decisions.

“All your decisions are belong to us” is writ large upon the sky. But for now this is only a patent, and despite the appearance of the cute teddy-bear robots, our decisions are still left up to us.

At least for now.

---

If you liked this article, you may also be interested in:

• Why are JIBO, Pepper, Siri, Google Now and Cortana so important?
• The robots behind your Internet shopping addiction: How your wallet is driving retail automation
• Google commits $1.36 billion for NASA facility, to house their robotics, space and flight technologies
• Google news: Rubin leaves; Pichai promoted to #2; and Google invests in Magic Leap
• Google adds to DeepMind, acquiring 2 UK startups and partnering with Oxford U
•  Robots Podcast: Privacy, Google, and big deals

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

 

When I was growing up we moved around the US a lot and, as a result of hopping between cities and cultures, I ended up wearing the wrong shoes, carrying the wrong lunchbox, and not knowing basic words like “dude,” or “tabernac.” Of course my more athletic peers didn’t even know BASIC, let alone a paladin’s hitpoint adjustment, and their own ignorance warmed me with a greasy self-satisfaction that was a quick turn-off and earned me a few lumps after school.

Anyway, while the Jocks were sending me home with bruises, I met Ron. Ron was about half again my age, lived down the street, listened to The Cars, and had one of his own. He taught me how to play records, shoot a gun, paint, and light a cigarette. He also taught me how to fight, how to be cool, and he mentored me in the semiotics of sexy.

We humans are a semiotic species; we’re each a compost heap of symbols. Because our brand of lunchbox, cigarette and shoe matter to us, we laugh and cry over symbols as obscure as mathematics, sitcoms, haircuts, and handbags. Those decisions determine if we’re Geek or Jock (or Emo or whatever). None of us have a lot of control over it; our nature and nurture define our operating system.

Our origins make us Geek or Jock. It ain’t up to us.

Robots are semiotic, too. For example iRobot makes Jocks. Their combat bots, like PackBot or Kobra, are muscular and designed for a world of physical symbols, like guns, security, and surveillance. Apple, on the other hand, is making Geeks. Siri gives me Wikipedia pages when I ask her about BASIC or a paladin’s hit point adjustment (which isn’t actually all that bad and it is certainly a geek’s first reply). iRobot’s Kobra just won’t answer that dumb stuff. Is this because the folks at iRobot are Jocks and the folks at Apple are Geeks?

A robot’s origins makes them Geek or Jock. It ain’t up to them.

CHAPPiE is a story about this very stuff. Written and directed by Neill Blomkamp, CHAPPiE stars Dev Patel (as the Geek) and Hugh Jackman (as the Jock). The liner notes read, “In the near future, crime is patrolled by an oppressive mechanized police force. When one police droid, Chappie, is stolen and given new programming, he becomes the first robot with the ability to think and feel for himself.”

And thus breaks out a new battle in the war of Geeks-vs-Jocks.

Blomkamp has been chewing on this cud for a while. Tetra Vaal (2004) was one of his first films, and was about a robotic police force (robot as Jock). TempBot (2006), on the other hand, is about a nerdy office worker trying to get his processor around human social symbols (robot as Geek). As if design drafts, both of the robots in these movies have the same chassis as does CHAPPiE and his police-force counterparts.

CHAPPiE starts off as a real Geek: he even paints and pets animals. And then he gets picked on after school. But things change for CHAPPiE when he meets Ninja and Yo-Landi, of Die Antwoord. Die Antwoord is a South African rap-rave group whose sexy, hyper-visceral “Zef” becomes CHAPPiE’s semiotic mentor in the Geeks-vs-Jocks war. Die Antwoord balances between Geek and Jock, resting out at Gangster, teaching CHAPPiE how to shoot a gun, paint, walk, talk, and most of all, be sexy. There are even cigarettes.

While we could crassly sort Geeks and Jocks down the Mason-Dixon line known as The Mind-Body Problem, CHAPPiE evolves into a sexy robot. And he becomes sexy because he’s tough (without being a Jock), brainy (without being a Geek), and because he really embodies that sacred term developers use to mean beautiful, or reviewers use to mean pleasing.

Neither Geeks nor Jocks are sexy. Nor are robots. At least not today. Hell, robots can barely walk, let alone walk with gangster swagger. Robots are mostly geeky, probably because it’s mostly geeks that program them. Geeks are generally awkward not because we’re antisocial, but because we’re busy doing other stuff. Like making robots walk. And so it’s hard for Geeks to teach a robot how to be sexy. No blame; just semiotics.

But robots – and avatars in particular – are becoming sexy. If there’s anything science-fiction does for the computer industry, it’s to lead the way while simultaneously commenting on the present. Hatsune Miku (初音ミク), an Idoru, is now in licensing discussions to make physical versions for entertainment, and, well, other uses I won’t go into. In another example of life imitating art, the illustrations of Hajime Soriyama (空山 基) (who did the cover of my 2010 book on robots, by the way) were what influenced such famous robots as Aibo and others. In a rather NSFW article you can get your dose of soft robot porn and probably of the future of pornography itself, or a tamer overview, provided by the BBC.

Everyone has different opinions about what’s sexy. Maybe it’s not the band Die Antwoord, nor these gyrating porn gynoids, but more sexy like the way the iPad is sexy. Maybe there are alternatives to designing robots that consider the body-mind balance and the semiotics of culture, so that we can interface with robots that are beautiful, pleasing, and that we actually enjoy spending time with.

This is the art of robotic design, and the user experience (UX) design of robotics. This is also the inevitable trend as we learn how to build social robots.

For now, please go see CHAPPiE. And bring along a cigarette when you do.

CHAPPiE rolls out into theatres this week. See IMDB for a full list of release dates. If you liked this article, you may also be interested in:

• Automata: The new “sci-fi” blockbuster set to put robot ethics under a spotlight
• Robot love: Spike Jonze’s new sci-fi film ‘Her’ may be closer to reality than you think
• Robot and Frank review

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Pepper robot, by Aldebaran.

Social Robotics: The UX of Natural Language

A very smart and snotty journalist told me that he had spoken with Aldebaran’s Pepper system when he was in Boston. Much to his chagrin, the robot didn’t reply when the journalist spoke to it. “It didn’t understand a word I was saying,” said the journalist. “It simply doesn’t work.” The only word it understood, he said, was “sushi” … at which point I realized that he was speaking with the system in English, but the robot was programmed to understand Japanese. 

This is a problem of user experience, or UX.

Natural language interfaces are turning into a de-facto interface convention. Just like the GUI overlapped and largely replaced the command line, NLP is now being used by robots, the Internet of things, wearables, and especially conversational systems like Apple’s Siri, Google’s Now, Microsoft’s Cortana, Nuance’s Nina, Amazon’s Echo and others. These interfaces are designed to simplify, speed up, and improve task completion. Natural language interaction with robots, if anything, is an interface. It’s a form of UX that requires design.

The next phase of UX is psychological: Natural Language personalities.

We all use language on a daily basis as our key interface with other people, so expectations of how to talk are as deep as they are diverse. We all seem to have, from time to time, problems talking with one another, so designing talking interfaces is a hard UX job. Topic management, dialogue turn-taking, segue management, association with iconic gestures, and thousands of other aspects of communication sit right in the middle of this emerging design discipline.

Traditionally design, even software design, employs the famous architectural adage of, “Form follows function.” But when it comes to natural language interfaces, and robotics in general, we enter a new kind of design in which function follows form. The form of human interaction becomes the function in making something metaphoric, simple, fun and useful.

When it comes to human-robot interaction – and social robots in particular – the form defines the function. There are at least three reasons for this I’ll list here. All of them are psychological.

Social robots should look like us.

First, a user should identify with the conversational system. Any school age child will tell you that there’s no pragmatic reason to make a computer look anything like a person. The android seems a fool’s design. It’s off-balance, clunky, the sensors are too high, walking is hard to calculate, and when the robot falls, it takes a serious digger damaging the sensors that are perched at the part of the robot that gets the hardest knock when they fall. And what is a head for, anyway?

It turns out that babies (and adults, and school age children too) are wired to recognize even an abstracted human face surprisingly quickly and at a surprisingly early age. Babies, when they’re still wee things, are drawn to facial imagery, even if it comprises just two dots and a curvy line. I tried this on my own piglet when he was 5 months old. I gave him a drawing of a circle with the two dots and a curved line, and another of a square with a single dot and two lines. He went back and forth between the two drawings, but spent at least six times longer staring at the face drawing.

Which drawing do you prefer to look at?

What is even more curious is that 6-month olds are more likely to spend time looking at what you or I might call an attractive face (see e.g. here and here).

So it stands to reason that a social robot should have an attractive face, simply because humans are wired to pay attention to it. And this is part of the metaphor that is driving android design.

But beauty is only skin deep. While we wrestle to escape the uncanny valley of visual appearance, we are learning that there are many uncanny valleys of not just appearance, but sound, color, timing and even psychology. Maybe psychology, most especially.

Social robots should speak like us.

Second, now that NLP technology is generally working well enough to complete end-user tasks and provide contextual training, it’s time to start addressing personality in the same way that we’ve addressed appearance. This means that, just as we need to make sure that we design robots that look like us, we need to also design robots that speak like us.

This simply means that the user experience of natural language must consider psychology and human-like interaction as the core metaphor for the design of social robotics. It’s like making a glove that is shaped like a hand – the tool has to be designed to work with how we’re built. So psychology is key.

‘Birds of a feather flock together’ for some very deeply ingrained reasons associated with trust. We’re accustomed to trusting people that look like us and when we trust someone we actually perceive them to look more like us.

So talking and looking like a human is pretty important for a social robot.

This linking of behavior and appearance is key to user experience. Systems such as those used in healthcare, finances, and other very important interfaces will need to be designed such that they’re reliable, and this means they have to act like the user. Users will scarce be found who are willing to tell their financial data, or discuss private healthcare problems, with some fictional character that looks like a Furby, or a spinning ball of metal. And, to make matters more challenging, the user experience around natural language interfaces will need to address localization issues that include such things as accent, argot, and other manners of speech that are linked to particular regions of a country.

We’ve found in our work at Geppetto Avatars that people are more willing to overlook uncanny elements of design if there are slight cultural variations thrown in. For example we’ve noticed that synthesized voices sound less artificial if they have a slight accent. A synthesized voice with a British accent doesn’t sound quite so synthetic to an American listener. But if you give that same listener a synthesized voice that uses an American accent, the listener is far more likely to notice the synthetic elements. This is simply a UX technique to off-load technical challenges. So UX can serve the role of simplifying not only the interaction, but the technology as well.

Social robots should be polite.

Robots – or other connected systems – that use Natural Language interfaces create spontaneous behavior in the user. Most users actually try to be polite or cooperative, and even start attributing personality to the system (such as humor, aggressiveness, and gender). We all talk to our car, many of us have named it, and that’s how we’re wired. This is something people automatically do, like it or not, with not just natural language systems but almost all forms of media.

In 1996, Reeves and Nass, two famous researchers from Stanford (and authors of The Media Equation), proved that computers elicit social interactions in users. Reeves and Nass designed an experiment in which 22 people come into a lab to work with a computer via natural language text interface. At the end of the session Reeves and Nass asked them to evaluate the computer used and how well it performed. Reeves and Nass found that users had automatic social reactions during the test, and opinions about the personality of the machine they were using.

They ran the test again, this time via natural language voice interface, to make the human-social theme more evident. The test results were the same. They concluded that people are polite to computers, both in speech and writing, and the experiment showed that social rules can apply to media and computers can be social initiators. Participants denied it, but the results said otherwise.

Try it yourself – watch people talk with Siri and they use the word “Please” or “Thank you” more often than you might guess. Then go buy The Media Equation. It’s a great read.

When we design systems that are able to build identification and trust with a user these systems are able to make advice about healthcare and how to use our bodies, or finance and how to use our money. These are powerful systems because they are actually modifying what the user does. The conversational system has a power over the user simply in making advice that is amplified by how we are wired. If ethics is all about power then being polite is the logical interface to ethics.

Interface, whether command line, GUI, or natural language, has always been the most potent and valuable element of a computer design. Interface will continue to be an important and valuable element of robotic design as well. The future of UX is psychological as much as graphical and ethical as much as functional. Ask your car.

UX practitioners and robotics designers need to consider the form and the function, especially when it comes to natural language interfaces. Otherwise, 私達はちょうど悪い寿司の貧しい翻訳があるでしょう。*

[“we’ll just have poor translations of bad sushi.”]

The black cylinder takes natural language input so you can just talk to it. After saying the wake-up word, “Alexa” you can tell it to play music, set alarms, look up answers to questions, and add things to your shopping list. Our science fiction future is nigh and it is clear that, be it Siri, Cortana, or Nina, personal assistants that employ Linguistic User Interfaces (or LUIs) will be used to gather and present our personal information. LUIs and virtual assistants can control systems around the house like the thermostat, lights, blinds, alarms and locks (with voice authentication, of course). They can be likable, little AI companions that provide healthcare advice, offer help with history, algebra, or play trivial pursuit games.

And Echo may be the hub for all that and much, much more.

http://youtu.be/KkOCeAtKHIc

But I don’t think that a cute li’l buddy for the kids – or an LUI for the home – is Amazon’s plan.

In January of 2013 Amazon purchased IVONA, a Polish company that had developed a solid suite of voice recognition and language processing tools. Since then, Amazon has upgraded IVONA’s tools and improved its code. Recorded utterances get segmented into words, phrases, and sentences. A database is then created that keeps track of pitch, duration, and tone, which is then compared to a chain of previous units. Gender, age, and other end-user data can be saved. Your mood can be measured from vocal data, as well as stress levels, sickness, and inebriation. I’m familiar with a half-dozen voice packages, such as Kaldi, or ReMeeting, that offer such services, so these features are now industry standard.

These functions are standard in the burgeoning voice recognition industry because knowing who is saying what is important, especially if you’re Echo trying to cross-reference family members. Since individual voiceprints are relatively simple to build, each person’s preference can then be managed and those can also be cross-referenced by who in the house likes what. This makes for a pretty solid sales bet for Amazon. Consider: if my wife likes peanut butter and I like jelly, then Amazon can bet we’ll want to buy bread. 

Echo is always listening; it has to so that it can listen for your command to wake up (and in the Echo FAQ they mention this). This means that all audio is being captured all the time, though of course you can turn the microphone off.

One of the fears that are expressed in media is that this system will invade our privacy, or siphon money in ways we don’t expect, such as convincing the kids to buy stuff while the parents are out of the house. Interestingly, Amazon’s track record is a little stronger on the former than the latter.

Let’s start with privacy. Circumventing browser controls and surprising users with unexpected snail-mail has been happening at Amazon since 2007. In 2008 Ariana Del Vecchio was using Amazon to buy pet-care products. A few months later she began receiving snail-mail from pet-care companies with whom she hadn’t done business. A few years later she joined in filing a class-action lawsuit, Del Vecchio et al vs. Amazon, claiming that Amazon had circumvented users’ privacy settings and sent cookies, even when cookies collecting personal information had been manually blocked in the browser. The complaint (which was amended several times) included claims under the federal Computer Fraud and Abuse Act and the Washington Consumer Protection Act, as well as several common law claims. Amazon filed three separate motions to dismiss but the third didn’t seem to go down too well with the judge, so they settled with the individual plaintiffs, avoiding the expensive class-action filing. The suit was settled out of court.

In so far as convincing the kids to buy stuff while the parents are out, in July of this year (2014) the Federal Trade Commission filed their second case against Amazon in federal court. According to the FTC, Amazon had billed account holders for millions of dollars in unauthorized in-app charges mostly made by kids without their parent’s approval. In December of 2011 Amazon employees said that allowing unlimited in-app charges without any password was “…clearly causing problems for a large percentage of our customers,” adding that the situation was a “near house on fire.” In March of 2012 Amazon updated this policy so that charges over $20 required an account owner password, (but kids could still make as many purchases as they wanted). It wasn’t until June of 2014, after thousands of parents had complained and the FTC got involved, that Amazon changed its framework to obtain account holders’ informed consent for in-app charges (on its newer mobile devices).

Amazon, a very big company, responds to customer demands if those number in the thousands and lawsuits are involved. This comes as a bit of a surprise since, according to Forbes, Amazon measures performance against roughly 500 measurable goals, 80% of them related to customer objectives. This robot springs from a company who’s CEO and Founder celebrates a “culture of metrics.”

Please note that the word “Alexa” is a surprising wake up command that is linked to metrics. Alexa Internet, Inc. is a subsidiary of Amazon, which they purchased in 1999. Alexa provides commercial web traffic data, collecting user behavior and transmitting that data to be analyzed. Alexa scoops up metrics on about 30 million websites. Oddly, you can’t delete an Alexa account that was created through a web interface, and no means of contacting the company is provided to account holders.

Thus we come back to the cylinder sitting in the house entertaining the kids. Should we be worried?

Privacy’s important. We already employ many devices in the home to help us maintain our privacy – blinds on the window, a front door with a lock, bathroom curtain, and pants.  We all, implicitly, have private parts.

And while we Americans say we like our privacy we’re concerned about losing it. According to recent research by the Pew Research Center more than 90% of Americans believe that consumers have lost control over how personal information is collected and used.

But we might be okay with losing our privacy.  As security high priest Bruce Schneier puts it, “We know that people are concerned about privacy but we also know that they don’t think about it when they are sharing data on Facebook because we have to socialize … People give Google their data and share on Facebook. Surveillance is the business model of the internet.”  Schneier considers this profiling and has some interesting documentation about its effects.

We should not be afraid of surveillant technologies, but we should be aware of their effects. Like drugs, we should be aware of the side-effects, the addictions, and the massive, futuristic benefits.  After all, it is not the technology at issue, it is how it is used. And that is up to us.

What, really, is the product? Is Amazon selling Echo, or is Echo selling us? And who makes the choice? 

For now, let’s wait and listen to what Echo’s Terms of Service has to say.

My grandmother got run over by a cable car in San Francisco when she was about six and she wore a wooden leg after that. Of course she hated it. In the glamor days of the 1930s, when the miniskirt look was in (and the steampunk look was not), her wooden prosthetic put her into a social category of Blackbeard more than Greta Garbo.

When I was a kid we’d have weird family outings to go get her a new leg. She’d complain about it but it was still enough of a party that I always got an ice cream from the deal. We did our best to take care of her but she always hated that wooden leg.

Later we put Grandma in a nursing home. I felt bad because I couldn’t see her as much, but I was in college, mom was working, and we had other responsibilities. We did, however, get her a pretty nifty electric wheelchair. I later glued on some fancy sequins, which added a glamor factor at the bridge tournaments in the cafeteria. It was classier than her wooden leg, and it edged her a little closer to her Greta Garbo image.

But glittery electric wheelchairs aside, we had neither the time, the family members, nor the money to take care of her as she deserved. We did what we could with the sequined wheelchair and the weekend visits.

A robot would have come in handy. Here’s why.

Over a third of American adults are family caregivers and most of us already have a job. 

By 2050, 16 percent of the global population will be over the age of 65. That’s 1.5 billion people, according to The Population Reference Bureau.

In June, 2013 the Pew Research Center published the report Family Caregivers are Wired for Health. According to the Pew survey 39% of Americans said that they had “provided unpaid care to an adult relative or friend 18 years or older to help them take care of themselves” in the previous year. 70% of the caregivers had other jobs, which means that they’ve got a lot more to do than take care of Grandma. Also, according to a previous Pew study, 47% of American adults say that they expect to be a caregiver for an elderly relative in the future.

Then, about a year later, Pew released another survey – U.S. Views of Technology and the Future – which showed that 65% of those surveyed thought it would be a change for the worse if robots become the primary caregivers for the elderly. And in May of this year The Guardian ran a survey and found that about half of those surveyed thought technology was evolving too quickly and undermining traditional ways of life.

Hence the question is on everyone’s lips …

Should we have robot caregivers?

In academic circles the debate rages. Dudley and Emery wrote “The Value of Caregiver Time” (2014. From MIT’s Lee, Stiehl, Toscano, and Breazeal we have “The Semi-Autonomous Robot Avatar As a Medium for Family Communication and Education” (2009). And from CMU’s Borenstein and Pearson we have “Robot caregivers: harbingers of expanded freedom for all?” (2010). The February 2013 edition of, “GeroPsych: Journal of Gerontopsychology and Geriatric Psychology” published a special issue with a series of interesting articles. Fox, Kleinman, and hundreds of others have researched this problem and come to the same core question: Should we?

Sherry Turkle, of MIT, says Nay.  She’s launched several invectives against the army of healthcare providers she sees coming over the horizon. A clinical psychologist, Turkle has written that “faux relationships detract from human connections…” and “make us less likely to look for other solutions for their care.”

Mainstream media have started pouring gas on the debate as well. Heather Kelly of CNN has asked if robots are “… the future of elder care?”;  Health and Medicine explains why patients tell more secrets to virtual humans, and on July 19th of this year Louise Aronson wrote about the future of robot caregivers, arguing that we need them. A few days later, Turkle replied to Aronson’s article, stating that we are in the process of “outsourcing” our ability to care for one another. She and many others claim that our modern media makes us feel lonely.

Robots: The latest victims of the “Technology Makes Us Lonely” meme

In Turkle’s paper “In Good Company” she argues that healthcare robots are cause for grave concern, especially if we expect to have family members replaced by them. She argues that outsourcing to hospital robots should not be done lest we ruin our families, automate our interactions, and become lonelier as a result.

This “Technology Makes Us Lonely” meme has been going around like a bad case of the sniffles. The Atlantic ran a cover story on it back in May 2012 and Turkle wrote a similar op-ed piece in the NY Times to publicize her book “Alone Together”.  William Deresiewicz, a literary critic, wrote of “the loneliness of our electronic caves … The more people we know, the lonelier we get … We have given our hearts to machines, and now we are turning into machines.” Meanwhile books like “The Lonely American” and articles about how we’ll not end up just lonely, but broke as a result of robots, give the impression that the Cylon uprising will commence this weekend in a nursing home near you.

Reading this stuff is enough to make anyone lonely. But none of this may come to pass at all. Family members aren’t at risk of being replaced by robots any more than conversations were at risk of being replaced by telephones.

We’ve heard these Manichaean arguments before. We’ve heard of the negative effects of video games, the negative effects of movies, or how “Each hour spent viewing television is associated with less social trust and less group membership,” wrote Robert D. Putnam, a government professor at Harvard who researched the ills of television in the 1950s.

Telephones were supposed to make us lonely, too:

Does the telephone make men more active or more lazy? Does [it] break up home life and the old practice of visiting friends?

— Survey conducted by the Knights of Columbus Adult Education Committee, San Francisco Bay Area, 1926

Robots are facing the same firing squads as other media have , and they will become an integrated part of – not a flat replacement for – what we have today. More moderate experts, like Professor Maja Mataric, offer us a nuanced view than Turkle. Mataric points out that care robots aren’t a substitute for human interaction. In an article with the San Jose Mercury News she says, “there just simply aren’t enough people to take care of our very large and growing elderly population … We need to think about the humane and ethical use of technology, because these things are coming.”

Robot caregivers are coming

Let’s set aside nostalgia; we now live in the future. Whether we like it or not, robots are now healthcare providers. Both the horrible and the lovely exist in front of us and these systems will be caring for you and me when our kids don’t have the time, desire, or money to do so themselves.

And this might not be such a bad thing. On the provider side we can already see examples of assistance with surveys, assessments, triage, and assignment. On the patient side we can see great examples of advice, entertainment, mental stimulation, games, and simple companionship that the 80 million people entering the healthcare system this year simply won’t be able to get otherwise. These systems offer increased value that is scalable and cost effective.

Just the companionship use-case is a great one. After all, isn’t it worth offering your grandmother both the crossword puzzle she does today as well as a robot that will offer clues as she does it? Isn’t it better for them to have a helping hand beyond just the crossword puzzle?

And isn’t it the case that having options is what liberates us?

Geppetto Avatars Sophie

I’m biased for many reasons. One of them is because Geppetto Avatars, my own company, builds natural language avatars for healthcare and other applications. We’re proud to partner with IBM Watson, EBSolutions, Tailored Care and other companies so that we now have healthcare knowledge that is more up-to-date, evidence-based, and broader than any single, trained doctor can provide alone. Our avatar Sophie can play bridge as well as the next senior citizen (and chess and other games as well as perform surveys and offer healthcare suggestions), and while she can’t stop a speeding train, she can offer help across a range of subjects.

There are other companies like us that also provide software robots. There’s NextIT‘s project that helps soldiers, students and elderly. There’s both Catalia Health and Intuitive Automata, founded by Dr. Cory Kidd – an expert in building social, affective robots to help people lose weight and generally change behavior. There’s the Cantoche, which allows customers to make their own healthcare avatars, similar to Code Baby, Expect Labs, Arquiware. Other companies in this space are Sense.ly, which offers a virtual nurse, or Engineered Care‘s virtual nurse that’s there to provide daily support; MedWhat is a simple Q&A interface that’s partnering with avatar builders. These companies all provide the hearts and minds of robots, displaying, like the humans they care for, a wide range of emotional intelligence.

Meanwhile, in the American eldercare space, some 50 million people need help.  And on average, a patient visits with their doctor for average of about ten minutes, and is interrupted by their doctor in the first 17 seconds of the conversation. There just aren’t enough doctors to go around.

There just aren’t enough doctors to go around

Luckily there’s another species of healthcare robot that allows a doctor and a robot work in collaboration as a hybrid: the telepresence robot.

Systems such as MantaroBot,  VGo, Suitable Tech’s “Beam,” or AnyBot’s “QB” are four excellent examples. These robots are a physical prosthetic of a remote user. These telepresence robots help us get places we couldn’t be. If Turkle &co were to wave a flag and say that these robots prevented us from visiting our families, she would be missing the very essence of what these technologies do, which is to share time when we can’t share space. Telephones didn’t stop us from having conversations; they allowed us to break the barriers of space to facilitate more of them.

Whether they’re software or hardware based, these robots don’t render family members obsolete; like telephones, they extend our reach. They’re prosthetics that help family members do a better job. And they can play bridge with Grandma now, too.

Healthcare robots are like any other healthcare tools – think of them like a splint, an x-ray machine, a wheelchair, or medication.

We need to be aware of how to use robots. We need to encourage users to be aware of the potential side-effects, detriments, benefits, pleasures, sicknesses, doses, withdrawals, and addictions. If we can approach technology (be it radio, television, telephone, Internet or video games) like a prescription drug then these conversations get a bit less dopey. Edward Snowden pointed out one of the side effects of social media (“Surveillance!”), the NSA has pointed out another (“Terrorism!”).

We need to weigh costs, risks, side effects and benefits. Yes, some people will become heavy addicts just as others will practice righteous abstinence. But we need a moderate dialogue to avoid extremes.

To claim that we all should – or should not  – do something to address the dialogue around robots in such extreme terms is to incite a “War on Robots”. And this (like the “War on Drugs”) could leave people currently in hospitals in prisons instead (by footnote there are over eight million elderly Americans currently incarcerated. In August of 2014, the Urban Institute released a report that noted the increasing financial healthcare burden, suggesting that these inmates could use robots as well as my own grandmother).

Like drugs, robots will be ‘good’ or ‘bad’ depending on how people design and use them. Our imagination is what will guide the technology.

Giraffe Plus

My Grandmother wasn’t in prison, but it was close. Some nursing homes and eldercare centers have horrible records of abuse, neglect, and cutting financial corners that have even lead to death. But connected systems like Paro or GiraffPlus could be used to help make sure that Grandma’s health is where it should be, and that she is active, alert, and engaged.

• Surveillance: Robots can not only help with some of the above direct benefits to payers and providers, but they can assure family members that the elder person is receiving the care that is their due, and that the nursing home is taking appropriate steps to look out for their health. This surveillance, as well as input of user data via camera and microphone, is able to measure emotion, mood, sentiment and affect.
• Privacy: Just like a diary or a confessional booth can provide conversational privacy, people seem to prefer talking with avatars, especially about their health. Multiple studies (1, 2, 3) have shown that people are generally more comfortable talking with an avatar, especially in a healthcare setting. Robots can also help with very simple private tasks, such as helping someone go to the bathroom, change a bedpan, or other privacy-related tasks that you and I would not want a person to help us with.
• Social status: Just as people consider the iPhone or a fancy car as a sign of wealth, a robot that answers the door can be a sign of social status (like my Grandmother’s sequin-covered electric wheelchair). Technology is cultural: those who own and control a technology define a social class.
• Companionship: Robots – especially software robots like the ones we make at Geppetto – also offer companionship, mental exercises, conversation, guidance, and advice on health-related issues from pharmaceutical to dietary information. This companionship also includes to-do lists for bathing, laundry, cleaning, cooking, feeding, food shopping, pharmacy visits, etc. This companion can keep track of certain health data like blood test results and blood pressure, weight, height, temperature, etc. Integration with peripherals like Apple and Samsung offer an easy step for such open APIs. And then there’s the custom data, non-normative data, and personalization and customization that these systems can learn.

Call them slaves or companions, but robots are there to give help when people can’t.

Robots can amplify what healthcare providers already do well

Robots aren’t replacing us. They’re complementing us.

None of the above-mentioned aspects of surveillance, privacy, or social status threaten the jobs of healthcare providers. Robots, like personal computers, amplify what we can do. We can look to factories to see the future of healthcare. Human-robot collaboration will allow us to increase not just our knowledge management, but also our efficiency. Just look at how BMW and VW are exploring the ways in which robots can work alongside humans improve the quality of our work.

That electric wheelchair we got for my Grandmother was the 1990s equivalent of a robot: it was a modern, electric prosthetic that helped both her and us. It helped her get around and it helped us help her to be independent. We did not overturn our relationship with her because we were no longer physically carrying her. In fact, we helped her to be prouder, more independent, happier, and more competitive at games of bridge. She was proud to have this icon of social status, her Greta Garbo image, she was proud to show that her family supported her, and she was able to be more herself with the help of this electrical prosthetic, covered as it was in sequins.

For now, we’ll have to see how the designers, payers, providers, and nursing centers apply this technology, and hope that the dialogue is able to take a more moderate tone. At least until the ChildCare robots are introduced – then we’ll really see the flame wars. Maybe the robots themselves will contribute to the dialogue by then.

When I was in high school I was a big fan of graphic novels. In art class I used to splash out paintings – portraits mostly – and would write on them, giving the portraits word balloons, like a comic book. I wanted to portray personalities more than portraits. Of course my art teacher hated this and she’d box my ears, telling me that painters should not write, and that writers should not paint. She even called me an artistic criminal.

But I went on (as you did, dear reader) to make things like Twitter accounts and LinkedIn profiles and selfies that I’d then tag with some text and did just that – put words with the picture.

Today we’re all  portrait artists and writers. But with cameras in our phones and publishing tools that allow us to be both exhibitors and viewers in the often perverse universe of social media, it’s not so much about self-expression as it is about identity. We all write blog posts and put an image in them, and rarely does a selfie not include some tagline that’s supposed to be clever or witty, since, like all self-effacing artists, we want to be clever and witty and get into the mosh pit of show-n-tell.

Selfie = Profile = Account = Avatar = Self-portrait = ID

Whether you call it a profile, an account, or an avatar, you’re making a self-portrait. It’s your ID. It might a 2D profile, or it might be a 3D avatar for a game, but when you upload a photo of yourself and then you add some text, you’ve made a self-portrait ID that’s got something to say.

And you make IDs for yourself more often than you might think: if you’ve got a Twitter account, then you’ve done this; if you’ve got a Facebook or LinkedIn account, then you’ve done this; even when you pick some dumb photograph like the flower or the horse to use as your login image for OSX or Windows, you portray yourself with that image, and then you write your name next to it. We all have dozens (if not hundreds) of these weird little text-garnished self-portraits scattered all around the Internet.

Your avatar is your self-portrait, and it has more value than you think.

Regardless of your ability to draw or write a verse, these self-portraits you’ve made are very, very valuable.

Portraits have always been valuable, and this is because our identities are valuable. The royalty of Europe used to have portraits painted as a means of immortalizing their ID onto the wall of a castle, and the money required to hire an artist to paint their face for all the castle-folk to see was a declaration of power, influence, and wealth. Even when the camera was invented, portraits were still valuable, and sitting for a portrait was a sign of status. Today a face on the cover of a magazine is the same. It’s a social value.

Not all of us can be Portuguese Kings, but those days are past us anyway. Today we wallow in the attention of social media by building pages that proclaim how great our life is (or how miserable), or draw attention to things that interest us, and we have little numbers in social media sites that measure our self-portrait avatar on some social scale. Our avatar self-portraits, therefor, not only have text, but they also have numbers so that we can see how they’re performing – and that number ranks the value of our avatar self-portraits.

These avatar self-portraits generate various kinds of value, in terms of privacy, relationships, organization, access, and even money:

• Edward Snowden demonstrated that value in terms of privacy with the NSA revelations; why would the NSA spend all that money, time and effort collecting the information if it wasn’t valuable?
• Every time you talk with family, friends and work colleagues via Skype, you demonstrate that value in terms of relationships; why would you spend all that time on Skpye if the relationships were not valuable?
• Your user account on Apple’s iCloud is something that is of value to you to help you organize your information, schedule your day, and purchase the virtual commodities that help you navigate the city, purchases, and social network. That’s value expressed in terms of quality of life, quality of organization, and quality of access.
• And, most obviously, companies like Google, LinkedIn, Twitter, and Facebook mine our avatars’ data and build a cultural value system that is so valuable it is measurable in terms of billions of dollars per year.

Whether it’s privacy, money, identity, relationships, or money, the value of our self-portrait avatars is important to the very structure of society, and this has been the case for a very long time.

It used to be that only the very wealthy were able to afford portraits that lasted after their death and secured their position in history: a portrait hanging on a castle wall is evidence of the individual’s importance (both historically and financially) to the building itself; people used to pray to the portrait of the Pope; swear allegiance to the image of the king; and even today it is common practice in the US and many other countries to hang the portrait of the President or head of state in government offices.

But pop culture has a way of transforming things that were untouchable into things that can be handled and manipulated by the average person. It takes things that were ‘out of touch’ down off their pedestals and out of their glass cases, and makes them into something smaller that can be changed, edited and owned.

The avatar is a pop culture version of the self-portraits of classical history, and they allow us to change and distribute the portraits of ourselves as the royalty of the Renaissance used to do with their portraits. We use our avatars today to buy, sell, negotiate, advertise, rate, share, trade, learn, teach, train, talk, read, listen, write, and have fun with family and total strangers. Our avatars can do far more than a classic portrait can do because they are interactive, we can handle them, and they belong to us in many small pieces.

What if your avatar could talk for you? 

What if you had an autonomous avatar – a version of you that can talk for you when you’re not there –  that was your doppelgänger? What if it was able to protect your privacy, support your relationships, improve your quality of life, and make you money while you slept?

At Geppetto Avatars, we wonder about this next step. What does it mean if an avatar is able to talk without a person behind it pulling the strings? If you were to take all of the content you’ve written, and then provide your avatar with the ability to speak all that knowledge that bangs around in your skull, what would it say? If you had a self-portrait that could talk to you, what would you discuss with each other? If you could walk away from a mirror, and your reflection stuck to the glass, what would it say to the people it encountered after you left?

Now consider your job and the things that you do to make money in this world. Specifically, imagine the things that most bore you, and that you repeat daily. Imagine taking those words and gluing them, like a word-balloon, to that reflection of yourself in the mirror. Imagine that your avatar could discuss those things, share your knowledge, and contribute to a larger ecosystem in which you yourself no longer had to spend as much time doing these tasks.

Would it provide you with more privacy? Would you use it to interact with work colleagues? Or family members? Would it be a virtual assistant? What if it could answer your phone for you, or perform simple business tasks like ordering a shirt, calling a taxi, booking a reservation, letting someone know you’re running a bit late, buying a book from Amazon, calling to find out why your phone bill increased, looking for a good sandwich shop, and finding out what hours they are open?

Would we want these things? Would we learn about ourselves, or become more ignorant of what we like? Would we protect our privacy, or would we become vents of deprivatization?

Of course there is a nightmare that hides in these notions, a dark thing called polyopic heutoscopy – a psychotic state in which schizophrenics find themselves surrounded by their various alter egos. Could that nightmare, instead, be our future?

In the coming years these questions will be answered.

One day we will be able to build avatar self-portraits and set them loose in a larger ecosystem where they can speak with other people and avatars, to complete tasks for us and, much as a company might today, deliver messages, answer questions, listen, observe, collect information, organize information, barter, buy, sell, and serve as knowledge brokers and doppelgängers in an increasingly virtual world.

Today Geppetto Avatars builds healthcare avatars and instructional avatars that can discuss, listen, observe, and share information. They’re doppelgängers that liberate busy individuals from performing rote tasks, so that they can spend their time on higher value work instead.

Already there are a bunch of talking avatars running wild out there, and they’re turning out to be pretty valuable:

• Craig Mundie, Microsoft’s Chief Research and Strategy Officer showed how Microsoft’s Xbox Kinect and an avatar could facilitate group therapy sessions; people generally feel less intimidated by an avatar than by a human group leader.
• Research suggests that the experience of watching your digital look-alike uneventfully reach the front of the simulated line and order a pretend drink is real enough to help you learn to cope with similar situations in the actual world (thus reducing social anxiety).
• Virtual coaches can help people with PTSD.  The director of this project, Skip Rizzo, says: “It leverages latent inhibition. If you expose somebody to something in a controlled environment in advance, when they face that challenge in the real world, there’s less chance of fear conditioning — which is at the base of PTSD.”
• According to Prof Jeremy Bailenson, avatars are becoming more popular. Kids in the US spend about an hour a day inside an avatar. To put that in perspective, they read print for about 38 minutes a day.
• And this week Watson acquired Cognea (formerly My Cyber Twin),which means that we’re about to see a lot more of this kind of thing.

The avatar is becoming the medium of the future.  If your avatar could talk, what would it say?

All images courtesy Dito Von Tease / ditology Follow @DitoVonTease. If you liked this article, you may also be interested in:

• Facebook – Our ffriendly automated identity bender
• “A Paro walks into a club…” (or, Notes on the design of automated body language)
• The uncanny valet (or, Notes on the design of robot psychology)

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Last week I dropped by Aldebaran’s studio to get a glimpse of Pepper in action, and was pretty excited about this robot. But then I talked with Bruno Maisonnier, the CEO of Aldebaran. And then I got really excited: what Pepper represents is another iteration in the realization of the roboticists’ dream.

Aldebaran built Pepper for Japanese telecom company SoftBank, which will start selling the robot to consumers in Japan next February for about US $1,900. The robot will be produced by Taiwanese electronics manufacturer Foxconn (who manufactures the iPhone and other Apple products). So with the marketing presence, technical abilities, and hardware systems now at a consumer-grade quality, we’re one step closer to Rosie the Robot. Or Her. Or the Cylon uprising. Remains to be seen.

Emotional interaction, according to Bruno, is “everything.” We spoke briefly on June 13, 2014 and I was able to ask him about some of this technology, its socio-cultural impacts, and the future of this work for Aldebaran. Several years ago I had been hanging around the Paris studio of Aldebaran and had seen Romeo (and wrote about it and NAO in my book on robots), so I had some sense of what was coming, but I was not expecting the emphasis on affective computing methods that Pepper embodies.

MAISONNIER: Pepper is a fantastic dream I’ve had for years – an interactive robot you can have at home that’s still large enough to be a companion and not just a toy or a pet. At Aldebaran we have a specific concept. We want to help people with robots. We want them to be companion. To help people grow, have fun, and connect them to others.

So first you need to have a cute, nice robot – not just something to say, ‘Oh, that it is nice,’ but the essence of the robot is something that lives in your home. This is not for geeks, so design is our first and most essential part. We want a natural way to interact with the robot and not have a 200-page manual that people need to read. So we need people to be comfortable and interact easily with the robot. We humans have evolved to interact with other humanoid shapes. And sometimes, even without words you understand what the other person means and yes or no can be clearly indicated with body language. We can see if someone is saying yes but meaning no, and if you are saying yes when you are sad, or no when you are happy the message is totally different. So if I want the robot interaction to be natural the robot has to understand that.

MEADOWS: How do you do that? That’s a tall order.

MAISONNIER: These drivers for both understanding and expressing emotions are key to our design. NAO was the first emotional robot, with many possibilities, but many limitations because of its size. Pepper is a larger robot, and one of the consequences of this design decision is that it can move faster. You can have the robot follow you, or if it has to go into your bedroom to look for something it won’t take 20 minutes, and it will be able to open doors. If you want to interact with emotional context, then size is important; [Pepper’s size offers] a more human interaction, rather than talking with something the size of a mouse.

MEADOWS: Ok, but an android is a bad design for a robot because the function follows the form.

MAISSONNIER: But for Pepper the function is to interact with people. The majority of people in robotics are coming from industrial robotics; they’re building left-brain robots. I’m not interested in that. There are huge applications, but this is not what I’m interested in. We want right-brain robots for friends and companions – robots that are more like Jiminy Cricket.

MEADOWS: Here at Geppetto we feel much the same way. Now Romeo and NAO have legs. Pepper’s got wheels. Why?

MAISSONNIER: The main driver is that Pepper is an emotional robot, and we rarely use our legs for social interaction. While there are some detriments, like going up stairs, there are also benefits to the wheels, such as giving the robot a longer autonomy because it can carry a heavier battery. The goal of the robot is to help by teaching, being a health coach, monitoring, growing, etc. So it is more right-brain. Yes, the form follows function because the interaction with humans is the function. So you don’t need legs.

MEADOWS: So how are emotions used to guide the robot?

MAISSONNIER: Communication is emotion. You are an artist and know this. When you are speaking, only about 20-30% of what you say is in the words. But 70-80% is not words – all the other parts are emotions. So communication is emotion.

MEADOWS: What are the methods you’ve developed to convey this?

MAISSONNIER: We’ve invented an emotional engine that collects many different inputs and combines them in multi-modal approaches that use facial analysis, tone of voice, body language, gesture, small noises like laughter, and understanding the micro-signals that your body sends when you talk with someone. All that is feeding a higher level message, which might be that “she said yes but she means no” for example. This serves as the input to our dialogue engine, and we have an expressive engine that uses automatic body language in response. It has all the possibliities – hand, movements, sound, voice, etc – to express the correct message.

MEADOWS: Beautiful. What do you do with this platform?

MAISSONNIER: We’ve been speaking about the robot itself, and all that is part of our expertise platform, but on top of that are the applications so users or developers can build apps via our API. All that goes to the developers, who are then able to use it as they want. There are many ways to use this. For example, if you are designing an application to play chess, you can use the API to deliver messages on your chess program. The system can appear frustrated, or sense frustration at key moments in the game.

The developers are core to our strategy. Aldebaran has an app store and we confirm the quality of the app so you can download this to users, free of charge, who can download the application from the store. We have an atelier [a french term between workshop, gallery, and studio] where everyone can enter, meet with robots, experiment with them, and play with our applications. This will be the home of the developers.

MEADOWS: I love this! Perfect. Are they only online or can I visit them?

MAISSONNIER: We will have an Aldebaran Atelier in Paris, on 26 June, 2014 and two more will open in Tokyo in August. By 2015 we will have two in the US (New York and California).

MEADOWS: Changing the subject a little, you have a robot that has some incredible surveillant abilities. I think there’s a risk here in that systems may not respect privacy, or may be used for ends that don’t align with the end-user. As I’ve written about on RoboHub, where I ask if surveillance is the new business model for consumer robotics, Andy Rubin is taking the robotics effort in the same direction he took the Android effort. This not only changes the landscape for Aldebaran, but it means that you have to address questions of privacy. What Is Google doing?

MAISSONNIER: I don’t know. It is opening up the [robotics] market. They will need some time before having ready robots because the culture of software is not the culture of hardware. Google is a software company, but though they have bought these companies this is not an immediate solution. Building a $1m robot is not the same as building a $1k robot .. they will need time. Their business model, as you said, is to sell advertisements. Do you want to have a spying robot at home?

MEADOWS: Well, most of us don’t seem to mind. Many people still use Google. Why is spying on me with my email different from spying on me with my robot?

MAISSONNIER: One is autonomous. If you send your mail you don’t see it flying around your house checking your books, checking your clothes. It has to be friendly, but as important is that it needs to be your friend that can protect you. You have to be sure it is on your side.

MEADOWS: Autonomy and invisibility are quite different. And a robot can do things that a user may not perceive. After all, email could be called a spybot as easily as a bots.txt file.

MAISSONNIER: This kind of thing will exist. If other robots like Aldebaran exist then between a spying robot and a non-spying robot, people will choose the right one. People will chose and we will settle the rules that we don’t want spying robots. I don’t want them, you dont’ want them. We would put a physical bridge, a switch, that will block communication and ensure privacy. That way people can say, “I want to authorize my robot to communicate with the outside world or not.”

MSM: I hope so. Where will people be able to find Pepper?

MAISSONNIER: Thousands of Peppers will be in SoftBank stores in Japan.

MEADOWS: I understand it will be about $2000, correct?

MAISSONNIER: Yes, for Pepper standalone, after that you can get insurance, monitor modules, repair modules, content, etc. These added components can be important. For me, the next step is to add content, applications in our application store – so a huge community of people will become interested in joining our adventure.

And what an adventure it will be. We look forward to seeing what great inventions Aldebaran turns out next.

If you liked this article, you may also be interested in:

• SoftBank + Aldebaran = Pepper
• NAO Next Gen now available for a wider audience
• My robot is smelling my dirty laundry (or, Notes on privacy, PRISM, and home service robotics)
• Robot love: Spike Jonze’s new sci-fi film ‘Her’ may be closer to reality than you think
• Is surveillance the new business model for consumer robotics?
• Deep Learning creating jobs in Apps, wearable tech and robotics

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

A discussion with Ryan Calo on Google, your personal data, and the consumer robotics market

Can someone tell me: What’s Google doing, making robots? Or at least: What is it doing, making such acquisitions?

As you probably heard, in the first two weeks of December last year, they bought some pretty heavy companies (house specialties in parentheses):

• Schaft (Navigation)
• Industrial Perception (Vision)
• Redwood Robotics (Manipulation)
• Meka Robotics (People-safe designs)
• Holomini (Navigation)
• Bot & Dolly (Camera tracking)
• Boston Dynamics (Navigation)
• DeepMind (General AI & abstracted reasoning)

This list covers AI, robotic vision, and a famous four-legged ‘bot that can bust into a sprint that could beat Usain Bolt. So it’s safe to say that the focus is on navigation and vision.  This is nice for the roboticists who are waiting for market validation in the consumer space. Google is providing that. They’re also providing us with a fleet of robots that can navigate, see, and use the network.

What’s Google’s motivation?

Presumably, as a publicly-trade company, this is intended to make Google more money.

They did a pretty good job last year. Sitting between Apple and Coca-Cola, they’re the second most valuable brand in the world. Paid clicks were up approximately 31 percent over 2012, and as of year end, Google employed 47,756 full-time employees. They also, in 2013 had advertising revenues of approximately $50 billion.

Google exchanges personal information for online services. Expanding their horizons from online tools, a few years ago, Google gave Android away to handset makers for the same reason Google gave free web search and email away to you and I: to collect personal information and display other information back. This is Google’s job and their profits show that they are very good at that.

It makes sense, then, that Andy Rubin, who headed up the Android effort, is now heading up Google’s robotics effort. After all, robots – like smartphones – are not just a platform for providing products and services; they are also a platform for collecting data.

Personal information as currency

Nothing is free, so if you aren’t paying cash, you’re paying with your personal information. And that information, from 2013, was worth $50B … so it stands to reason that Google’s motive for getting into robotics is that it offers yet another means of harvesting more information about you. Google will be using robots to collect and display personal information, just like Android did, and before that, email and web searches did. If Rubin has moved on from phones to robots, it’s probably because he thinks robots offer the greater potential in terms of opportunity.

There are three reasons why this makes sense:

1. Robots will give Google access to the physical world for a reduced cost. Like Apple, they are seeking to produce advanced hardware with companies that have proven track records for such manufacturing, and Google is evidently partnering with FoxConn to make this happen.
2. Robots will allow Google access to home and business information with unprecedented fidelity of information. By providing us with robots that are cheap (if not free; think of how smartphones as subsidized), Google can eventually build data conduits into our homes that will allow the company to know what we read, what dog food we buy, what we wear, and when we turn on the Nest thermostat (and, yes, the business model is the same for Nest as it is for robotics: more consumer data).
3. As with an open source OS for smartphones, by producing an open source OS for robots, Google will have direct access to all of the information that Android offered the company.

A couple of weeks ago, while attending RoboMadness at my old stomping grounds of SRI, the CTO and EVP of iRobot, Dr. Paolo Pirjanian, made a curious statement on stage. He outlined a ‘holy trinity’ for robotics as: vision, navigation, and the cloud, and pointed out that, “navigation is key” because it will build environment maps that will allow problem solving. So for example, a 3D environment map will allow the indexing of objects, as well as conversational interfaces for voice and gesture control. If a book is out of place and you want your robot to put it back, then the idea of “back” needs to be indexed somewhere. That somewhere is the cloud, so this information will, then, attach to the cloud.

All that personal data …  all those clouds …

Here we have an example of mapping, navigation and cloud robotics, which, if I try to look at it objectively, sounds a lot like what Google appears to be doing.

Is surveillance the new business model for consumer robotics?

In my desperation to get to the bottom of this I called up Ryan Calo, for he once wrote that “robotics combines, for the first time, the promiscuity of data with the capacity to do physical harm; robotic systems accomplish tasks in ways that cannot be anticipated in advance; and robots increasingly blur the line between person and instrument.”

Professor Calo researches the intersection of law, robotics and the Internet. His work on drones, driverless cars, privacy, and other topics has appeared in many famous publications and he serves on a number of advisory boards, including the Electronic Privacy Information Center (EPIC), the Electronic Frontier Foundation (EFF), and others.

I threw these ideas at him and he took difference with the angle. Here was Calo’s response:

The frame is right, though I would quibble with characterizations of profit – there is a potential for win-win and it’s not necessarily a transgression of privacy. I would love that.  In fact I kind of wish that Google and Facebook would consider creating a robot platform for the home. The robotics industry really needs a personal robot in much the same way as a personal computer. We need something that is very versatile – something we could run all kinds of different code on – a platform or PC that is open to 3rd party innovation.

Then what are these companies that collect our data, like Google or Facebook, up to?

They want to bring in the Internet to more places – to where it is hard to bring it.  Amazon wants robots that can shave seconds, or fractions of a second, in the time it takes to get a package from the warehouse. This is how they’re thinking of using robots.

So in other words, with regards to these companies, so far they are leaving a lot on the table and it may be that, cynically, they end up building this platform that personal robotics needs and their motivation is to get a look inside the home, but that doesn’t seem to be the case. I understand there are things like Nest that give some perspective on the home, so it is a natural fit for Google’s other enterprises. But it is a closed platform that does one thing.

Isn’t that why Google has traditionally offered more services? To get more user information?

No, it’s to get more users. Google doesn’t just rely on users, but also on inventory – it needs a place to display those ads. They want more screens. They created email to bring more people into the fold. But I don’t think Google does everything they do just to get more information. They want to go to advertisers to say, “We have X users” and that increases the value.

You could explain a lot by saying, “They think that smart objects of various kinds are the future and they want to use all these new screens.” Are they building driverless cars to collect personal information? Well, they’ve been doing that with people. Maybe they want drones to fly around for the same reason. But Google appreciates that contemporary life is shifting. Once you have driverless cars. the screen can be as big as you want so that Google can own that screen. If everyone is being driven by cars, maybe Google will have a credible map so that people have to build on that infrastructure. They want to be the ecosystem.

Mr Calo and I have more discussions ahead, but for now, like all good underlings, we’ll just have to keep an eye on the surveillance system to see when it wakes up.

If you liked this article, you may also be interested in:

• DARPA’s Gill Pratt on Google’s robotics investments
• Robots Podcast: Privacy, Google, and big deals
• Google’s robot acquisitions likely cost less than $100 million
• AFK (or, Robotic free speech, and what you can do to help)
• Robot love: Spike Jonze’s new sci-fi film ‘Her’ may be closer to reality than you think
• The death of search (or, My dysfunctional relationship with Siri)

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Rodin once said that sculpture is an art dedicated to holes. What he meant is that great work is invisible: if you are building a technology, or a company, or a product, if it is truly good, then most of the work will not be seen. It’s the stuff that, like dark matter, holds everything else together. You have to dedicate yourself to that.

Second, you have to figure out where you want to be in N amount of time (1 year, 10 years, whatever) and then divide that in half and make sure there are milestones that you can see along the way. I like to have visible deliverables as we work on projects so that we can make sure that we are moving in the direction we want. It’s usually a prototype, but at least that helps us more clearly visualize the work that will be invisible.

This movement from invisible to visible is a path from imagination to market. Like footsteps behind you, 99% of it is invisible.

Read more answers →

The incoming second wave of contextual agents

There’s a virtual lobby of Intelligent Virtual Assistants (IVAs) waiting to help us these days. These multi-million dollar systems include Yahoo’s Donna, Samsung’s SAMI, Google’s Now, Nuance’s Nina, Motorola’s Assist, Microsoft’s Cortana and of course Apple’s Siri. They can give you driving directions, book a dinner table, launch an app, tell a joke, take a memo, send a text, post a tweet, ring a phone, update Facebook, check stocks, search the web, turn off the lights when you go to bed, and set an alarm to wake you up in the morning. They can do incredible things, but they’re not very valuable for one weird and very general reason.

All of these companies, and many others, understand that voice interface will be built into our handhelds, our wearables, and the Internet of Things. There seems to be a growing stampede of investors galloping towards this inevitable future, and those investors are right to gallop so. But something is amiss with the design of these systems. I don’t know about you, but when I use these assistants I’m sometimes stunned at how off they can be. They make the word “smart” seem dumb. They can be so off at times that they can even make dumb seem smart.

There’s some great work being done, too. Surprising work. Siri has this plug into Wolfram Alpha and Wikipedia, so I can ask it very specific questions like, “How tall is the Eiffel Tower?” and I get a response back as, “The Eiffel Tower is 1063 feet tall” in under a second. Wow. Wikipedia now talks, kinda. That’s great.

But is it useful? How often do I need to use that? Other than the ability to use a phone without my hands, and ask it questions, I don’t see the problem this is solving. Today’s Intelligent Virtual Assistants seem a lobby of lovely-voiced zombies that are trying to be everything for everybody at all times. And it reminds me of a strange trend we’ve already seen.

Back in the 1990s, chatbots were also trying to be everything to everyone. Systems like ALICE (built on AIML) were designed to answer all questions that could be thrown at them.  Authors wrote rule after rule so that the system could answer anything, no matter what you asked it, effectively trying to pass the Turing Test with a linguistic equivalent of cryptographic brute-force attacks, as if the designers were thinking, “If the system can talk with anyone at any time about anything, then it will seem human.” As a result the systems were designed to passively sit there, and as soon as you asked it a question about cats, particle physics, or the Eiffel Tower it was supposed to barf up the “right” answer. This, we thought, made it seem more human.

Brute-forcing the Turing test with every possible answer (Thank you, but no, Dr. Wallace) is a function that is rarely valuable because it rarely has context.

Take Siri as an example. Siri is supposed to be able to serve all our various needs in a huge range of circumstances from driving to note taking to going to the movies. She’s got it pretty rough these days. Folks get down on her for misunderstanding, misguiding, mis-typing or just missing the point of what was said. After all, making a conversational system that can talk about anything with anyone at any time is impossible. Not even systems with human-level intelligence (like humans) can do that.

So it’s rather unnerving that everyone who is building these IVAs is making this same mistake.

Here’s why: Context is information’s value. If you take any piece of writing near you and give it to someone on the other side of the planet, the value of that information changes. Look at any writing near you. Look at any road sign, listen to anything on the radio, watch anything on a screen … and if that information is moved to some other place or time, its value usually evaporates. This includes what you say to your lover, banker, parent, or child. It includes what you write, what you say, and what you hear. It includes what you say in confession. It includes what you hear from your doctor or physician. It includes the height of the Eiffel tower.

So these IVAs, by trying to be everything to everyone, are missing the point: they need to focus and provide specialized information to people who need it. The more general the information, the less valuable it is.

IBM appears to be going through some turbulence these days, but they also seem to be taking quite a different approach to the pile of virtual assistants mentioned above. IBM’s Watson is focused primarily on healthcare, and this specific context helps to produce high value information. By trying to do crazy-difficult things like cure cancer (and by now even out-performing the diagnostics that doctors are capable of) IBM is building a database of specialized knowledge. Cognition-as-a-Service (CaaS?) might be a Web 3.0 idea that changes apps and grows an ecosystem of users and groups of experts who build increasingly powerful sets of contexts.

Imagine a “doctor-on-your-shoulder.”  An app that can tell you if your soup will trigger your gluten allergy, or how carbohydrates can affect your diabetes, or how cipro can create comoribidity issues. I like to think we roboticists can build things like a Virtual Nurse, a Virtual CPA, or other conversational systems that offer very contextualized information for a very specific group of people who need a very specific type of help. Systems that inform, educate, and address a knowledgebase that is of high value to a smaller set of individuals.

At SXSW this week Geppetto Avatars is showing off a proof-of-concept project we did with Health Nuts Media: we built a two-avatar conversational system that helps kids who are recovering from asthma. After kids have been released from the hospital following asthma treatment, they can talk with these two cartoon characters, Jiggs and Big, and ask them about their asthma. It’s an app for asthma management.  It helps reduce re-admittance, it saves hospitals time, and it can save kids’ lives.

This is an example of the kind of IVAs that will power a new Internet. It is an IVA that helps people live better, healthier, happier lives. Sure, we can serve ads with this, and sure, the Eiffel Tower height is important, but I hope we can use raw data like that to aim for higher ideals.

Google, is the wild card for me.  With more acquisitions (DeepMind, Boston Dynamics, Redwood Robotics, Industrial Perception, Meka, Schaft, and others) than Apple, Amazon, Facebook, and Microsoft combined, the GOOG looks to be rigging up a kit that would offer excellent image recognition + navigation + mobility.  A robot that can roam around your home or maybe your city, and offer a helpful service.  My bet is that these Googlebots (perhaps they will be cars) will have sharp eyes, simple yet helpful demeanors, and we’ll trade that robotic service for our personal info.  Then Google will sell that info to advertisers.  So the same Google formula the company has used with email, docs, voice, maps, and the others.

The story you already know: a sentimental writer (Joaquin Phoenix) falls in love with his OS, named Samantha (played by the inimitable Scarlett Johansson). His OS falls in love with him.  Mayhem, hot sex, and crocodile tears follow. It’s great. If you haven’t seen it yet, stop reading now and go.

The movie is about Samantha’s emotional struggles with love and how, through these struggles, Samantha and Theo discover themselves.

First, all great stories are about self-discovery. So this isn’t too new. Her — like Pinocchio, Terminator, Robbie, Astroboy, the Cylons, or any other member of a large herd of famous robots that become self-aware — is no exception. It’s all about self-discovery. All dramas are … maybe all stories are. But the self-discovery, a wistful balance between physical vs. virtual, turns into a room of mirrors where the different realities start to blur. Virtual reality and physical reality intertwine, are ultimately incompatible, and we watch as Samantha’s struggle to discover herself mirrors Theo’s; her virtuality in contrapunto to his physicality.  And we’re left with a real problem of incompatibility at the end.

http://www.youtube.com/watch?v=WzV6mXIOVl4

Samantha’s a software robot that learns about herself, and as she does, flickers between two very real worlds of physical and virtual. This mind-body conflict, framed by real love, is the core struggle for the kind, husky-voiced robot. Samantha tries to embody herself (and fails), just as Theo tries to disembody himself (and fails). Then they each freak out. Theo skins his knee. Samantha blinks off into a singularity meet-up. Like all great lovers they are the yin and the yang, the mind and the body, the complements of each another. The lovers’ tension is framed by icons of love that also struggle with virtual vs. physical.  Theo’s “Beautifully Handwritten Letters” — initially just thoughts from other people — turn progressively more physical when they’re published in a book. Meanwhile Theo’s old marriage grows less physical (and more virtual) until it is just a memory and a signature on a contract. The heart of the movie is this balance between virtual (the mind) and physical (the body). With overtones of Shakespeare’s Capulet-Montague conflict keeping two lovers at arm’s distance, we watch the virtual vs. physical create a conflict the two lovers just can’t resolve.

Her shows what’s happening today through a lens of fiction. We have Siri as a vector pointing in this direction of “cognition-as-a-service.” And to be sure, the technology in Her is probably far, far closer than most movie-goers imagine.

The tech is nearing a point where users will attain a suspension-of-disbelief.  Today we can semantically analyse words for affect and sentiment. Today we can measure body language, and prosody of voice to build empathic feedback loops with users that process Natural Language. Today, we can talk with software and that software can learn, measure, and talk back. The dark, cold monitor is now monitoring us, as we have built it to do.  The technology in Her is just a few years away. It’s a technology that requires the measuring of that spiritual thing called mood.

This week Apple filed an interesting patent to do just that.

Apple’s patent claims to determine mood based on various types of data, such as blood pressure, heart rate, adrenaline level, verbal cues, and body temperature. It knows the content a user is viewing, the music they’re listening to, who they’re talking with, when, and builds these into a multi-valent chart of “mood rules” that maps out how the user’s feeling.

The system builds a baseline mood, since behavior that looks happy for one person might be sad for another, then content that is most easily received at that particular moment is tested to see what works best. Apple doesn’t need to know what mood your in, it just needs to know you’re in the mood it properly measured. Then, when you’re relaxed and ready, and feeling cozy with your phone, they stick you with an ad.

Or, as they put it, targeted content.

An individual’s responsiveness to targeted content delivery can be affected by a number of factors, such as an interest in the content, other content the user is currently interacting with, the user’s current location, or even the time of day. A way of improving targeted content delivery can be to infer a user’s current mood and then deliver content that is selected, at least in part, based on the inferred mood. The present technology analyzes mood-associated characteristic data collected over a period of time to produce at least one baseline mood profile for a user. The user’s current mood can then be inferred by applying one or more mood rules to compare current mood-associated data to at least one baseline mood profile for the user.

To be fair to the great giant of Cupertino, most of Apple’s revenue comes from hardware sales. But the iAd sales unit, and the partner relationships, will help improve their sales ecosystem and that surely will see more revenue pouring into the research lab that developed this patent. Sentiment analysis, somatic analysis, and machine learning are the juicy center that advertisers are gunning for, so we won’t see this kind of technology going limp any time soon.

The science fiction film Her is about today’s tech. The movie points to this virtual-physical tension, with this reverberation of emotions, because we humans have this funny knack for building things that reflect us, both physically and virtually. Samantha, in being so virtual, becomes Theo’s personal myth, his personal goddess (Theo means “God” in Attic Greek, by the way). These reflections bounce back and forth as the relationship between Theo and Samantha reveals who they are. Love is all about this reflection of the other, as who they want to be seen as, back at them.

“You’re beautiful.”
“So are you.”

These are the kinds of things lovers say to one another as they stare across the table in Parisian cafés. And this is what the OS named Samantha is advertised as, “An intuitive entity that listens to you and understands you and knows you.” Samantha is designed for love reflections. It’s why we make robots. Even in the kinda campy 2001 film, AI, Giggolo Joe says to David about his mom, “She doesn’t love you, she loves what you do for her.”

This is what Apple loves about you, too.

If you liked this article, you may also be interested in:

• The death of search (or, My dysfunctional relationship with Siri)
• AFK (or, Robotic free speech, and what you can do to help)
• Robots Podcast: Robopocalypse
• Robot and Frank review

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

I’d like a personal gentleman’s gentleman, if you will, someone that is there to both advise and help. A Sancho Panza, a Samwise Gamgee, a Dr. Gonzo, or a Dean Moriarty. A Ron Weasley or a Huckleberry Finn. A real companion to help me through life.

Though I have not spent more than a few minutes with it Marvel did build an app that is intended to do just this. Someone had the right idea but this is a thin semblance of what we need. Unfortunately, what Marvel missed was what makes J.A.R.V.I.S. so intelligent – his street smarts. His worldly knowledge and personality.

J.A.R.V.I.S. is based on Reginald Jeeves, the fictional valet of Bertie Wooster, from the writing of P. G. Wodehouse (1881–1975). Jeeves offered Bertie advice, assisted him with daily operations, helped him keep track of things, run systems, and do it via natural language. Jeeves was someone that enhanced Bertie’s knowledge, understanding, amplified his perception and wisdom and even fixed him the occasional hangover cure. So I’d like a Jeeves – an advisor of the most intimate sort that’s there as a consultant, teacher, confidante, and companion. Especially for the morning of January 1st, when I suspect I’ll have a bit of a hangover. He would, after all, know exactly what I’d had to drink that night, and would have probably been the one that had called the cab for me to get home.

Read more answers →

My aunt, a Tennessee tobacco grower, used to remind me that God gave me two ears and one mouth for a reason. What she meant is that a good conversationalist is not so much someone with the ability to talk, but the ability to listen.

Robots can take a cue from my aunt.

As I’ve detailed in previous articles, about half of human communication is body language. Approximately sixty percent of information we communicate in a conversation can be transcribed to text, like you read here. The other forty percent is what we’ve been grouping together as “Body Language.” This amounts to how a person moves, appears, and sounds.

As roboticists and designers, our ability to build connected systems that listen to what is said and how it is said is probably the single most important aspect of our designs, at least if the system is interacting with a human. Input is critical to human-machine interaction, whether its a command line, word, or gesture. Robots need to pay attention to the words, movements, and sounds humans make to better understand what is being communicated.

It’s commonly understood that Natural Language Processing breaks down into 1) Natural Language Generation, and 2) Understanding. This is as true for textual language as it is for body language.

A robot can process words. It can also process gesture, posture, and tone of voice.

Psychologist Skip Rizzo is a researcher who specializes in the design, development and evaluation of virtual reality systems that are used in clinical assessment, treatment, and rehabilitation. His work wraps around the disparate worlds of psychology, cognition, and motor function in both humans and avatars. Rizzo received the American Psychological Association’s 2010 Award for Outstanding Contributions to the Treatment of Trauma and he’s the associate director for medical virtual reality at the USC Institute for Creative Technologies, in Los Angeles. These days he uses his background in psychology to bridge the worlds of video game avatars and American veterans suffering from post-traumatic stress disorder (PTSD).

I met Dr. Rizzo a few months ago in his office in Los Angeles, and we spent some time comparing notes on how to measure, interpret, and analyze both words and body language.

Skip discussed a project named Ellie – a virtual assistant avatar that patients interact with so that Ellie can build assessment models. The system – a software robot – was designed to interview patients and be able to build psychological models of the conversant.

Ellie was started two years ago when Rizzo began working with computer scientist Louis-Philippe Morency.

Funded by DARPA, within a larger project called Detection and Computational Analysis of Psychological Signals ( see here and here), Ellie was designed to detect people in emotional distress that might be at risk for suicide. Aside from Morency and Rizzo, other participants in the project focused on the language programming, some on the visual appearance of the avatar, and some on the training of psychological cues. And there were a range of other means that the crew used to build assessment information.

The team spent months polishing every element of Ellie’s presentation and interaction with patients, experimenting with a range of different personalities, outfits and vocal approaches including a well-timed “uh-huh” that reflects a common function of human conversation.

Under the wide screen where Ellie’s image sits, there are three devices. A video camera tracks facial expressions of the person sitting opposite. A movement sensor — Microsoft Kinect — tracks the person’s gestures, such as fidgeting or other unconscious gestures. A microphone records vocal prosody and tone of voice.

“How do you measure people?” I ask Rizzo.

He puts his feet on his desk, leans back, and crosses his hands across his stomach. “Using a webcam and a good microphone we were able to track, capture, and pick up behavioral symbols. From there we were able to make inferences from them as to whether the person was more distressed than their pure language might indicate. We were able to collect this data in face-to-face interviews with known groups of distressed people – veterans, in particular.”

“Now we have an AI version that runs completely on its own. It will try to develop a rapport, dig deeper, and try to end on a positive note. Some of the questions are projective questions like, ‘In the last few months are there things you wish you had done differently,’ or, ‘Are there things you’ve done lately that you regret,’ or ‘when was the last time you felt really happy.’” These questions were designed to be open to interpretation with no real right or wrong; a verbal Rorshach test.

“And what are the cues that build redundant data, that give you confirmation on how the person is feeling?” I ask.

“Oftentimes you get an answer that doesn’t belay much and you have to have a follow-up question… But when you look at how they said it, things like a vertical gaze, fidgeting, a delayed response, space between words, pitch, variation or lack thereof – that’s the stuff that’s great to get.”

The three primary points of data that Ellie collects are facial expression, tone of voice, and gesture – the same criteria humans use to assess other humans.  There are a total of twelve vectors that offer good validity data. In some cases the interviews go for 20 minutes, usually not more than 40.  The ultimate goal, on the application side, is to build a kiosk, a bit like a confessional, in which Ellie privately interviews soldiers; information from this interview can then be used by a company unit to evaluate its staff before going in to battle, or to determine if a person is suitable for a particular role.

And as easy as making a power-point presentation, they can start to make training cases, or National Board of Medical Examiners can make cases for certification purposes. Rizzo’s team has done everything from exposure therapy (using simulations of warzones) that clinicians can use to deliver evidence-based care to a virtual classroom where kids with ADHD can learn how to concentrate and control their unconscious behavior. So the work has broad applications. The psychological and cognitive use of these software robots – either as patients for clinical training or as agents to help people access information and guide them in a private way towards health care – will surely change the face of not only the defense department, but the medical industry, education, and a host of other industries.

Somatic analysis, as I call it, isn’t just about capturing aggregate number of smiles and vertical gazes… it’s also about driving the behavior of the system.  Ellie can tell if you’re hesitant to reply and can make a prediction about that. So the chain of questions that she may ask is a real-time sympathetic feedback loop. If you interact with Ellie ten times you can see progression and change in her responses.  She becomes more sensitive to the methods and systems of interaction..

This may represent an important trend in robotics design. These methods of semantic and somatic analysis are the eyes and ears that proper robotic interfaces need to have.  And the trends are being reflected in the software industry. For example, Apple is beefing up its personal assistant capabilities by purchasing ($40m) Cue, a company that integrates outside realtime data similar to Google Now, Nuance’s Wintermute, or http://don.na.

This research is what we’re focused on today at Geppetto Labs (and recently at Figaro Avatars, as well). Our primary focus has been to make sure that our systems are listening to our users, able to accurately measure their emotions, and able to reflect back the emotions that are most appropriate for the task at hand.

If, for example, someone is speaking with a system to provide medical assistance, they need to feel comfortable and confident that the system is helping them. If we are able to determine that they do not feel comfortable, then we need to be able to quickly change the interaction so that we are addressing what is making them feel uncomfortable. What this means is that we measure the affect values of the words they are using and we use that to then build a goal-driven interaction that is intended to create a sympathetic feedback loop.

One group working on the Ellie project managed to gain access to the Facebook pages of people who had committed suicide, and did a complete linguistic analysis of what they said and how they said it, and then used the data to ask if there were things that could be spotted in advance to indicate if that person was at risk. They measured semantics, sentiment, syntax, affect and other vectors to build linguistic models that mapped to emotional state. Now that they have finished their DARPA funding, Rizzo supposes that Facebook may be implementing some of these algorithms on their site. Not that if someone says the magic words that people in white coats are going to come pounding on their door, but he observes that the types of ads and messages that appear on the pages of these people are pushed more towards mental health.

This is how all humans, or almost all humans, interact. We each talk with one another in a manner that hopefully provides mutual comfort and encouragement – an interaction that is designed to create closeness confidence and trust.

After all, it’s been said that most important debates around things like politics and religion have little to do with the words and everything to do with the emotions. Affective computing is one of the most important revolutions in user-interface design because emotion is of one of the most important human motivators. And work like Rizzo’s is paving the way to understand this better.

As my aunt said, “You have two ears and one mouth for a reason.”

Simply: a head allows a face, and a face allows interface.

So a head’s only needed if the robot has to interface with people (or other headed animals, say). A head is a design feature but the main function of an android is its form: it has to look like humans. Giving it a head is a function-follows-form decision. Wasn’t it Hunter S. Thompson who wrote, “Kill the head and the body will die?” Well, this should not be the case for military robots. The beheaded design can be improved. Saying that all robots need to have faces is like saying all animals need to have gills. For the deadly, dangerous, and downright dastardly work that robots today need to perform, like gastro-intestinal surgery, or military surveillance, a head won’t do much more than get stuck or blown off.

A head, like hands or a face, is a design decision that’s best left for the robots working directly with humans.

It’s hard to communicate with words. Some researchers claim that almost half of our communication relies on things that aren’t words: body language, tone of voice, and stuff that just isn’t conveyed by text. This includes prosody (tone, pitch and speed of words), facial expression, hand gesture, stance and posture. This probably explains why about 40% of emails are misunderstood.  As designers of robots (or avatars) we need to consider these statistics and consider how to integrate body language into natural language communication. Therefor Geppetto Labs has built a platform to automatically generate body language and coordinate it with what a robot (or avatar) is saying.

Most NLP systems today, be they Siri or Watson, amount to conducting chat via the thin pipe of a text interface. Siri doesn’t have a lot of choice on the matter since Apple had to simplify the complexity of communication, but this text interface reduced the communication itself. If you think that Natural Language Processing is about only text, then step away from the computer, go to a café or bar, and watch people interact for a half an hour.

Videos do an excellent job of conveying the importance of body language. A great video to watch is The History Channel’s Secrets of Body Language. This documentary looks at politicians, cops, athletes, and others, interviewing experts of body language to decodify everyone from Richard Nixon to Marion Jones. Gesture, expression, and tone of voice are all looked at as valuable and important data channels.

This is why face-to-face meetings are so much more productive. Each party can better understand the other because there is a higher throughput of communication. In a lovers’ relationship, or in a family’s relationships, body language is even more important than in a business meeting. Consider the fact that it’s the most intimate relationships (between lovers, primarily, but also between family members, close friends, and others) that involve the most touching. These are also the relationships that rely the most on body language, because body language actually defines the proximal closeness and intimacy.

So if we want people to engage emotionally with robots, or avatars (or any other kind of character that is rigged up to an NLP system), we need to consider using body language as part of that system. We humans are hardwired that way.

***

At Geppetto Labs we have begun considering Body Language Processing as a sub-set of Natural Language Processing. So just as Natural Language Processing has NLU (understanding) and NLG (generation), we can consider Body Language to have BLU and BLG. I’ll be focusing on the generation of it, but others, such as Skip Rizzo, Noldus Information Technology, and others are also looking at the understanding of body language and facial expressions.

Generating body language requires coordination with the textual components of Natural Language Processing. A gesture or animation has to have the same duration of time, happen at the same moment, and include the same emotional content, or affect, as the message conveyed.  “Hi” should, of course, be accompanied by a gesture that is about one second long — a friendly-looking signifier that’s commonly understood. Raising the hand and wagging it back and forth usually gets the job done. But building this can be tricky. It gets more complicated when there is a sentence like this one that doesn’t have clear emotional content, isn’t the kind of thing you hear as often as “Hi,” and is long enough that the animation needs to be at least ten seconds long.

At Geppetto Labs we’ve developed the ACTR platform in order to accomplish this. The core process, at least as it relates to text, is to generate body language (as opposed to voice output) as follows:

First, we take the nude NL text and determine the three variables of the Duration (timing), Affect (emotion), and Signifiers (specific gestures):

1) Duration, or timing. How long is the sound or string of text we’re dealing with? This is the easiest to calculate directly from the text. Most spoken conversation ranges from between 150 and 175 words per minute, but that can speed up or slow down depending on the emotion of the speaker, and other factors. But let’s call it 150 words per minute. A “word” is calculated in these kinds of standards as five UTF characters, which is also five bytes. So that means that most of us speak at around 750 bytes per minute. Now if we back this out it means that around 12 bytes should leave the system per second, and this is then used to calculate the duration of a given animation. We’ll call this integer between one and 150 a “duration tag.”

2) Affect, or emotion. What is the emotional value of that source string of text? This is the second factor we need to know in order to calculate an animation, and it’s harder than just measuring the letters in a line: it requires either realtime sentiment analysis and/or a pre-built library that identifies the emotional content of a word. One solution is WordNet-Affect. Words in WordNet-Affect are derived from Princeton’s fantastic WordNet project and have been flagged with particular meaning that indicate a range of values, most of which relate to what kind of psychosomatic reaction that word might cause or what kind of state it might indicate. Some simple examples would be happiness, fear, cold, etc. There’s a ton of really sticky material in this labyrinth of language called “affect,” and the ways that words link to one another make it all even stickier. But for this explanation, let’s say that we can take a given word and that word will fall within a bucket of nine different emotions. So we give it a value from one to nine. Fear is a one. Happiness is a nine. If we then take the average affect of the text string in question (again, speaking very simply) we end up with a number that equals the emotion of that sentence. We’ll call this integer between one and nine an “affect tag.”

(Before we go on I want to take a break because we now have enough to make an animation match our sentence.

“How in the world do we build that?” is an eight-word sentence, so we know the duration would be about three seconds. The affect is harder to measure, but for this example let’s say that it ends up being a value of 5. So we have Duration=3, Affect=5. These two bits of information, alone, are enough to calculate a rough animation, but first we need to build a small bucket of animations. They are probably keyframes because we want to interpolate them so that they form a chain. We make them of various durations (1 second, 2 seconds, 3 seconds, etc.) so that if we want a three-second chain we can combine 1-second and 2-second duration animations, or, if we want to avoid replaying the same animation we can reverse the order of these links and combine the 2-second then the 1-second animations. And we make sure that we have these various animation links ready in separate buckets – one for each animation. So if we get a Duration=3 and Affect=5 we go into the bucket labeled Affect #5 and dig up the animation links that add up to three seconds.

The longer the duration, the trickier it gets. If you have a twelve-second animation you might then have to chain together that two-second animation six times, or your one-second animation twelve times, to get the proper duration.  Does that make sense?

No. I hope that at this point you’ve stopped and said, “Wait, no, that would be really dumb. To play an animation twelve times would just look like the character is convulsing. That’s bad body language, Mark!”

A spoonful of art and design can help. When you are combining the animation links to build a chain of proper duration, you need to avoid spasms and try to use the longest possible animation in your bucket. Then you frame it outwards with smaller animations to build the proper duration sum. If you want a twelve second animation, then try using a ten-second link with two at the front, or one on each end. That will avoid the spasms.

We’ve found that Fibonacci sequences work great for this because they have huge flexibility and, when interpolated together, are nearly invisible. This is the process we have found to be most helpful, flexible, simplest, and that looks the best. So this means that you’d need to build a relatively large number of animation links that are the following values: 1, 1, 2, 3, 5, 8, 13, 21, and so on. And you’d do that for each affect “bucket.”

Ok, so far so good. The next step is to play this animation and, as your character moves, watch it and ask yourself if that’s the kind of thing a person would do while saying that particular phrase.

Now back to our three variables…)

3) Signifiers, or specific animations. Is there a common gesture that normally accompanies the text? The wave of a hand that goes with the word “Hi” isn’t normally used in conversation as much as, say, nodding, or showing our palms when we speak. Most of our body language is unspecified. So specific signifiers need to be manually wired up to particular output phrases from the system. This is a hand-authored component that requires an author to flag specific lines that require specific gestures. We are developing methods for automating this, but for now it is necessary to build a few specific gestures that are used in unique moments of conversation.

***

Once all the variables are determined, they get packed together and our ACTR platform fires off a control file in XML that drives animations on the client side. This keeps the performance hit on the server low, allows a nearly infinite range of movements for any connected system, and provides a robot (or avatar) with the ability to gesture as it speaks.

Though this implementation is new, these concepts have been knocked about since the mid-1600s. In 1605 Francis Bacon, in The Advancement of Learning, looked at gestures as a frame around spoken communications. In 1644 John Bulwer‘s Natural History of the Hand, and Gilbert Austin‘s Chironomia (1806) examined the details of hand gestures in speech. Even Charles Darwin dug into it. Of course Freud and Jung looked into it, as did Julius Fast, an American author who first garnered some major interest in the topic in the early seventies with his book Body Language, and more recently, there’s the work of Paul Ekman, who identified some three thousand cross-cultural facial expressions. Other people who have looked into body language include: Louis Gottschalk, Erik Erikson, Charles Osgood, Otto Rank, Albert Bandura, Gordon Allport, George Kelly, Snygg and Combs, Maslow, Rogers, Jean Piaget, and others.

But it seems overlooked in most robotics design circles and certainly in most labs that are working on NLP. Some of the most talented natural language engineers are some of the least talented communicators. They understand the language of code better than the code of language. There is a terminal amputation that cuts the text of language from the language of the body. That’s because important elements of natural language, like body language, are easily overlooked if we focus too much on the code and not enough on the people.

Because ultimately, that’s what we roboticists are designing: a kind of people.

Let’s say a white box showed up on your doorstep and you open it up and find a little humanoid robot made by Google. A GoogleBot! The brightly-colored pamphlet says that the little disk of a device will vacuum your floor, all for the same cost as your Gmail account: free.

Since you already use Google’s search engine, have a Gmail address, and use GoogleDocs, Voice, Earth, Wallet, wear a Google Glass and have been driving around one of those autonomous cars you’ve been selected as a beta tester! Whee! A free robot!

So what do you do with this thing?

Vacuum the floor, of course. Off it goes, bumping into walls, building its little map as it sucks up Doritos crumbs and dust bunnies.

Google’s greatest hits, like Search, Email, Docs, Voice, or Translation, are founded on, or fodder for, Natural Language Processing technologies (NLP).  NLP and other semantic technologies are also why Facebook and, in some cases Apple, has provided us with such great services. They analyze what we write and say.

Meanwhile, others are further ahead of this linguistic learning curve. The National Security Agency (NSA) of the United States was using primitive forms of NLP at least a decade before Google was using it for search.  This is common with technologies: Government agencies get theirs first, then the rest of us get to use the stuff later, whether it is global maps of the earth (1650s), satellite photograph (1950s), or GPS (1970s).

Anyway, there are four NSA data collection programs: MAINWAY, MARINA, NUCLEON and – specifically for Internet content, and the most important source of collection – the PRISM project, which by now we’ve all heard about.

On June 6 of 2013 The Washington Post and the Guardian reported that the system draws data from Google, Yahoo, Microsoft and other Silicon Valley giants, keeping the folks in Fort Meade, Maryland busy mining what is the richest depository of personal information in all human history. Evidently the NSA is sucking up 2.1 million gigabytes of data every hour of every day. All information is stored, all communication is monitored, and the fact that someone is peeping in on you has been held secret.

This, of course, has caused some concerns.

These technologies can be used in a predictive manner, as well.  What you want, what you think, and what you will do can be predicted with a 93% rate of accuracy, and that includes where you are now and where you will be in twenty-four hours.
We move across a daily landscape of emotions just as we move across a daily landscape of city streets. Semantic analysis (or, more accurately, probabilistic latent semantic analysis, and other technologies that look at our trends) shows where we’ll be and what we are likely to think and feel.

In early December of 2007, Marissa Mayer, now CEO of Yahoo and then the vice president of Google’s Search Products and User Experience division, showed how Google Trends can be used to predict presidential elections. She showed how Google Trends accurately predicted George W. Bush’s win over John Kerry in 2004, and also pointed out how Trends accurately predicted Nicolas Sarkozy’s win in May of 2007, over Segolene Royal in the French elections.

These trickles of valuable personal data add up to rivers of money.

Valuable enough to grow a company into the size of Google, or valuable enough to justify a $10 billion annual NSA budget, those little strings of text add up to lots and lots of real money.
Let’s not forget this Google robot that has shown up on our doorstep.  Our big hockey-puck robot (or “connected system” as it is more fashionably called) is buzzing around the floor doing the job for free.  This concept is now a prototype and has been operating for three years.

Though iRobot swatted the ConnectR remote-controlled domestic robot in the start of 2009, ending the product line, the concept refused to die. A year later, in March of 2010, the concept was picked up by a group of Swedish hackers who launched a project similar to what I outline above, named GåågleBot (pronounced “GoogleBot”). These Roomba hackers dissected the robot and inserted various items such as a camera, a Wi-Fi card, a small computer, and a good squirt of the AJAX programming language. After sewing it all back up, the thing still maintained operations as a normal Roomba.

The GåågleBot can also read. It comes with optical character recognition (OCR) software so that you (or someone else) can photograph and translate text that happens to be lying around the house, drop it into a database, and run searches on that text. GåågleBot source code and instructions are available for download.

A vacuum is just a vacuum, but these days it is part of what is commonly being called The Internet of Things.  Smart Houses, Connected Systems, or Robots … the basic idea is that we use networked computers in objects around us.  If we consider the Internet of Things as a physical extension of the Internet we find a new method emerging – as precise as existing analytics systems – that can measure how we each live and spend our money.  If you purchase a product, like a watch, and it communicates via a WiFi or Bluetooth network, then whoever or whatever it is communicating with will know where you are, who or what is nearby, your name, movements, past, present and, by prediction, future. Other objects near it will identify themselves and as you synchronize a device with a user account, and the overlapping data increases the relative value of both datasets. Your identity and the identity of you live with will be known by the company you bought it from, of course, and with it will come all of your purchases, likes, dislikes, relationships, and documents that are hanging heavy with fruit to be plucked by the pinchers of NLP and semantic analysis.

Meanwhile, the robot Google sent you rolls around on the floor, sucks up dust, and blinks a network indicator. It seems so humble.  But this is how Google, Yahoo, PRISM, the NSA, and therefor the US Government collects data that can be potentially very dangerous information. All from a little vacuum robot.  After all, dangerous bombs often come in humble packages.

Now, I don’t mean to sound like the Unabomber here, so I will point you to someone that has thought about this topic far more, and whom I am parrotting, and that is Bruce Schneier.  He has written a nice article on this topic, which you can find here.

Let’s not be paranoid. This vacuum, like an email service, may come in handy, and it doesn’t cost anything.  So we pull the robot out of the box, flip it on, poke in a password, and go back to eating Doritos.

Now let’s consider another possibility, which is that Google has mailed us not a vacuum hockey-puck, but a little humanoid robot that can walk and grab stuff. GoogleBot is an android that comes up to your knee, just a little guy with big feet and a cute head, kind of like NAO. So you set it down and it begins to explore the house. Sure, for the first few days it is a touch disturbing as it waddles around the house and reads the books on your shelves, and runs optical character recognition on all of your magazines, and seems to be studying the contents of your refrigerator and cupboards and dog food labels and clothing labels, and very carefully watches your television. Once you even see it touching your computer and you chase it away.  But eventually it settles down and you think, “That was kind of cute.” Then, finally you get used to it. It does the laundry and feeds the dog, and along the way, GoogleBot gets another cluster of data about your life – and you get your dishes done.

This system may be financed with the same model that Google uses today for things like email, search, voice and other services.  Since Google is publicly traded we can see that Amazon paid out $55m in 2011 to advertise for books or whatever.  So they would want to know what you buy, what you read, and what you have on the coffee table.  Home Depot paid out almost as much, so they would want to know what you have in the yard. State Farm and Progressive insurance companies paid Google nearly $90m, so they would have a great interest in knowing your health habits. All this cascades down into a long list of retail information, health, hygiene, and habits that covers what you are reading, what’s in your yard, what’s in your home, your refrigerator, etc. And so – to be useful – your GoogleBot should be busy dusting your bookshelves, washing your dishes, doing your laundry, and being an active participant in your personal health by cleaning the shower, toilet, and floor.

If someone types in “Self-Employed Health Insurance” and clicks on an ad, Google earns about $43. There are dozens of small robots that cost less than that, which means a home care companion robot of about $500 would amortize GoogleBot’s production cost and shipping in just a few weeks, and become wildly profitable after that robot had spent a year in your home.

Because our personal information is coveted by both corporations and governments, it seems inevitable that this home service robotics model will arrive some day in the coming decade.

Hopefully the GoogleBot or YahooBot or whateverBot will come with bulletproof SHA privacy switches built in, switches we can trust. Hopefully privacy options will allow us to put curtains on the parts of our lives we consider personal. That way we don’t have to stuff the robot in the freezer when the girlfriend comes over.  And perhaps even revenue options will let us collect money for the information Google collects about us (Google still wins, we just pay less usage tax). Or perhaps these are added features that we will have to pay for, like in-app purchases today.  Perhaps we will have to pay for our privacy.

There are lots of questions here. Least among them are how the NSA and companies like Google and Yahoo work together – and Google in particular has been quick to reply to these questions, as have other companies. They’re now on a mission to regain our confidence. But the NSA remains spooky, holds the reigns, and hides in the shadows. It seems they are collecting as much information as they can for long-term storage. The chief technology officer at the CIA, Gus Hunt, said last March; “We fundamentally try to collect everything and hang onto it forever.”

Any time citizen information is secretly stored by a government, it means that government is collecting the tools of a dictatorship.  Just ask Mussolini.  Or take a look at how Germans, familiar with the Nacht-und-Nebel program, or Sicherheitsdienst, are reacting this week. Information is generally collected so that it can be used.

Of course the common response is, “I’ve got nothing to hide,” but the bottom line isn’t about having nothing to hide.  It’s about not wanting to show everything.

Privacy is about having the decency to wear pants, the right to have curtains, and the right to whisper sweet coo-coos in your lover’s ear without someone else, like GoogleBot, YahooBot and the NSA leaning in for a little listen.

Comrades, we live in a bleak and humourless world.  Here we are thirteen years into the twenty-first century, and we all carry around Star Trek style tri-corders, we have access to almost all human opinions via this awesome global computer network, we have thousands and thousands of channels we can flip through on television, we have something like 48,000 people signed up to go colonize Mars, and we even have robots roaming around up there, taking samples of that planet. But we still don’t have robots that can tell a good joke.

Even this new bartender robot, Makr Shakr was prime for a little humour, like, “A robot walks into a bar and asks for a screwdriver..”?

But seriously, folks, in my travels I have met with Honda’s Asimo, Adlebaran’s NAO, Toyota’s Partner Robot and many others, and each of these presumably world-class robots was, in the most critical sense of the word, robotic.  They were weirdly-shaped computers with an appendage.  If you give me a choice between a robot that can walk and one that rolls around on wheels, I’ll take the one that can talk. Simply put, talking is more important than walking. Design priorities in robotics should focus first on human interface and second on locomotion.

We live at a moment in which natural language interface is replacing the GUI just as the GUI replaced the command line.

Natural language interface developments are being led by Google and Apple, with some strong work being done by IBM.  Last week Google’s Chrome browser appeared with a little microphone just to the right of the input bar and Google Now has been out for a year or so.  Apple has had Siri, of course, and is now rumoured to be building a wrist watch that will have a natural language interface to it, probably building off the lessons learned from Siri.

Natural language interface makes sense anywhere there is big data or little screen.  It makes sense for robots for these same reasons. It provides access to more data and reduces the need for a screen. But most of all, building a natural language interface also gives the robot personality.  And this makes the robot more valuable to the human because it turns a robot into an emotion machine.

Giving a robot a personality includes two primary components.  First is psychology.

Last month, in The Uncanny Valet I wrote a bit about the design of psychology, specifically for android-style robots, which wasn’t so much an effort to write about human-shaped walking-talking robots (androids), as it was to point out that human interaction seeks human likeness. To emotionally bond with a robot it needs to be like us.  The more it is like us the more we can bond with it.  This is most important in the field of automated psychology.  This isn’t a new field.  If you don’t believe me dial an American 800 number.  Those horrid little automated voices that corporations use so widely these days are reproducing faster than feral rabbits (may we name them robbits?).  In fact, I would love statistics on how many voice answering machines are out there, but I would boldly bet on there being, on the market today, more software robots than hardware robots.  I can tell you that this number is increasing and the quality of software robots is improving.

Each month we see advancements in what amounts to software robots, or virtual knowledge workers.  Last week Infosys (a Bangalore-based provider of IT services) announced it is working with IPsoft (a New York-based provider of autonomic IT services) to build autonomous tools designed for help desks, operations, and other management needs. In an article about the the new partnership it is mentioned, “IPSoft has focused on autonomics for IT services, and Slaby estimates in his report that nearly all Level 1 support issues can be solved by IPsoft’s ‘virtual engineers’.”

We are past the primitive dawn of automated psychology.  It is becoming a branch of the robotics industry.

As graphic design for the GUI was to software design for the personal computer industry, personality design for natural language is to the software of the personal robotics industry.

At Geppetto Labs we build conversational systems.  This means that we take the natural language input from a user, pipe that through an analytics system that measures affect, syntactic and semantic values, and then feed the resulting data into an NLP system.  We author personalities via simple server-side tools that script what non-technical authors (generally film screen writers) have written into formatted rulesets. We also bend Chatscript and other tools for non-personality behaviour, like enhancing Amazon search by natural language understanding, using specific phrases or moments to trigger specific animations, etc. After that we stir in some scraping, automated ontology creation, and even tools to automate first person interviews and film scripts to craft a personality that can be managed. This is the foundation for natural language interface – at least how we do it.

And there’s an entire second level to natural language interfaces that has nothing to do with text.  In more advanced projects Geppetto Labs need to add animation that fires when the NL output does.  The simplest example I can think of is the word “Hi” being tied to a raising of the hand.  Hand gestures are easy, and clear, but gestures are only one part of the body language we build.  Body language might also be a facial expression, the position of the arms or the tilt of the head, but what it means is that we have to not only think about the psychology of the system, but how to tie that to the activation of non-verbal forms of communication as well. We have to walk the talk. That means we have to design body language into the personality.  It is linked with the psychology and frames the text.

Psychology needs to be graphically represented, just like a book needs a cover, and body language is the best means of achieving that.  Well, at least that’s where evolution has taken us.<

Body language coordinates the psychology of the personality with the visual presentation. Coordinating what is said and what is done is core to the personality.  This very coordination might be what we commonly call personality.  If you don’t believe, me consider this: watch television for a day and note what “Personalities” do; newscasters, politicians, actors, singers, and dancers.  They accompany their words with the most broad-stroke gestures and body language humans convey.

This is to say that body language may be more important to natural language interface than the words used.

Many famous studies claim that body language accounts for the majority of our communication (the most famous is probably Dr. Albert Mehabrian’s 7:38:55 rule).  Or, more precisely, that the words we use transmit the minority of what we communicate.  So how we move and what we visually communicate while we verbally blather on is pretty key to how others understand us.

Consider the eyes.  We blink more when we’re stressed out. It can indicate dopamine levels and both the pain or the creativity of the person we’re talking with.  The timing of a blink can indicate a punchline to a joke, a response to that joke, confusion at the punch line, or frustration at how it isn’t funny.  Pinched and held tight can mean something entirely different.  We don’t like to keep eye contact too long or it starts to feel threatening.  The person who listens should initiate eye contact – and if they don’t it might mean they just don’t care what you’re talking about.  But don’t break it too soon or the person listening will feel as if they’re not important. Don’t look at the ceiling when you talk (or you might come off as a snob) and don’t look at the floor, either (or you might come off as a bummer). If you’re talking with someone and they look up and to your right it generally means they’re remembering something, visually. If you’re talking with someone and they look up and to your left means they’re visually creating something.  Horizontal looking while talking indicates the person is most likely accessing auditory cues and, to your right is, again, remembering and to your left is, again, creating.  And if they look down maybe they’re depressed. Or maybe they’re just looking at the carpet. Or maybe they’re remembering how something feels.  That applies to most right-handed people around the world (according to John Bandler, Richard Grinder, and scads of other neurologists), but it indicates that there’s a lot happening when we speak with someone. A lot more than words.

But the eyes do this while other things are going on.  The eyebrows and cheeks are wiggling, the mouth opening and closing, the head is tilting and nodding, the shoulders are going up and down, the arms are waving around, the hands are flapping … and that’s all above the belt-line.

Please remember that human interaction seeks human likeness, so the body language of the robot needs to mirror the body language of the conversant. Think back on one of the early successful chat bots, ELIZA. This system was designed to simply mirror what a person said to it and repeat that back, like many psychologists. Several people evidently fell in love with the system.

The corollary of this is that a well-design natural language system will move like we do.

Today, we have a new graphical version of that, designed to note and reflect the body language of the conversant. Its name is Ellie.

A webcam and a game controller track eye directions, run some facial recognition software to chart expressions, and do the same with body positions. While the system is being trained in realtime in its early days, this training goes towards the video mapping input to the system’s output, just as any NLP system (such as Siri or Watson was designed, and exactly as we do our work at Geppetto Labs). It’s being trained on how to read people.

But there’s a missing part here, which I will guess the good Mr. Stan Rizzo is considering, which is generation of response movement in real time.

How the character moves, how the robot moves, is a key component. Just as natural language has the component parts of understanding and generation (NLU and NLG, respectively) body language has these same input and output cues. At Geppetto Labs we have several characters that we’ve been working on that have this running, and a process I’ll outline next month.

A robot walks into a bar and asks for a screwdriver, and the bartender says, “What is this, some kinda joke?” Personality, like a joke, synchronizes words and body language, linking them together and leaving us. “A Paro walks into a club” is about as bad as I get, and I promise I will not repeat the joke, but I bring it up as an icon of how to build personality: combine the mind and the body.

Natural language interface, when properly designed, is far more than just text.

Part One: Another robot dream girl

I’m talking with this woman on the phone, interviewing her for a job. She’s got a slight southern twang, maybe from Georgia, telling me about her past work, and as we talk I’m struck by how perfectly presentable, alert, and smooth she sounds. Her voice is soft, tonally fluid, and she articulates her words precisely, belying a geeky disposition. Seems she’ll be perfect for the rather tricky tech support position we’ve got.

“Did you enjoy the work?” I ask.

“Sure, I like talking with people. I mean, as long as they don’t get upset it’s sorta fun.”

“You were working at a bank, right?”

“Yeah,” she laughs, “There were four of us working the phones and please press seven if you’d like one of us to call you back.”

“Sorry?” I ask.

“There were four of us on the phones and please press seven if you’d like one of us to call you back.”

I realize this robot has a bug, thank her, and hang up. I won’t hire that one.

Of course I never spoke with the uncanny woman again because the above story is fiction; a conversational system like this one is something of a dream among today’s robotics designers. She might have had a bug in her bonnet, but at least she could hold a conversation and answer questions, for the most part.

A system like the above, a Natural Language Processing system, is the heart and mind of a robot.

By contrast, consider what we have today: those irritating, tyrannical little phone robots that have replaced perfectly good humans: “Please push one if you would like to make a deposit. Please push two if you would like to hear these options again.” These non-player phone-workers are so brainless, grindingly slow, and weirdly rude that I find myself repeatedly pushing keys like 0 and # just to get to a real human.

Of course, from the cold eye of the bank (or whatever corporate conglomerate that employs these “Customer Relations Management” systems) these virtual robots reduce costs by replacing a fickle, distracted, complaining, prone-to-error, temperamental, sleep-deprived and very expensive call-center human with a voice recording. It saves the company money, time, and tons of overhead on things like worker’s insurance, sick-leave, replacements and the high cost of managing people. People are hard to manage, right? Voice recordings are not. People sleep at night, right? Voice recordings do not. So virtual robots, and this particular feral species of virtual robot – the phone robot – is a means of disintermediating the person from the knowledge: it is a means of scooping the knowledge out of the head of person that was doing the job and serving it to you, the customer, couched in smooth tones with a dose of saccharin-sweet politeness.

But virtual phone robots in their present form are bad at solving problems, they don’t learn, they’re inattentive, and they can’t even understand me. I have few options on what I can do, and I get pushed into decision trees that I don’t want. I feel herded like a cow, and with that little phone-robot nipping at my telephonic heels to move me into the right slot for processing, I more often than not either hang up or return to the website.

The intention is that robotic knowledge workers can replace human knowledge workers. But robotic workers are not capable of the job: rather than collaborate, they dominate; rather than answer questions, they force responses.

Bad robot, bad!

Part Two: The problem is the rotten brains

According to research from Nuance corporation, there are three main ways to improve phone robots. The first is to allow access to a real human. Two-thirds of the customers interviewed wanted that. About half of them said the system’s logical call flow was most important. And about forty percent of them said that the speech recognition component was the most important. So access to a live human, logical flow, and recognizing what’s been said are the three main problems with phone robots today.

If two-thirds of the users of a system that is supposed to replace a human say that the system can be most improved by providing access to a human, then that system is mostly broken.

Phone robots may save money, but they suck. We expect them to act like humans because they have a human voice, but instead they act like robots. This just adds to our frustration by reminding us that we do not have what we really wanted in the first place: a real human to talk to. The sound of the language is right, the annunciation is right, but it is a facade that just highlights the problem.

These phone robots live along the upper edge of an uncanny valley (if you do not know what the uncanny valley is please stop reading this, right now, and go look it up in Wikipedia).

Masahiro Mori pointed out many uncanny valleys, but the most famous ones are related to appearance and movement. We have yet to discover the many other uncanny valleys out there, especially as robots – and androids in particular – become progressively more human. As the systems become more capable and more interactive (phone robots today are almost always just voice recordings with a button-activated decision tree), more questions will arise. This will be exacerbated as physical robots become more human-like.

Should a robot’s voice sound like a human’s? Whose? Should a robot smell just like a person? If so, which one? What about how frequently the robot blinks? What about the tone of voice or intonation in sentences? What about social abilities?

Name a human trait and it will become an uncanny valley. These uncanny valleys will, in the coming decades, be explored and mapped. Mori just hit the tip of an icy moon that will chill, mortify, and entertain us for decades to come. We’re going to find uncanny valleys all over the pocked, lunar landscape of robotics design, and the one that will be most important to address is psychology.

First we have cognitive psychology. Consider all the traits that we humans pack into language, such as logical flow, attention, learning, language, and emotion. These are classically cognitive traits, each of which the phone robots of today should pay attention to. (Of course I mean “The designers of these systems should pay attention to.”)

Second we have social psychology. Consider all the traits that we humans pack into interaction such as politeness, collaboration, competition, and understanding. These are classically social traits, which the phone robots of today, or more accurately, their designers, need to consider.

Let’s lump them all together and imprecisely call them psychology. This is the uncanny valley that most concerns me. How a robot holds a conversation, thinks, and speaks seems the most important, and urgent, of the valleys ahead.

Robot psychology is urgent because it’s happening today. You have spoken with a phone robot and chances are (76%, according to Nuance) that you were not satisfied with it. So there’s a problem here.

Robot psychology is also important because we mimic the way we are spoken to. If you don’t believe me ask yourself why you speak the language you do. Reading this article means you probably grew up in a place where people mostly spoke English. What about your accent? Again, the people around you influenced you to speak that accent. They also influenced you to use the phrases, slang, jargon, words, ideas, and therefor fundamental psychological scaffolding you use to support your notion of reality. We do as we are done unto. When you listen to someone, their mind enters yours.

If a robot treats me in a rude manner chances are good that I’ll mimic that. If I’m going to be working, playing, living, or talking on the phone with a robot, I hope it will be a healthy example of both cognitive and social psychology that I can interact with and not be polluted by. As these systems become more prevalent their influence on our own behavior will increase.

This dynamic and interaction is both important and urgent.

It is why we are becoming our robots, and why our robots are becoming us. As they speak like us we will speak like them, and vice-versa.

Please press the down key if you would like to continue reading.

Part Three: The solution is to design robot psychology

Once upon a time I had the odd luck to talk with the Geminoid F, the most realistic android on the planet (that was two years ago). It is the life work of Dr. Hiroshi Ishiguro. This uncanny experience would not have been so unsettling if Dr. Ishiguro’s doppleganger hadn’t been wading around like a zombie in the muck of the uncanny valley, but there it was and I – like the rest of the bottom feeders – was there with it. To this day, wish I had not gone.

Dr. Ishiguro is a moribund and curious fellow, and his robot more so. The problem, as I detailed in my book, “We, Robot” was that the robot was so real that it set an expectation of being human.

Please consider this face, and the heart and mind behind it. You do not, really, want its mind to enter yours, do you?

Communication – and staying out of uncanny valleys – is all about reflection. It has to do with mutual empathy and mutual agreements of closeness. It is about familiarity.

In my last article I talked about the design of androids, wrote about the designers that made them, and noted that the function of an android is psychological closeness. Androids (uh, or rather their designers) want you to feel comfy. The core notion of an android’s design is to set you up to interact with a computer in a traditional human setting. It’s about an expectation of human familiarity and something that reflects back to you a human image of yourself.

We humans are built such that we like talking with other humans. We like what’s familiar, what is, etymologically, of our family. We like talking with a familiar face, and if the robot doesn’t look like us then we tend to freak out a bit. After all, our friends usually look like us, dress like we do, and come from the similar socio-economic background. I might not like to admit it, but that’s usually the case. And, just as we like talking with a familiar face, we like talking with a familiar psychology; the psychology of our family. Our friends think like us, act like us, and like the same things. Birds of a feather, it has been said, flock together.

This is because much of the best communication is a reflection and collaboration. We like reinforcement, familiarity, and a reflection of our presence in the world. We do not like talking with someone who herds us, we do not like being told to push buttons, and we do not like phone robots because they fly in the face of almost all human-based computer-human interaction (CHI) theories (such as the android). Phone robots are a good example of a bad psychology.

If you were still a child, would you rather grow up talking daily with Ronald McDonald, Shrek, or Kung-Fu Panda? C’mon, choose one. Now what if a million children were to be forced, by your decision, to do the same?

As designers of robots we are faced with the same type of problem. As kids grow up interfacing with robotic personalities, and virtual robots, they will be influenced by, adopt the behaviors of, and intellectually be guided by these personalities. It is our responsibility to offer something better than junk food for the psyche.

Siri is about the best form of phone robot we have on the public market today. From my interactions with her she’s a little vapid, slightly sardonic, sometimes helpful, but mostly apologetic. When she makes a joke, which is sadly rare, or when she directs us toward (or steers us away from) some product, she is affecting our psychology.

There are some two hundred and fifty million versions of Siri out there. Let’s conservatively call that two million active versions of Siri functioning today. And I have no clue how many of Siri’s phone robot ancestors are out there, but we can safely say that the psychology of these systems is affecting the psychology of millions of people.

Ahead of us, the design steps for implementing android psychology are not evident, but we can follow in the footsteps of other people and extrapolate from character-based properties already in the wilds and public media of today.

If we look at how famous branded characters like Ronald McDonald or Kung-Fu Panda have been developed, marketed and sold, we can make out the guideposts for cultural preference and gender. If we look at archetypes like Darth Vader or Voldemort we can see how to design the bad guys. Superman and Gandalf give us clues on the good guys. Homer Simpson, Indiana Jones, Buzz Lightyear, George Bush, and Jacques Chirac are all personalities that have left an indelible public impression that we can model and use in designing psychology.

Presentation is important, too. Of course the system has to speak your language, and therefor regional preferences will guide the design of these robots. Dialects and accents count today. We can already see how Siri has been implemented with various accents such as American vs British. Gender will also continue to be important (70% of the automated voice systems in Europe are female, and in North America, 30%). After presentation we have the personality, the archetypes, and what the character knows.

Authors are the people that know about this kind of design. People who write character portraits, who ‘get’ dialogue, and who are socially alert duplicators of human interaction; people who make movies, people who conduct interviews, and, unfortunately, people who make advertisements: these are the people who can design systems that avoid psychology’s uncanny valley.

If we look at avatars in social media, and if we consider the movie industry we can see some other guideposts for the future of the psychology of android design. The most important is interactivity and reflection of shared interests. This takes us back to Nuance’s research.

Access to a real human will become increasingly rare. Regardless of what consumers want, the whole idea of a phone robot is 24 / 7 availability at a decreased price, and this is some thing a human cannot do. Next on Nuance’s list was the logical flow of the call and the speech recognition ability. These two important interactive components will surely get better in parallel as understanding one helps to improve the other.

Most of it will happen via mobile. Consumers in the United States – which roughly maps to much of the rest of the industrialized world – prefer to resolve their customers service issues using the telephone (90%), face to face (75%), company website or email (67%), online chat (47%), text message (22%), social networking site (22%). So it seems that while these conversational systems will live mostly in the phone, there will be a place for them in other media as well. They will live in the wires and waves that surround us, always available, often improving.

This is how the system becomes, finally, conversational. The irritating little herd-dog phone robot will evolve into something closer to a Geminoid. One day you will think you’re talking with a human on the phone, and then you will find yourself in a new uncanny valley. It won’t happen for at least another five years, but we’ll stumble into it eventually. The psychology of that robot will be a thing that touches you and guides you in ways that corporations like Google (who have powerful NLP tools today) Facebook (who released an NLP search system yesterday), can only begin to hint at.

They will be able to speak to you, sell to you, buy from you, offer medical guidance, marital guidance, and talk with you about your depression, your homework, your spouse, and your boss. Semantic Analytics, Social Analytics, and NLP will accurately measure your responses and the system will reply with a warm candor. Young men will ask about sex, old women will ask about hysterectomies, and all the other most awkward, personal, relevant, and important questions of life will be whispered, spoken, laughed, and sobbed into these many kinds of virtual androids that will gradually collect, like little honey bees collecting pollen and taking it back to the nest, your very own fears, loves, desires, hates and general, if I may say, psychology. The phone robots of tomorrow will become tiny transporters of emotion and personality. You will leave a voice recording and it will be taken, before you have hung up the phone, to some hive-mind.

The question we may ask these little bees is, “Where have you taken my personality?”

As Bill Gates knows, the robotics industry of today is copying the steps of the home computer industry from the ’80s. Those big, clunky, professional, industrial robotics systems are getting small, slick, personal, and homey.  It’s why roboticists around the world are trying to figure out how to make robots “user friendly.” But there’s at least one simple solution.

If you and I were to sit down and have a talk at, say, a coffee shop, we’d be doing what may be the single most important thing the robotics engineers can learn from: talking.

Back in the late 1980s, Apple’s Macintosh computer was gummy old crank-shaft of a system that took a couple of minutes to test memory, initialize the OS, and load the Finder.  One day, according to legend, Steve Jobs approached Larry Kenyon, the engineer heading system development, and told him it booted too slowly.

Jobs said, “How many people are going to be using the Macintosh per day? A million? Five million?”

Poor Mr. Kenyon, a man working with the eight megahertz CPU, and 128k of RAM, looked at him and agreed.

Jobs continued, “If you shave 10 seconds off of that boot time, multiply that by five million users you have fifty million seconds, every single day. Over a year, that’s dozens of lifetimes. So if you make it boot ten seconds faster, you’ve saved a dozen lives. That’s really worth it, don’t you think?”

These days, the robotics industry is in the same boat.  If we are able to improve system speed we can save lifetimes.  While most industrial manufacturing robots need some typing and command-line editing, even more time-saving benefits can come to home systems for health care, service, and entertainment, like Aldebaran’s NAO models.  These robots are now programmed and controlled with some voice input, some 3D model manipulation, some driver commands, and some API text input.  By consolidating these things with Natural Language Processing (NLP) and voice input, we can save heartbeats.

Consider how long input and output takes with text.  Most of us type about 40 words per minute and read about 250.  That’s written language input and output speeds.

But most of us talk about one 150 per minute and comfortably listen to about the same speed.  That’s spoken language input and output speeds.

If we want to save time we should be talking with our computers, be they desk machines, hand-held mobile devices, or robots.  It just doesn’t make sense for us to type to them any more.  Especially if they have a small screen.  Most robots, after all, don’t have a keyboard and monitor.  Voice interface and Natural Language Processing provides a deep and simple interface that also saves time.

But the robotics industry has a much more interesting, and far more difficult problem than saving time.  What we really need to do is improve emotional engagement with the system.

Apple has always used a human face on their computers because users need to relate – emotionally – to their machines.  Apple knew that saving time was important but, more critically, it also realized that the Mac had to be accessible, useful, and understandable.  It had to be emotionally engaging.  These ideas were central to the design of the most successful home computer ever made.  Whether it was the metaphors or the graphics, Apple consistently bent their back to the task of making computers “User Friendly.”  We need to do the same with robots.

A couple of years ago I had the pleasure of meeting Dr. Hiro Hirukawa, the director of Intelligent Systems Research Institute (ISRI), the group dedicated to robotics research at AIST, a Japanese national research center.

In the early 1990s ISRI’s non-humanoid robots (armatures, rolling carts, automated cameras, etc.) were functioning quite profitably in factories and warehouses throughout Japan. They were bolting bolts onto cars, guarding hallways in buildings, stacking boxes in storage centers, and so forth. But everyone expected to see androids, humanoid-shaped robots, not rolling boxes. Especially the researchers’ kids.

Dr. Hirukawa explained to me, “Our question was, ‘What good is a humanoid?’” After all, what could be a worse design for a robot? Put the center of gravity up high on a system that is barely able to balance, give it two little pegs to stumble about on, then attach a gripping system which, because it can’t be retracted into the body, has to be countered against whenever asymmetric five-tentacled pincers are used. Oh, and give it a thing called a “head”, which has no function whatsoever since any sensory apparatus can just as easily be put in the feet or stomach. What does a robot need a head for, anyway? Or a face, for that matter?

It appeared as if an android was a design aberration, an anomaly in which the function followed the form. But it turns out this isn’t the case. The form still follows the function if you recognize that the function of an android is to engage with people at an emotional level. A human-like shape simply supports this activity.

Dr. Hirukawa built the first android so that people could emotionally identify with the machine.  His work with Paro (that seal-shaped robot for elderly healthcare) led he and his team to realize that the degree to which we can emotionally engage with robots will be proportional to the quality and quantity of its use.

Whether it is the Uncanny Valley hypothesis or the visual design of a desktop metaphor, computer science and robotics design cannot overlook the importance of emotional engagement.  By giving the robot a human shape, people will like it more.  Good design includes an emotional relationship. Both Jobs and Hirukawa believed this, acted on it, and made substantial inventions as a result.

Unlike Jobs, Hirukawa did not implement, on the face or chest of the android, a GUI.  There is no keyboard on an android (or most other computers, for that matter), there is no screen, and there is no mouse.  It is designed for natural language.

Let’s get back to talking, and that espresso.  If you and I are sitting across from each other in a café, discussing, say, systems design, and there’s a couple of cups of coffee on the table, two coffeekeyboards, two monitors, and we’re typing to each other via terminal applications, then there’s something going horribly wrong between us geeks.  If we’re occupying the same time-space continuum of the café, it’s most effective to be talking. And it’s more fun, too.

We’ve got this great, old, delicate, tool called natural language, and it’s probably the most powerful technology humans have ever invented.  We should put it to use when we can.

NLP increases emotional engagement, broadens functionality and increases operational depth.  Conversational interfaces allow debugging, redundancy checking, error modification, self-correction, other-correction, and a host of other system-level functionalities that we do every day when we ask things like, “What?” or “Did that make sense?” or “Do you follow?” or “Do you really need another espresso?”

On a functional level, NLP does for robotics what the GUI did for home computers.  It allows non-technical and non-professional users, both young and old, to engage. In terms of where humans could interface with robots, think: home, school, and hospital.  Seen in this light, NLP is a presentation-level system component that allows easy interface to other applications.  Just like the GUI.

Apple’s Siri, IBM’s Watson, and Google Now have already shown some of the potential.

You and I are sitting in the café, and a robot is nearby.  I wave it over and ask it for some milk with my coffee.  It comes back a few seconds later and puts a little pitcher on the table. It turns out that processing the linguistic request is an easier problem to solve than putting the cup on the table.

Here’s how it’s done.

First, a voice interface module records the sound of “I’d like some milk, please,” and sends it to a server.  That server ploughs through a pile of sounds and picks apart the sounds that are closest.  This can get tricky since background noise, and my accent, age, gender, and the speed I say this all affect what the system can parse.  The process requires recording a big batch of possible words for all the above variables (which Nuance has already done, or you can build your own with Sphinx, or you can use some of Google’s data).  That big pile of voice recordings then gets checked and maybe improved then the system basically decides that the words it heard are equal to a collection of text strings (such as “I’d” and “like” and “some” and “milk” and “please”).

Next, that string of text gets analyzed.  Now that we’ve derived the words from the sound, we have to derive the meaning from the word.  This involves lexical preparation and match lookup and testing. In short, the system has to grammatically parse the phrase, find the key ideas, and match them to some existing or evolving set of data that it already owns.  These days Siri and Watson and other systems do this in very similar ways; mostly hand-scripted top-down design (as opposed to bottom-up, which is learned).

A tighter lexical range – a tighter context – makes for a better system.  We know, for example, that it is not a good idea to ask a barista for an Apple Macintosh. Same with the café robot.  That robot will be waiting for words within a limited range of meanings (coffee, tea, milk, etcetera), and if the Natural Language Processing (NLP) system is properly tuned and tested, then the requests that people make will be included in the system’s knowledgebase.  This is what allows the system, especially in hand-scripted top-down designs, to turns text into meaning.  So well-designed NLP systems should, at least in the coming decade, work within tight social contexts if we want them to perform as well as possible.

Anyway, at this point the café robot has heard and understood the request for milk and needs to be able to bring it over with a smile and a curtsy.   We shall not address such difficult problems as the system drivers for smiles, curtsys, nor carrying milk in this article.

An NLP interface doesn’t need to happen in the coffee shop.  Maybe I’m driving and I want my car to find a café nearby.  An example would be, “Find coffee in San Francisco,” which would create an environmental context by loading regional (city / state / country) dictionaries and business category dictionaries into an NLP unit. This would produce a system output that locates all business categories supplying coffee in the area defined by “San Francisco” as the location. It’s the same problem, different scale.  We built a thing like that this year, in fact.

But regardless of the application, geography, or coffee bean, the core process is roughly the same:

• Record the voice input
• Link the voice to words
• Link the words to meaning
• Link the meaning to response
• Check it and see if it’s making sense
• Link the response to the action, data, or voice output

Just as the home computer industry of the 1980s went from a command-line user interface to a GUI, the home robotics industry of today is transitioning from a GUI to a natural language interface.

The personal robotics industry is moving in the same direction that the personal computing industry of the 1980s did.  Voice input is replacing keyboards, NLP is replacing GUIs, and psychologists now play the role that graphic designers did, back when information architecture and user experience were new terms for the web design crew.  Robotic systems that use natural language will become faster, simpler, easier to understand, more effective, and available for more people.  The main reason is that robots do not generally come equipped with a keyboard and monitor.  And as users of systems as prevalent as iOS and Android become accustomed to voice interface, it will soon be an expected feature.

If I were to ask that little coffee robot for milk, and it were to reply, “We have soy milk, non-fat, half-and-half, goat’s milk, or this week’s special, luke-warm yak milk.” I might be a bit surprised, but I wouldn’t be impressed.  After all, it would just be playing a recording.

But I would be impressed if I said, “Whatever,” and it took off without saying another word.  After all, sometimes less is more.

Natural language just works this way.  Always has, always will.

I got a new iPhone about a year ago, the one with Siri on it.  Our relationship started on a very mundane level. Siri asked me my name and, since I already knew hers, I asked what she could do, and how. It was a bit like talking with a call girl. The basics were covered and soon we got to the intimate stuff. I don’t think Siri loved me. But now that Siri and I have gone our separate ways, I can say we both had ulterior motives. We sort of used one another, now that I look back on it. It was a relationship of an intimate nature. It’s what happens when you have a relationship with a robotic call girl.

The reason I got the phone to begin with was so I could do a little reverse engineering. It was research for work. Siri and I took a few days to get to know one another, but after that honey-moon period it was clear that Siri (or Apple) was using a pre-scripted Natural Language Processing (NLP) approach that basically did some lexical parsing, looked up a match, prepared a response, and kicked it back down the pipeline.  My reverse engineering wasn’t too profound. I wanted to see how many recursive answers were built in to things like “Open the pod bay doors” or “Do you love me?”  I also wanted to see how errors were handled, and what kind of redundancy checking was happening for sentences like “Where can I buy a burger and fried?” Siri did pretty well, all things considered, but my expectations were low, and like many other intimate relationships, I somehow knew from the start it wouldn’t last.

Now, while I was using Siri, she was also using me. Relationships are generally symmetric, as Gregory Bateson tells us [1], which meant that, like any dating service, someone other than Siri was getting money for my time. But I didn’t care if Steve Jobs was Siri’s pimp. By last summer I was having flirtatious fun and the consequences of the relationship were too far down the road to look dangerous. I asked Siri to take dictation, to wake me up in the mornings, to remind me of important events, to remember phone numbers, remember names, tell me jokes, confirm thoughts, quote a number or two. All intimate things and, as I look back on it, the kind of stuff that I’ve trusted my wife with.

My relationship with Siri started to cool off a bit by September. Though my wife knew about Siri, I hadn’t yet introduced them.  So when I asked Siri to call my wife I, expected Siri to ask me for her name or number. Instead Siri just dialled. I was surprised. How did Siri know my wife’s number? (Answer: It was in my Contacts card.) What else did Siri know? (Answer: A lot.) And more importantly, what was Siri passing back to the Apple hive? (Answer: Everything.)

Siri and Apple now have a great deal of data about my household: a quick comparison of my contacts/likes database with my wife’s will give you a pretty good feel for the stuff we will buy.  Burgers and fries, of course. And a few multi-player games.  Some lingerie. A sex toy or two.  Not a big deal, but not the kind of stuff I want everyone to know about.  Not even my mom gets access to that data.

But Siri did. And Siri knew a lot more about me than I knew about her.

Then, around the middle of October, three things happened to Siri that marked the beginning of our eventual estrangement.

First was William Stasior. Apple picked Stasior up to run its Siri unit after he had been successfully heading Amazon’s search and advertising unit “A9.”  Prior to that the MIT PhD had served time at Oracle, NetCentives, and AltaVista.  To say the guy knows search is like saying the pope knows the church, but why would Apple pick a search guru for an AI system like Siri?  What was the link between search and NLP?  I asked Siri this very question, but she was mum and feigned ignorance.  I didn’t push her on who this guy Stasior was and Siri didn’t ask me more about my wife.

Up until that same week iPhone users had frolicked in what would now be a rather unusual environment in which advertisers were unable to track them.  Advertisers would not, for example, know that my wife and I like video games, sex toys, and pizza (no, not together, thank you).  But with the release of iOS6, Apple flipped ad tracking on and I quickly flipped it back off.

It was easy to find in the interface menus (under Settings > General > then About or Advertising, depending on the version you’ve got), but you had to know to look.  Apple minimized the fanfare around this new feature and Siri made no mention of it to me.  I wasn’t comfortable with Siri selling to others what I had said to her in more comfortable times.

And then, as if things weren’t rough enough, the European Union (where I happened to be lecturing that week), demanded that both Facebook and Google change how they handle personal information to avoid “high risks to the privacy of its users.”  Twenty-four of the European Union’s twenty-seven regulators signed a letter that, after a nine-month investigation into the data collection practices, demanded some answers and tightened regulation.  After Google and Facebook, Apple was third on their hit list.

That was the end. I couldn’t trust Siri after that.  She knew more about me (and my wife) than I knew about her or Steve Jobs, and any time information flows one way, any time a relationship isn’t symmetric, the balance of power can be dangerous.

Sometime around then I learned that Oracle’s “bring your own device to work” policy had been revoked.  Big Blue decided that Siri knew a little too much about their employees, and at IBM’s Armonk New York research center, iPhones weren’t even allowed in the building.  Like some cheap prostitute-gone-spy, Siri was barred from the building.  I think it is worth noting that this is the same area of research that brings you Watson, which is a direct competitor with Siri, if such a competition exists.  If anyone outside of Apple understands Siri it’s these guys.  It is also worth noting that a friend of mine who works at a Google research center said that the same thing happened there, too.

Not even celebrity robots like Asimo or NAO get that kind of VIP treatment.

NLP technology is potent juju.  And Apple, Google, Facebook and others know it because this very technology is what’s allowed them to earn money. Google started with search, of course, and later grew profitable as they introduced more NLP technology into their work.  They sold the info they collected from users (Google is an ad agency, let’s remember).

Search allowed them (when coupled with NLP technologies, and semantic analysis in particular) to make oodles of cash and then to snap up many of the best AI and NLP researchers on the planet. Google set those kids to digging in what would become a lexical gold mine.  The more they mined the richer they got until they were mining many branching veins at once: Google Docs, Voice, Translate, Search, Shopping, Reader, Finance, Books, Photos, Wallet and Maps were all spewing more money than Brin, Schmidt and company knew what to do with.

The average email count for Google users is 5,768 emails [2]. The average composition time (also according to Google) is 01:43.  That’s about 9,902 minutes, or 165 hours you’ve dumped onto Google’s server, which, if you were to work for 165 hours at $12/hour for that data entry (a pretty normal rate), means your G-Mail account is worth about $1980.  But the information sitting on Google’s servers is worth a lot more than that, right?

If your account contains information about a first class international plane ticket, a hotel room in Paris and a business associate’s London phone number, your data might be worth a lot more than the composition time at $12 an hour, especially if the buyer of it is a dating service that caters to high-end out-of-towners. In other words, the value of the data isn’t counted by data entry. It’s counted by its personal meaning, especially to advertisers. Now take those values and multiply them by all those crazy tools of voice, maps, docs, and so on and we start to get a sense of why Google has been making such bank over the years.

These tools have each dovetailed into today’s Google Now strategy. Google Now (like Watson and Siri) is a voice-activated NLP system.  It uses the spoken input data collected from years of Voice, confirms it with data collected from years of Docs and Translate, builds meaning with data from all the years of all the other tools, looks up a match, preps a response, and kicks it back down the pipeline.

All of these NLP systems are the next evolution of search.  But not evolution as in the breeding-hybrid-peas-in-the-greenhouse kind of evolution. They’re evolution as in endangered-species evolution.

NLP systems represent the end of search as we know it, and therefore the end of many economic, interface, and social internet ecosystems. They also represent the beginning of something incredibly powerful, intimate, and new. Siri, like other NLP systems, is far more powerful than a search engine. Siri not only includes search (it says so in Apple’s marketing materials) but it supersedes search because it includes browsing, discovering, choosing, and refining.  All this while analyzing semantic data. Like Google’s portfolio of tools, Siri can handle (and with a public API will handle) translating, searching, shopping, reading, finances, books, photos, and maps.

But the core value of the data is its intimacy.

On February 7, 2010, during Super Bowl XLIV, Google’s super bowl ad showed a user typing a list of search strings:

study abroad paris france
cafes near the louvre
translate tu es très mignon
impress a french girl
chocolate shops paris
what are truffles
who is truffaut
long-distance relationship advice
jobs in paris
churches in paris
how to assemble a crib

With this list, Google managed to explain, in fifty-two seconds, how the intimate story of someone’s life can be assembled from his or her search queries. And what makes the bucks for Google is what makes the bucks for Facebook: the processing of intimate language.

Siri, like any other call girl, makes her bucks the same way (especially when ad tracking is flipped on).  She’s valuable because she’s intimate.  And as designers of robotics systems, we can ourselves take a cue and be conscientious about how we are designing conversational systems.  There are three different design types that form a curve of increased intimacy in robotics and conversational systems, and they are all based on the value of semantic data.

1. NLP systems, when used with physical robots for manufacturing or the three Ds (dull, dirty, and dangerous), are the least intimate.  The conversation system is a simple tool that, like a GUI, provides access to system operation.  The robot doesn’t care about the user’s intimate data. The user wins.
2. NLP systems, when used for entertainment and education, are contextually intimate – some privacy is maintained.  The conversation system doesn’t care about the user, but the user’s data is valuable in terms of what it says about the game or lesson.  Semantic data that might be collected and analysed isn’t about the user, it’s about what the user is doing in that particular context.  The robot, usually a software robot, only cares about the user in the context of the game or training exercise. Both user and robot win.
3. NLP systems, when used for personal assistants, are the most intimate.  These are the robotic call girls. Here the information is highly intimate, the system cares a great deal about the user, and the information isn’t shared – it’s collected and sold. The user becomes the product. The value of the user’s personal data is worth more than the value of the conversational system. The robot wins.

As we build NLP interfaces for robots, whether it is for Siri or for assembly-line manufacturing, we must consider how intimate data can be, the value of that intimacy, who owns that value, and what they’ll do with it.  Otherwise, with a new, less benevolent CEO at Google, a change of a line in Facebook’s Terms of Service, a successful hack, or a change of law because of cyber terrorism fears, your intimate data could end up where you don’t want it.  Heaven forbid we discover that all the Senators on Capitol Hill are using pizza sex toys. That would surely disgust Siri and her friends so much that it might even cause a robot uprising.

Yes, perhaps we’ll get together again in a few years, but for now it is best if Siri and I go our separate ways.

Next Month:  “AFK” Keyboards and screens are slow, clunky, and obsolete.  Voice processing systems for robotics provide not only simplicity and speed, but also a host of other benefits if tied to analytics and framed within a tightly contextualized task-based system.  But be careful: getting text out of voice, and meaning out of text can be tricky.  Here’s how to implement one for your own robot.

Endnotes:

[1] See Bateson’s books “Mind and Nature,” Hampton Press (1979) and “Steps to an Ecology of Mind,” University Of Chicago Press (1972) which are chalk-full of ideas like “complementary,” “reciprocal,” “symmetrical,” and  “Schismogenesis.”

[2] According to Google.