Alelo’s virtual role-play technology creates safe learning environments for students to practice real-life conversations. Customized feedback can be provided during and after interaction to analyze performance.

Cyberlearning project teaches language and cross-cultural communication with avatars and robots.

Communication skills are critical in a global economy.

Since communication often occurs across cultures, people must know more than just the vocabulary and grammar of a foreign language. They must also have cultural awareness–a recognition of the “dos and don’ts” that each culture maintains–to communicate smoothly, effortlessly and with confidence.

Virtual role-play – where learners engage in simulated encounters with artificially intelligent agents that behave and respond in a culturally accurate manner – has been shown to be effective at teaching cross-cultural communication.

“You learn by playing a role in a simulation of some real-life situation,” said W. Lewis Johnson, a former professor at the University of Southern California and co-founder of Alelo Inc., a company that makes cyberlearning tools. “You practice communication with some artificial intelligent interactive characters that will listen and respond to you, depending on what you say and do. It helps develop fluency, but it also helps to develop confidence.”

With an award from the National Science Foundation (NSF), Johnson and his team at Alelo (a Hawaiian word that means “language” or “tongue”) developed a range of products to teach cross-cultural communication using virtual worlds. Alelo’s tools have already achieved success in military training and English-as-a-second-language programs and are now being applied to a wide range of learning applications.

“We found in our military training that when soldiers use this approach, once they get into a foreign country–let’s say they’re sitting down with local leaders–they feel as if it’s a familiar situation to them, even though they’ve never experienced it in real life before,” Johnson said. “That degree of comfort is extremely powerful.”

As an example, Johnson showed a role-playing scenario that orients officers to the etiquette expected at a Chinese banquet. (The goal: Don’t get drunk!)

“How do you show proper respect for your host in participating in these toasts but not fall down drunk in the process?” Johnson asked. “That’s an effective communication problem. Role-playing teaches some of the communication skills that you could use if you find yourself in this situation.”

Importantly, Alelo’s virtual role-playing tool gives one the ability to practice conversation and etiquette until you get them right.

To develop a version to teach English as a second language (ESL) to those in the United States, the researchers interviewed immigrants to determine what cultural issues they found most problematic. The instruction they created–available in multiple languages–explains how to handle different situations that one might face as a new immigrant in the United States and provides tips on culturally appropriate behavior.

Students can practice on a range of devices, allowing them to use the program to prepare for classroom activities, as follow-up to classroom activities or as needed in the field. The ESL program even offers a virtual coach who provides help and feedback.

In 2012, Alelo partnered with the Commonwealth of Virginia to develop and test virtual role-play learning products for conversational practice in Chinese.

Tammy McGraw, the former director of digital innovations and outreach at the Virginia Department of Education, had met Johnson at a Wired conference years earlier and, although Alelo was demonstrating cyberlearning technologies for soldiers, McGraw was struck by how the virtual characters were able to impart both linguistic and cultural lessons.

Moreover, the software could address the shortage of qualified teachers able to teach Chinese, Arabic and other less popular languages, while allowing the state to increase the number of students who were able to take classes that were over-enrolled.

The RALL-E project combines Robokind’s advanced social robotics and Alelo’s virtual role-play technology. Using automated-speech recognition and customized acoustic models, RALL-E can understand and respond to Mandarin Chinese speakers of all levels.

The first Web-based course was tested last year via Virtual Virginia, a statewide virtual school program and received overwhelmingly positive ratings from the students.

“The virtual classes enabled students to be more self-directed than teachers could normally support,” McGraw said.

Recently, McGraw and Johnson have been adapting the insights learned in virtual role-playing research for a new platform: a robo-tutor.

With support from NSF, Alelo has taken RoboKind’s lifelike social robot and equipped it with the ability to converse in Chinese. The technology gives learners abundant opportunities to practice their conversational skills and lowers the entry barrier for difficult languages such as Chinese.

“It’s non-threatening,” McGraw said. “The robot is not going to be offended if you say the wrong thing. He’s not going to laugh at you if you pronounce Chinese incorrectly. But he also has the ability to engage learners through active interaction.”

Though still in the prototype phase, they imagine RoboKind could be used in the classroom, with kids coming up to engage with it at will. It could also be used outside of formal class as a personal tutor. Teachers at Thomas Jefferson High School of Science and Technology in Alexandria, Va., signed on to try out the computer-based pilot.

By combining artificial intelligence and robotics with insights from virtual role-playing research, Alelo is pioneering new avenues of personalized education.

“The question we’re trying to answer is: what are the aspects of the technology that are most important for mastering communicative skills?” McGraw said.

“We have two different realizations of virtual role-play technology. They have different strengths, different weaknesses. What are they going to be good for? Which is good for what? Doing this research, we hope to get some answers to those questions.”

Related Websites
Alelo: http://www.alelo.com/

When I was in high school I was a big fan of graphic novels. In art class I used to splash out paintings – portraits mostly – and would write on them, giving the portraits word balloons, like a comic book. I wanted to portray personalities more than portraits. Of course my art teacher hated this and she’d box my ears, telling me that painters should not write, and that writers should not paint. She even called me an artistic criminal.

But I went on (as you did, dear reader) to make things like Twitter accounts and LinkedIn profiles and selfies that I’d then tag with some text and did just that – put words with the picture.

Today we’re all  portrait artists and writers. But with cameras in our phones and publishing tools that allow us to be both exhibitors and viewers in the often perverse universe of social media, it’s not so much about self-expression as it is about identity. We all write blog posts and put an image in them, and rarely does a selfie not include some tagline that’s supposed to be clever or witty, since, like all self-effacing artists, we want to be clever and witty and get into the mosh pit of show-n-tell.

Selfie = Profile = Account = Avatar = Self-portrait = ID

Whether you call it a profile, an account, or an avatar, you’re making a self-portrait. It’s your ID. It might a 2D profile, or it might be a 3D avatar for a game, but when you upload a photo of yourself and then you add some text, you’ve made a self-portrait ID that’s got something to say.

And you make IDs for yourself more often than you might think: if you’ve got a Twitter account, then you’ve done this; if you’ve got a Facebook or LinkedIn account, then you’ve done this; even when you pick some dumb photograph like the flower or the horse to use as your login image for OSX or Windows, you portray yourself with that image, and then you write your name next to it. We all have dozens (if not hundreds) of these weird little text-garnished self-portraits scattered all around the Internet.

Your avatar is your self-portrait, and it has more value than you think.

Regardless of your ability to draw or write a verse, these self-portraits you’ve made are very, very valuable.

Portraits have always been valuable, and this is because our identities are valuable. The royalty of Europe used to have portraits painted as a means of immortalizing their ID onto the wall of a castle, and the money required to hire an artist to paint their face for all the castle-folk to see was a declaration of power, influence, and wealth. Even when the camera was invented, portraits were still valuable, and sitting for a portrait was a sign of status. Today a face on the cover of a magazine is the same. It’s a social value.

Not all of us can be Portuguese Kings, but those days are past us anyway. Today we wallow in the attention of social media by building pages that proclaim how great our life is (or how miserable), or draw attention to things that interest us, and we have little numbers in social media sites that measure our self-portrait avatar on some social scale. Our avatar self-portraits, therefor, not only have text, but they also have numbers so that we can see how they’re performing – and that number ranks the value of our avatar self-portraits.

These avatar self-portraits generate various kinds of value, in terms of privacy, relationships, organization, access, and even money:

• Edward Snowden demonstrated that value in terms of privacy with the NSA revelations; why would the NSA spend all that money, time and effort collecting the information if it wasn’t valuable?
• Every time you talk with family, friends and work colleagues via Skype, you demonstrate that value in terms of relationships; why would you spend all that time on Skpye if the relationships were not valuable?
• Your user account on Apple’s iCloud is something that is of value to you to help you organize your information, schedule your day, and purchase the virtual commodities that help you navigate the city, purchases, and social network. That’s value expressed in terms of quality of life, quality of organization, and quality of access.
• And, most obviously, companies like Google, LinkedIn, Twitter, and Facebook mine our avatars’ data and build a cultural value system that is so valuable it is measurable in terms of billions of dollars per year.

Whether it’s privacy, money, identity, relationships, or money, the value of our self-portrait avatars is important to the very structure of society, and this has been the case for a very long time.

It used to be that only the very wealthy were able to afford portraits that lasted after their death and secured their position in history: a portrait hanging on a castle wall is evidence of the individual’s importance (both historically and financially) to the building itself; people used to pray to the portrait of the Pope; swear allegiance to the image of the king; and even today it is common practice in the US and many other countries to hang the portrait of the President or head of state in government offices.

But pop culture has a way of transforming things that were untouchable into things that can be handled and manipulated by the average person. It takes things that were ‘out of touch’ down off their pedestals and out of their glass cases, and makes them into something smaller that can be changed, edited and owned.

The avatar is a pop culture version of the self-portraits of classical history, and they allow us to change and distribute the portraits of ourselves as the royalty of the Renaissance used to do with their portraits. We use our avatars today to buy, sell, negotiate, advertise, rate, share, trade, learn, teach, train, talk, read, listen, write, and have fun with family and total strangers. Our avatars can do far more than a classic portrait can do because they are interactive, we can handle them, and they belong to us in many small pieces.

What if your avatar could talk for you? 

What if you had an autonomous avatar – a version of you that can talk for you when you’re not there –  that was your doppelgänger? What if it was able to protect your privacy, support your relationships, improve your quality of life, and make you money while you slept?

At Geppetto Avatars, we wonder about this next step. What does it mean if an avatar is able to talk without a person behind it pulling the strings? If you were to take all of the content you’ve written, and then provide your avatar with the ability to speak all that knowledge that bangs around in your skull, what would it say? If you had a self-portrait that could talk to you, what would you discuss with each other? If you could walk away from a mirror, and your reflection stuck to the glass, what would it say to the people it encountered after you left?

Now consider your job and the things that you do to make money in this world. Specifically, imagine the things that most bore you, and that you repeat daily. Imagine taking those words and gluing them, like a word-balloon, to that reflection of yourself in the mirror. Imagine that your avatar could discuss those things, share your knowledge, and contribute to a larger ecosystem in which you yourself no longer had to spend as much time doing these tasks.

Would it provide you with more privacy? Would you use it to interact with work colleagues? Or family members? Would it be a virtual assistant? What if it could answer your phone for you, or perform simple business tasks like ordering a shirt, calling a taxi, booking a reservation, letting someone know you’re running a bit late, buying a book from Amazon, calling to find out why your phone bill increased, looking for a good sandwich shop, and finding out what hours they are open?

Would we want these things? Would we learn about ourselves, or become more ignorant of what we like? Would we protect our privacy, or would we become vents of deprivatization?

Of course there is a nightmare that hides in these notions, a dark thing called polyopic heutoscopy – a psychotic state in which schizophrenics find themselves surrounded by their various alter egos. Could that nightmare, instead, be our future?

In the coming years these questions will be answered.

One day we will be able to build avatar self-portraits and set them loose in a larger ecosystem where they can speak with other people and avatars, to complete tasks for us and, much as a company might today, deliver messages, answer questions, listen, observe, collect information, organize information, barter, buy, sell, and serve as knowledge brokers and doppelgängers in an increasingly virtual world.

Today Geppetto Avatars builds healthcare avatars and instructional avatars that can discuss, listen, observe, and share information. They’re doppelgängers that liberate busy individuals from performing rote tasks, so that they can spend their time on higher value work instead.

Already there are a bunch of talking avatars running wild out there, and they’re turning out to be pretty valuable:

• Craig Mundie, Microsoft’s Chief Research and Strategy Officer showed how Microsoft’s Xbox Kinect and an avatar could facilitate group therapy sessions; people generally feel less intimidated by an avatar than by a human group leader.
• Research suggests that the experience of watching your digital look-alike uneventfully reach the front of the simulated line and order a pretend drink is real enough to help you learn to cope with similar situations in the actual world (thus reducing social anxiety).
• Virtual coaches can help people with PTSD.  The director of this project, Skip Rizzo, says: “It leverages latent inhibition. If you expose somebody to something in a controlled environment in advance, when they face that challenge in the real world, there’s less chance of fear conditioning — which is at the base of PTSD.”
• According to Prof Jeremy Bailenson, avatars are becoming more popular. Kids in the US spend about an hour a day inside an avatar. To put that in perspective, they read print for about 38 minutes a day.
• And this week Watson acquired Cognea (formerly My Cyber Twin),which means that we’re about to see a lot more of this kind of thing.

The avatar is becoming the medium of the future.  If your avatar could talk, what would it say?

All images courtesy Dito Von Tease / ditology Follow @DitoVonTease. If you liked this article, you may also be interested in:

• Facebook – Our ffriendly automated identity bender
• “A Paro walks into a club…” (or, Notes on the design of automated body language)
• The uncanny valet (or, Notes on the design of robot psychology)

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

It’s hard to communicate with words. Some researchers claim that almost half of our communication relies on things that aren’t words: body language, tone of voice, and stuff that just isn’t conveyed by text. This includes prosody (tone, pitch and speed of words), facial expression, hand gesture, stance and posture. This probably explains why about 40% of emails are misunderstood.  As designers of robots (or avatars) we need to consider these statistics and consider how to integrate body language into natural language communication. Therefor Geppetto Labs has built a platform to automatically generate body language and coordinate it with what a robot (or avatar) is saying.

Most NLP systems today, be they Siri or Watson, amount to conducting chat via the thin pipe of a text interface. Siri doesn’t have a lot of choice on the matter since Apple had to simplify the complexity of communication, but this text interface reduced the communication itself. If you think that Natural Language Processing is about only text, then step away from the computer, go to a café or bar, and watch people interact for a half an hour.

Videos do an excellent job of conveying the importance of body language. A great video to watch is The History Channel’s Secrets of Body Language. This documentary looks at politicians, cops, athletes, and others, interviewing experts of body language to decodify everyone from Richard Nixon to Marion Jones. Gesture, expression, and tone of voice are all looked at as valuable and important data channels.

This is why face-to-face meetings are so much more productive. Each party can better understand the other because there is a higher throughput of communication. In a lovers’ relationship, or in a family’s relationships, body language is even more important than in a business meeting. Consider the fact that it’s the most intimate relationships (between lovers, primarily, but also between family members, close friends, and others) that involve the most touching. These are also the relationships that rely the most on body language, because body language actually defines the proximal closeness and intimacy.

So if we want people to engage emotionally with robots, or avatars (or any other kind of character that is rigged up to an NLP system), we need to consider using body language as part of that system. We humans are hardwired that way.

***

At Geppetto Labs we have begun considering Body Language Processing as a sub-set of Natural Language Processing. So just as Natural Language Processing has NLU (understanding) and NLG (generation), we can consider Body Language to have BLU and BLG. I’ll be focusing on the generation of it, but others, such as Skip Rizzo, Noldus Information Technology, and others are also looking at the understanding of body language and facial expressions.

Generating body language requires coordination with the textual components of Natural Language Processing. A gesture or animation has to have the same duration of time, happen at the same moment, and include the same emotional content, or affect, as the message conveyed.  “Hi” should, of course, be accompanied by a gesture that is about one second long — a friendly-looking signifier that’s commonly understood. Raising the hand and wagging it back and forth usually gets the job done. But building this can be tricky. It gets more complicated when there is a sentence like this one that doesn’t have clear emotional content, isn’t the kind of thing you hear as often as “Hi,” and is long enough that the animation needs to be at least ten seconds long.

At Geppetto Labs we’ve developed the ACTR platform in order to accomplish this. The core process, at least as it relates to text, is to generate body language (as opposed to voice output) as follows:

First, we take the nude NL text and determine the three variables of the Duration (timing), Affect (emotion), and Signifiers (specific gestures):

1) Duration, or timing. How long is the sound or string of text we’re dealing with? This is the easiest to calculate directly from the text. Most spoken conversation ranges from between 150 and 175 words per minute, but that can speed up or slow down depending on the emotion of the speaker, and other factors. But let’s call it 150 words per minute. A “word” is calculated in these kinds of standards as five UTF characters, which is also five bytes. So that means that most of us speak at around 750 bytes per minute. Now if we back this out it means that around 12 bytes should leave the system per second, and this is then used to calculate the duration of a given animation. We’ll call this integer between one and 150 a “duration tag.”

2) Affect, or emotion. What is the emotional value of that source string of text? This is the second factor we need to know in order to calculate an animation, and it’s harder than just measuring the letters in a line: it requires either realtime sentiment analysis and/or a pre-built library that identifies the emotional content of a word. One solution is WordNet-Affect. Words in WordNet-Affect are derived from Princeton’s fantastic WordNet project and have been flagged with particular meaning that indicate a range of values, most of which relate to what kind of psychosomatic reaction that word might cause or what kind of state it might indicate. Some simple examples would be happiness, fear, cold, etc. There’s a ton of really sticky material in this labyrinth of language called “affect,” and the ways that words link to one another make it all even stickier. But for this explanation, let’s say that we can take a given word and that word will fall within a bucket of nine different emotions. So we give it a value from one to nine. Fear is a one. Happiness is a nine. If we then take the average affect of the text string in question (again, speaking very simply) we end up with a number that equals the emotion of that sentence. We’ll call this integer between one and nine an “affect tag.”

(Before we go on I want to take a break because we now have enough to make an animation match our sentence.

“How in the world do we build that?” is an eight-word sentence, so we know the duration would be about three seconds. The affect is harder to measure, but for this example let’s say that it ends up being a value of 5. So we have Duration=3, Affect=5. These two bits of information, alone, are enough to calculate a rough animation, but first we need to build a small bucket of animations. They are probably keyframes because we want to interpolate them so that they form a chain. We make them of various durations (1 second, 2 seconds, 3 seconds, etc.) so that if we want a three-second chain we can combine 1-second and 2-second duration animations, or, if we want to avoid replaying the same animation we can reverse the order of these links and combine the 2-second then the 1-second animations. And we make sure that we have these various animation links ready in separate buckets – one for each animation. So if we get a Duration=3 and Affect=5 we go into the bucket labeled Affect #5 and dig up the animation links that add up to three seconds.

The longer the duration, the trickier it gets. If you have a twelve-second animation you might then have to chain together that two-second animation six times, or your one-second animation twelve times, to get the proper duration.  Does that make sense?

No. I hope that at this point you’ve stopped and said, “Wait, no, that would be really dumb. To play an animation twelve times would just look like the character is convulsing. That’s bad body language, Mark!”

A spoonful of art and design can help. When you are combining the animation links to build a chain of proper duration, you need to avoid spasms and try to use the longest possible animation in your bucket. Then you frame it outwards with smaller animations to build the proper duration sum. If you want a twelve second animation, then try using a ten-second link with two at the front, or one on each end. That will avoid the spasms.

We’ve found that Fibonacci sequences work great for this because they have huge flexibility and, when interpolated together, are nearly invisible. This is the process we have found to be most helpful, flexible, simplest, and that looks the best. So this means that you’d need to build a relatively large number of animation links that are the following values: 1, 1, 2, 3, 5, 8, 13, 21, and so on. And you’d do that for each affect “bucket.”

Ok, so far so good. The next step is to play this animation and, as your character moves, watch it and ask yourself if that’s the kind of thing a person would do while saying that particular phrase.

Now back to our three variables…)

3) Signifiers, or specific animations. Is there a common gesture that normally accompanies the text? The wave of a hand that goes with the word “Hi” isn’t normally used in conversation as much as, say, nodding, or showing our palms when we speak. Most of our body language is unspecified. So specific signifiers need to be manually wired up to particular output phrases from the system. This is a hand-authored component that requires an author to flag specific lines that require specific gestures. We are developing methods for automating this, but for now it is necessary to build a few specific gestures that are used in unique moments of conversation.

***

Once all the variables are determined, they get packed together and our ACTR platform fires off a control file in XML that drives animations on the client side. This keeps the performance hit on the server low, allows a nearly infinite range of movements for any connected system, and provides a robot (or avatar) with the ability to gesture as it speaks.

Though this implementation is new, these concepts have been knocked about since the mid-1600s. In 1605 Francis Bacon, in The Advancement of Learning, looked at gestures as a frame around spoken communications. In 1644 John Bulwer‘s Natural History of the Hand, and Gilbert Austin‘s Chironomia (1806) examined the details of hand gestures in speech. Even Charles Darwin dug into it. Of course Freud and Jung looked into it, as did Julius Fast, an American author who first garnered some major interest in the topic in the early seventies with his book Body Language, and more recently, there’s the work of Paul Ekman, who identified some three thousand cross-cultural facial expressions. Other people who have looked into body language include: Louis Gottschalk, Erik Erikson, Charles Osgood, Otto Rank, Albert Bandura, Gordon Allport, George Kelly, Snygg and Combs, Maslow, Rogers, Jean Piaget, and others.

But it seems overlooked in most robotics design circles and certainly in most labs that are working on NLP. Some of the most talented natural language engineers are some of the least talented communicators. They understand the language of code better than the code of language. There is a terminal amputation that cuts the text of language from the language of the body. That’s because important elements of natural language, like body language, are easily overlooked if we focus too much on the code and not enough on the people.

Because ultimately, that’s what we roboticists are designing: a kind of people.