Source: Viv Labs

Dag Kittlaus, founder and CEO of Siri, Inc., produced and uploaded an app into the Apple App store February 2010. The app quickly rose to 1st in the ‘Lifestyle Section.’ In March, Steve Jobs called Kittlaus to start a dialogue which led to Apple acquiring Siri shortly thereafter in April.

Siri cost Apple around $200 million. In addition to the acquisition of the technology, the purchase was also a smart competitive move because Siri was working on adapting the app to work on Nokia and Android phones. At the time they sold the company to Apple, Siri had 24 employees. As the team moved into the Apple apparatus, and for the next 18 months, Kittlaus headed the team at Apple that applied Siri technology to the iPhone 4S which was launched in October 2011, the month Steve Jobs died.

The Siri app was free for iPhone, iPad, and iPod Touch users. The algorithms that made the app work, however, were the product of years of defense-sponsored research at SRI International. SRI spun out Siri, Inc. in 2008 to commercialize their work. This included the three Siri founders: Kittlaus, Tom Gruber, CTO, and Adam Cheyer, VP Engineering. They named the company “Siri,” a Norwegian girls’ name. Silicon Valley venture capital firms Menlo Ventures and Morgenthaler Ventures poured $24 million into the technology that Kittlaus, Gruber and Cheyer built into the Siri app.

After Jobs died, Kittlaus resigned and went back to Chicago to be near his family. Liv, his mother, was a native of Norway, and met Dag’s father, Karl, when he was serving with the U.S. Air Force in France. The couple settled in Palos Heights, Ill., a Chicago suburb, and had two sons. Whenever possible, Liv took them to visit her parents and sister in Norway. Doug went to Norway and completed his MBA at BI Norwegian Business School, in Oslo and stayed in Norway for seven more years. He began his career in technology at the Scandinavian telecom giant, Telenor Mobile. Returning to the United States, he worked at Motorola for five years and then went to work with SRI and the Siri project.

Source: YouTube/TechCrunch

In 2012 Kittlaus and a small group – including Cheyer and Brigham – created a new startup, Viv Labs and their AI Viv, named after Kittlaus’ mother but also the Latin root meaning live. In May 2016, Kittlaus demonstrated Viv at a TechCrunch Disrupt event in New York. The demonstration showed that Viv was a breakthrough accomplishment in providing a single conversational user interface (UI) to recognize our words, determine our intent, and then write code to effect that intent. At the demonstration, Kittlaus said that search wasn’t going anywhere but as Viv and other new AI assistants become more capable and become the primary resource for users, people really won’t want to go back to the old way.

Now enter Samsung, 5G, and the Internet of Things. On October 5, 2016, Samsung announced that it had acquired Viv Labs for an undisclosed amount. As part of the acquisition, the Viv founding team will work closely with Samsung’s Mobile Communications business but continue to operate independently.

“Unlike other existing AI-based services, Viv has a sophisticated natural language understanding, machine learning capabilities and strategic partnerships that will enrich a broader service ecosystem,” said Injong Rhee, CTO of the Mobile Communications business at Samsung Electronics. “Viv was built with both consumers and developers in mind. This dual focus is also what attracted us to Viv as an ideal candidate to integrate with Samsung home appliances, wearables and more, as the paradigm of how we interact with technology shifts to intelligent interfaces and voice control.”

In addition to their phone business, as we near 2020, when the new 5G telecom systems begin to be deployed, starting first in Korea in 2018 and Asia in 2019 and 2020, appliance makers such as Korean Samsung foresee the interaction of their devices with the Internet in all sorts of ways. Machine-to-machine technology, including robots, self-driving cars and other cognitive computing systems that are able to make decisions and solve problems without human intervention, is here now but growing in capabilities daily, and will soon be engrained in our daily lives as the capabilities of the 5G networks kick in. Human-machine-interaction, such as that offered by Viv, will be facilitated by 5G.

[NOTE: Samsung isn’t the only buyer of this type of UI technology. Last month Google acquired API.ai, a 2011 startup providing a virtual digital assistant. In December 2014, the company pivoted to focus on becoming a developer platform for natural language processing (NLP). Their platform enables developers to create their own voice apps across a range of end markets, including the mobile device, wearables, robotics, automotive, and smart home markets.]

{NOTE 2: In a move similar to Samsung’s acquisition of Viv to enable integrating it with Samsung home appliances, Amazon just partnered with KITT.ai, another developer of conversation-driven interfaces for developers, to make it easier for developers to use voice-controlled products using the Alexa Voice Service.].

What is 5G?

5G is the fifth generation of wireless technology and offers big advances in the number of simultaneous connections that mobile networks can support – up to a million connections per 1/3 mile. Apart from providing massive capacity, 5G users will see significantly improved network speeds and low latency, well beyond the capabilities of 4G. Data speeds on 5G are estimated to reach 20 times faster than current 4G networks. Estimates suggest that 5G users would be able to download a full-length, high-definition movie in just one second. In addition, 5G is expected to come with an ultra-low latency rate of less than 1 millisecond. Existing 4G networks have a latency of 10 milliseconds. Latency is defined as the amount of time it takes for a packet of data to get from one point to another. The ultra low latency also means that advanced applications, such as autonomous driving, can be supported by 5G networks with precision.

Bottom line

Personal assistants are all about simplifying the process of performing daily tasks. They are what you would expect from a knowledgeable executive assistant to maximize productivity and minimize your involvement, a person who really knows you and doesn’t require you to explain things again over and over. The list of vendors is impressive: Amazon, Google, Facebook, IBM, Baidu, Microsoft, Apple, SoftBank, all the car companies, smaller companies like Jibo, and many large consumer product providers – including Samsung.

To date, none have achieved that sweet spot where end users are fully satisfied. The holdup has been the software, the need for extensive coding, and the ability to identify “intent” from sounds recognized as words. The breakthrough technology that Viv is offering is a way to change the way programmers work with computers and developers offer their products. Programmers will no longer have to instruct step-by-step procedures in rigid code, but instead they’ll just have to describe the process and intent and the AI will develop the code – it will write the programs needed on the fly. And developers will just need to add Viv as their conversational UI.

Source: Viv Labs

Artificial intelligence that programs itself on the fly – from the team that developed Siri at SRI and Apple – was demonstrated this week in New York and it quickly performed complex spoken tasks, live and onstage. 

At the recent TechCrunch Disrupt event, held in New York City May 9-11, Dag Kittlaus unveiled Viv, the new single-source cross-platform voice assistant for the thousands of apps available to us. Viv describes itself as an AI platform that enables developers to distribute their products through an intelligent, conversational interface. Kittlaus expects to offer Viv SDKs to developers in the fall and hopes to fully launch by the end of 2016.

In the video below of his 14-minute presentation and demo, with another 10 minutes of Q&A afterwards, one can easily imagine how Viv will soon become integral to the way we use and operate our apps. Viv provides a paradigm shift in the way we affect our daily tasks — the very shift that others are attempting to provide in the form of personal assistants. It is a breakthrough accomplishment in providing a single conversational user interface (UI) to recognize our words, determine our intent, and then write code to affect that intent. As a side benefit, the video is very informative about the process of AI. [Forward to 4:00 and watch for 10 minutes just to see the demos and skip the intro and Q&A.]

Jibo, Pepper, Alexa, Duer, Cortana, Google, Watson and many others (startups are popping up every few days) are all in various stages of presenting and bringing to market their versions of personal assistants – some physical, others virtual.

But the Viv demo – done live – was speedier than any and Kittlaus explained why: because as more and more apps become available, and as more and more users want to use some of those apps, a personal assistant must scale to handle all those functions and the larger the scale, the more complex it becomes, the more lines of code that must be written, and it ultimately will slow down. Conversely Viv has a dynamic program generator that writes solution-based code from the intent of the user. In other words, Viv’s AI parses through the words you say to determine intent and then writes only the software that meets the specific task requested and does it quickly. The video also shows, but doesn’t detail, how Viv determines intent (goals) from the words. It breaks down the conversation into goals and values within thereby instructing the dynamic program generator which apps are required and what the parameters are for those apps.

Is Viv going to eliminate Google as the middleman? Kittlaus says that search isn’t going anywhere but as Viv and other new AI assistants become more capable and become the primary resource for users, people really won’t want to go back to the old way.

Siri, co-founded by SRI’s Dag Kittlaus and Adam Cheyer and by Tom Gruber, was spun-out from the SRI International Artificial Intelligence Center and acquired by Apple for $200 million in 2010. Kittlaus left Apple in 2011; Cheyer left in 2012.

BOTTOM LINE

Personal assistants are all about simplifying the process of performing daily tasks. They are attempting to be what you would expect of a knowledgeable executive assistant working alongside to maximize your productivity and minimize your involvement, a person who really knows you and doesn’t need you to reexplain things over and over. The list of vendors is impressive: Amazon, Google, Facebook, IBM, Baidu, Microsoft, Apple, SoftBank, all the car companies, many large consumer product providers and smaller companies like Jibo. To date, none have achieved that sweet spot where end users are fully satisfied.

The holdup has been the software, the need for extensive coding, and the ability to identify “intent” from sounds recognized as words. The breakthrough technology that Viv is offering is a way to change the way programmers work with computers and developers offer their products. Programmers will no longer have to instruct step-by-step procedures in rigid code, but instead they’ll just have to describe the process and intent and the AI will develop the code – it will write the programs needed on the fly.  And developers will just need to add Viv as their conversational UI.

[Viv uses Nuance for voice recognition. Nuance is a member of the ROBO Global Robotics and Automation Index (of which I am a Co-founder)].

Source: YouTube/TechCrunch

Viv is an AI personal assistant launched at TechCrunch Disrupt NY. It promises to radically simplify our interface with everything. Viv is everything that Siri cofounders — Dag Kittlaus and Adam Cheyer — wanted to build with Siri, but couldn’t. Viv is a few years more advanced than Siri, but the big difference is that Steve Jobs preferred to keep Siri in a walled garden and not allow open partnerships.

Viv is a global platform that enables developers to plug into and create an intelligent, conversational interface to anything.

Cheyer and Kittlaus are focusing on creating a developer ecosystem so that Viv will be the way every device interacts with you in the future. They believe no one company can bring everything you need. But, one personal assistant can interact with all the companies and services that you need.

Viv does ‘conversational commerce’ with breathtaking speed and accuracy. I think you can forget about deep linking as the next mobile/online paradigm. Kittlaus described Viv as “the next major marketplace and channel for offering content, commerce and services.”

Viv will launch a gradually expanding developer program later in 2016. And as thousands and thousands of developers add to the Viv intelligence, the power of Viv expands. While the demo focused on showing Viv’s strengths on a phone, Viv is clearly intended to be the interface for all of our smart connected devices. And robots.

In fact, I’ll bet that my future home robot will be running on Viv.

Is there a downside?

What are could possibly go wrong in a future world full of personal assistants?

Well, while Dag Kittlaus was launching Viv at TechCrunch Disrupt NY, cofounder Adam Cheyer was in San Jose on the CHI2016 panel: “On the Future of Personal Assistants” with Phil Cohen,VoiceBox Technologies; Eric Horvitz, Microsoft Research; Rana El Kaliouby, Affectiva; and Steve Whittaker, University of California at Santa Cruz.

Firstly, the panel noted that the term ‘personal assistant’ was deceptive as we are transitioning from helpful but passive assistants, such as Siri, to active agents, such as Echo, Viv and perhaps Jibo, or other home robots in the future.

Assistants are just helpful. Agents go out into the world and perform actions.” said Phil Cohen, VoiceBox.

There are also huge potential implications in the transition from personal assistants to family assistants. Amazon’s Echo is an example of a family assistant, where the device is no longer in our hands or pockets, but sits in a room and interacts with a range of people.

How do we deal with multiple devices in various areas of our lives, ie. from car to kitchen to work to bed, and then, how do devices interact across our individual family members? Will we be creating functional or dysfunctional family agents? Navigating these issues of privacy, trust and boundaries should be at the forefront of our development.

Eric Horvitz, Microsoft, described the ‘penumbra – or bubble of data and devices’ that travels with each individual. We expect these agents to collaborate with us, but who else do our agents collaborate with in order to provide us services?

The next step is to consider how our agents can in the future interpret our emotional states. What inferences can be made, and shared, with whom? What control can be effected by these agents.

Rana El Kaliouby, Affectiva said “your fridge will understand your emotional state and help you make better decisions, if you want to change your behavior. That’s a very powerful promise.”

Cheyer was clear he doesn’t want “a plethora of different assistants… I want one assistant who can learn my preferences over time and do a better job.”

Although the intelligence might be in the cloud, the form factor of the device will give very different feel to how we interact, whether it’s a screen, TV, or a robot. And as we start to embody our assistants we change the relationship significantly.

Horvitz told an anecdote about going on holidays and turning back to bring Alexa along. Kind of like a member of the family.

Steve Whittaker, UCSC said that “we can use the metaphor of a house as an approach to designing different categories of personal assistants. A house is a collection of resources for family activities. At some level they are shared facilities for a family, but it is segmented.”

“Cortana developers are already tackling ideas of primary agency,” said Horvitz. “When I walk into a room, can I use someone else’s agent? And then is it my agent or some one else’s? And what level of shared agent? How do we develop the trust?”

“Both technically and ethically, how much access to the underlying personal information can someone else’s agent have?” added El Kaliouby.

Phil Cohen, VoiceBox questioned whether agents needed to track our beliefs, desires and intentions, or underlying emotions.

“What value or service is augmented intelligence providing in people’s lives? What tasks should a system be designed to address and for whom? Especially when a device is ambient (taking action on its own) not simply being addressed.”

 

Cheyer said that a useful assistant has to be transparent and allow you to change your mind about what is being shared. Whittaker said that while that might be the case for personal agents it would be very complex for family agents.

Horvitz pointed out that there are many elegant designers in the CHI community who can meet the challenge, as long as we focus on high value services that are tolerant of some uncertainty or error, so that there is a low cost of failure.

El Kaliouby added that we faced a moral obligation to give permission to some algorithms to take action. For example, some agents are showing ability to spot depression and suicidal states from things like social media posts. But what permission to they have to take action and what actions?

Whittaker pointed out that while spotting suicidal behaviors is a clear social good, generally people aren’t that good at identifying and tracking emotional states, so at what point does mood tracking create useful and actionable information?

Horvitz would like to see us address time management for people, seeing it as a huge problem that we’ve barely begun to scratch the surface of.

Cheyer sees time management or life coaching as just one of the domains that a system like Viv will start to have impact on. Although we’ll need better understanding of how we interact with AI and devices to be able to get to the next level: “Everyone is talking about the strengths of AI, but it’s not really going to solve all our problems. Where Siri was pretty dumb, Viv has light weight planning ability. Viv is able to do complex tasks with several steps which is a lot closer to the way we really ask for things. It’s not going to be doing the life planning things that Eric was asking for. But we’re working on taking actionable insights out of the unstructured data of our lives. And in the end that’s a very exciting development across all domains.”

In a room full of interactionists and ethicists, it’s no surprise that the first question to the panel was: “Is it good for society to be creating a slave class?”

This opened the door on a discussion that ranged from the ethics of ‘nagware’ up to lethal force. At which point the challenge was made for designers shift away from negativity and to create delightful solutions that would enhance and improve our lives. People-centered rather than task-centered design.

Cheyer concluded by bringing the discussion back to Viv. “Every 10 years if you look back we’ve had a new paradigm for interacting with our devices, from desktops, to web, to mobile. And each one had its problems. But I think the time is right for the assistant/agent paradigm and it’s going to be far more impactful than everyone realizes.”

Horvitz added, “The only thing I can promise for sure in the next 5 to 10 years is that we will have impacts and effects that no one has predicted.”

*note: quotes are not verbatim and have been edited for clarity.

Source: Times of Isreal

A consortium of Chinese companies, municipalities and investors — including Siasun, the largest robotics firm in China and the city of Guangzhou — have inked a deal to set up a $20 million R&D center in Israel dedicated to developing robot technology that China can use to modernize and automate its industrial base.

The investment follows the September establishment of a Sino-Israeli Robotics Institute (SIRI) as the centerpiece of a new $2 billion industrial park in Guangzhou.

The new Israeli R&D center will develop technology and robots worked on by Chinese and Israeli researchers in Israel. Products developed will be mass produced in China. In addition to industrial robots, the institute will develop robot “servants” – waiters, cleaners, security guards and customer service agents, and robotic assistants for construction sites, where finding manpower to do heavy work is becoming difficult.

Imagine an assistant that knows your schedule, contacts, interests and location, and that can communicate with you to help you stay informed, connected and on top of your game. This is more personalized than your PC, tablet or smartphone and, depending on whether physical or virtual, is in a pleasing, personal and compatible form. If these assistants catch on, they will begin anticipating and handling many of the little decisions that fill our days. The potential for this form of artificial intelligence has tech companies vying to lead the pack.

Apple’s Siri, Google’s Google Now, Microsoft’s Cortana, Facebook, and Baidu are all in various states of pursuing virtual assistants. But JIBO, Pepper, Cubic, EmoSpark and others are just now coming to market and are offering a physical component – a robotic presence – which they hope will enhance the experience.

2015 is the year physical products will be coming to market and available for experimentation and testing. Pepper ships in the summer in Japan, JIBO ships preorders in Q3 as does Cubic in the fall and EmoSpark in the summer.

These types of personal assistants rely on multiple activities including: speech recognition to determine and parse through spoken commands; predictive technology to anticipate what a user will likely do next; and determining and using contextual data (biographic data and situational information, e.g.: location, time and date). Apple, Google and Baidu – all of which attempt to provide this service – want their intelligent interface to fit all front ends, to be the universal contact point. They see a plentiful revenue stream if they are successful. Others want Apple and Google to redirect users to company sites with apps that understand their own content better, e.g.: directing requests for pizza to Dom, the digital front-end for taking Dominos Pizza orders.

Many are waiting to see how the competition plays out. Car companies and other smart product providers, fearing a variety of consequences outside of their control, aren’t enamored of letting Apple or Google (or any of the others) run their voice-powered front-end systems … thus their current intense race for in-house and startup development. But if Google or Apple or ??? gets to market first with a true digital assistant that people embrace and buy – one that can be virtual or physical – car companies may need to make their systems adapt to that system because it’s what everyone wants to use.

The key to the outcome of this race is whether a general purpose AI will be able to steer people through their digital world, or whether users would rather navigate to applications that are specialists (such as American Airlines or Dominos Pizza).

Natural language recognition – and the ability to analyze user desires from that language – is a very competitive science these days. Nuance, the biggest provider of voice and language solutions to the auto industry and a collaborator with IBM up until recently, is now competing with IBM’s Watson team and also Apple, which is rumored to be working on their own language processing system instead of using Nuance. A look at the What’s Next section of Nuance’s website is, to quote them, “The intersection of Science Fiction, super-pi, and technology innovation.” But a more descriptive quote from that page seems more appropriate: “Intelligent self-service delivers a better customer experience.” That’s what this race is all about.

• The second wave of social robots is here
• Amazon’s Echo: A point-of-sale even the kids can relate to
• The next era of conversational technology
• The death of search (or, My dysfunctional relationship with Siri)

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Voice recognition is a timely science not only because of all the human-automobile communication offered by car companies, but also by smart devices and their growing use of voice interfaces. It won’t be long until we operate many of our appliances by voice command.

Most present-day offerings require you to follow a menu, a convoluted non-intuitive form of communication. Few have the technology to enable consumers to ask for information quickly, easily, and without needing time-consuming complicated dialogues – but many companies are working on solving these problems.

Apple and Siri are at the forefront of handling free form conversational speech – and Apple’s acquisition of Novauris, a UK speech recognition company – will help in that pursuit. Terms and amount paid for the transaction – which occurred late last year – are unknown, but Novauris’ co-founder Melvyn Hunt confirmed the sale to a TechCrunch analyst.

Talent in this field is scarce and many companies have chosen to acquire that talent – and their software, patents and intellectual property (IP) libraries – for their various voice recognition quests.

These are a few of the companies developing and offering all sorts of consumer conversation recognition systems and context understanding systems from which instructions and responses are developed:

• Nuance (which recently acquired Vlingo)
• Evi by TrueKnowledge (which was acquired by Amazon)
• Alfred (which was acquired and then shut down by Google)
• Cue (which was acquired and then shut down by Apple)
• Novauris Technologies (acquired by Apple)
• Siri (which uses Apple and Nuance technology)
• Okay Google by Google

Notice the pattern? Apple, Google, Amazon and Nuance appear to be hot on the trail of being able to capture conversational instructions, convert the words into context, determine the instructions required, and then process those instructions. Speed is helped by experience, and Apple, Google, Amazon and Nuance are gathering that experience and processing hundreds of millions of spoken words every day.

It really won’t be long until you and your refrigerator are having a meaningful conversation.

The incoming second wave of contextual agents

There’s a virtual lobby of Intelligent Virtual Assistants (IVAs) waiting to help us these days. These multi-million dollar systems include Yahoo’s Donna, Samsung’s SAMI, Google’s Now, Nuance’s Nina, Motorola’s Assist, Microsoft’s Cortana and of course Apple’s Siri. They can give you driving directions, book a dinner table, launch an app, tell a joke, take a memo, send a text, post a tweet, ring a phone, update Facebook, check stocks, search the web, turn off the lights when you go to bed, and set an alarm to wake you up in the morning. They can do incredible things, but they’re not very valuable for one weird and very general reason.

All of these companies, and many others, understand that voice interface will be built into our handhelds, our wearables, and the Internet of Things. There seems to be a growing stampede of investors galloping towards this inevitable future, and those investors are right to gallop so. But something is amiss with the design of these systems. I don’t know about you, but when I use these assistants I’m sometimes stunned at how off they can be. They make the word “smart” seem dumb. They can be so off at times that they can even make dumb seem smart.

There’s some great work being done, too. Surprising work. Siri has this plug into Wolfram Alpha and Wikipedia, so I can ask it very specific questions like, “How tall is the Eiffel Tower?” and I get a response back as, “The Eiffel Tower is 1063 feet tall” in under a second. Wow. Wikipedia now talks, kinda. That’s great.

But is it useful? How often do I need to use that? Other than the ability to use a phone without my hands, and ask it questions, I don’t see the problem this is solving. Today’s Intelligent Virtual Assistants seem a lobby of lovely-voiced zombies that are trying to be everything for everybody at all times. And it reminds me of a strange trend we’ve already seen.

Back in the 1990s, chatbots were also trying to be everything to everyone. Systems like ALICE (built on AIML) were designed to answer all questions that could be thrown at them.  Authors wrote rule after rule so that the system could answer anything, no matter what you asked it, effectively trying to pass the Turing Test with a linguistic equivalent of cryptographic brute-force attacks, as if the designers were thinking, “If the system can talk with anyone at any time about anything, then it will seem human.” As a result the systems were designed to passively sit there, and as soon as you asked it a question about cats, particle physics, or the Eiffel Tower it was supposed to barf up the “right” answer. This, we thought, made it seem more human.

Brute-forcing the Turing test with every possible answer (Thank you, but no, Dr. Wallace) is a function that is rarely valuable because it rarely has context.

Take Siri as an example. Siri is supposed to be able to serve all our various needs in a huge range of circumstances from driving to note taking to going to the movies. She’s got it pretty rough these days. Folks get down on her for misunderstanding, misguiding, mis-typing or just missing the point of what was said. After all, making a conversational system that can talk about anything with anyone at any time is impossible. Not even systems with human-level intelligence (like humans) can do that.

So it’s rather unnerving that everyone who is building these IVAs is making this same mistake.

Here’s why: Context is information’s value. If you take any piece of writing near you and give it to someone on the other side of the planet, the value of that information changes. Look at any writing near you. Look at any road sign, listen to anything on the radio, watch anything on a screen … and if that information is moved to some other place or time, its value usually evaporates. This includes what you say to your lover, banker, parent, or child. It includes what you write, what you say, and what you hear. It includes what you say in confession. It includes what you hear from your doctor or physician. It includes the height of the Eiffel tower.

So these IVAs, by trying to be everything to everyone, are missing the point: they need to focus and provide specialized information to people who need it. The more general the information, the less valuable it is.

IBM appears to be going through some turbulence these days, but they also seem to be taking quite a different approach to the pile of virtual assistants mentioned above. IBM’s Watson is focused primarily on healthcare, and this specific context helps to produce high value information. By trying to do crazy-difficult things like cure cancer (and by now even out-performing the diagnostics that doctors are capable of) IBM is building a database of specialized knowledge. Cognition-as-a-Service (CaaS?) might be a Web 3.0 idea that changes apps and grows an ecosystem of users and groups of experts who build increasingly powerful sets of contexts.

Imagine a “doctor-on-your-shoulder.”  An app that can tell you if your soup will trigger your gluten allergy, or how carbohydrates can affect your diabetes, or how cipro can create comoribidity issues. I like to think we roboticists can build things like a Virtual Nurse, a Virtual CPA, or other conversational systems that offer very contextualized information for a very specific group of people who need a very specific type of help. Systems that inform, educate, and address a knowledgebase that is of high value to a smaller set of individuals.

At SXSW this week Geppetto Avatars is showing off a proof-of-concept project we did with Health Nuts Media: we built a two-avatar conversational system that helps kids who are recovering from asthma. After kids have been released from the hospital following asthma treatment, they can talk with these two cartoon characters, Jiggs and Big, and ask them about their asthma. It’s an app for asthma management.  It helps reduce re-admittance, it saves hospitals time, and it can save kids’ lives.

This is an example of the kind of IVAs that will power a new Internet. It is an IVA that helps people live better, healthier, happier lives. Sure, we can serve ads with this, and sure, the Eiffel Tower height is important, but I hope we can use raw data like that to aim for higher ideals.

I got a new iPhone about a year ago, the one with Siri on it.  Our relationship started on a very mundane level. Siri asked me my name and, since I already knew hers, I asked what she could do, and how. It was a bit like talking with a call girl. The basics were covered and soon we got to the intimate stuff. I don’t think Siri loved me. But now that Siri and I have gone our separate ways, I can say we both had ulterior motives. We sort of used one another, now that I look back on it. It was a relationship of an intimate nature. It’s what happens when you have a relationship with a robotic call girl.

The reason I got the phone to begin with was so I could do a little reverse engineering. It was research for work. Siri and I took a few days to get to know one another, but after that honey-moon period it was clear that Siri (or Apple) was using a pre-scripted Natural Language Processing (NLP) approach that basically did some lexical parsing, looked up a match, prepared a response, and kicked it back down the pipeline.  My reverse engineering wasn’t too profound. I wanted to see how many recursive answers were built in to things like “Open the pod bay doors” or “Do you love me?”  I also wanted to see how errors were handled, and what kind of redundancy checking was happening for sentences like “Where can I buy a burger and fried?” Siri did pretty well, all things considered, but my expectations were low, and like many other intimate relationships, I somehow knew from the start it wouldn’t last.

Now, while I was using Siri, she was also using me. Relationships are generally symmetric, as Gregory Bateson tells us [1], which meant that, like any dating service, someone other than Siri was getting money for my time. But I didn’t care if Steve Jobs was Siri’s pimp. By last summer I was having flirtatious fun and the consequences of the relationship were too far down the road to look dangerous. I asked Siri to take dictation, to wake me up in the mornings, to remind me of important events, to remember phone numbers, remember names, tell me jokes, confirm thoughts, quote a number or two. All intimate things and, as I look back on it, the kind of stuff that I’ve trusted my wife with.

My relationship with Siri started to cool off a bit by September. Though my wife knew about Siri, I hadn’t yet introduced them.  So when I asked Siri to call my wife I, expected Siri to ask me for her name or number. Instead Siri just dialled. I was surprised. How did Siri know my wife’s number? (Answer: It was in my Contacts card.) What else did Siri know? (Answer: A lot.) And more importantly, what was Siri passing back to the Apple hive? (Answer: Everything.)

Siri and Apple now have a great deal of data about my household: a quick comparison of my contacts/likes database with my wife’s will give you a pretty good feel for the stuff we will buy.  Burgers and fries, of course. And a few multi-player games.  Some lingerie. A sex toy or two.  Not a big deal, but not the kind of stuff I want everyone to know about.  Not even my mom gets access to that data.

But Siri did. And Siri knew a lot more about me than I knew about her.

Then, around the middle of October, three things happened to Siri that marked the beginning of our eventual estrangement.

First was William Stasior. Apple picked Stasior up to run its Siri unit after he had been successfully heading Amazon’s search and advertising unit “A9.”  Prior to that the MIT PhD had served time at Oracle, NetCentives, and AltaVista.  To say the guy knows search is like saying the pope knows the church, but why would Apple pick a search guru for an AI system like Siri?  What was the link between search and NLP?  I asked Siri this very question, but she was mum and feigned ignorance.  I didn’t push her on who this guy Stasior was and Siri didn’t ask me more about my wife.

Up until that same week iPhone users had frolicked in what would now be a rather unusual environment in which advertisers were unable to track them.  Advertisers would not, for example, know that my wife and I like video games, sex toys, and pizza (no, not together, thank you).  But with the release of iOS6, Apple flipped ad tracking on and I quickly flipped it back off.

It was easy to find in the interface menus (under Settings > General > then About or Advertising, depending on the version you’ve got), but you had to know to look.  Apple minimized the fanfare around this new feature and Siri made no mention of it to me.  I wasn’t comfortable with Siri selling to others what I had said to her in more comfortable times.

And then, as if things weren’t rough enough, the European Union (where I happened to be lecturing that week), demanded that both Facebook and Google change how they handle personal information to avoid “high risks to the privacy of its users.”  Twenty-four of the European Union’s twenty-seven regulators signed a letter that, after a nine-month investigation into the data collection practices, demanded some answers and tightened regulation.  After Google and Facebook, Apple was third on their hit list.

That was the end. I couldn’t trust Siri after that.  She knew more about me (and my wife) than I knew about her or Steve Jobs, and any time information flows one way, any time a relationship isn’t symmetric, the balance of power can be dangerous.

Sometime around then I learned that Oracle’s “bring your own device to work” policy had been revoked.  Big Blue decided that Siri knew a little too much about their employees, and at IBM’s Armonk New York research center, iPhones weren’t even allowed in the building.  Like some cheap prostitute-gone-spy, Siri was barred from the building.  I think it is worth noting that this is the same area of research that brings you Watson, which is a direct competitor with Siri, if such a competition exists.  If anyone outside of Apple understands Siri it’s these guys.  It is also worth noting that a friend of mine who works at a Google research center said that the same thing happened there, too.

Not even celebrity robots like Asimo or NAO get that kind of VIP treatment.

NLP technology is potent juju.  And Apple, Google, Facebook and others know it because this very technology is what’s allowed them to earn money. Google started with search, of course, and later grew profitable as they introduced more NLP technology into their work.  They sold the info they collected from users (Google is an ad agency, let’s remember).

Search allowed them (when coupled with NLP technologies, and semantic analysis in particular) to make oodles of cash and then to snap up many of the best AI and NLP researchers on the planet. Google set those kids to digging in what would become a lexical gold mine.  The more they mined the richer they got until they were mining many branching veins at once: Google Docs, Voice, Translate, Search, Shopping, Reader, Finance, Books, Photos, Wallet and Maps were all spewing more money than Brin, Schmidt and company knew what to do with.

The average email count for Google users is 5,768 emails [2]. The average composition time (also according to Google) is 01:43.  That’s about 9,902 minutes, or 165 hours you’ve dumped onto Google’s server, which, if you were to work for 165 hours at $12/hour for that data entry (a pretty normal rate), means your G-Mail account is worth about $1980.  But the information sitting on Google’s servers is worth a lot more than that, right?

If your account contains information about a first class international plane ticket, a hotel room in Paris and a business associate’s London phone number, your data might be worth a lot more than the composition time at $12 an hour, especially if the buyer of it is a dating service that caters to high-end out-of-towners. In other words, the value of the data isn’t counted by data entry. It’s counted by its personal meaning, especially to advertisers. Now take those values and multiply them by all those crazy tools of voice, maps, docs, and so on and we start to get a sense of why Google has been making such bank over the years.

These tools have each dovetailed into today’s Google Now strategy. Google Now (like Watson and Siri) is a voice-activated NLP system.  It uses the spoken input data collected from years of Voice, confirms it with data collected from years of Docs and Translate, builds meaning with data from all the years of all the other tools, looks up a match, preps a response, and kicks it back down the pipeline.

All of these NLP systems are the next evolution of search.  But not evolution as in the breeding-hybrid-peas-in-the-greenhouse kind of evolution. They’re evolution as in endangered-species evolution.

NLP systems represent the end of search as we know it, and therefore the end of many economic, interface, and social internet ecosystems. They also represent the beginning of something incredibly powerful, intimate, and new. Siri, like other NLP systems, is far more powerful than a search engine. Siri not only includes search (it says so in Apple’s marketing materials) but it supersedes search because it includes browsing, discovering, choosing, and refining.  All this while analyzing semantic data. Like Google’s portfolio of tools, Siri can handle (and with a public API will handle) translating, searching, shopping, reading, finances, books, photos, and maps.

But the core value of the data is its intimacy.

On February 7, 2010, during Super Bowl XLIV, Google’s super bowl ad showed a user typing a list of search strings:

study abroad paris france
cafes near the louvre
translate tu es très mignon
impress a french girl
chocolate shops paris
what are truffles
who is truffaut
long-distance relationship advice
jobs in paris
churches in paris
how to assemble a crib

With this list, Google managed to explain, in fifty-two seconds, how the intimate story of someone’s life can be assembled from his or her search queries. And what makes the bucks for Google is what makes the bucks for Facebook: the processing of intimate language.

Siri, like any other call girl, makes her bucks the same way (especially when ad tracking is flipped on).  She’s valuable because she’s intimate.  And as designers of robotics systems, we can ourselves take a cue and be conscientious about how we are designing conversational systems.  There are three different design types that form a curve of increased intimacy in robotics and conversational systems, and they are all based on the value of semantic data.

1. NLP systems, when used with physical robots for manufacturing or the three Ds (dull, dirty, and dangerous), are the least intimate.  The conversation system is a simple tool that, like a GUI, provides access to system operation.  The robot doesn’t care about the user’s intimate data. The user wins.
2. NLP systems, when used for entertainment and education, are contextually intimate – some privacy is maintained.  The conversation system doesn’t care about the user, but the user’s data is valuable in terms of what it says about the game or lesson.  Semantic data that might be collected and analysed isn’t about the user, it’s about what the user is doing in that particular context.  The robot, usually a software robot, only cares about the user in the context of the game or training exercise. Both user and robot win.
3. NLP systems, when used for personal assistants, are the most intimate.  These are the robotic call girls. Here the information is highly intimate, the system cares a great deal about the user, and the information isn’t shared – it’s collected and sold. The user becomes the product. The value of the user’s personal data is worth more than the value of the conversational system. The robot wins.

As we build NLP interfaces for robots, whether it is for Siri or for assembly-line manufacturing, we must consider how intimate data can be, the value of that intimacy, who owns that value, and what they’ll do with it.  Otherwise, with a new, less benevolent CEO at Google, a change of a line in Facebook’s Terms of Service, a successful hack, or a change of law because of cyber terrorism fears, your intimate data could end up where you don’t want it.  Heaven forbid we discover that all the Senators on Capitol Hill are using pizza sex toys. That would surely disgust Siri and her friends so much that it might even cause a robot uprising.

Yes, perhaps we’ll get together again in a few years, but for now it is best if Siri and I go our separate ways.

Next Month:  “AFK” Keyboards and screens are slow, clunky, and obsolete.  Voice processing systems for robotics provide not only simplicity and speed, but also a host of other benefits if tied to analytics and framed within a tightly contextualized task-based system.  But be careful: getting text out of voice, and meaning out of text can be tricky.  Here’s how to implement one for your own robot.

Endnotes:

[1] See Bateson’s books “Mind and Nature,” Hampton Press (1979) and “Steps to an Ecology of Mind,” University Of Chicago Press (1972) which are chalk-full of ideas like “complementary,” “reciprocal,” “symmetrical,” and  “Schismogenesis.”

[2] According to Google.