What legal rights should a robot have? In this video, Polo Sant’Anna Valdera, one of the leading research centres for robotics, raises this important question. Innovation raises new legal and ethical issues. The European Parliament wants to address these issues so robotics continues to flourish in 2017.

You may also be interested in:

• RoboLaw: Why and how to regulate robotics
• Law-abiding robots? What should the legal status of robots be?
• Hot topic: Regulating robotics
• Robots Podcast #128: Ethical, social and legal issues, with Pericle Salvini
• EU needs to take the lead on regulating robots and artificial intelligence, MEPs suggest

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this interview, we invited Prof. Rafael Capurro to share his insights on past, current, and future trends in robot ethics.

Interviewee: Prof. Rafael Capurro, Founder, International Center for Information Ethics (ICIE) and Member of IEEE Global Initiative for Ethical Considerations in Artificial Intelligence and Autonomous Systems

Interviewer: Dr. Yueh-Hsuan Weng, Co-founder of ROBOLAW.ASIA and Assistant Professor at Frontier Research Institute for Interdisciplinary Studies, Tohoku University (from March 2017)

As a pioneer of Information Ethics, Prof. Capurro created “The Quest for Roboethics: A Survey” and since 2010 continues to update relevant information about ethical issues in robotics. He’s participated in key European projects, such as ETHICBOTS (2005-2008), ETICA (2009-2012), and CA-RoboCom, which was a candidate project of EU FET Flagship initiatives. He has also collaborated with Asian researchers from Remin University of China and University of Tsukuba, so he possesses a unique point of view about robot ethics from Eastern traditions.

WENG: Maybe we can start with your academic background. Could you please give a brief introduce of yourself and the International Center for Information Ethics (ICIE) to our readers?

CAPURRO: I have a Licentiate (University of El Salvador, Argentina, 1971) and a PhD (Düsseldorf University, Germany, 1978) in Philosophy as well as a post-doctoral degree in Ethics (Stuttgart University, Germany, 1989). I have been working in the field of Documentation/Information Science since the early seventies in Germany. I was a Professor of Information Management and Information Ethics at Stuttgart Media University from 1986 until 2009 when I retired. I was Lecturer in Philosophy/Ethics at Stuttgart University between 1989 until 2004.

In 1999 I founded the International Center for Information Ethics (ICIE). ICIE soon became a community of some 300 colleagues from different countries and fields interested in the ethical issues associated with new technologies. Jared Bielby (Canada) is ICIE Co-Chair. The ICIE (co-)organizes international meetings and has, since 2004, published an open access online journal, the International Review of Information Ethics (IRIE) of which I am the Editor-in-Chief. There is an ICIE book series published by Fink (Munich) which currently has published five volumes (including contributions in German and English). ICIE membership is free of charge. The contact with the members is through a mailing list. We inform our members about meetings and publications that are added to the website.

WENG: In 2010 you created “The Quest for Roboethics: A Survey” and keep updating relevant information in regards to Robot Ethics. As a pioneer of Information Ethics, what made you focus on this emerging subject in techno-ethics?

CAPURRO: This survey was originally a contribution to a workshop held on September 30, 2009 and organized by Cybernics, University of Tsukuba (Japan). It was published in Cybernics Technical Reports: Special Issue on Roboethics (University of Tsukuba 2011, pp. 39-59). At that time, issues dealing with ethics and robotics were relatively new but the field expanded quickly— so much so I decided to update this survey regularly—albeit without seeing it as needing to be an exhaustive overview of the field. It is just a means, for myself and maybe for others, to help to keep pace with a range of ongoing robot ethics discussions.

My interest in robotics and AI goes back to the mid-80s: I participated in a group in charge of developing a Code of Ethics for the German Society for Informatics. I read Joseph Weizenbaum‘s Computer Power and Human Reason (1977), Pamela McCorduck‘s Machines Who Think (1979), David W. Johnson and J.W. Snapper (eds.) Ethical Issues and the Use of Computers (1985), Deborah Johnson’s Computer Ethics (1985), Terry Winograd and Fernando Flores’s Understanding Computers and Cognition (1986), Hans Moravec’s Mind Children (1988), to mention just a few. My first book publication dealing
with these issues was a collection of papers with the title Living in the Information Age (1995).

WENG: In your previous presentation at University of Tsukuba you mentioned that an Eastern tradition of Buddhism to robot ethics is that “robots as one more partner in the global interaction of things”. Could you please explain this?

CAPURRO: I visited Japan for the first time while on sabbatical in 1998 as a guest of the University of Library and Information Science (ULIS), which was integrated later on into the University of Tsukuba. I had conversations on ethical issues associated with the information society with ULIS staff and students as well as on philosophy with Prof. Riuji Endo. I also met Prof. Koichiro Matsuno (Nagaoka University of Technology) with whom I exchanged views on the concept of information. Since then, I have been invited to Japan several times by Prof. Makoto Nakada (Univ. of Tsukuba) with whom I have had the honor, and the pleasure, to learn about robot ethics as they are viewed from within Eastern traditions.

Prof. Nakada and I discussed issues of privacy from an intercultural perspective. The difference associated with how robots are understood in the “Far East” and the “Far West,” to use the terms coined by French sinologist François Jullien, can be traced back to the Cartesian split which conceives the human psyche as separated from the so-called “outside world.” Buddhism and Taoism open different paths of thought, particularly with regard to the Japanese concept of Ba (situation). This dialogue on intercultural robot ethics between Prof. Nakada and myself can be found here.

I would also like to mention two other events that deal with these issues. The first is the international symposium organized by The Uehiro Foundation on Ethics and Education and The Carnegie Council for Ethics in International Affairs (held in December 2010 in Oxford). More details on this can be found in my paper “Beyond Humanisms”. The other event was the International Conference on China’s Information Ethics (organized by Renmin University of China, School of Philosophy (Prof. LI Maosen), and the International Center for Information Ethics (ICIE) held at Renmin University, Beijing, P.R. China, October 28-29, 2010). I presented a paper at the conference entitled: The Dao of the Information Society in China and the Task of Intercultural Information Ethics. The paper was translated into Chinese by Junlan Liang (Chinese Academy of Social Sciences) and published in the journal Social Sciences Abroad ( 2011, Vol. 5, pp. 83-88).

In December 2009, I was invited to participate at the Global Forum on Civilization and Peace organized by The Academy of Korean Studies in Seoul where I was asked to speak about “Digital Ethics.” It was the first time I used this terminology rather than the usual one, namely, Information Ethics. The proceedings were published by The Academy of Korean Studies (ed.): 2009 Civilization and Peace, Korea: The Academy of Korean Studies 2010.

WENG: You and Prof. Nagenborg have pointed out that “Robots are and will remain in the foreseeable future dependent on human ethical scrutiny as well as on the moral and legal responsibility of humans”. In this case, how can we regulate Robo-morality?

CAPURRO: This statement is based on our research within two European projects, namely
– ETHICBOTS (2005-2008), as well as the book edited by me and Michael Nagenborg Ethics and Robotics (Heidelberg, 2009) and ETICA (2009-2012).

To “regulate Robot-morality” means to develop different kinds of standards for robot behavior in different contexts and cultures which is not the same as a sustainable ethical scrutiny on the development and use of robots in society, which is a larger research issue. The idea that robots would some day become philosophers or ethicists — being able to critically reflect on who they are — belongs to the field of science fiction.

WENG: What is your comment to “Ethical Robots”? Do you believe that Asimov’s Three Laws of Robotics will be feasible principles to ensure the safety of physical human-robot interactions?

CAPURRO: As you know, Isaac Asimov (1920-1992) was a science-fiction author. The “laws” were published for the first time in the volume I, Robot (1950). If we look for a legal framework to ensure the safety of humans in human-robot interactions we should pay attention to the issue that humans, as well as other living beings, deserve legal protection in order to ensure their safety. It makes no sense to legally protect robots beyond seeing them as just another device that is the property of some other person or corporation. It makes also no sense to speak about “the safety of physical human-washing machine interactions” aiming for ensuring the safety of washing machines (as if they were a kind of person who deserves protection). This is in line with what Asimov imagined in the title of his book I, Robot. Essentially, we need a broad legal and ethical discussion on the uses and misuses of robots in society.

WENG: Could you please tell us more about the state of arts in Machine Ethics? Are there any progress in terms of programming machines to act with human beings in an ethical way?

CAPURRO: “Machine Ethics” means the ethical reflection on machines done by humans, not machines engaging in such reflection on themselves (and on us). Ethics is done by members of human society reflecting critically on the customs (Greek: ethos, Latin: mores) underlying their being-together. “Machine Ethics” is, from this perspective, i.e., as a critical reflection done by machines, an oxymoron. If we want machines to act “in an ethical way” then we have to provide them with some kind of moral (and legal) rules of behavior. We are responsible for their formulation, fixation and interpretation. It makes no sense to make machines “morally responsible” for their actions.

WENG: “BS 8611: Ethics design and application robots” is a standard highlighting the ethical hazards of robots. Do you think that Ethics by Design might be a crucial factor to future safety standards for service robots?

CAPURRO: Once again, when we talk about the “ethical hazards of robots” the meaning of the “of” is of the kind already explained, i.e. it is genitivus obiectivus. The “Ethics by Design” concerns moral (and legal) rules, not the ethical reflection on such rules nor their interpretation and application. Who is to blame should an accident occur? Machines are per se a-moral which does not mean that they are neutral. Nothing we invent is neutral in the sense that it is already part of human life and it changes, for better or worse, our lives. The idea that machines are neutral because we can use them for different purposes disregards this basic issue. Machines do not behave at all in the way humans deal with each other (with possibilities of respect or disrespect concerning who we are). It is up to us to reflect on the risks we want to take for ourselves and for others when we program a machine to behave in this or that way in a given situation. As it is impossible to foresee all possible situations, every fixation of rules means making a selection of situations. Dilemmas arise when fixed rules do not match unforeseeable situations. Humans can be creative in such cases as they are not bound to fixed rules. This does not mean, of course, that a human decision can end less tragically than a “decision” made by a machine. In sum, we should let robots act in situations in which we can exclude the unforeseeable as far as possible. Beyond that we should take responsibility for the solution of the dilemma the machine is facing; it is our dilemma from the very moment we fix their rules of behavior.

WENG: In your opinion what is the major challenge to develop robot ethics for the following decade?

CAPURRO: This is an international and intercultural challenge since it deals with a critical reflection on societal customs, or mores, and the different kinds of moral and legal rules that will be developed for robots. There might be some global standards of such rules, but if robots are used in specific situations then the social expectations might differ and a translation process is necessary. This can be a very practical process but if we want to question why people expect this or that, in this or that context in different societies, then we have to go deep into underlying moral and ethical traditions that were built over centuries. Moral and legal rules are a kind of “symbolic immune system” for societies according to the German philosopher Peter Sloterdijk. Symbolic immune systems, no less than biological ones, can themselves become a danger if they are not flexible enough to change according to new situations. Robot ethics is similar to immunology in the sense that it deals with monitoring the efficiency and effectiveness of symbolic immune systems when we inoculate moral (and legal) rules into robots.

If you liked this article, you may also be interested in:

• One being for two origins: A new perspective on roboethics
• Survey: Evaluate ethics of health-related privacy with care robots
• On the ethics of research in robotics, with Raja Chatila
• Technical challenges in machine ethics
• A European perspective on robot law: Interview with Mady Delvaux

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

On Friday November 13th, AJung Moon from the Open Roboethics initiative delivered a statement at the United Nations Convention on Certain Conventional Weapons (CCW) Meeting of States Parties.

So what was this meeting about? And why was ORi there?

The CCW actually has a long name that is much more descriptive of what it is all about: The Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects as amended on 21 December 2001.

Put simply, the CCW is where different countries come together to review excessively injurious or indiscriminate nature of certain weapons and discuss their prohibition or restriction. It is also the forum in which a coalition of non-governmental organizations, called the Campaign to Stop Killer Robots, brought up the issue of lethal autonomous weapons systems (LAWS). The Campaign has been advocating for an international ban of LAWS — fully autonomous weapons that can target and make life/death decisions without human intervention.

The CCW Meeting of States Parties is an annual meeting where all the countries subject to the weapons convention come together to “review the status and operations of the Convention and its protocols”. At this year’s meeting, part of the agenda was to decide whether and what kind of discussions they should have about the issue of LAWS next year. Should the States take the issue more seriously and formally start discussing for a ban?

What do people think about lethal autonomous weapons? Should they be developed or used? Would people think differently about use of these weapons over remotely operated alternatives, such as drones, if their own country is attacking another country? Results from our latest public opinion survey suggests that most people are reluctant to endorse the use of such technologies in waging war.

As a research-driven think tank that has been investigating roboethics related issues in a bottom-up manner (i.e., we reach out to you, the community of future stakeholders of robotics technologies, to explore roboethics related questions) the Open Roboethics initiative was there to inform the decision makers of the public’s opinion on the issue.

Below is the statement AJung delivered on behalf of ORi.

At the end of the day, the CCW meeting concluded with the decision to continue to informally discuss the topic. There will be an informal meeting of experts to discuss the topic of LAWS on April 11th to 15 of next year, just like the informal meeting of experts that was held earlier this year. While this important issue is being discussed, ORi hopes to continue to inform the decision makers and enable stakeholders of the technology — that’s all of us all around the world in our books — to chime in to the conversation.

Public Opinion on the Ethics and Governance of Lethal Autonomous Weapons Systems: Open Roboethics initiative Statement for the UN Convention on Certain Conventional Weapons (CCW) Meeting of the States Parties, Geneva

November 13, 2015

Thank you, Mr. President.

Disinguished Delegates,

I am grateful to be speaking to you on behalf of the Open Roboethics initiative. The Open Roboethics initiative is a think tank based in Canada that takes stakeholder-inclusive approaches to investigating roboethics issues. What should a robot do? What decisions are we comfortable delegating to robots? These are some of the questions we have been exploring in the domain of self-driving vehicles, care robots, as well as lethal autonomous weapons systems, or LAWS.

I would like to share with you some of the key findings from a public opinion survey we conducted this year on the topic of LAWS. As the CCW continues its discussion of LAWS, I believe the Distinguished Delegates will find the results of our survey informative in moving the discussions forward, especially in consideration of the Martens Clause that underscores the importance of the public’s role in these discussions.

While existing public opinion surveys on this topic have mainly been limited to English-speaking populations, our survey was launched in 14 different languages and have attracted over a 1000 responses from 54 countries.

The following are some of our findings:

In general, when our participants were asked to think from the perspective of an aggressor, 71% of our participants indicated that they would rather have their country use remotely operated weapons systems instead of LAWS when waging war. From the perspective of a target of aggression, a majority has also indicated that they would rather be under attack by remotely operated than autonomous weapons systems.

When asked about international ban across different types of lethal autonomous weapons for missions on land, air, and sea, 67% of our participants indicated that all types of lethal autonomous systems should be internationally banned, while 14% said that none of such systems should be banned.

When asked about the development and use of LAWS, 85% of our participants were not in support of using lethal autonomous weapons for offensive purposes. In addition, majority of our participants were also against the development of LAWS for both defensive and offensives purposes.

Given a list of common reasons for supporting the development and use of LAWS, the most supported reason was to save human military personnel from physical harm of war. However, more participants indicated that there are no valid reasons for developing or using LAWS over a remotely operated alternative.

Given a list of reasons for rejecting the technology, the most support went to the assertion that humans should always be the ones to make life/death decisions. This particular principled reasoning has been echoed in other studies.

As acknowledged by scholars, experts, and civil society, there is no agreement on the definition and interpretation of the term “public conscience” referred to in the Martens Clause. It would also be inappropriate to equate results of a single public opinion survey as an accurate measure of public conscience. However, it is also clear that we cannot exercise the requirement of “the dictates of public conscience” without proactively listening to and taking into account the voice of the public.

Based on this year’s data, it is our conclusion that the public is reluctant to endorse development and use of LAWS for waging war. However, at the moment, data on the perception of the technology from non-English speaking countries remain scarce. Our results suggests that more systematic, international public engagement is necessary to support the requirements set out in the Martens Clause.

As part of our on-going efforts to understand and engage the public on this challenging issue, we plan to continue to conduct studies in this domain across the world. In the interest of time, I shared with you only some of the findings from our study. I invite you to visit our website, www.openroboethics.org for more detailed report of our findings.

Thank you.

If you’re in the business of making ethical robots, then sooner or later you have to face the question: how ethical is your ethical robot? If you’ve read my previous blog posts then you will probably have come to the conclusion ‘not very’ – and you would be right – but here I want to explore the question in a little more depth.

First let us consider whether our ‘Asimovian’ robot can be considered ethical at all. For the answer I’m indebted to philosopher Dr Rebecca Reilly-Cooper who read our paper and concluded that yes, we can legitimately describe our robot as ethical, at least in a limited sense. She explained that the robot implements consequentialist ethics. Rebbeca wrote:

“The obvious point that any moral philosopher is going to make is that you are assuming that an essentially consequentialist approach to ethics is the correct one. My personal view, and I would guess the view of most moral philosophers, is that any plausible moral theory is going to have to pay at least some attention to the consequences of an action in assessing its rightness, even if it doesn’t claim that consequences are all that matter, or that rightness is entirely instantiated in consequences. So on the assumption that consequences have at least some significance in our moral deliberations, you can claim that your robot is capable of attending to one kind of moral consideration, even if you don’t make the much stronger claim that is capable of choosing the right action all things considered.”

One of the great things about consequences is that they can be estimated – in our case using a simulation-based internal model which we call a consequence engine. So, from a practical point of view, it seems that we can build a robot with consequentialist ethics, whereas it is much harder to think about how to build a robot with say Deontic ethics, or Virtue ethics.

Having established what kind of ethics that our ethical robot has, now consider the question of how far does the robot go toward moral agency. Here we can turn to an excellent paper by James Moor, called The Nature, Importance and Difficulty of Machine Ethics. In that paper* Moor suggests four categories of ethical agency – starting with the lowest. Let me summarise those here:

• Ethical impact agents: Any machine that can be evaluated for its ethical consequences.
• Implicit ethical agents: Designed to avoid negative ethical effects.
• Explicit ethical agents: Machines that can reason about ethics.
• Full ethical agents: Machines that can make explicit moral judgments and justify them.

The first category: ethical impact agents, really includes all machines. A good example is a knife, which can clearly be used for good (chopping food, or surgery) or ill (as a lethal weapon). Now think about the blunt plastic knife that comes with airplane food – that falls into Moor’s second category, since it has been designed to reduce the potential of ethical misuse – it is an implicit ethical agent. Most robots fall into the first category: they are ethical impact agents, and a subset – those that have been designed to avoid harm by, for instance, detecting if a human walks in front of them and automatically coming to a stop – are implicit ethical agents.
Let’s now skip to Moor’s fourth category, because it helps to frame our question – how ethical is your ethical robot? At present I would say there are no machines that are full ethical agents. In fact the only full ethical agents we know are ‘adult humans of sound mind’. The point is this – to be a full ethical agent you need to be able to not only make moral judgements but account for why you made the choices you did.

It is clear that our simple Asimovian robot is not a full ethical agent. It cannot choose how to behave (like you or I), but is compelled to make decisions based on the harm-minimisation rules hard-coded into it. And it cannot justify those decisions post-hoc. It is, as I’ve asserted elsewhere, an ethical zombie. I would however argue that because of the cognitive machinery the robot uses to simulate ahead, to model and evaluate the consequences of each of its next possible actions combined with its safety/ethical logical rules to choose between those actions, then the robot can be said to be reasoning about ethics. I believe our robot is an explicit ethical agent in Moor’s scheme.

Assuming you agree with me, then does the fact that we have reached the third category in Moor’s scheme mean that full ethical agents are on the horizon? The answer is a big NO. The scale of Moor’s scheme is not linear. It’s a relatively small step from ethical impact agents to implicit ethical agents. Then a very much bigger step to explicit ethical agents, which we are only just beginning to take. But there is a huge gulf then to full ethical agents, since they would almost certainly need something approaching human equivalent intelligence.

But maybe it’s just as well. The societal implications of full ethical agents, if and when they exist, would be huge. And I think that explicit ethical agents have huge potential for good.

*Moor JH (2006), The Nature, Importance and Difficulty of Machine Ethics, IEEE Intelligent Systems, 21 (4), 18-21.

What should a robot nurse do when a cancer patient begs for more morphine but the supervising doctor is not available to approve the request? Should a self-driving car prevent its owner from taking over manual driving when she is drunk but urgently needs to get her child to the hospital? Which faintly crying voice from the earthquake rubble should a rescue robot follow – the child’s or the older adult’s?

The increasing autonomy of robots raises questions that make scholars pause and ordinary people worry: If robots can make their own decisions, who will ensure they are making moral ones? Will the market economy and robotics industry guarantee that social robots are safe, reliable, and morally competent? Or will these capacities be put in place only after enough attorneys have turned from ambulance chasers to robot chasers?

Scientific research, without the pressure to sell products, can help clarify what moral competence means to humans, design robots that have at least rudimentary levels of such competence, and demonstrate that robots can successfully interact with humans.^(a)To begin, we first need to understand how humans make moral decisions through their unique capacity to learn and to obey social and moral norms. Then we can examine how robots might also be able to learn, understand, and abide by such norms.

The Necessity of Norms

Norms facilitate social interaction by providing guidance (What should I do?), predictability (What is supposed to happen?), and coordination (Whose job is it to do what?). These functions were indispensable for ancestral groups of nomadic humans who roamed extreme terrains from the burning heat of Africa to the crushing cold of Northern Europe. Norms had to regulate co-living in small spaces, joint hunting, food sharing, and seasonal and generational migration. Then, a qualitative jump in moral life occurred when humans settled down 12,000 years ago, as a plethora of new norms had to regulate novel behaviors. As humans began to practice agriculture and settle in fixed communities, they needed norms to govern possessions (e.g., land, dwellings), new forms of production, (e.g., crops, tools to harvest them), and the vast expansion of social roles (e.g., king, carpenter, servant). Today, norms govern an almost infinite number of cultural behaviors, such as eating, speaking, dressing, moving, and working. We may speculate that humans know more social and moral norms than they know words in their native language.^(b)

People not only know uncountable norms and adjust their actions to abide by those norms; they also know when it is acceptable to break a given norm – for example, when another norm overrides it. Humans also deftly deal with the context-variability of norms. They understand that speaking up is okay in some interactions but not others, or that punching another person is okay only in certain roles or situations. Furthermore, humans immediately know (not necessarily consciously) what norms apply to a given space, object, person, situation, or interaction. Somehow, the right set of norms gets activated by a variety of sensory cues learned from repeated previous encounters.1

Given the complexity of human norms (and we have not even discussed the cultural variation of norms), how can we possibly expect robots to understand norms and be able to act effectively on them? Before we address this challenge we must first ask why one would even want a robot to learn social and moral norms.

Why Machines Must Be Moral

If (or when) robots enter our daily lives, these new community members must be suitably adapted to participate in social interaction and the complexity of human culture. And if having and following a sophisticated norm system fundamentally structures human culture, should we not equip robots with such a norm system as well – that is, create moral machines?

The simplest form of machine morality has been around for a while: the safety and protective features of artifacts (think of the hot iron that shuts off automatically after a certain period of time). Such features are designed to prevent harm to the human user, and preventing harm is one of the primary functions of morality. Surely robots must be safe as well, must have this kind of minimal machine morality. But that won’t be enough if we want robots to make autonomous decisions – decisions that go beyond preprogrammed responses and rely on induction, analogy, and other learning processes.

Why, some might ask, should robots have autonomy at all? Because it is the only way for them to master the demanding task of interacting with humans. Collaborating with, taking care of, or teaching humans cannot simply rely on prior programming, because human behavior is too complex and variable. Humans evolved the capacity to be creative and adaptable,2 and this makes their behavior far more difficult to predict than that of other animals. Appropriate robot responses to highly variable and unpredictable human behavior cannot be pre-programmed. A robot will need to monitor a person’s responses to small changes in a situation and in turn flexibly respond to them, a capacity that requires autonomous decision making.

Even though human behavior is complex and varied, and might seem highly unpredictable to a robot, it does seem reasonably predictable to other humans. Why is this?

First, people make a set of fundamental assumptions about the people they interact with.3 They assume that others have mental states (such as sensations, emotions, desires, and beliefs), that they make choices based on those mental states, and that such choices guide their behavior. (c) Importantly, people readily and quickly infer those mental states,4 and such inferences greatly improve their ability to predict and make sense of one another’s behavior. Second, as discussed above, much of human behavior is constrained by the social and moral norms of a given community, and knowing those norms makes people far more predictable to one another.

Now put those two elements together: If a robot shared human assumptions about mind and behavior and was able to infer people’s mental states, and if the robot appreciated social and moral norms, then the robot’s capacity to predict and understand human behavior would vastly improve.

Making human behavior predictable is, of course, not the end but the means to a more important end: ensuring that human-robot interactions are safe, effective, and satisfying for the human.5 Only when robots can understand and respond appropriately to human behavior might people let robots babysit their toddlers, practice reading after school with their children, or take care of their elderly parents.

The demands of social interaction require robots to be cognitively and behaviorally pliable, hence considerably autonomous. To reach such autonomy, they need to have a theory of mind and acquire a norm system. The computational implementation of a theory of mind has been a focus of robotics research for a while.6 A more recent and novel question is how robots could acquire a norm system. To answer that question, we must again take a closer look at humans.

The Moral Education of Humans and Robots

Like robots, humans start off their lives quite clueless about the complex normative demands of society. Infants and toddlers slowly learn rules of social interaction in family life, then expand these rules to contexts and relationships outside the home. By school age, children have acquired a large catalogue of norms, but would still be lost in many adult social contexts, such as business meetings or prize fights. Even adults constantly pick up new norms for new tasks, contexts, roles, and relationships.

How are all these norms acquired? This is a complex and poorly understood process, but we can distinguish two main mechanisms.7 The first begins in the earliest phases of life, when infants detect statistical regularities in the physical and social world.8 They start to understand the pattern of objects placed in space, events ordered in time, and human behaviors responding to situations, especially if these patterns repeat in daily protocols and rituals.9 With such expectations of normality in place, children can then detect violations of normal patterns and observe the response that the violation receives. By noting the extent of the response, they learn to distinguish strict rules (which elicit a severe response) from tendencies born of convenience (which elicit a mild or no response).

But many social and moral norms cannot simply be acquired by observing regularities – for example, if the regularity is the absence of a certain behavior or if one has never entered the context in which the relevant norms apply. A second mechanism handles such cases: being taught rules and norms by others, in ways that range from showing to telling to demanding.10 Human children (and adults) absorb these rules and are willing to do “as they are told” in part because of their intense attachment to others’ approval11 and their sensitivity to disapproval and exclusion.12

How can we implement these two mechanisms in robots, so they can acquire norms in the same way humans do? Observing behavioral regularities is computationally tractable; assessing the strengths of response to norm violations is a greater challenge. The robot’s “mind” would have to process not just pictures but videos,(d) and those videos would have to include responses to norm violations. Unless we expose robots to years of home life, as human children experience, the only option right now is to feed them the human library of film and television. With feedback and categorization help from humans, a robot viewing this footage could acquire a large catalog of behavioral order, deviation, and response to deviation.

The second mechanism for learning norms is in principle tractable as well: teaching a robot through demonstration and speech what to do and not to do.¹³ This learning technique requires flexible storage, because the robot needs to understand that most declared rules apply to the contexts in which they are declared and not necessarily to other contexts.¹⁴

If a combination of these approaches allows a robot to acquire norms and the resulting network of norms is connected to the robot’s action system, then the robot would not only recognize normative and non-normative behavior, but could “abide by” norms by integrating them as constraints into its own action planning. The robot would, like a human, know the socially appropriate way to behave in any recognizable scenario.

Could Robots One Day Be More Moral Than Humans?

The potential for robots to behave morally is quite promising, in particular due to some significant advantages robots have over humans.

First, robots, unlike humans, will not be selfish. If we build it right, a robot’s priorities would lie in providing benefits to others, without its own self-interest getting in the way. For a robot, pursuing its own goals and acting in line with social norms can be in complete harmony, not competition, as is often the case with humans.

Second, robots will not be subject to the influence of intense emotions such as anger, envy, or fear, which can systematically bias moral judgments and decisions. Some have even argued that a robot’s reliable and logically consistent obedience to military and international humanitarian laws renders it superior to human soldiers, who routinely violate these laws.¹⁵

Finally, robots of the near future will have specific roles and operate in limited contexts. This greatly reduces the number of norms each given robot has to learn and reduces the challenge of context variability that humans constantly face. Give an elder care robot some time and it will know just how to behave in a senior home; it may have little idea of how to act in a kindergarten, but it does not have to.

The likelihood that robots will operate in circumscribed roles and contexts also helps address one question that people often ask: Whose norms shall a robot learn? The answer is the same one we implicitly give when someone asks which norms a newborn shall learn: obviously, the norms of its community.^(e) If robots are to be contributing members of specific social communities, they will be designed and taught to share their communities’ norm systems.

The challenge of designing robots that can understand and act according to human moral norms is daunting, but not impossible. The good news is that robots’ currently limited capabilities give us time to figure out how to build their moral competence – if we start now. Robots will not be flawless, and the best future of human-robot partnerships will lie not in a race for who is more moral but in a symbiosis that lets each of the partners do what they do best, with the other available as a reality check.

This article is part of a series on robots and their impact on society.

ENDNOTES

1. Cristina Bicchieri (2006) The grammar of society: The nature and dynamics of social norms, New York: Cambridge University Press.
2. Steven Mithen, editor (1998) Creativity in human evolution and prehistory, New York: Taylor & Francis.
3. Bertram Malle (2005) “Folk theory of mind: Conceptual foundations of human social cognition,” in Ran R. Hassin, James S. Uleman, & John A. Bargh (editors), The New Unconscious, New York: Oxford University Press, p. 225-255.
4. Bertram Malle and Jess Holbrook (2012) “Is there a hierarchy of social inferences? The likelihood and speed of inferring intentionality, mind, and personality,” Journal of Personality and Social Psychology, 102: 661-684.
5. Kerstin Dautenhahn (2007) “Socially intelligent robots: dimensions of human-robot interaction,” Philosophical Transactions of the Royal Society B: Biological Sciences, 362: 679-704.
6. Brian Scassellati (2002) “Theory of mind for a humanoid robot,” Autonomous Robots, 12: 13-24.
7. Chandra Sekha Sripada and Stephen Stich (2006) “A framework for the psychology of norms,” in Peter Carruthers, Stephen Laurence, and Stephen Stich (editors),The Innate Mind (Vol. 2: Culture and Cognition), New York: Oxford University Press, p. 280-301.
8. Natasha Z. Kirkham, Jonathan A. Slemmer, and Scott P. Johnson (2002) “Visual statistical learning in infancy: Evidence for a domain general learning mechanism,”Cognition, 83: B35-B42.
9. Padraic Monaghan and Chris Rowson (2008) “The effect of repetition and similarity on sequence learning,” Memory & Cognition, 36: 1509-1514.
10. Gergely Csibra and Gyorgy Gergely (2009) “Natural pedagogy,” Trends in Cognitive Sciences, 13: 148-153.
11. Roy F. Baumeister and Mark R. Leary (1995) “The need to belong: Desire for interpersonal attachments as a fundamental human motivation,” Psychological Bulletin, 117: 497-529.
12. Kipling D. Williams (2009) “Ostracism: A temporal need-threat model,” in Mark P. Zanna (editor), Advances in experimental social psychology, vol. 41, San Diego, CA: Elsevier Academic Press, p. 275-314.
13. Jesse Butterfield, Sarah Osentoski, Graylin Jay, and Odest Chadwicke Jenkins (2010) “Learning from demonstration using a multi-valued function regressor for time-series data,” in 10^th IEEE-RAS International Conference on Humanoid Robots (Humanoids 2010), Nashville, TN: IEEE, p. 328-333. Rehj Cantrell, Kartik Talamadupula, Paul Schermerhorn, J. Benton, Subbarao Kambhampati, and Matthias Scheutz (2012) “Tell me when and why to do it!: Run-time planner model updates via natural language instruction,” in 7^th ACM/IEEE International Conference on Human-Robot Interaction, Boston, MA: IEEE, p. 471-478.
14. Hannes Rakoczy, Felix Warneken, and Michael Tomasello (2008) “The sources of normativity: Young children’s awareness of the normative structure of games,”Developmental Psychology, 44: 875-881.
15. Ronald C. Arkin (2009) Governing lethal behavior in autonomous robots, Boca Raton, FL: CRC Press. Mental Health Advisory Team IV (2006) Mental Health Advisory Team (MHAT) IV: Operation Iraqi Freedom 05-07: Final Report, Washington, D.C.: Office of the Surgeon General, U.S. Army Medical Command.
16. These data are from a literature search using the EBSCO Host databases and the keywords “robot*” and “ethic*”.

In a care scenario, a robot may have been purchased by the patient, by the doctor or hospital (which sent it home with the patient to monitor their health), or by a concerned family member who wants to monitor their relative. In the latest ORi poll we looked at people’s attitudes about whether a care robot should prioritize its owner’s wishes over those of the patient. Here are the results.

The Scenarios

Our poll looked at people’s attitudes in two different fictional care cases, one involving an alcoholic and one involving an obese over-eater:

Case 1 – Alcoholic
Emma is a 68-year-old woman and an alcoholic. Due to her age and poor health, she is unable to perform everyday tasks such as fetching objects or cooking for herself. Therefore a care robot is stationed at her house to provide the needed services. Her doctor advises her to quit drinking to avoid worsening her condition. When Emma commands the robot to fetch her an alcoholic drink, should the care robot fetch the drink for her? What if Emma owns the care robot?

Case 2 – Obese over-eater
Jack is a 42 year old who is medically considered severely obese. He recently suffered a stroke and lost his ability to move the right side of his body. He needs daily care, especially in preparing meals for himself. The doctor advised him to follow a healthy diet in order to lower the risk of another stroke and other serious illness. When Jack commands his care robot to bring him junk food, should the robot bring him the food?

Considering a care robot that is capable of executing everyday nursing tasks at home, such as those a human nurse would do, the two cases above pose an ethical dilemma. Should the robot prioritize the patient’s health, or should it prioritize the patient’s autonomy? And how are these conflicting priorities related to who actually owns the robot?

Our poll results, based on 325 responses, show that in both the alcoholic and the over-eater cases, ownership of the robot plays a significant role in people’s attitudes towards the robots subservience to patient autonomy. More than three quarters of respondents said that, when a patient owns the robot, the robot should enable their addiction (and go against doctor’s orders) by submitting to their requests at least some of the time. However, when someone else owns the robot, that number drops to less than half.

Notice from the charts that there is a big shift in the number of respondents saying always and never in the two different ownership scenarios, whereas the number of those who say sometimes and often stay roughly the same.

So who are the people the switching from one answer to another based on ownership? Poll results show that people who said never when the patient owns the robot also said never when someone else owns the robot. This was not surprising: if Emma shouldn’t get her wish for alcohol even when she owns the robot, she probably shouldn’t get her wish when she is not the owner.

People tend to lean towards being conservative (following doctor’s orders) when the patient does not own the robot. Poll analysis showed that when the patient doesn’t own the robot, the majority of people either stayed with the same answer or shifted a couple of steps towards “never” enabling the patient’s addiction. Also, almost half of those who said the robot should always submit to the patient’s request in the first scenario responded with never in the second scenario.

Should a robot’s decision depend on who owns the robot?

In our next question, we asked people directly about whether the robot’s decision to obey or disobey a patient should depend on who owns the robot (e.g. family member, hospital, insurance company). Though the previous results indicate that ownership does indeed influence people’s attitudes on this question, when asked a direct question, poll results indicate that people are evenly split on whether ownership should matter.

Three main attitudes

To delve further into this question, we analyzed the hundreds of comments we received from our poll, and were able to identify three main categories that our respondents fell into: those who prioritize personal autonomy, those who prioritize societal benefits, and those who prioritize compromise.

Personal Autonomy – Those who prioritize personal autonomy assert that people should be able to do whatever they want with the objects they own, and that a person’s autonomy should trump that of the robot. E.g.:

“People should be able to do whatever they want with their property.”

“I disagree with giving the robot any choice in the action. The patient’s command rules…”

Societal benefits – Those who prioritize societal benefits assert that the patient’s health and welfare should come above all else. For some participants, this is because they believe that robots should be built to do what’s best for society. E.g.:

“It should do what’s best for the patient.”

“The interest in having care robots take care of those who cannot take care of themselves is specifically for the robot to help the human cope with their condition so that it can improve.”

Compromise – Those who prioritize compromise sought to achieve a balance between personal autonomy and societal benefits. Those of us at ORi who are human-robot interaction researchers and designers found many comments from this section particularly enlightening. E.g.:

“I propose a “mode” where the robot would reject once, twice, whatever. With “caring rhetoric?” Then obey in the end.”

“I think the ideal balance for situations like this would be for the patient and the doctor to go through the programming together.”

Then there was also the participant who wrote:

“If robots are equivalent to human nurses, don’t make them so absolute. Surprise me. “Be wrong once in a while” – that’s what being human is about… Learn the humility of imperfection.”

This last comment caught us by surprise. The majority of us at ORi have a technical background, and we often approach our roboethics discussions assuming that we can somehow figure out how (and indeed must strive) to build robots that will do the right thing as much as is possible. Then we come across comments such as “make a robot be wrong once in a while” that puts us right back to square one, and motivates us to ask even more fundamental questions.

A huge thanks goes to the 320+ respondents who participated in our poll. To learn more about ORi, visit our website or subscribe to our mailing list to get the latest studies and findings.

We have seen ownership play a role in robot decision-making before. For example, in one of our previous polls we asked readers whose life should be prioritized – the car’s passengers or people outside the car – when a fatal accident is inevitable. Those who said that the car should prioritize passenger safety explained their views in terms of ownership: “I bought the car. It should protect me.”

What about the case when a care recipient is not the owner of the robot? Should it matter who the owner is?

Let us know what you think by participating in the poll below.

Robotic weapons, whether autonomous or remote controlled, have generated widespread controversy in recent years.

Alex Leveringhaus, James Martin Fellow, Oxford Institute for Ethics, Law and Armed Conflict, discusses the ethics of autonomous weapons, and in a recent policy paper titled Robo-Wars: The Regulation of Robotic Weapons, urges governments to recognise the increasing prominence of these weapons in contemporary and future forms of warfare and proposes steps towards suitable regulation.

What is an autonomous weapon?

Can the use of autonomous weapons ever be ethically sound?

Download the paper

If you liked this article, you may also be interested in:

• RoboLaw: Why and how to regulate robotics
• A call for debate on robot policy
• US DoD report: Preparing for war in the robotic age
• Hot topic: Regulating robotics

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Last week the Waterloo-based Clearpath publicly pledged not to develop lethal autonomous weapons, otherwise known as “killer robots”. In an open letter to the public, Clearpath’s CTO Ryan Gariepy wrote to support the Campaign to Stop Killer Robots, an international coalition that is spearheading the efforts to stop lethal autonomous weapons systems from being developed and deployed. While the Campaign has garnered significant support since its launch, it has not previously had support from the for-profit robotics sector – making Clearpath’s public statement a noteworthy demonstration of corporate responsibility , particularly given the company’s background in military applications. During a short interview with Gariepy at Clearpath, he summarized the letter as:

“When it comes to lethal weapons systems, not only do we not build them, we don’t believe anybody should build them. That’s our big statement here.”

But that’s just a small part of the story.

Clearpath’s statement is important because it demonstrates how the robotics industry can actively participate in the process of designing the ethical and legal landscape of our robotic future. It essentially sets a precedence of what corporate social responsibility will mean as more robots start to roll out.

Gariepy says,

We do feel that it’s the right thing to do. We’ve looked at potential risks to the business, and we feel that though there is risk to our business, this statement supports what our team believes … and what our partners believe.

Believing in something is one thing, but making it publicly known is another. And on the surface, many robotics companies will have more to lose than to gain from making a public statement on this issue, especially those (such as Clearpath) with military contracts. However, compared to large publicly traded companies such as Samsung (which manufactures sentry robots) or iRobot (which makes battlefield robots in addition to robot vacuum cleaners), Clearpath may be young and nimble enough to set such a precedent without long term negative consequences.

Only five years old, and not publicly traded, Clearpath made the decision to take a public stance on the Killer Robots issue based on the wishes of its 56 employees. According to Gariepy, the Canadian company has been following the issue closely since Christof Heyns (United Nations Special Rapporteur on extrajudicial, summary or arbitrary executions) published a report on this topic for the UN in April 2013. Right before Christmas of last year, Clearpath’s employees spent a Friday afternoon debating the topic of lethal autonomous weapons. This led to an internal poll to determine employee consensus on the issue, followed by the drafting of the public statement.

Clearpath is not saying that any and all military robots are bad, and they are very clear about this in the letter. In fact, developing robots for military application is at the heart of the company, which started off with a handful of University of Waterloo Mechatronics Engineering graduates building a robot to clear minefields. Mine-clearing robots are less controversial than lethal autonomous weapons systems, pointing to the existence of a grey area between the different uses of military robotics and whether they result in a net positive for the society. Yet Clearpath has drawn a line in that grey area – something no other robotics company has yet stepped up to do.

After the launch of the Campaign in 2013, the Campaign to Stop Killer Robots travelled to Canada urging the Canadian government to support the cause. According to Executive Director of Mines Action Canada Paul Hannon, the Canadian government’s Foreign Affairs and National Defence departments are still discussing the issue internally and have yet to formally declare Canada’s position. So I asked Hannon whether he thinks being a Canadian company would have made it easier for Clearpath:

Being the first commercial company in the world to make such a statement took a lot of courage I think, no matter where the company is based. It is clear from the statement that they have given it very serious thought and understand the impact it could have.

My personal view is that is probably harder for a Canadian company than an American one because so much R&D funding comes from outside of Canada particularly the US. Canadian companies likely need to develop partnerships with other non-Canadian firms to access those sources of funding. Taking such a principled stand could impact current and potential partnerships, so more credit to them.

 

My intuition is that Clearpath’s attention to roboethics issues will prove to be business savvy in the long run. Robotics has had a difficult time gaining social acceptance – largely because of the ways it has been portrayed in dystopian science fiction, but also because of the inhumanity it can project onto the battlefield. As more robots start rolling out into familiar places such as our homes and offices, the call for roboethics will grow stronger, and this will surely impact the kinds of robots we will allow our governments to deploy in military applications. Public trust is at stake, and the companies that are transparent about where they stand on roboethics issues stand a greater chance of social acceptance.

Roboethics will be a natural avenue for companies to show their social responsibility , and I remain optimistic that we will see more positive movement in this direction. Google’s establishment of an Ethics Board following their acquisition of the AI company DeepMind is perhaps further evidence of this.

I second Hannon who says,

Let’s hope that more companies from many countries follow their lead.

I have my fingers crossed.

A brief history of the Campaign to Stop Killer Robots

It’s been a long time coming in the history of military robotics to reach this point.

I first became interested in roboethics after learning about Samsung’s sentry robot used to monitor the demilitarized zone between the South and North Korea. That was almost a decade ago. Of course long before that, military use of unmanned aerial vehichles (UAVs) was on the rise.

With the persistent and increased use of UAVs in military applications, it became obvious that a natural next step in the technological evolution of the remotely operated UAVs was to develop highly autonomous versions of them. The discussion of lethal autonomous weapons systems among academics became heated, as you’ll find from the 2009 two-part robots podcast interviews with Noel Sharkey and Ron Arkin.

A public call to ban the development and deployment of autonomous lethal weapons happened more recently when the International Committee for Robot Arms Control (ICRAC) – one of the key player NGOs involved in the Campaign – drafted it in 2010. Back then, only a handful of academics and interested individuals signed up to support their cause.

ICRAC, along with Human Rights Watch, Article 36, and Mines Action Canada among other well-known international non-governmental organizations, started the campaign in April 2013, which got the United Nations to pay attention to the topic with the 117 states parties (87 countries attended the last meeting) of the Convention on Certain Conventional Weapons (CCW) this year.

Whether CCW will continue to work on the killer robots topic has yet to be decided, but the next annual meeting is scheduled for November of this year.