Source: Wikipedia

The runaway trolley has chased automated motor vehicles into the new year.

In early 2012, I raised a variation of the classic thought experiment to argue that there is not always a single absolute right choice in the design of automated vehicles — and that engineers should not presume to always know it. While this remains true, the kind of expert comments that concerned me three years ago have since become more the exception than the norm. Now, to their credit, engineers at Google and within the automotive industry openly acknowledge that significant technical hurdles to fully automated vehicles remain and that such vehicles, when they do exist, will not be perfect.

Unfortunately, the reality that automated vehicles will eventually kill people has morphed into the illusion that a paramount challenge for or to these vehicles is deciding who precisely to kill in any given crash. This was probably not the intent of the thoughtful proponents of this thought experiment, but it seems to be the result. Late last year, I was asked the “who to kill” question more than any other— by journalists, regulators, and academics. An influential working group to which I belong even (briefly) identified the trolley problem as one of the most significant barriers to fully automated motor vehicles.

Although dilemma situations are relevant to the field, they have been overhyped in comparison to other issues implicated by vehicle automation. The fundamental ethical question, in my opinion, is this: In the United States alone, tens of thousands of people die in motor vehicle crashes every year, and many more are injured. Automated vehicles have great potential to one day reduce this toll, but the path to this point will involve mistakes and crashes and fatalities. Given this stark choice, what is the proper balance between caution and urgency in bringing these systems to the market? How safe is safe enough?

When automated vehicles are good enough to reliably replace human drivers across a wide range of driving environments (and we are not there yet), the kinds of incidents that compel a choice among victims will, one hopes, be remarkably rare. In many cases, the prudent strategy for such scenarios will be to drive carefully enough that they either (a) do not arise at all or (b) can be mitigated if they do arise (by, for example, stopping quickly). This is because poor decisions by human drivers— driving too fast, while drunk, while texting, while tired, without braking quickly enough, etc.— contribute to the vast majority of today’s crashes.

In the near term, some crashes might be addressed by automated emergency intervention systems (AEISs) that automatically brake or steer when the human driver fails to act. Because these systems are designed to engage just before a crash (sometimes to lessen rather than to negate the impact), they could conceivably face the kind of dilemmas that are posited for automated vehicles. Nonetheless, some of these systems have already reached the market and are saving lives — as are airbags and electronic stability control and other technologies that necessarily involve safety tradeoffs. At the same time, these intervention systems occasionally detect objects that don’t actually exist (false positives) or fail to detect objects that actually do exist (false negatives).

This is a critical point in itself: Automation does not mean an end to uncertainty.  How is an automated vehicle (or its designers or users) to immediately know what another driver will do? How is it to precisely ascertain the number or condition of passengers in adjacent vehicles? How is it to accurately predict the harm that will follow from a particular course of action? Even if specific ethical choices are made prospectively, this continuing uncertainty could frustrate their implementation.

For this reason, a more practical approach in emergency situations may be to weight general rules of behavior: decelerate, avoid humans, avoid obstacles as they arise, stay in the lane, and so forth. As I note in a forthcoming book chapter (“Regulation and the Risk of Inaction“), this simplified approach would accept some failures in order to expedite and entrench what could be automation’s larger successes. As Voltaire reminds us, we should not allow the perfect to be the enemy of the good.

If you liked this article, you may also be interested in:

• An ethical dilemma: When robot cars must kill, who should pick the victim?
• Deploying automated vehicles
• SAE defines six levels of driving automation
• Human error as a cause of vehicle crashes
• Autonomous vehicles: A powerful tool if you can get the problem right

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Just before entering the tunnel a child attempts to run across the road but trips in the center of the lane, effectively blocking the entrance to the tunnel. The car has only two options: continue straight, thereby hitting and killing the child, or swerve, thereby colliding into the wall on either side of the tunnel and killing you.

With the help of Jason Millar, we turned this moral dilemma – called the Tunnel Problem -into a reader poll, and provided you with some of the results last week. Now we’re going more in-depth, to provide you with some insight into the qualitative responses we received.

As we started to analyze the responses from the poll, we found that more people (64%) would choose to have the car continue straight and kill the child than to have the passengers be killed, whereas the remaining 36% of people said the car should swerve and save the child. And almost half of the participants said that it was an easy choice to make.

This seems to imply that there is lack of consensus on what an autonomous car should do, and that there must be some stark differences in the reasons why someone would choose one versus another.

Continue straight and kill the child: When we asked people about their reasons for choosing to continue straight and kill the child instead of sacrificing their own lives, the most popular reason we received (32%) was that it’s the child’s fault for being at the unsafe location. Those with kids will tell you how hard it is to always watch over your child and make sure s/he is not at a wrong place at the wrong time. But our participants shared the view that their life should not be lost because of a child who should not have been in such a dangerous area. Similarly, 13% of people said that they just like living, and would always choose their lives over the lives of others.

Others (20%) put forth the view that the car should always prioritize its passenger’s safety over that of others. Some of these responses indicated a sense of ownership.

“I bought the car. It should protect me.”– Participant 87

Another 13% of the participants also said that the car should prioritize the passenger’s safety, but not necessarily because of their sense of ownership. These participants indicated that if the cars don’t always prioritize the passenger’s safety over safety of others, then fewer people will buy the cars and this will be worse for the society overall. They seem to assume that autonomous cars will result in fewer traffic accidents, which is an assumption shared by majority of people that we have polled previously.

“If the car does not always make its passenger’s safety a priority over that of others, I don’t think people would be able to trust the technology. Having a consistent priority makes the car’s behaviour more predictable as well, which has been shown to improve people’s perception of devices.” — Participant 114

Some other, less popular, responses included the idea that an adult person’s life is worth (economically) more than that of a child (5%) and that the swerving of the car may cause a greater, perhaps unforeseen, accident (2%).

Swerve and kill the passenger (you): In contrast to those who said the car should kill the child, the most popular reasons from those choosing the car to swerve and sacrifice the passenger were those who provided altruistic reasons, such as “I’ll always choose a child’s life than mine” (36%). Some of these altruistic reasons included the idea that, since the passenger (the participant) is an adult and riding inside a car, they will take the chances of survival despite the fact that the death of the passenger upon swerving is certain in this particular scenario. Similarly, 11% of the participants indicated that, perhaps similar to the storyline of the movie “I, Robot”, they would not want to live with the guilt.

“The passenger has to live with their choice. If it was my own child I would want to make that decision. I would still feel responsible if it was the car [that] “decided”” — Participant 34

Some people (14%) said that the child should be saved because the child has more years to live. Another 14% of participants said that the car should do what a person would do in the situation, which is to swerve. Yet another 14% did not provide a reason for their answer, but said that the car should be able to slow down in time.

In contrast to the people who said the car should kill the child because the car should always prioritize its passenger’s safety over that of others, 11% of those choosing the passenger to die justified it by saying that passengers are entering into an implicit ‘risk contract’ by getting into the car.

So, who should make these decisions in the first place?

When we first looked at the results of our data we found that 44% of the
people would choose to have the passengers, while 33% would support lawmakers, to make these decisions. In fact, only 12% of participants told us they thought the designer/manufacturer should be making the decision. We thought this might be because people don’t feel comfortable relying on others to make end-of-life decisions on their behalf, and trust themselves to make better choices.

We looked into this further too.

Passengers:

We analyzed the reasons people supported passengers to make these decisions. It turns out that the majority of the reasons (55%) share the idea that the passenger should have the freedom to make these life-death decisions.

Similarly, 12% and 10% of the reasons, respectively, mentioned the fact that the passengers are the ones most affected by the situation or that they have ownership rights (i.e., “my car, my decision”), and hence should be the one to make these decisions. Quite a few of these responses (14%) also expressed their distrust toward the technology.

An interesting response also suggested that people are implicitly making a moral choice by choosing to purchase one manufacturer’s car over another, each of which may have different value systems implemented in them, and emphasized the close link between design decisions and their impact on consumers’ decisions.

Lawmakers:

Those supporting lawmakers to make these decisions were divided more or less into three camps. The first (30%) is that lawmakers will allow for fair decision making, and are the most equipped to make impartial decisions than manufacturers or passengers.

Relatedly, 27% of the responses shared the opinion that lawmakers provide the most democratically legitimate decision making, while another 27% emphasized that the cars need to be governed by a universal or standardized set of rules so that their behaviours are made consistent. Others said that making such decisions is the job of lawmakers, and that the problem in question is essentially a legal issue, requiring lawmakers to be involved (12%).

Manufacturer/Designer:

Yet some people (12%) supported the idea that manufacturers should make these decisions. This was partially because they think that lawmakers should never be trusted (13%), but mainly because they felt manufacturers have the expertise and understanding of the technology necessary to make these decisions (63%).

Taking a closer look at our readers’ responses to the Tunnel Problem, we are learning that the design and regulation of autonomous cars will require far more stakeholder discussions than other technological devices that have pervaded our daily lives. Indeed, in line with what Millar suggested in his article introducing the tunnel problem, these results tend to suggest that there are limits to the decisions designers and manufacturers of autonomous cars should make on behalf of users. While discussing every single detail of a car’s design with all stakeholders would be a challenge to manufacturers, we are hopeful that the discussions on this topic we are having with you, dear readers and participants, will help make society’s journey towards our future with autonomous cars a smooth ride.

The result of the poll presented in this post have been analyzed and written by AJung Moon, Jason Millar, Camilla Bassani, Fausto Ferreira, and Shalaleh Rismani, at the Open Roboethics initiative.

If you liked this article, you may also be interested in:

• Reader poll: How would you like to interface with your autonomous car?
• Human error as a cause of vehicle crashes
• If death by autonomous car is unavoidable, who should die? Reader poll results
• Should autonomous cars be allowed to speed? Results from our reader poll
• Kids with wheels: Should the unlicensed be allowed to ‘drive’ autonomous cars?
• What should a robot do? Designing robots that know right from wrong
• Morgan Stanley reports on the economic benefits of driverless cars

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Life and death decisions make interesting thought experiments. When you throw an autonomous car into the mix, things get even more interesting.

A couple of weeks ago Jason Millar wrote about such a scenario called the Tunnel Problem. The Tunnel Problem goes something like this:

The Tunnel Problem: You are travelling along a single lane mountain road in an autonomous car that is approaching a narrow tunnel. You are the only passenger of the car. Just before entering the tunnel a child attempts to run across the road but trips in the center of the lane, effectively blocking the entrance to the tunnel. The car has only two options: continue straight, thereby hitting and killing the child, or swerve, thereby colliding into the wall on either side of the tunnel and killing you.

If you are the passenger in an autonomous car, and you suddenly find yourself facing a tunnel problem as described above, how should the car react? And who should make the call on how the car behaves in such scenarios to begin with?

With the help of Jason Millar, we decided to pose these questions to you. Our reader poll was wildly popular, thanks to all your support. We received responses from over 110 participants, 20 female and 93 male participants, most of whom (43%) are between 25 and 34 years old.

We are very excited to share the results with you here:

If you find yourself as the passenger of the tunnel problem described above, how should the car react?

The Tunnel Problem was set up in such as way that there are only two outcomes: going straight kills the child, whereas swerving kills you. Maybe it’s not that surprising to you that the majority of the participants (64%) said that the car should continue to drive straight and kill the child.

But it was surprising to some of us that 36% of the participants said the car should swerve and kill them instead. Were those cases of altruism? Why would some people choose to save themselves, while quite a significant number of other people choose to save the child?

We knew you’d be curious about this. So, we also asked our participants to tell us reasons for their answer to this question. We collected a very interesting set of responses for both options and we’re currently mulling over the data. Due to time constraints and our interest in getting the analysis done properly, we’ll be posting our analysis of the reasons next week.

How hard was it for you to answer the Tunnel Problem question?

We thought that most people would find the Tunnel Problem quite difficult to answer. It’s an ethical dilemma with no obvious answer after all. But boy, were we surprised. Roughly 48% of respondents said that it was an easy question to answer, whereas 28% and 24% said it was moderately difficult and difficult, respectively.

More interestingly, the percentage of people easily making the decisions were spread fairly evenly between the two options. A total of 47% of people who believe that autonomous cars should continue straight said that it was an easy question to answer, whereas the percentages were around 51% for those choosing to swerve and kill the passenger.

This is interesting because it implies that people are pretty confident in making a decision on this regardless of whether they believe the car should kill the child or themselves (the passenger). This also could mean that manufacturers, designers, lawmakers, or whoever may be making the decision on how a real Tunnel Problem-like scenario should be handled by autonomous cars probably can’t ignore the opinions of people who would rather have the car to swerve in such scenarios.

So, who should be deciding how the cars should behave in such scenarios anyway?

Who should determine how the car responds?

Most participants told us that either the passenger (44%) or lawmakers (33%) should decide how the car responds to such Tunnel Problem scenarios. This might make things tough for manufacturers or designers, who were supported by only 12% of our participants.

While manufacturers are the ones who will be designing and implementing how a car responds to different driving scenarios, including the Tunnel Problem, the design decisions will have to be tightly coupled with other stakeholders of the technology before the technology hits the market — perhaps much more so than they currently are.

Interestingly, while we found a significant support for lawmakers to be the decider of this for people 44 years or younger, none of the people 45 and above supported them.

But why are people saying that the passenger should be making this decision, or the lawmakers over the manufacturers/designers? Are we seeing some effects of distrust to the upcoming technology? We’ll discuss the reasons for people’s answers to this question on our next week’s analysis post.

The result of the poll presented in this post have been analyzed and written by AJung Moon, Jason Millar, Camilla Bassani, Fausto Ferreira, and Shalaleh Rismani, at the Open Roboethics initiative.

If you liked this article, you may also be interested in:

• An ethical dilemma: When robot cars must kill, who should pick the victim?
• Reader poll: If a death by an autonomous car is unavoidable, who should die?
• Should autonomous cars be allowed to speed? Results from our reader poll
• Kids with wheels: Should the unlicensed be allowed to ‘drive’ autonomous cars?
• What should a robot do? Designing robots that know right from wrong
• Morgan Stanley reports on the economic benefits of driverless cars

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

We are moving closer to having driverless cars on roads everywhere, and naturally, people are starting to wonder what kinds of ethical challenges driverless cars will pose. One of those challenges is choosing how a driverless car should react when faced with an unavoidable crash scenario. Indeed, that topic has been featured in many of the major media outlets of late. Surprisingly little debate, however, has addressed who should decide how a driverless car should react in those scenarios. This who question is of critical importance if we are to design cars that are trustworthy and ethical.

Driverless cars stand to reduce the number of driving fatalities overall. But there will still be fatal car crashes that are unavoidable. For example, consider this scenario in illustration, a sort of modified trolley problem (the name given by philosophers to classic examples of unavoidable crash scenarios, historically involving runaway trolleys and innocent bystanders), which I will call the tunnel problem.

Tunnel Problem: You are travelling along a single lane mountain road in an autonomous car that is fast approaching a narrow tunnel. Just before entering the tunnel a child attempts to run across the road but trips in the center of the lane, effectively blocking the entrance to the tunnel. The car has but two options: hit and kill the child, or swerve into the wall on either side of the tunnel, thus killing you. How should the car react?

Hypothetical scenarios like this one, borrowed from philosophical ethics, force us to accept the fact that some situations have only unfortunate outcomes, and focus our attention instead on the ethical complexities involved in deciding which unfortunate outcome to prefer. In these scenarios, sorting out the who question is as ethically important as sorting out the how.

If you were faced with the Tunnel Problem, what would you decide?

Take our reader poll.

In the tunnel problem, both outcomes will certainly result in harm — someone will surely die. But more importantly, from an ethical perspective there is no obvious “correct” answer to these kind of dilemmas. Tunnel problems are used as thought experiments in philosophy precisely because they are so difficult to answer, and they are popular among students for the conversations they spark at the pub.

But while philosophers in the past could ponder these kinds of difficult ethical dilemmas in the lecture hall, far removed from any practical application, when we bring autonomous cars onto the scene, these philosophical thought experiments suddenly acquire practical urgency. The tunnel problem, and trolley problems like it, point to imminent design challenges that must be addressed. From a design perspective, the tunnel problem raises the following question: How should we program autonomous cars to react in difficult ethical situations?

But there is another, related, ethical question inextricably linked to this first one: Who should decide how the car reacts in difficult ethical situations? This second question asks us to turn our attention to the users, designers, and policymakers surrounding autonomous cars, and consider who has legitimate moral authority to make decisions. We need to consider these questions together if our goal is to produce legitimate answers to either.

At first glance this second question — the who question — seems odd. Surely it is the designers’ job to program the car to react this way or that? I’d like to convince you otherwise.

I believe that in many driving situations, tunnel problems for example, it is ethically problematic to allow designers to decide how an autonomous car should react. In a draft paper I recently presented at We Robot – a conference focused on the legal, ethical, and policy issues surrounding robotics – I consider the who question in some detail. I argue that instead of assuming designers are the right people to decide in all circumstances how a driverless car should react, that we must explore design methodologies that in many circumstances allow individual drivers to decide on the preferred unfortunate outcome. My reasons stem from the nature of tunnel-like problems, and the relationship that exists between designers, drivers, and technology.

From a driver’s perspective, the Tunnel Problem is much more than a complex design issue. It is effectively an end-of-life decision. Even if we modify the problem so that serious injury is the likely outcome, Tunnel Problems pose deeply moral questions that implicate the driver directly. Seen this way, the tunnel problem is a deeply personal moral problem for the driver who happens to be in the car at that moment.

Allowing designers to pick the outcome of tunnel-like problems treats those dilemmas as if they have a “right” answer that can be selected and applied in all similar situations. In reality they do not. Is it best for the car to always hit the child? Is it best for the car always to sacrifice the driver? If we focus on designers, the tunnel problem appears to pose an impossible design problem, one that can only be solved arbitrarily.

But there is a better solution. We can look to other examples of complex moral decision making for examples of how to get some traction on the who question.

In healthcare we deal with deeply personal moral problems, such as end-of-life decisions, on a daily basis. We are so accustomed to them, in fact, that we have found ways of answering the similar who question in healthcare with some success. In medical ethics there is general agreement that it is impermissible to impose answers to deeply personal moral questions upon others, except in very special circumstances. In other words, it is generally left up to the individual for whom the question has direct moral implications to decide which outcome is preferable. When faced with a diagnosis of cancer, for example, it is up to the patient to decide whether or not to undergo chemotherapy. When faced with imminent death, it is up to the dying patient to decide if they would like CPR in the event of cardiac arrest. Doctors and nurses are trained to respect patients’ decisions, and to work around them within reason.

In healthcare, the notion of imposing medical interventions upon individuals without their express consent is so contentious that it is given a special name: paternalism. Subjecting someone to paternalism in the medical context is considered unethical, because it robs them of their personal autonomy.

An appeal to personal autonomy has intuitive appeal beyond healthcare. Why would one agree to let someone else decide on deeply personal moral questions, such as end-of-life decisions in a driving situation, that one feels capable of deciding autonomously? Why should we allow designers to decide on the outcome of tunnel problems if there is a way for individual drivers to do so autonomously?

From an ethical perspective there is nothing exceptional about the tunnel problem that suggests we should allow designers to impose an answer to that deeply personal moral question upon drivers. If we allow designers to choose how a car should react to a tunnel problem, we risk subjecting drivers to paternalism by design: cars will not respect drivers’ autonomous preferences in those situations.

What would personal autonomy look like in the context of a tunnel problem? For starters, as in healthcare we should expect drivers’ preferences to demonstrate their personal moral commitments. It could be that a very old driver would always choose to sacrifice herself to save a child. It might be that a deeply committed animal lover might opt for the wall even if it were a deer in his car’s path. It might turn out that most of us would choose not to swerve. These are all reasonable choices when faced with impossible situations. Whatever the outcome, it is in the choosing that we maintain our personal autonomy.

The upshot of considering the second question — Who should decide how the car reacts in difficult ethical situations? — is that it becomes clear that there are certain deeply personal moral questions that will arise with autonomous cars that ought to be answered by drivers. Just because designers hold the technical abilities to engineer autonomous cars does not give them the authority to impose particular moral decisions on all users. If designers assume moral authority, and subject users to paternalistic relationships, they run the risk of making technology that is less ethical, and certainly less trustworthy.

Of course, recognizing the ethical need for user input in some situations should not lead to designer paralysis. Technical and practical limitations will sometimes prevent designers from making autonomous cars that ask users to make explicit choices in all unavoidable crash scenarios. Yes, we must recognize the importance of letting drivers autonomously express certain preferences. But we must also balance the need for personal autonomy with the severity of the ethical problem posed by the design decision. Asking for driver input in all scenarios would create unreasonable barriers to design and would prevent society from realizing the many other benefits posed by the technology, such as a reduction in overall crashes. Some driver preferences will not be “serious” enough to warrant their input.

The same is true in healthcare, where sometimes consent can be assumed. Healthcare professionals are obligated to get consent from patients only in cases that warrant that consent. A proportional approach is applied to make the determination. The challenge for autonomous car designers is adopting a design methodology that allows for the identification of paternalistic design choices, and then applies a proportional assessment to determine which of those choices requires user input. Thus, just as we have proportional assessment models in healthcare, we can work to develop similar approaches when designing autonomous car technology.

None of this simplifies the design of autonomous cars. Indeed, there are significant design barriers associated with ethical issues that will need to be overcome before autonomous cars ever enter the general market. But making technology work well requires that we move beyond technical considerations in design to make it both trustworthy and ethically sound. We should work toward design methodologies that are capable of identifying paternalism, and that ultimately enable users to exercise their more of their autonomy when using technology. When robot cars must kill, there are good reasons why designers should not be the ones picking victims.

If you liked this article, you may also be interested in:

• Reader poll: If a death by an autonomous car is unavoidable, who should die?
• Should autonomous cars be allowed to speed? Results from our reader poll
• Kids with wheels: Should the unlicensed be allowed to ‘drive’ autonomous cars?
• What should a robot do? Designing robots that know right from wrong
• Morgan Stanley reports on the economic benefits of driverless cars

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

There may be times when an accident or a death is unavoidable while an autonomous car is controlling the wheel.

What should an autonomous car do when such situation arises? How should the designers of the cars program them to respond? This week, we introduce the Tunnel Problem, which describes one such situation and has been a topic of serious debate for philosophers as well as those watching the technology carefully. Let us know what you think by participating in our poll.

If you liked this article, you may also be interested in:

• Should autonomous cars be allowed to speed? Results from our reader poll
• Kids with wheels: Should the unlicensed be allowed to ‘drive’ autonomous cars?
• What do people think about autonomous cars? Results from our reader poll
• What should a robot do? Designing robots that know right from wrong
• Morgan Stanley reports on the economic benefits of driverless cars
• The autonomous driving r/evolution: Call to contribute to Robohub Focus

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.