Tesla Model S autopilot-software. Source: Tesla

NHTSA released the report from their Office of Defects Investigation on the fatal Tesla crash in Florida last spring. It’s a report that is surprisingly favorable to Tesla. So much so that even I am surprised.

While I did not think Tesla would be found defective, this report seems to come from a different agency than the one that recently warned comma.ai that:

It is insufficient to assert, as you do, that the product does not remove any of the driver’s responsibilities” and “there is a high likelihood that some drivers will use your product in a manner that exceeds its intended purpose.

The ODI report rules that Tesla properly considered driver distraction risks in its design of the product. It goes even further, noting that drivers using Tesla Autopilot (including those monitoring it properly and those who did not) still had a decently lower accident rate for mile than drivers of ordinary cars without autopilot. In other words, while the Autopilot without supervision is not good enough to drive on its own, the Autopilot even with occasionally lapsed supervision that is known to happen is still overall a safer system than not having the Autopilot at all.

This will provide powerful support for companies developing autopilot style systems, and companies designing robocars who wish to use customer supervised driving as a means to build up test miles and verification data. They are not putting their customers at risk as long as they do it as well as Tesla. This is interesting (and the report notes that evaluation of Autopilot distraction is not a settled question) because it seems probable that people using the Autopilot and ignoring the road to do e-Mail or watch movies are not safer than regular drivers. But the overall collection of distracted and watchful drivers is still a win.

This might change as companies introduce technologies which watch drivers and keep them out of the more dangerous inattentive style of use. As the autopilots get better, it will become more and more tempting, after all.

Tesla stock did not seem to be moved by this report. But it was also not moved by the accident or other investigations — it actually went on a broadly upward course for 2 months following the announcement of the fatality.

The ODI’s job is to judge if a vehicle is defective. That is different from saying it’s not perfect. Perfection is not expected, especially from ADAS and similar systems. The discussion about the finer points of whether drivers might over-trust the system are not firmly settled here. That can still be true without the car being defective and failing to perform as designed or being designed negligently.

Tesla Model S autopilot-software. Source: Tesla

In this article, Brad Templeton provides a rundown of different approaches for validation of self-driving and driver assist systems, a recommendation to Tesla and others to have countermeasures to detect drivers not watching the road, and permanently disable their Autopilot if they show a pattern of inattention.

The recent fatality for a man driven by the Tesla “autopilot” system has ignited debate on whether it was appropriate for Tesla to allow their system to be used as it was.

Tesla’s autopilot is a driver assist system, and Tesla tells customers it must always be supervised by an alert driver ready to take the controls at any time. The autopilot is not a working self-driving car system, and it’s not rated for all sorts of driving conditions, and there are huge numbers of situations that it is not designed to handle and can’t handle. Tesla knows that. But the public, press, and Tesla customers forget that and there are many Tesla users who are treating the autopilot like a real self-driving car system, and who are not paying attention to the road — and Tesla is aware of that as well. The press made this mistake as well, regularly writing fanciful stories about how Tesla was ahead of Google and other teams.

Brown, the driver killed in the crash, was very likely one of those people, and if so, he paid for it with his life. In spite of all the warnings Tesla may give about the system, some users
do get a sense of false security. There is an ongoing debate if that means driver assist systems are a bad idea.

There have been partial self-driving systems that require supervision since the arrival of the cruise control. Adaptive cruise control is even better, and other car companies have released autopilot like systems which combine adaptive cruise control with lane-keeping and forward collision avoidance, which hits the brakes if you’re about to rear-end another car. Mercedes has sold a “traffic jam assist” like the Telsa autopilot since 2014 that only runs at low speeds in the USA. You can even go back to a Honda demo in 2005 of an autopilot like system.

With cruise control, you might relax a bit but you know you have to pay attention. You’re steering and for a long time even adaptive cruise controls did not slow down for stopped cars. The problem with Tesla’s autopilot is that it was more comprehensive and better performing than earlier systems, and even though it had tons of things it could not handle, people have started to trust it with their lives.

Tesla’s plan can be viewed in several ways. One view is that Tesla was using customers as
“beta testers,” as guinea pigs for a primitive self-drive system which is not production ready — and that this is too much of a risk. Another is that Tesla built (and tested) a superior driver assist system with known and warned limitations and customers should have listened to those warnings.

Neither is quite right. While Tesla has been clear about the latter stance, with the knowledge that people will over-trust it, we must face the fact that it is not only the daring drivers who are putting themselves at risk, it’s also others on the road who are put at risk by the over-trusting drivers — or perhaps by Tesla. What if the errant car had not gone under a truck, but instead hit another car, or even plowed into a pedestrian when it careened off the road after the crash?

At the same time, Tesla’s early deployment approach is a powerful tool for the development and quality assurance of self-drive systems. I have written before about how testing is the big unsolved problem in self-driving cars. Companies like Google have spent millions with a staff of paid drivers testing their cars for 1.6 million miles. This is massively expensive and time consuming, and even Google’s money can’t easily generate the billions of miles of testing that some feel might be needed. Human drivers will have about 12 fatalities in a billion miles, and we want our self-driving cars to do much better. Just how we’ll get enough verification and testing done to bring this technology to the world is not a solved problem.

While we don’t want a repeat of the things that led to the death of this driver, there may be greater value to the field by having fleets of volunteers help verify these vehicles and if it can be done safely, the world probably needs it. But even if there is no way to do volunteer testing of systems that are still being refined — and they will still be in the being refined state for many years to come — there is still the question of whether you can have systems that need human supervision and intervention that get so good that humans ignore warnings and stop bothering to supervise them.

Here’s a look at a variety of approaches to testing, verifying, refining and learning:

In Simulator

The only way many billions of miles will be done is in a simulator, especially since you want to re-test all new software builds as much as you can before deploying. In a simulator you can test all sorts of risky situations you could never readily test in the real world. That’s why many years ago I laid out a plan for an open source simulator because I think this would be a great thing for everybody to contribute to. An open source simulator would soon feature every crazy situation anybody has even seen or thought of, and it would make all cars safer.

But simulation is just simulation, and it only features what people have thought of. You need more.

On test tracks

The most conservative, but most expensive and limited approach, is a private test track. And almost everybody starts on a closed testing ground because you need to when things are just getting going. They provide the confidence to eventually go out on the road — and go out on the road you must. You can really only do a fraction of a percent of your testing on test tracks, even with robot slave cars to drive along with. You just won’t encounter enough variation in terrain and driving situations, and you can only test a few cars at a time.

On the plus side, on test tracks you can use unmanned cars, reducing some costs, and you can run 24 hours/day.

There is a hybrid of simulator and test track called “Vehicle in the Loop” which puts the car on a dyno in a building while robots whiz around to provide relative motion. That requires less real estate and is safer, but automated test track operation is probably better at this point.

On the road with trained professional safety drivers

All companies eventually need to go out on real roads and deal with real situations. They do it by having paid professionals, usually with special training, who sit behind the wheel watching the vehicle and ready to take the controls at any sign of doubt. It is also common to have a second person who is monitoring the computer systems and sensor scans to make sure all is operating properly, and to tell the safety driver to take over if something is wrong. In fact, safety drivers are required for testing self-driving cars (but not autopilots/cruise software) by most of the state laws about self-driving cars.

Companies like Google tell their safety drivers to take control at any sign of doubt. They don’t want them to let the car keep driving to see if it would have braked properly for the children crossing the road. Later, they can use the simulator to replay what the car would have done if the safety driver had not intervened. If the car would have done something wrong, it’s a bug to be fixed at high priority.

Still, even trained safety drivers can make mistakes. Google has reported one accident where their systems made a mistake. The system decided that a bus it was sharing the lane with would let it in to traffic, but the bus did not. The safety driver didn’t intervene because he also judged the bus would let the car in.

In addition, when the systems are very good, safety drivers can also become complacent and not intervene as quickly as they should.

On the road with regular drivers but limited function

You could test a self-driving system by having it do only some of the driving. For example, you could make a system which steers but does not control the brakes. A driver who knows that she has to hit the brakes when approaching something is not going to stop paying attention. With this system you can note when the human hit the brakes, and see if the system also would have, and who was right. Chances are any surprise braking would be combined with taking the wheel. Of course, you would also learn when the driver grabbed the wheel.

WEpods: The first autonomous vehicle on Dutch public roads. Source: True Form/YouTube

We have the first report of a real Tesla autopilot crash. To be fair to Tesla, their owner warnings specify very clearly that the autopilot could crash in just this situation. In the video, there is a stalled car partly in the lane and the car in front swerves around, revealing little time for the driver, or the autopilot, to react.

The deeper issue is the way that the improving quality of the Tesla Autopilot and systems like it are lulling drivers into a false sense of security. I have heard reports of people who now are trusting the Tesla system enough to work while being driven, and indeed, most people will get away with this. And as people get away with it more and more, we will see people driving like this driver, not really prepared to react. This is one of the reasons Google decided not to make a system which requires driver takeover ever. As the system gets better, does it get more dangerous?

Declaration of Amsterdam

Last month, various EU officials gathered in Amsterdam and signed the Declaration of Amsterdam that outlines a plan for normalizing EU laws around self-driving cars. The meeting included a truck automation demo in the Netherlands and a self-drive transit shuttle demonstration. It’s a fairly bland document, more an expression of the times, and it sadly spends a lot of time on the red herring of “connected” vehicles and V2V/V2I, which governments seem to love, and self-driving car developers care very little about.

Let’s hope the regulatory touch is light. The reality is that even the people building these vehicles can’t make firm pronouncements on their final form or development needs, so governments certainly can’t do that, we must be careful that attempts to “help” but may hinder. We already have a number of examples of that happening in draft and real regulations and we’ve barely gotten started. For now, government statements should be limited to, “let’s get out of the way until people start figuring out how this will actually work, unless we see somebody doing something demonstrably dangerous that can’t be stopped except through regulations.” Sadly, too many regulators and commentators imagine it should be, “let’s use our limited current knowledge to imagine what might go wrong and write rules to ban it before it happens.”

Speech from the throne

It was a sign of the times when her Majesty the Queen, giving the speech from the throne in the UK parliament, laid out elements of self-driving car plans. The Queen drove jeeps during her military days, and so routinely drives herself at her country estates, otherwise she would be among the set of people most use to never driving.

The UK has 4 pilot projects in planning. Milton Keynes is underway, and later this year, a variation of the Ultra PRT pods in use at T5 of Heathrow airport — they run on private tracks to the car park — will go out on the open road in Greenwich. They are already signing up people for rides.

 Car companies thinking differently

In deciding which car companies are going to survive the transition to robocars, one thing I look for is willingness to stop thinking like a traditional car company which makes cars and sells them to customers. Most car company CEOs have said they don’t plan to keep thinking that way, but what they do is more important than what they say.

In the past we’ve seen Daimler say it will use their Car2Go service (with the name Car2Come probably likely to cause giggles in the USA) as a way to sell rides rather than cars. BMW has said the same about DriveNow. (And now GM has said this about its partnership with Lyft.) Daimler is also promoting their moovel app which tries to combine different forms of mobility, and BMW is re-launching DriveNow in Seattle as ReachNow which adds peer-to-peer carsharing and other modes of transportation to the mix.

Of course, these are tiny efforts for these big companies, but it scores big over companies still thinking only in the old ways. I’m looking at you, most car companies.

BWM also announced the iNext electric flagship sedan will offer self-drive in 2021.

Florida tells cities — think about self-driving

In 2010, I put out a call for urban planners to starting thinking about robocars and mostly it fell on deaf ears. Times are changing and this month Florida told its cities they should include consideration of this and other future transit in their updated long-term plans.

It is a tough call. Nobody’s predictions about the future here are good enough to make a firm plan and commit billions of dollars. At the same time, we are starting to learn that certain plans — especially status quo plans — are almost certainly seriously wrong. We may not have a perfect idea on what to spend city money on, but we can start to learn what not to do.

Fortunately, transportation is becoming a digital technology, which means you can change your plans much faster than physical infrastructure plans. Robocars like bare pavement, as stupid as can be. The ‘smarts’ go in the cars, not in the roads or the cities. So if you change your mind, you just have to reprogram your cars, not rebuild your city. Which is good, because you can’t rebuild your city.

China gets in the game

China is the world’s number one car manufacturing company. A lot of people don’t know that. Last year I visited the Shanghai auto show and it was a strange trip to walk through giant hall after giant hall of automakers you have never heard of before.

Self-driving car action in China has been slow. Right now everybody has a focus on just making regular cars for the rising middle class that is buying them as fast as they can. Wealthier Chinese usually buy foreign brands, although those cars are often made in China even though they have a VW or Buick nameplate.

Baidu has been working on cars for a couple of years and promises a pilot project in 2018. Recently, Chinese automaker Changan did an autopilot demo driving 1,000 miles. China has had its own annual academic version of the Darpa Grand Challenge for several years as well.

There is a strong chance that companies like Uber, Apple or Google, when they want to get their cars made, could go to Chinese manufacturers. Now that they’re getting practice at making their own technology. China is also a major source of electric vehicles, with future robotaxis likely to be small and electric. The manufacturers and suppliers with the most experience at making such vehicles are likely to be the winners.

Apple buys into Didi

Speaking of which, Apple just put a billion dollar investment into Didi. You may not know Didi, but it is the dominant phone-hail service in China with a much larger market share than Uber. It’s one of the few places Uber has lost in the market, but of course it’s China. As the auto industry moves to being about selling rides rather than cars, it’s an interesting move by Apple, which rarely does outside investments like this.