Source: commaai

Comma.ai, the brash startup attempting to make a self-driving system entirely from a neural network has announced it will cancel the “comma one” add-on box it has planned to sell to owners of certain Honda vehicles. The box stuck on the rear-view mirror and used the car’s own bus commands to provide an autopilot similar to those offered by car makers, with lane-keeping and adaptive cruise control.

Of particular importance is the letter from NHTSA to comma.ai which I suggest you read. This letter creates several big issues:

1. There are many elements of this letter that would also apply to Tesla and other automakers which have built supervised autopilot functions.
2. Of particular interest is the paragraph that says: “it is insufficient to assert, as you do, that the product does not remove any of the driver’s responsibilities” and “there is a high likelihood that some drivers will use your product in a manner that exceeds its intended purpose.” That must be very scary for Tesla.
3. I noted before that the new NHTSA regulations appear to forbid the use of “black box” neural network approaches to the car’s path planning and decision making. I wondered if this made illegal the approach being done by Comma, NVIDIA and many other labs and players. This may suggest that.
4. We now have a taste of the new regulatory regime and had it existed before systems like Tesla’s Autopilot, Mercedes Traffic Jam Assist, and Cruise’s original aftermarket autopilot would never have been able to get off the ground.
5. George Hotz of comma declares, “Would much rather spend my life building amazing tech than dealing with regulators and lawyers. It isn’t worth it. The comma one is cancelled. comma.ai will be exploring other products and markets. Hello from Shenzhen, China.”

To be clear, comma is a tiny company taking a radical approach, so it is not a given that what NHTSA has applied to them would have been or will be unanswerable by the big guys. Because Tesla’s Autopilot is not a pure machine learning system, they can answer many of the questions in the NHTSA letter that comma can’t. They can do much more extensive testing that a tiny startup can’t. But even so, a letter like this sends a huge chill through the industry.

It should also be noted that in Comma’s photos the box replaced the rear-view mirror, and NHTSA had reason to ask about that.

Source: commaai

George’s declaration that he’s in Shenzen gives us the first sign of the new regulatory regime pushing innovation away from the United States and California. I will presume the regulators will say, “We only want to scare away dangerous innovation” but the hard truth is that is a very difficult thing to judge. All innovation in this space is going to be a bit dangerous. It’s all there trying to take the car — the 2nd most dangerous legal consumer product — and make it safer, but it starts from a place of danger. We are not going to get to safety without taking risks along the way.

I sometimes ask, “Why do we let 16-year-olds drive?” They are clearly a major danger to themselves and others. Driver testing is grossly inadequate. They are not adults so they don’t have the legal rights of adults. We let them drive because they are going to start out dangerous and then get better. It is the only practical way for them to get better, and we all went through it. Today’s early companies are teenagers. They are going to take risks. But this is the fastest and only practical way to let them get better and save millions.

“…some drivers will use your product in a manner that exceeds its intended purpose”

This sentence, though in the cover letter and not the actual legal demand, looks at the question asked so much after the Tesla fatal crash. The question which caused Consumer Reports to ask Tesla to turn off the feature. The question which caused MobilEye, they say, to sever their relationship with Tesla.

The paradox of the Autopilot is this: The better it gets, the more likely it is to make drivers over-depend on it. The more likely they will get complacent and look away from the road. And thus, the more likely you will see a horrible crash like the Tesla fatality. How do you deal with a system which adds more danger the better you make it? Customers don’t want annoying countermeasures. This may be another reason that “Level 2,” as I wrote yesterday is not really a meaningful thing.

NHTSA has put a line in the sand. It is no longer going to be enough to say that drivers are told to still pay attention.

Black box

Comma is not the only company trying to build a system with pure neural networks doing the actual steering decisions (known as “path planning”.) NVIDIA’s teams have been actively working on this, as have several others. They plan to make commentary to NHTSA about these elements of the regulations, which should not be forbidding this approach until we know it to be dangerous.

It is challenging. It’s hard to do QA on neural networks. You can examine any single state of them, but not really understand them. You can fix the errors they make, but not know how you fixed it or whether your fix is going to work in other cases. On their own that sounds too scary, but the problem is they are outperforming other algorithms at many of the problems they are being applied to.

If you have two systems:

1. A conventionally coded system which you fully understand, that has one safety incident per 100,000 miles,
2. A black box machine learning system that has one safety incident per 150,000 miles, but you have minimal understanding of.

Which is the one that is better to put on the road? It’s not a no-brainer.

Scaring people away

George Hotz is about as brash as they come, so some might discount his bluster about moving his projects to China. But this is a real thing. When regulations come, they tend to favour the big players. The big players have large QA teams and compliance departments. They can afford them, and they move conservatively because of them. Big players help write the regulations to favour their way of doing things — sometimes selfishly but sometimes just because it is the only thing they know.

The USA and California led the way in robocars in part because it was unregulated. In the USA, everything is permitted unless it was explicitly forbidden and nobody thought to write “no robots” in the laws. Progress in other countries where everything is forbidden unless it is permitted was much slower. The USA is moving in the wrong direction.

Countries like China are another animal altogether. There, everything is forbidden unless a smart autocrat allows it but once the smart (or stupid) ruling official makes it happen, the other red tape can vanish. It is not the best system overall, but it can win at things they put their minds to.

Tesla

Has Tesla received such a letter? What would they have done if they received it in their early days? While Tesla and Elon Musk love the Autopilot project, the hard truth is Tesla’s biggest problem is going to be making enough Model 3s to meet demand and doing it profitably. Taking out Autopilot isn’t going to cut their sales at all — they have too many sales and not enough cars. Only their forward vision and guts will keep them going if NHTSA does the same thing to them. I do believe Tesla can answer questions like those in the demand letter more easily than comma.ai. But not trivially, and they almost surely will need to bump the countermeasures.

The long awaited list of recommendations and potential regulations for robocars has just been released by the National Highway Traffic Safety Administration (NHTSA), the federal agency that regulates car safety and safety issues in car manufacture. Normally, NHTSA does not regulate car technology before it is released into the market, and the agency, while it says it is wary of slowing down this safety-increasing technology, has decided to do the unprecedented — and at a whopping 115 pages.

Broadly, this is very much the wrong direction. Nobody — not Google, Uber, Ford, GM or certainly NHTSA — knows the precise form these cars will have when deployed. Almost surely something will change from our existing knowledge today. They know this but still wish to move. Some of the larger players have pushed for regulation. Big companies like certainty. They want to know what the rules will be before they invest. Startups thrive better in the chaos, making up rules as they go along.

NHTSA hopes to define “best practices” but the best anybody can do in 2016 is lay down existing practices and conventional wisdom. The entirely new methods of providing safety that are yet to be invented won’t be in such a definition.

The document is very detailed, so it will generate several blog posts of analysis. Here I present just initial reactions. Those reactions are broadly negative. This document is too detailed by an order of magnitude. Its regulations begin today, but fortunately, they are also accepting public comment. The scope of the document is so large, however, that it seems extremely unlikely that they would scale back this document to the level it should be at. As such, the progress of robocar development in the USA may be seriously negatively affected.

Vehicle performance guidelines

The first part of the regulations is a proposed 15 point safety standard. It must be certified (by the vendor) that the car meets these standards. NHTSA wants the power, according to an Op-Ed by no less than President Obama, to be able to pull cars from the road that don’t meet these safety promises.

Framework for vehicle performance guidance. Credit: NHTSA, Federal Automated Vehicles Policy

• Data Recording and Sharing
• Privacy
• System Safety
• Vehicle Cybersecurity
• Human Machine Interface
• Crashworthiness
• Consumer Education and Training
• Registration and Certification
• Post-Crash Behavior
• Federal, State and Local Laws
• Operational Design Domain
• Object and Event Detection and Response
• Fall Back (Minimal Risk Condition)
• Validation Methods
• Ethical considerations

As you might guess, the most disturbing is the last one. As I have written many times, the issue of ethical “trolley problems” where cars must decide between killing one person are another are a philosophy class tool, not a guide to real-world situations. Developers should spend as close to zero effort on these problems as possible since they are not common enough to warrant special attention. If not for our morbid fascination with machines making life or death decisions in hypothetical situations. Let the policymakers answer these questions if they want to; programmers and vendors don’t.

For the past couple of years, this has been a game that’s kept people entertained and ethicists employed. The idea that government regulations might demand solutions to these problems before these cars can go on the road is appalling. If these regulations are written this way, we will delay saving lots of real lives in the interest of debating which highly hypothetical lives will be saved or harmed in ridiculously rare situations.

NHTSA’s rules demand that ethical decisions be “made consciously and intentionally.” Algorithms must be “transparent” and based on input from regulators, drivers, passengers and road users. While the section makes mention of machine learning techniques, it seems in the same breath to forbid them.

Most of the other rules are more innocuous. Of course, all vendors will know and have little trouble listing what roads their car works on, and they will have extensive testing
data on the car’s perception system and how it handles every sort of failure. However, the requirement to keep the government constantly updated will be burdensome. Some vehicles will be adding streets to their route map literally ever day.

While I have been a professional privacy advocate, and I do care about just how the privacy of car users is protected, I am frankly not that concerned during the pilot project phase about how well this is done. I do want a good regime — and even the ability to do anonymous taxi — so it’s perhaps not too bad to think about these things now, but I suspect these regulations will be fairly meaningless unless written in consultation with independent privacy advocates. The hard reality is that during the test phase, even a privacy advocate has to admit that the cars will need to make very extensive recordings of everything they can, so that any problems encountered can be studied and fixed and placed into the test suite.

50 state laws

NHTSA’s plan has been partially endorsed by the self-driving coalition for safer streets (whose members include big players Ford, Google, Volvo, Uber and Lyft.) They like the fact that it has guidance for states on how to write their regulations, fearing that regulations may differ too much state to state. I have written that having 50 sets of rules may not be that bad an idea because jurisdictional competition can allow legal innovation and having software load new parameters as you drive over a border is not that hard.

In this document, NHTSA asks the states to yield to the DOT on regulating robocar operation and performance. States should stick to registering cars, rules of the road, safety inspections and insurance. States will regulate human drivers as before, but the feds will regulate computer drivers.

States will still regulate testing, in theory, but the test cars must comply with the federal regulations.

New authorities

A large part of the document just lists the legal justifications for NHTSA to regulate in this fashion and is primarily for policy wonks. Section 4, however, lists new authorities NHTSA is going to seek in order to do more regulation.

Some of the authorities they may see include:

• Pre-market safety assurance: Defining testing tools and methods to be used before selling
• Pre-market approval authority: Vendors would need approval from NHTSA before selling, rather than self-certifying compliance with the regulations
• Hybrid approaches of pre-market approval and self-certification
• Cease and desist authority: The ability to demand cars be taken off the road
• Exemption authority: An ability to grant rue exemptions for testing
• Post-sale authority to regulate software changes
• Much more

Other quick notes:

• NHTSA has abandoned their levels in favour of the SAE’s. The SAE’s were almost identical of course, with the addition of a “level 5” which is meaningless because it requires a vehicle that can drive literally everywhere, and there is not really a commercial reason to make a car at present that can do that.
• NHTSA is now pushing the acronym “HAV” (highly automated vehicle) as yet another contender in the large sea of names people use for this technology. (Self-driving car, driverless car, autonomous vehicle, automated vehicle, robocar etc.)

More analysis is on the way.

The National Highway Traffic Safety Administration (NHTSA) took an important step towards interpreting safety standards in ways that could make self-driving cars a reality for the public: it said that computers controlling those cars can be considered drivers just like humans.

The NHTSA said that it would write guidelines for self-driving cars within six months. Their answer was specifically given in response to a detailed request from Google, and was posted online so all could see.

Their answer to over 20 questions was that the SDS (self-driving computer and AI system) is the driver. But when it came to eliminating brake pedals, the NHTSA said that safety regulations and laws would need to be changed before brakes could be removed from the passenger compartment.

The question of who is responsible for a self-driving car is an important starting point for legal, insurance, and safety considerations. Setting the computer (and thus the maker of the computer and software) as the “driver” and as the responsible person/entity, is the first step along a long pathway toward establishing all the new rules for self-driving cars – including eliminating the need for brake pedals controlled by the human driver.

In a recent speech in Washington DC, the president of Volvo Cars, Hakan Samuelsson, said that the US is currently “the most progressive country in the world in autonomous driving… but it risks losing its leading position” because of the lack of Federal guidelines for the “testing and certification” of autonomous vehicles. Unanswered ethical and legal concerns, particularly when it comes to liability for driverless car accidents, are also stalling development. Thus the NHTSA's promise to write guidelines for self-driving cars within six months, and their defining the SDS as a driver, are both good signs.