Here’s my annual summary of the top stories of the prior year. This time the news was a strong mix of bad and good.

Read the text story on Forbes.com at Robocars 2022 year in review.

And see the video version here:

California has released the disengagement reports the law requires companies to file and it’s a lot of data. Also worth noting is Waymo’s own blog post on their report where they report their miles per disengagement has improved from 5,600 to 11,000.

Fortunately some hard-working redditors and others have done some summation of the data, including this one from Last Driver’s Licence Holder. Most notable are an absolutely ridiculous number from Apple, and that only Waymo and Cruise have numbers suggesting real capability, with Zoox coming in from behind.

The problem, of course, is that “disengagements” is a messy statistic. Different teams report different things. Different disengagements have different importance. And it matters how complex the road you are driving is. (Cruise likes to make a big point of that.)

Safety drivers are trained to disengage if they feel at all uncomfortable. This means that they will often disengage when it is not actually needed. So it’s important to do what Waymo does, namely to play back the situation in simulator to see what would have happened if the driver had not taken over. That playback can reveal if it was:

• Paranoia (as expected) from the safety driver, but no actual issue.
• A tricky situation that is the fault of another driver.
• A situation where the vehicle would have done something undesired, but not dangerous.
• A situation like the above, but dangerous, though nothing would have actually happened. Example — temporarily weaving out of a lane when nobody else is there.
• A situation which would have resulted in a “contact” — factored with the severity of the contact, from nothing, to ding, to crash, to injury, to fatality.

A real measurement involves a complex mix of all these, and I’ll be writing up more about how we could possibly score these.

We know the numbers for these events for humans thanks to “naturalistic” driving studies and other factors. Turns out that humans are making mistakes all the time. We’re constantly not paying attention to something on the road we should be looking at, but we get away with it. We constantly find ourselves drifting out of a lane, or find we must brake harder than we would want to. But mostly, nothing happens. Robots aren’t handled that way — any mistake is a serious issue. Robocars will have fewer crashes because “somebody else was in the wrong place when I wasn’t looking.” Their crashes will often have causes that are foreign to humans.

In Waymo’s report you can actually see a few disengagements because the perception system didn’t see something. That’s definitely something to investigate and fix, but humans don’t see something very frequently, and we still do tolerably well.

A summary of the numbers for humans on US roads:

• Some sort of “ding” accident every 100,000 miles of driving (roughly).
• An accident reported to insurance every 250,000 miles.
• An accident reported to police every 500,000 miles.
• An injury accident every 1.5M miles.
• A fatality every 80M miles of all driving.
• A highway fatality every 180M of highway driving.
• A pedestrian killed every 600M miles of total driving.

Software disengagements

The other very common type of disengagement is a software disengagement. Here, the software decides to disengage because it detects something is going wrong. These are quite often not safety incidents. Modern software is loaded with diagnostic tests, always checking if things are going as expected. When one fails, most software just logs a warning, or “throws an exception” to code that handles the problem. Most of the time, that code does indeed handle the problem, and there is no safety incident. But during testing, you want to disengage to be on the safe side. Once again, the team examines the warning/exception to find out the cause and tries to fix it and figure out how serious it would have been.

That’s why Waymo’s 11,000 miles is a pretty good number. They have not published it in a long time, but their number of “necessary interventions” is much higher than that. In fact, we can bet that in the Phoenix area, where they have authorized limited operations with no safety driver, that it’s better than the numbers above.

The Bell Helicopter tiltrotor, ducted fan hybrid aircraft had a giant crowd when the hall was open.

My feet are aching, as usual, after 3 days on the CES show floor, and the question people always ask others there is “what have you seen that was interesting?”

I won’t say I didn’t see anything interesting, and I had a large number of rewarding conversations with all sorts of companies, making the trip very worthwhile, but I will say I saw less that was new and exciting than ever before. This may be a result of the show’s constant growth that meant in 3 days I still did not manage to get to 3 1/2 major rooms of the show, putting my focus on cars as I usually do.

A larger fraction of CES is not even remotely consumer electronics. In fact, the reason I go is largely the robocar related companies, none of whom are selling to consumers at this time, and most of which never plan to. And yes, the amazing new TVs from Samsung, LG and others are always a draw — LG always creates an astonishing video wall, and all companies were now displaying 8K TVs as a real product (even if a very expensive one for which no video sources are available, but which will be fantastic for still photos.)

My favourite part of the show, Eureka Park, even disappointed even though I gave it most of a day. The proportion of products which are poorly thought through is always high, but you search for gems that might someday change the world. I saw few Eureka moments.

Several fields, such as Iot/Smarthome were growing and maturing, with lots more companies, but at the same time, very little in smart home moves beyond the “Oh, that sounds nice” into truly appealing and useful. I’ve had computer controlled lights for decades. They are handy, and I would rather have them than not have them, but they aren’t essentials the way my phone and a dozen apps within it are, things I can’t do without any more.

There were, again, a ton of different LIDAR companies. What’s interesting about the LIDAR crowd is that they are almost all quite different, in many cases taking entirely different approaches to the technology — the output, the steering, the sensing, the band and even the core technology. I’ll have more coverage of that later. Of course these are not for consumers.

Other press were equally unexcited — I saw far less coverage of the “hot new thing” in the media this year as the show opened and progressed. This was the year of mostly incremental improvements.

Flying car style vehicles did generate excitement, but they were barely there. The most buzz was about Bell helicopter’s “Nexus” tilt-rotor hybrid vehicle on display. It’s a high-end vehicle, with 6 giant ducted fan tilt-rotors, and very much a prototype. Bell knows a lot about tilt-rotor from their work on the V-22 Osprey, a vehicle which has a rather checkered history, so it’s not sure how this will go. The few other companies doing flying vehicles (for people) had tiny booths showing nothing.

The Nexus is a hybrid-hybrid — both multirotor and fixed-wing flight, and both electric batteries and gasoline powered generator for power.

There was some interesting battery technology, including a small startup named GBatteries which claimed that by pulsing the charge voltage they could charge at vastly faster speeds — 5C, or 10 minutes from empty to 80%, and 5 minutes to 50%, without any more heat or battery damage than today’s fast charging (which takes about an hour.) Their demo was just on a drill, so it is yet to be proven in a car, but it would change some of the usability of battery devices and electric cars.

I saw other interesting battery tech but there is some irony – because of all the investment made by Tesla and others in standard lithium ion cells, these cells are now better than other, more innovative technologies, because of their mass production (driving down the cost) and the heavy R&D (improving charge times and reducing battery degradation.) The lithium ion cell was created to be lightweight for laptops where weight is absolutely crucial, but now it’s everywhere, from cars to scooters to even grid storage.

There was an uptick of mobility products (or at least I am paying the most attention to them) beyond cars, including scooters, delivery, bike riding and more.

In spite of what most are writing, it was a year of much progress.

A number of other summaries of 2018 in robocars have called it a bad year, the year it all went south, even the year the public realized that robocars will never come.

In fact, 2018 was the year the field reached a new level of maturity, as its warts began to show, and we saw the first missteps (minor and major) and the long anticipated popping of some of the hype.

As predicted by Gartner’s famous “hype cycle” any highly-hyped technology goes through a “trough of disillusionment” after the initial surge.I see several reasons why the trough is happening now:

• The public is starting to understand some realities which have not been very well conveyed to them, though they were known by the major teams:
  • This is a very hard task
  • It is geographic in nature, due to the need of mapping and local driving rules, and so it begins in limited areas and grows from there.
  • The amount of QA needed to get to unmanned operation is immense, and if you have money, there is no reason to remove safety drivers until you’re quite sure.
  • The so called “level 5” isn’t on any serious roadmap, and may never happen
• Cars showed up at the peak of Gartner’s very own chart a couple of years ago, so it’s just about time in their rulebook
• It’s very typical in software for dates to slip.People predict when they hope they can make a target, which is really the beginning of the time window in their estimate.
• Some people, such as Elon Musk, have made irresponsibly optimistic predictions, leading the public to imagine their drive-everywhere robocar is coming next year.
• If you follow Gartner’s literal curve, they had robocars at the top in 2015.So the trough doesn’t really even need a reason.But I think there will be multiple peaks and troughs.

In reality, the plan at Waymo and many other companies has always been to build a car that serves a limited set of streets — a small set of easy streets — at the start, and then starts growing the network of streets, and eventually towns with streets, as time goes by.

There is going to be a big surge once the technology reaches a level where the remaining problems are no longer technological as logistic.That is to say, when the barrier to expanding to new streets and cities is the detailed work of mapping those streets, learning the local rules and working with local governments.That’s when the “land rush” happens.The limiting factor there is time and talent more than it’s money.

But none of that happens until the cars are ready for deployment, and until they are, they will be tested as prototypes with safety drivers in them.Even the first prototype services, like Waymo’s and Zoox’s and others, will have safety drivers in them.

The Uber fatality — the top story of the year

No question the big story this year was the death of Elaine Herzberg as the result of a compound series of errors and bad practices at Uber.The story is notable for many reasons, including of course how it happened, but also in the public’s reaction. For a long time, I’ve been assured by many skeptics that the first death would mean the end of the robocar dream.The public actually thinks the first deaths were in Teslas (they weren’t) and Tesla stock went up after they took places. The Uber fatality was real, and did teach us that teams are capable of more negligence than I had thought. While it did scale up public distrust, and Uber did shut down their program for at least a year, the overall effect still seems modest.(The larger effect will be much greater intolerance for the next fatality, the one that would have been the first.)

Here’s some of my many posts on Uber this year:

• It certainly looks bad for Uber
• How does a robocar see a pedestrian and how might Uber have gone wrong?
• Comparing the Uber and Tesla fatalities with a table
• Uber reported to have made an error tuning perception system
• NTSB Report implies serious fault for Uber in fatality
• Did Uber really botch the switch to one driver and a timeline of the failure
• Almost every thing that went wrong in the Uber fatality is both terrible and expected
• Don’t watch TV while safety driving

Story #2 — Waymo’s non-launch

Waymo remains the clear leader in the field, so the next top story has to be about them, but sadly it’s the story of their first miss — promising to launch in 2018 and feeling forced to do a “launch” that was really just a formalization of existing activity. I believe that Uber is partly to blame here, in that it did use up a lot of the public’s tolerance for errors, especially in the Phoenix area.Waymo soft launches in Phoenix, but…

The better story for Waymo, however, was their first totally unmanned operations earlier in the year.This also disappointed people because these unmanned operations were on a much more limited scale than people originally imagined, but it’s still a major milestone.It means Waymo’s team convinced the lawyers and board that the systems were good enough to take this risk, even if only in a limited area.
Waymo goes totally unmanned, arbitration and other news

• Uber and Waymo settle lawsuit in a giant victory for Uber
  The Waymo-Uber battle, triggered by Anthony Levandowski, was concluded this year.
• Anthony Levandowski, Jiajun Zhu and Dave Furguson, all formerly of Google Car, make big announcements Anthony even came back, and other alumni of Waymo/Google Car had their milestones.Nuro’s launch of a truly unmanned road robot is also significant.
• Aurora’s manifesto is worth a readAurora, lead by former Waymo leader Chris Urmson, laid out their excellent philosophy well.
• Recent Waymo announcements are slightly underwhelming
• Waymo’s left turns frustrate other drivers

Flying Cars

This was also the year that “flying cars” also known as e-VTOL aircraft, “took off.”It’s now clear the engineering problems are close to solved, though many social and logistic problems remain.These vehicles are at the stage robocars were 10 years ago, and the excitement is building.Sebastian Thrun, the modern “father of self-driving cars” and the man who first got me excited about them, has switched his efforts to flying.I’ll be writing more on this in the coming year.

Other notable news

In chronological order, not order of importance

Lidars and other sensors
• Tons of LIDARs at CES 2018Sensors were a big story this year
• New “Shared Mobility Principles” have too much 2018 thinking.
• FMCW LIDAR is coming, what does it mean?
• Nice summary of LIDAR technologies — is it a “crutch?”
• Robocars enter phase two as different strategies aboundWhat approaches are the teams taking?
• The GM/Cruise robocar interior is refreshingly spartan What robocar interiors will look like.
• All about sensors: Advanced radar and more for the future of perception
• Local Motors hopes to win with 3D printed robocars A different, small volume approach
• Cruise getting a ticket is a very positive story Cruise got too close to a pedestrian, or did it?
• Tesla model X fatality in Silicon Valley had Autopilot turned on We learn more about what Tesla autopilot isn’t.
• Tesla makes a custom neural network chip, is that wise?A first taste of robocar processor battles.
• Google gets a bunch of news — and that is news.Also, contract manufacturing
• Tim Kentley-Klay suddenly ousted from Zoox See also my analysis of Zoox, written just before this.
• New NHTSA regulations, Waymo miles and big deals

My essays on the issues

The main focus of this site are my essays on the issues and future of robocars.Here are the ones from this year I think you will find most valuable.

• Designing a metric to measure robocar safety — what does insurance teach? I examine the hard problem of measuring the safety level of a robocar
• Making tunnels for robocars would be vastly cheaper than subways for trains like SF’s new Central SubwayOnce we have robocars, we should rethink even what we put in private right-of-way like subway tunnels
• The paradox on robocar accidentsA fundamental change of thinking for understanding trends in accidents
• NHTSA/SAE’s “levels” of robocars may be contributing to highway deaths
• Driving without a map is another example of being cheap rather than being safeA mistake I think a few of the major teams are making.This is not the time to be cheap, this is the time to be safe as quickly as you can.
• Safety Drivers for Robocars — the issues and rationaleIn light of Uber’s mistakes, how should we think of the role of the safety driver?
• No, ads won’t pay for your robotaxi ride — but your employer might, and that has big consequencesDebunking one of the sillier ideas out thre, but realizing the basic concept may mean employers pay for commutes, which affects a lot of the economics of travel.
• The road trip robocar and tourist robocarWhat a robocar meant for tourists and road trips might be like.
• Is BRT the best answer for bewildered city planners?City planners should future-proof their urban plans by using BRT instead of LRT or Subways.
• Is the Zoox plan for a custom car the path to robotaxi domination?I talk about Zoox’s radical plan and wonder if it’s smart.
• Sharing the ride:Less sharing is better for transit, more sharing better for carsBreaking up the biggest myth of transit planning, that bigger is better.
• The death of parking services has already begunRobocars change parking greatly.Uber has already started that change.
• Calculating all the externalities of drivingWhat we need to do to really plan transportation well.
• The Spot Market in ParkingA plan for how parking will get sold in the future.
• Cars will go to the chargers, you don’t need to bring the chargers to the carsSolving the charging infrastructure problem for electric cars.
• What does airline competition tell us about robotaxi competition?
• Sleeper carsThe first of two essays on cars designed for long-haul overnight trips.
• Sleeper cars and the unexpected efficiency of solo transport
• The dance between pedestrians and robocarsHow should cars and pedestrians interact?
• The future timeline of robocars — 2020s land rush, 2030s maturityThe answer to that ever-popular question of “When will the robocars arrive?”
• Will people with robocars hire them out as taxis when they are not using them?Breaking apart the economics of sharing your private robocar.
• What processors will be important in robocars?Is the battle for the chip market important?
• What happens to human driven cars in the robocar world?No, we don’t kick the humans off the road for a long time.
• Waymo is first, but is Cruise second, and how can you tell?Why it’s hard to rank the teams.
• Study claims robocars increase pollution; they could not be more wrong
• The utilitarian math overwhelming says we should be aggressive in robocar development.How do we do that?A look into the moral calculus of taking risks in order to save lives.

Waymo car drives in Tempe

Waymo announced today they will begin commercial operations in the Phoenix area under the name “Waymo One.” Waymo has promised that it would happen this year, and it is a huge milestone, but I can’t avoid a small bit of disappointment.

Regular readers will know I am a huge booster of Waymo, not simply because I worked on that team in its early years, but because it is clearly the best by every metric we know. However, this pilot rollout is also quite a step down from what was anticipated, though for sensible reasons.

1. At first, it is only available to the early rider program members. In fact, it’s not clear that this is any different from what they had before, other than it is more polished and there is a commercial charging structure (not yet published.)
2. Vehicles will continue to operate with safety drivers.

Other companies — including Waymo, Uber, Lyft and several others — have offered limited taxi services with safety drivers. This service is mainly different in its polish and level of development — or at least that’s all we have been told. They only say they “hope” to expand it to people outside the early rider program soon.

In other words, Waymo has missed the target it set of a real service in 2018. It was a big, hairy audacious target, so there is no shame or surprise in missing it, and it may not be missed by much.

There is a good reason for missing the target. The Uber fatality, right in that very operation area, has everybody skittish. The public. Developers. Governments. It used up the tolerance the public would normally have for mistakes. Waymo can’t take the risk of a mistake, especially in Phoenix, especially now, and especially if it is seen it came about because they tried to go too fast, or took new risks like dropping safety drivers.

I suspect at Waymo they had serious talks about not launching in Phoenix, in spite of the huge investment there. But in the end, changing towns may help, but not enough. Everybody is slowed down by this. Even an injury-free accident that could have had an injury will be problematic — and the truth is, as the volume of service increases, that’s coming.

It was terribly jarring for me to watch Waymo’s introduction video. I set it to play at one minute, where they do the big reveal and declare they are “Introducing the self driving service.”

The problem? The car is driving down N. Mill Avenue in Tempe, the road on which Uber killed Elaine Herzberg, about 1,100 feet from the site of her death. Waymo assures me that this was entirely unintentional — and those who live outside the area or who did not study the accident may not recognize it — but it soured the whole launch for me.

Cruise’s spartan interior

A recent Reuters story suggests Cruise is well behind schedule with one insider saying “nothing is on schedule” and various reports of problems not yet handled. This puts doubt into GM’s announced plan to have a commercial pilot without safety drivers in operation in San Francisco in 2019.

The problem for me, and everybody else, is that it’s very hard to judge the progress of a project from outside. This is because it’s “easy” to get a basic car together and do demo runs on various streets. Teams usually have something like that up and running within a year. Just 2 years in, Google had logged 100,000 miles on 1,000 different miles of road. Today, it’s even easier.

As such, you can see lots of miles logged, and you can take a test ride where nothing eventful happens, and be impressed, but learn very little.

You learn more if you can get detailed statistics, but even there it’s difficult. Only the people on the team in charge of measuring quality have a really good sense of it, and sometimes not even them. Even high quality vehicles will be having minor problems from time to time, temporarily not perceiving things, or being too conservative in their driving. After all, humans have tons of minor problems all the time. We’re always temporarily not paying attention, missing minor things, drifting in lanes, hard braking for surprises and more, but we still make the bar.

This is one reason I declared Waymo’s (very limited) deployment with no safety drivers as the big story of 2017. To do that, their team had to get the lawyers and board to sign off on that risky decision. I wasn’t there, but I will guess that the Alphabet board itself had to sign off on that decision. Which means they had to have a pretty good case.

As such, Cruise’s announced plan for 2019 was the 2nd boldest plan announced with any credibility. (I have never given that much credit to Elon Musk’s announcements on Tesla full self driving, and even Tesla has dialed them back. Elon’s a good guy but he is a little fast and loose with the predictions.)

I believe that before Cruise actually deploys like this it will probably require a GM board level approval, but possibly not. GM’s name and reputation will be on the line.

As such, the most compelling information about how confident a team is in their vehicle is real “put your money and reputation on the line” moves. Of course, the smaller startups do not have the same sort of reputations to put on the line.

Since there are not very many such declarations, we must look for other things. The sheer amount of driving is one metric. The reported disengagements required by California are a popular thing to look at, but each company has a different methodology about how to count them and what they mean.

In addition, I and other skilled people can make judgements by getting a chance to get to really know a team, and ride in the car while watching un-canned visualizations of the quality of its perception and other systems. We often see perception videos released (with bounding boxes and labels on obstacles) but these are usually vetted in advance. I have at times done just this with some of the cars I have ridden in. Perhaps I or others should start an independent service verifying claims. The problem, of course, is that the team should know its own numbers already. Since the planned regimen for regulation is self-certification, that’s what will matter most.

Cruise’s 2019 date is ambitious, as they are half the age of Waymo. Most other forecast dates are in the 2020s and not very real — they are just aspirations, and were not signed off on after some specific process. And we should expect any dates we get to be aspirations, “We hope we could do it by such-and-such date.” In reality, it is likely to slip, but nobody knows how much it will slip.

Toyota has set a hard deadline with the planned service for the Olympics, but while they can’t move the date of the games, they can change the parameters of the service. I think Waymo changed their parameters after the Uber fatality, which used up a lot of the public’s tolerance for error in Phoenix.

This is related to, but different from, the core question of a metric for proving safety which I have discussed a number of times. While the metric may be the final step to certifying a car for operation on the road, you don’t need the final metric to get a sense of where teams are.

But what of the driver of the regular car? What lies ahead for those who love driving and want to own a traditional car? I often see people declare that nobody will own cars in the future, and that human driving will even be banned. Is that realistic?

Nobody restricts human driving for quite some time

The transition to robocars must be gradual, one car at a time, at least in most of the world. That means lots of human driven cars on all the roads for decades to come.

Some people predict that human driving will quickly be banned. This won’t happen in most places simply because there will still be lots of places robocars don’t go because it’s not commercially viable to certify them there. In addition, there will be strong political opposition. At a rough guess, around 1/3rd of people never have a car accident in their lives. What is the justification in taking away their licences?

When I give talks on robocars, I usually get some people telling me they can’t imagine why anybody would drive or own a car in the future, and others declaring that only a minority will give up the fun, freedom and control of manual driving. The real answer will be a mix. Though to those who tell me that Americans love cars too much to ever give them up, I ask how it is that car-loving Californians can move to Manhattan and give up car ownership in 15 minutes.

Restricted zones

We might see the rise of robocar-only lanes in certain places. There might be a special highway lane, where faster driving is allowed, and platooning takes place.

More dramatic would be the designation of certain downtown areas as robocar only at certain times or all the time. It’s not unusual for there to be driving restrictions in downtowns, particularly “old city” downtowns with small streets as found in Europe. Sometimes downtown streets are converted to pedestrian malls as well. It’s possible to imagine the robocars being deemed well behaved enough to go into these restricted areas.

We might also see robocars allowed to access, under certain rules, the private right-of-way used by transit lines, particularly bus rapid transit paths. It’s also possible rail lines and tunnels could get partially paved to allow the robocars to use them when the transit vehicle is not. They can be trusted not to interfere, and they can also drive reliably on thin strips of pavement — like rails for tires — if necessary.

This is not taking anything away from regular cars, but rather it’s giving the robocars a privilege the traditional cars never got.

More threatening to the human driver might be time restrictions or congestion restrictions. Once robocars provably cause less congestion, we might see congestion taxes on human drivers, or limitations on human driving during congested times, just as we sometimes see for trucks.

New infrastructure only for robocars

Many years down the road, cities might realize that when building new infrastructure, there are advantages in making it only for robocars, or for certain classes of robocars, such as lightweight electric single passenger cars. You can build very small tunnels for them, or much cheaper bridges and elevated roadways. They can drive reliably on a thin lane, saving a lot of money. Your old car won’t go there, but then again, it can’t go there now.

It’s also possible we might see housing neighbourhoods that take advantage of some things robocars can do, like serve the homes only with a single lane back alley, which comes with occasional wide spots for cars to do perfectly timed passing of one another on the short lanes.

Like many people, I think that robocars will be electric. As we get more electric cars, they might get privileges on the road you don’t get if you drive a manual gas guzzler. This even includes things like driving into buildings.

The fun roads will still be fun roads

The vast majority of the changes due to robocars will be on urban streets and highways. Commuting roads. Even those who love to drive don’t tend to love the urban commute. The country roads, the scenic coastal and mountain routes — these will stay pretty much the same for a long time to come. In fact, you might like having a robocar drive you out of the city to places where it’s fun to drive, and then take the wheel (or switch to a sportscar) to enjoy those great drives.

New traffic control and virtual infrastructure

Robocars must mix with regular cars, but they are able to do a better job at helping reduce traffic congestion by obeying on-the-fly rules about road use. They make use of “virtual infrastructure” — things like maps that show everything about the roads and their rules.

Drivers of regular cars can participate in this as well, with just a smartphone. If the day comes when cities want to meter or charge for roads using computer networks, the robocars will be ready to do it, but so will the ordinary drivers as long as they have that phone.

Parking will dwindle, but get cheaper for robocars

Robotaxis don’t need to park that much — they would rather be out working on another ride. Private robocars will drop their masters off at the door and then go find the cheapest parking nearby. That parking doesn’t have to be right nearby, they will shop around in a way that human drivers can’t. And park themselves densely in valet style. So they won’t pay that much for parking. You’ll probably have to pay more since you need to park your car right where you’re going while they don’t. You also take up more space.

Many parking spaces on the street might be restricted to robocars at certain times. That’s because robocars don’t park, they “stand,” able to move away at any time. A robocar can stand in front of a driveway or even a fire hydrant. And we might see important streets that only let robocars park on them as rush hour approaches, because they can be trusted to reliably clear the parking lanes and make them open for traffic. Robocars can also double park where there is room, because they can get out of the way if the “blocked” car wants to get out.

Trucks

The existence of cheap delivery robots might change how much people want trucks or vehicles with lots of cargo space. If you know that, when you want to move a big thing, you can easily call up a delivery robot, you might reduce your desire for cargo space. On the other hand, if you want a van with everything in it all the time, you might still go for that.

Your insurance might go down a bit

The price of insurance is based on how frequently people like you have accidents. Unless you think you will have more accidents, your insurance won’t go up. In fact, as the roads get safer and more orderly, you might have fewer accidents. And late model manually driven cars will still be loaded up with new accident prevention features, some of them a result of robocar research. That means your insurance will get a bit cheaper. Of course, the robocars will drive more safely than you, and pay even less for the cost of accidents. If not, they’re doing it wrong.

Some kids never get a licence

Kids are alive today who won’t get a licence due to robocars. That’s easy because kids are alive today who are not getting licences due to transit and Uber. This will grow. As it grows, it will become more normal, which means more effort to accommodate them, and a bit more pressure against human driving in some areas and times. Parents will cut back on the idea of giving a teenager a car, preferring to offer them a “card” which provides robotaxi service. A time will come when preferring to manual drive will be looked on as a touch odd, and for the older generation and the enthusiast.

Still plenty of places without robocars

As I outline in my timeline for deployment, robocars will roll out on very different schedules based on where you are. There will be cities that have heavy robotaxi penetration in the 2020s, while other cities have next to none. There will be countries which have no robocars even in the 2030s.

That’s good news for people who want to give up car ownership and sell their existing car. Even if they might have a hard time selling a used car in a city where car ownership is decreasing, there will still be plenty of places, both in the USA and around the world, where the car can be sold.

In addition, robotaxi service will not be available in rural areas for a very long time. Out there you will find only traditional cars and privately owned robocars. There will be many places to enjoy driving and sell cars.

Conversely, there is talk that countries like China, which have been known to do bold projects and are building brand-new cities, might declare a city to be all-robocars before too long. While that’s possible, it is not going to affect how you drive outside that city.

Privately owned robocars will have a steering wheel available, though if it’s for rare use it may be more like a video game wheel. You’ll need it to drive places robocars are not rated to go, like off-road or the dirt road to grandpa’s house — or just the many places the robocar doesn’t have maps and safety certification for. This probably won’t be “fun” sports driving, but it could be. Taxis won’t have these wheels (except for use by staff if the vehicle fails) but might drop you off next to a vehicle that has a wheel if that’s needed to complete your trip.

Society may not tolerate your mistakes

The biggest threat to the lover of driving is that while you won’t get any more dangerous, you will be more dangerous by comparison. You might not get away with things you could get away with before when everybody was dangerous.

It’s possible that you might lose your licence after one or two accidents. Perhaps even after just one DUI or a few serious tickets. Today, society doesn’t want to take away people’s licences because that destroys their mobility. In the future, it won’t — you will still be able to get around. So society may not tolerate you making big mistakes.

This also might happen more to people as they get older. Again, today we don’t want to take away a senior’s licence even when they are half-blind, because that ends their mobile life. Not any more.

The more distant future

As time passes, a new generation will grow up without learning how to drive. A couple of generations from now, manual driving may be mostly a sport or affectation. Still allowed, but uncommon. After all, people still ride horses a century after they stopped being used for transportation.

In this far away time, driving may be seen mostly as a sport, not a means of transportation. Racetracks and special roads will still exist, but for fun, not travel.

This week’s hot story was again from Amir at The Information and there is even more detail in the author’s Twitter thread.

The short summary: Amir was able to find a fair number of Waymo’s neighbours in Chandler, Arizona who are getting frustrated by the over-cautious drive patterns of the Waymo vans. Several used the words, “I hate them.”

A lot of the problems involve over-hesitation at an unprotected left turn near the Waymo HQ. The car is just not certain when it can turn. There is also additional confirmation of what I reported earlier, that operation with no safety driver is still very rare, and on limited streets.

Unprotected turns, especially left ones, have always been one of the more challenging elements of day-to-day driving. You must contend with oncoming traffic and pedestrians who may be crossing as well. You may have to do this against traffic that is moving at high speed.

While I am generally not surprised that these intersections can be a problem, I am a little surprised they are one for Waymo. Waymo is the only operating car which features a steerable, high resolution long range LIDAR on board. This means it can see out 200m or more (in a narrower field of view.) This lets it get a good look at traffic coming at it in such situations. (Radar also sees such vehicles but with much less resolution.)

For Waymo, this is not a problem of sensors but one of being too timid. One reason they are operating in Phoenix is it’s a pretty easy area to be timid in. The instinct of all teams is to avoid early risks that lead to early accidents. That became even stronger in Arizona after the Uber fatality, which used up a lot of the public’s tolerance for such errors. As such, the “better safe than sorry” philosophy which was already present in Waymo and most teams has been strengthened.

The problem is, it needs to eventually be weakened. Timid drivers won’t make it in the real world. They won’t make it far at all in the non-tame places like Boston. Thus you look at a nasty trade-off:

• The more timid you are, the more problems you have and the more people driving behind you that you annoy.
• The less timid you are, the greater the risk of a mistake and an accident.

That leaves only a few options. The other drivers must adapt better to a timid driver, or the public must get more tolerant of a slightly higher (but still better than human) accident risk.

Hype around self-driving cars has pushed some into the public to expect computerized perfection. That’s not coming.

Changing how people react?

Believe it or not, it is not as impossible to change the behaviour of other drivers as you might think. In Manhattan not that may years ago, two things were very common. Endless honking, and gridlock. Both were the result of well ingrained aggressive habits of New York drivers. The city decided to crack down on both, and got aggressive with fines. It worked, and those things are both vastly reduced.

It’s less possible to adjust the patterns of pedestrians. Some recent articles have gotten a lot of attention from people suggesting this must happen. I think some of it will happen, but much less than hoped, and that will be the subject of another article.

A robocar should eventually get very good at driving decisions which involve physics, because computers are very good at physics. Unlike humans which have difficulty precisely timing when an oncoming car will get to them, robots could in theory do it with a very frightening precision, making turns through gaps that seem tiny to humans. This would frighten other drivers and cause problems, so it’s not something we’ll see today.

This isn’t the last story we’ll see of robocars frustrating other drivers. That’s particularly true if we unwisely keep them at the speed limit. My long term belief is that most of the traffic code should be eliminate for robocars and be replaced by, “it’s legal if you can do it safely and without unfairly impeding traffic.” The whole idea of a traffic code only makes sense for humans. With robots, since there will never be more than a few dozen different software stacks on the road, you can just get all the designers together in a room and work out what should be done and what is safe. Traffic codes and other laws are there to deal with humans who can’t be trusted to know the rules or even obey the rules they know. While companies can’t be trusted to do anything but look after their own interests, you can easily make it in their interests to follow the rules.

The newsletter The Information reports Uber’s investors are pushing Uber to sell its self-drive division to some other large player. The division has, of course, been nothing but trouble for Uber, and as I have noted several times, Uber is one of the few large players in this space that doesn’t have to build their own tech. They have the #1 brand in selling rides, and selling rides is what the robotaxi business is all about.

At the same time, Uber recruited a great team (though it has lost many of them.) First they recruited many of the best around CMU for their Pittsburgh ATC headquarters. CMU and Stanford are where most of the stars of the robocar world have come from. Then they “paid” $680M in stock for Otto, which was really an acqui-hire in many ways, at least if you believe court documents. Because that stock payout required certain milestones that weren’t met, the Otto stockholders did not get their money, but Uber had to make a large stock payout to Waymo as a result of the lawsuit over Anthony Levanowski’s actions.

So they have this team, but the team and the Uber name are tarnished by the fatality. But the right pickings from the team are still valuable to somebody who is falling behind, if you eliminate the factors that led to the fatality and take the Uber name off it.

At the same time, Uber gets a pledge that the car the team makes will be usable in the Uber network. They get guaranteed access to tech (if the new owner succeeds) though not the deep control that comes with owning it. They don’t need the deep control. In fact, it’s now better if they can stick another famous brand on it.

Nuro starts a delivery pilot

Aside from Starship (which I am involved in) the company to watch in the delivery space is Nuro. Nuro has started a delivery pilot in Scottsdale with Fry’s groceries. The pilot will be done with regular Prius and Leaf cars with Nuro’s self drive tech, and a safety driver, not with the small no-seat delivery pod Nuro is trying to build. That pod isn’t ready for the real streets, even the easy streets of Scottsdale.

Nuro is the company to watch in on-road delivery because its founders were among the top members of Google/Waymo’s team. I worked with them regularly when I was there and they are the best. They have also raised lots of money.

Driving on the road is much harder than the sidewalk, though. A road robot must be able to handle any situation on its own and get to a safe state if there is a problem no matter what. A sidewalk robot can stop in 30cm. If it encounters something it doesn’t understand, it can just stop and request help from HQ. Safety is very unlikely to be an issue, while a road robot could really hurt somebody in a mistake.

The $6 price for delivery in the pilot is strange, since that’s a common price for human driven delivery.

The first was with Walmart. Walmart has installed special parking spots in their lots, and will offer people free rides to Walmart to pick up online orders. Only some of the 400 Waymo “Early Riders” will participate, at first.

The other project involves last mile rides to some stations of the Valley Metro transit agency. This begins only with the transit employees, but will expand to passengers of “Ride Choice” a special heavily discounted taxi service for seniors and the disabled.

The far more interesting recent piece is this Bloomberg interview with a teen-aged Early Rider about her experiences. Indeed, learning just how people interact with the cars is one of the key goals of a project like this.

While I’m obviously a Waymo fan (and worked on the project in its early years) I must say these announcements are a bit underwhelming, compared to the big milestone which Waymo has teased for 2018 — rides for the general public, with no safety driver. It is disappointing, though understandable, that Waymo is doing these baby steps. The reality has always been that making a public service is a very hard problem. Waymo has a significant lead on the problem but nobody is fully there yet. I do believe we’ll see it happen in Arizona and a few other places this decade, but in some places it is still several years away.

The Walmart project seems particularly unexciting. In fact, since the ride is just to pick up groceries ordered online, it’s hard to imagine why this isn’t more a task for delivery robots. (I work in the delivery robot space with Starship Technologies.) The need for special parking spaces, instead of just taking people to the door, suggests that such navigation at the busy door of a Walmart may be on the more challenging list.

The transit project elicits several thoughts. I have always felt that paratransit is a fantastic use of robocars. Paratransit (transit service for the disabled, technically “on demand”) is terrible in the USA. You have to book it a day in advance and wait up to an hour, and it ends up costing cities an average of $30 per ride, nationwide. For the disabled, driving a car can be difficult or impossible, and even riding accessible transit can still be a pain. If anybody needs robocars with cheap taxi rides, it’s this community.

On the other hand, I feel that “last mile” applications are wimpy. While they certainly make transit more tolerable, direct service along the shortest door to door route should be what this is all about. As I outline in The Future of Transit this can still be sharing a vehicle, and still involve transfers (but with no waiting or deviation from your route) but the old ideas of lines and schedules and stops should stay in the 20th century.

This will however, be the closest thing to a service for members of the general public.

Another underwhelming revelation comes in the interview with the Early Rider. While Waymo is operating vehicles without safety drivers, she reports having only ridden once in such a vehicle even though she uses it very regularly. This tells us that Waymo’s deployment of no-safety-driver operation is still quite limited, and probably only in a limited set of conditions and routes. This again makes sense, but should temper our enthusiasm for Waymo’s progress slightly, though its lead is still almost overwhelming. Nobody else is close to operations without a safety driver on open roads.

Motion sickness generally results when your eyes and inner ear (balance) tell you different things. Looking down at your book, your eyes see it and the interior of the car not moving, but your motion sensors remind you that you really are. Looking out at the horizon puts your eyes and balance in agreement, and makes it better, but on a winding road, even that can be not enough.

Of course it happens on ships due to the waves and rarely on trains which make few turns and don’t bounce much.

I’ve been pondering this for a while, and recently came upon an interesting invention from a team at French automaker Citroën. They gave people some very geeky looking glasses with circular liquid “levels” both in front, and to the sides. These make an artificial horizon in two dimensions, no matter where you are looking. They claim it reduces motion sickness.

While they have to make them look better, the idea is an interesting one, and those who really suffer from this may not mind looking uncool, especially in a solo car.

Here are some other thoughts I have had over the years on addressing this problem:

Bank the car and the seats

A robocar might be made with a passenger compartment — or just a passenger seat — mounted on hydraulic actuators like a simulator pod. This would allow the vehicle to tilt the passenger left or right, or forward or backward, depending on how much the car is turning or accelerating. Done right, you could tilt the passenger so that the “net force” is always only up or down from their perspective. In other words, they would never feel the car tilt, accelerate or brake, they would only feel themselves get lighter or heavier, like you do in an elevator. Done perfectly, a ball placed on the seat would not roll during any of these turns.

Because a robocar knows the geometry of all planned turns perfectly, it could make this adjustment. This might combine with a vehicle that has independent wheel suspension so the whole vehicle is able to “lean” in turns, as a motorcycle does. In fact, if you want a narrow stance vehicle for single people, there is no choice but to lean in order to maintain stability.

Moving the whole passenger cabin is hard, it would be better if you just had to tilt the seat. You would need to make sure nobody could get their hand under the seat.

This would definitely be valuable in places where roundabouts are common, like much of Europe. Driving at high speed through roundabout after roundabout could make anybody sick. And robocars will be great at roundabouts, always judging their merge perfect and thus rarely slowing down much except when they are fairly busy.

Provide audible cues

Again, the car knows how it plans to turn. It can also know about upcoming bumps in the road and some of the need for braking. I would like to experiment with making a quiet background noise — perhaps so quiet you are not consciously aware of it. In spite of this, the noise would make subtle changes in tone when there are upcoming or current accelerations. This might be able to train the brain to get a different channel of information (through the inner and outer ear rather than the middle.) More likely, it could provide an unconscious signal that “it’s time to look up now” so people pause in their reading before the turn.

Screens on the walls or floor

I’ve seen concept cars with screens all over the doors. I don’t think people will want those for working, but you could but in screens to make parts of the body seemingly transparent, or simply provide an artificial horizon like these glasses. The ability to see that while staring at your device could help.

Softer suspension

At a far simpler level than a banking passenger compartment, a robocar doesn’t need the tight, driver-oriented suspension that lets you feel the road. You don’t want to feel the road. This allows a super-soft suspension. In fact, you can have a suspension that responds perfectly to road bumps, potholes and other problems because it knows they are there, either from the map or from the laser scanner which measured the geometry of the road. Some cars already have some of this functionality. In the future, it might be possible to drive a washboard dirt road and barely feel the bumps. That should also help with motion sickness.

High video screen

The simplest thing of all will be placement of the screen for watching movies up high, with windows behind it. For a long time, the backseat passengers have had video screens in the headrests, though this is now supplanted by everybody having their own tablet back there for games and video. This is, of course, not allowed in the front seat, but that will change. Perhaps you can’t read a book but you’ll probably be able to enjoy a video.

What ideas do you have for helping people not get queasy in a bouncing car?

Source: Uber

In discussion of the eventual cost of a robotaxi ride, I and others have forecast costs similar to the all-in cost of car ownership. Today that’s 40 to 60 cents/mile (plus parking) and for a one person electric “city car” it can be under 20 cents. Note that in building these costs I am looking at the full retail cost today including:

• All profit margins
• All marketing budgets of existing car, banking and energy companies

As such, it makes sense as a retail price for somebody who is selling you the whole enchilada — a ride from A to B with everything included. You need to add a bit for empty vehicle moves (about 10-15% of miles,) and the cost of parking the car when not in motion — but that parking is going to be cheap because cars don’t need to park exactly where you want to be dropped off.

One big number contradicts all this, however. Today, services like Uber and Lyft sell rides for about $500. In the past, it only costs around $2/mile. (I see a range from $500. In the past, it only costs around $1.50/mile on long rides to $2.50 or more on shorter rides. Super short rides have a $500. In the past, it only costs around $5-$500. In the past, it only costs around $7 minimum.) While they started at 20%, today they take 25% of this money for their own fees. In other words 50 cents/mile. And for that, the value they provide is the largely the app that connects riders with drivers.

Where does that more than 50 cents/mile go?

• Some of it goes to insurance for the trip, which really does cost by the mile
• A modest amount goes into marketing. These companies don’t market much to riders, but they do market to drivers
• A large fraction is going to incentives to drivers to recruit and keep them
• The rest mostly goes to paying their staff in a fast-growing business. Maintaining software and many other things. A bit for servers and operations.
• None of it goes to profit — in fact, they are losing money at this price, some say losing it fast.

Yet they do a tiny part of the actual process. The cost of simply arranging a ride should dwindle to almost nothing with time. It should become a flat fee per trip, not counting insurance, because none of their costs depend very much on the length of the trip — other than you can do fewer long trips in a day than short ones, so more fixed cost must be allocated to the long trips.

As noted, they are in growth mode, so a lot of their revenue — and their losses — are spent on this growth. In particular, incentives to recruit drivers.

The robotaxi company doesn’t have to recruit drivers or keep them happy. It just has to get capital to buy cars and pay the interest on that capital. That’s not a lot at today’s interest rates — around $500. In the past, it only costs around $1/day on that $500. In the past, it only costs around $10,000 single person car. And car ownership actually has a higher capital cost because the car sits idle much more of the time.

A robotaxi company needs an app, of course, and a fleet scheduling and positioning system. It needs to take requests from customers, and dispatch cars to them. It needs to do strategic repositioning of cars to lower wait times. As does Uber, though it puts the cost of this mostly on drivers, who are not paid when driving to you or to a surge zone.

Over time, these companies will cut their costs. Will they cut their price to drivers? A competitor trying to break into their market today would probably be very attractive to drivers if it charged only a flat fee per ride, rather than this percentage. But nobody has had much success in competition without taking the big money and big losses approach.

But it will happen. And to get our robotaxis down to the super low prices that are possible, it has to happen. But it won’t happen on day one, and as such early robotaxi services will probably cost $500. In the past, it only costs around $1/mile.

It will also be interesting to look at the cost of the hybrid robotaxi services Lyft and Uber will offer. These services will serve you with a robotaxi if your trip is a route the cars can drive. If you want a different route, they will serve you with a human. They can either charge $500. In the past, it only costs around $1/mile for the first route and $2/mile for the second, or perhaps can charge some price in between for all rides, subsidizing the human driven rides with the robot ones. This strategy makes sense when you are looking at subscription services, and services for rides other than your standard commute. Nobody will offer you a robot-priced subscription service unless your most common rides (ie. home to work) are in the robocar’s territory, and human drivers are only needed for your more random other trips.

The Tempe police released a detailed report on their investigation of Uber’s fatality. I am on the road and have not had time to read it, but the big point, reported in many press was that the safety driver was, according to logs from her phone accounts, watching the show “The Voice” via Hulu on her phone just shortly before the incident.

This is at odds with earlier statements in the NTSB report, that she had been looking at the status console of the Uber self-drive system, and had not been using her phones. The report further said that Uber asked its safety drivers to observe the console and make notes on things seen on it. It appears the safety driver lied, and may have tried to implicate Uber in doing so.

Obviously attempting to watch a TV show while you are monitoring a car is unacceptable, presumably negligent behaviour. More interesting is what this means for Uber and other companies.

The first question — did Uber still instruct safety drivers to look at the monitors and make note of problems? That is a normal instruction for a software operator when there are two crew in the car, as most companies have. At first, we presumed that perhaps Uber had forgotten to alter this instruction when it went form 2 crew to 1. Perhaps the safety driver just used that as an excuse for her looking down since she felt she could not admit to watching TV. (She probably didn’t realize police would get logs from Hulu.)

If Uber still did that, it’s an error on their part, but now seems to play no role in this incident. That’s positive legal news for Uber.

It is true that if you had two people in the car, it’s highly unlikely the safety driver behind the wheel would be watching a TV show. It’s also true that if Uber had attention monitoring on the safety driver, it also would have made it harder to pull a stunt like that. Not all teams have attention monitoring, though after this incident I believe that most, including Uber, are putting it in. It might be argued that if Uber did require drivers to check the monitors, this might have somehow encouraged the safety driver’s negligent decision to watch TV, but that’s a stretch. I think any reasonable person is going to know this is not a job where you do that.

There may be some question regarding if a person with such bad judgement should have been cleared to be a safety driver. Uber may face some scrutiny for that bad choice. They may also face scrutiny if their training and job evaluation process for the safety drivers was clearly negligent. On the other hand, human employees are human, and if there’s not a pattern, it is less likely to create legal trouble for Uber.

From the standpoint of the Robocar industry, it makes the incident no less tragic, but less informative about robocar accidents. Accidents are caused every day because people allow themselves ridiculously unsafe distractions on their phones. This one is still special, but less so than we thought. While the issue of whether today’s limited systems (like the Tesla) generate too much driver complacency is still there, this was somebody being paid not to be complacent. The lessons we already knew — have 2 drivers, have driver attention monitoring — are still the same.

“Disabled the emergency braking.”

A number of press stories on the event have said that Uber “disabled” the emergency braking, and this also played a role in the fatality. That’s partly true but is very misleading vocabulary. The reality appears to be that Uber doesn’t have a working emergency braking capability in their system, and as such it is not enabled. That’s different from the idea that they have one and disabled it, which sounds much more like an ill act.

Uber’s system, like all systems, sometimes decides suddenly that there is an obstacle in front of the car for which it should brake when that obstacle is not really there. This is called a “false positive” or “ghost.” When this happens well in advance, it’s OK to have the car apply the brakes in a modest way, and then release them when it becomes clear it’s a ghost. However, if the ghost is so close that it would require full-hard braking, this creates a problem. If a car frequently does full-hard braking for ghosts, it is not only jarring, it can be dangerous, both for occupants of the car, and for cars following a little too closely behind — which sadly is the reality of driving.

As such, an emergency braking decision algorithm which hard brakes for ghosts is not a working system. You can’t turn it on safety, and so you don’t. Which is different from disabling it. While the Uber software did decide 2 seconds out that there was an obstacle that required a hard brake, it decides that out of the blue too often to be trusted with that decision. The decision is left to the safety driver — who should not be watching TV.

That does not mean Uber could not have done this much better. The car should still have done moderate braking, which would reduce the severity of any real accident and also wake up any inattentive safety driver. An audible alert should also have been present. Earlier, I speculated that if the driver was looking at the console, this sort of false positive incident would very likely have been there, so it was odd she did not see it, but it turns out she was not looking there.

The Volvo also has an emergency braking system. That system was indeed disabled — it is normally for any ADAS functions built into the cars to be disabled when used as prototype robocars. You are building something better, and you can’t have them competing. The Volvo system does not brake too often for ghosts, but that’s because it also doesn’t brake for real things far too often for a robocar system. Any ADAS system will be tuned that way because the driver is still responsible for driving. Teslas have been notoriously plowing into road barriers and trucks due to this ADAS style of tuning. It’s why a real robocar is much harder than the Tesla autopilot.

Other news

I’ve been on the road, so I have not reported on it, but the general news has been quite impressive. In particular, Waymo announced the order of 63,000 Chrysler minivans of the type they use in their Phoenix area tests. They are going beyond a pilot project to real deployment, and soon. Nobody else is close. This will add to around 20,000 Jaguar electric vehicles presumably aimed at a more luxury ride — though I actually think the minivan with its big doors, large interior space and high ride may well be more pleasant for most trips. The electric Jaguar will be more efficient.

The NHTSA/SAE “levels” of robocars are not just incorrect. I now believe they are contributing to an attitude towards their “level 2” autopilots that plays a small, but real role in the recent Tesla fatalities.

Readers of this blog will know I have been critical of the NHTSA/SAE “levels” taxonomy for robocars since it was announced. My criticisms have ranged to simply viewing them as incorrect or misleading, and you might have enjoyed my satire of the levels which questions the wisdom of defining the robocar based on the role the human being plays in driving it.

Recent events lead me to go further. I believe a case can be made that this levels are holding the industry back, and have a possible minor role in the traffic fatalities we have seen with Tesla autopilot. As such I urge the levels be renounced by NHTSA and the SAE and replaced by something better.

Some history

It’s true that in the early days, when Google was effectively the only company doing work on a full self-driving car for the roads, people were looking for some sort of taxonomy to describe the different types of potential cars. NHTSA’s first article laid one out as a series of levels numbered 0 to 4 which gave the appearance of an evolutionary progression.

Problem was, none of those stages existed. Even Google didn’t know what it wanted to build, and my most important contribution there probably was being one of those pushing it from the highway car with occasional human driving to the limited area urban taxi. Anthony Levandowski first wanted the highway car because it was the easiest thing to build and he’s always been eager to get out into reality as soon as possible.

The models were just ideas, and I don’t think the authors at NHTSA knew how much they would be carving them into public and industry thinking by releasing the idea of levels in an official document. They may not have known that once in people’s minds, they would affect product development, and also change ideas about regulation. To regulate something you must define it, and this was the only definition coming from government.

The central error of the levels was threefold. First, it defined vehicles according to the role the human being played in their operation. (In my satire, I compare that to how the “horseless carriage” was first defined by the role the horse played, or didn’t play in it.)

Second, by giving numbered levels and showing a chart the future, it advanced the prediction that the levels were a progression. That they would be built in order, each level building on the work of the ones before it.

Worst of all, it cast into stone the guess that the driver assist (ADAS) technologies were closely related to, and the foundation of the robocar technologies. That they were just different levels of the same core idea.

(The ADAS technologies are things like adaptive cruise control, lanekeeping, forward collision avoidance, blindspot warning, anti-lock brakes and everything else that helps in driving or alerts or corrects driver errors.)

There certainly wasn’t consensus agreement on that guess. When Google pushed Nevada to pass the first robocar regulations, the car companies came forward during the comment phase to make one thing very clear — this new law had nothing to do with them. This law was for crazy out-there projects like Google. The law specifically exempted all the the ADAS projects car companies had, including and up to things like the Tesla Autopilot, from being governed by the law or considered self-driving car technology.

Many in the car companies, whose specialty was ADAS, loved the progression idea. It meant that they were already on the right course. That the huge expertise they had built in ADAS was the right decision, and would give them a lead into the future.

Outside the car companies, the idea was disregarded. Almost all companies went directly to full robocar projects, barely using existing ADAS tools and hardware if at all. The exceptions were companies in the middle like Tesla and MobilEye who had feet in both camps.

SAE, in their 2nd version of their standard, partly at my urging, added language to say that the fact that the levels were numbered was not to be taken as an ordering. Very good, but not enough.

The reality

In spite of the levels, the first vehicle to get commercial deployment was the Navia (now Navya) which is a low speed shuttle with no user controls inside. What would be called a “level 4.” Later, using entirely different technology, Tesla’s Autopilot was the first commercial offering of “level 2.” Recently, Audi has declared that given the constraints of operating in a traffic jam, they are selling “level 3.” While nobody sold it, car companies demonstrated autopilot technologies going back to 2006, and of course prototype “level 4” cars completed the DARPA grand challenge in 2005 and urban challenge in 2007.

In other words, no ordering at all. DARPA’s rules were so against human involvement that teams were not allowed to send any signal to their car other than “abort the race.”

The confusion between the two broad technological areas has extended out to the public. People routinely think of Tesla’s autopilot as actual self-driving car technology, or simply as primitive self-driving car technology, rather than as extra-smart ADAS.

Tesla’s messaging points the public in both directions. On the one hand, Tesla’s manuals and startup screen are very clear that the Autopilot is not a self-driving car. That it needs to be constantly watched, with the driver ready to take over at any time. In some areas, that’s obvious to drivers — the AutoPilot does not respond to stop signs or red lights, so anybody driving an urban street without looking would soon get in an accident. On the highway, though, it’s better, and some would say too good. It can cruise around long enough without intervention to lull drivers into a false sense of security.

To prevent that, Tesla takes one basic measure — it requires you to apply a little torque to the steering wheel every so often to indicate your hands are on it. Wait too long and you get a visual alert, wait longer and you get an audible alarm. This is the lowest level of driver attention monitoring out there. Some players have a camera actually watch the driver’s eyes to make sure they are on the road most of the time.

At the same time, Tesla likes to talk up the hope their AutoPilot is a stepping stone. When you order your new Tesla, you can order AutoPilot and you can also pay $5,000 for “full self driving.” It’s mostly clear that they are two different things. When you order the full self driving package, you don’t get it, because it doesn’t exist. Rather you get some extra sensors in the car, and Tesla’s promise that a new software release in the future will use those extra sensors to give you some form of full self driving. Elon Musk likes to made radically optimistic predictions of when Tesla will produce full robocars that can come to you empty or take you door to door.

Operating domain

NHTSA improved things in their later documents starting in 2016. In particular they clarified that it was very important to consider what roads and road conditions a robocar was rated to operate in. They called this the ODD (Operational Design Domain.) The SAE had made that clear earlier when they had added a “Level 5” to make it clear that their Level 4 did not go everywhere. The Level 5 car that can go literally everywhere remains a science fiction goal for now — nobody knows how to do it or even truly plans to do it, because there are diminishing economic returns to handling and certifying safety on absolutely all roads, but it exists to remind people that the only meaningful level (4) does not go everywhere and is not science fiction. The 3rd level is effectively a car whose driving domain includes places where a human must take the controls to leave it.

NHTSA unified their levels with the SAE a few years in, but they are still called the NHTSA levels by most.

The confusion

The recent fatalities involving Uber and Tesla have shown the level of confusion among the public is high. Indeed, there is even confusion within those with higher familiarity of the industry. It has required press comments from some of the robocar companies to remind people, “very tragic about that Tesla crash, but realize that was not a self-driving car.” And indeed, there are still people in the industry who believe they will turn ADAS into robocars. I am not declaring them to be fools, but rather stating that we need people to be aware that is very far from a foregone conclusion.

Are the levels solely responsible for the confusion? Surely not — a great deal of the blame can be lain in many places, including automakers who have been keen to being perceived as in the game even though their primary work is still in ADAS. Automakers were extremely frustrated back in 2010 when the press started writing that the true future of the car was in the hands of Google and other Silicon Valley companies, not with them. Many of them got to work on real robocar projects as well.

NHTSA and SAE’s levels may not be to blame for all the confusion, but they are to blame for not doing their best to counter it. They should renounce the levels and, if necessary, create a superior taxonomy which is both based on existing work and flexible enough to handle our lack of understanding of the future.

Robocars and ADAS should be declared two distinct things

While the world still hasn’t settled on a term (and the government and SAE documents have gone through a few themselves. (Highly Automated Vehicles, Driving Automation for On-Road Vehicles, Automated Vehicles, Automated Driving Systems etc.) I will use my preferred term here (robocars) but understand they will probably come up with something of their own. (As long as it’s not “driverless.”)

The Driver Assist systems would include traditional ADAS, as well as autopilots. There is no issue of the human’s role in this technology — it is always present and alert. These systems have been unregulated and may remain so, though there might be investigation into technologies to assure the drivers are remaining alert.

The robocar systems might be divided up by their operating domains. While this domain will be a map of specific streets, for the purposes of a taxonomy, people will be interested in types of roads and conditions. A rough guess at some categories would be “Highway,” “City-Fast” and “City-Slow.” Highway would be classified as roads that do not allow pedestrians and/or cyclists. The division between fast and slow will change with time, but today it’s probably at about 25mph. Delivery robots that run on roads will probably stick to the slow area. Subclassifications could include questions about the presence of snow, rain, crowds, animals and children.

What about old level 3?

What is called level 3 — a robocar that needs a human on standby to take over in certain situations — adds some complexity. This is a transitionary technology. It will only exist during the earliest phases of robocars as a “cheat” to get things going in places where the car’s domain is so limited that it’s forced to transition to human control while moving on short but not urgent notice.

Many people (including Waymo) think this is a bad idea — that it should never be made. It certainly should not be declared as one of the levels of a numbered progression. It is felt that a transition to human driving while moving at speed is a risky thing, exactly the sort of thing where failure is most common in other forms of automation.

Even so, car companies are building this, particularly for the traffic jam. While first visions of a car with a human on standby mostly talked about a highway car with the human handling exit ramps and construction zones, an easier and useful product is the traffic jam autopilot. This can drive safely with no human supervision in traffic jams. When the jam clears, the human needs to do the driving. This can be built without the need for takeover at speed, however. The takeover can be when stopped or at low speed, and if the human can’t takeover, stopping is a reasonable option because the traffic was recently very slow.

Generally, however, these standby driver cars will be a footnote of history, and don’t really deserve a place in the taxonomy. While all cars will have regions they don’t drive, they will also all be capable of coming to a stop near the border of such regions, allowing the human to take control while slow or stopped, which is safe.

The public confusion slows things down

Tesla knows it does not have a robocar, and warns its drivers about this regularly, though they ignore it. Some of that inattention may come from those drivers imagining they have “almost a robocar.” But even without that factor, the taxonomies create another problem. The public, told that the Tesla is just a lower level of robocar, sees the deaths of Tesla drivers as a sign that real robocar projects are more dangerous. The real projects do have dangers, but not the same dangers as the autopilots have. (Though clearly lack of driver attention is an issue both have on their plates.) A real robocar is not going to follow badly painted highway lines right into a crash barrier. They follow their maps, and the lane markers are just part of how they decide where they are and where to go.

But if the public says, “We need the government to slow down the robocar teams because of those Tesla crashes” or “I don’t trust getting in the Waymo car because of the Tesla crashes” then we’ve done something very wrong.

(If the public says they are worried about the Waymo car because of the Uber crash, they have a more valid point, though those teams are also very different from one another.)

The Automated/Autonomous confusion

For decades, roboticists used the word “autonomous” to refers to robots that took actions and decisions without having to rely on an outside source (such as human guidance.) They never used it in the “self-ruling” sense it has politically, though that is the more common (but not only) definition in common parlance.

Unfortunately, one early figure in car experiments hated that the roboticists’ vocabulary didn’t match his narrow view of the meaning of the word, and he pushed with moderate success for the academic and governmental communities to use the word “automated.” To many people, unfortunately, “automated” means simple levels of automation. Your dishwasher is automated. Your teller machine is automated. To the roboticist, the robocar is autonomous — it can operate entirely without you. The autopilot is automated — it needs human guidance.

I suspect that the public might better understand the difference if these words were split in these fashions. The Waymo car is autonomous, the Tesla automated. Everybody in robotics knows they don’t use the world autonomous in the political sense. I expressed this in a joke many years ago, “A truly autonomous car is one that, if you tell it to take you to the office, says it would rather go to the beach instead.” Nobody is building that car. Yet.

Are they completely disjoint?

I am not attempting to say that there are no commonalities between ADAS and robocars. In fact, as development of both technologies progresses, elements of each have slipped into the other, and will continue to do so. Robocars have always used radars created for ADAS work. Computer vision tools are being used in both systems. The small ultrasonic sensors for ADAS are used by some robocars for close in detection where their LIDARs don’t see.

Even so, the difference is big enough to be qualitative and not, as numbered levels imply, quantitative. A robocar is not just an ADAS autopilot that is 10 times or 100 times or even 1,000 times better. It’s such a large difference that it doesn’t happen by evolutionary improvement but through different ways of thinking.

There are people who don’t believe this, and Tesla is the most prominent of them.

As such, I am not declaring that the two goals are entirely disjoint, but rather that official taxonomies should declare them to be disjoint. Make sure that developers and the public know the difference and so modulate their expectations. It won’t forbid the cross pollination between the efforts. It won’t even stop those who disagree with all I have said from trying the evolutionary approach on their ADAS systems to create robocars.

SAE should release a replacement for its levels, and NHTSA should endorse this or do the same.

Each company faces a different challenge to fix its problems. For Uber, they need to improve the quality of their self-drive software so that such a basic failure as we saw here is extremely unlikely. Perhaps even more importantly, they need to revamp their safety driver system so that safety driver alertness is monitored and assured, including going back to two safety drivers in all situations. Further, they should consider some “safety driver assist” technology, such as the use of the system in the Volvo (or some other aftermarket system) to provide alerts to the safety drivers if it looks like something is going wrong. That’s not trivial — if the system beeps too much it gets ignored, but it can be done.

For Tesla, they face a more interesting challenge. Their claim is that in spite of the accidents, the autopilot is still a net win. That because people who drive properly with autopilot have half the accidents of people who drive without it, the total number of accidents is still lower, even if you include the accidents, including these fatalities, which come to those who disregard the warnings about how to properly use it.

That people disregard those warnings is obvious and hard to stop. Tesla argues, however, that turning off Autopilot because of them would make Telsa driving and the world less safe. For them, options exist to make people drive diligently with the autopilot, but they must not make the autopilot so much less pleasant such that people decide to not use it even properly. That would actually make driving less safe if enough people did that.

Why the Tesla crashed

A theory, now given credence by some sample videos, suggests the Telsa was confused by the white lines which divide the road at an off-ramp, the expanding triangle known as the “gore.” As the triangle expands, a simple system might think they were the borders of a lane. Poor lane marking along the gore might make the vehicle even think the new “lane” is a continuation of the lane the car is in, making the car try to drive the lane — right into the barrier.

This video made by Tesla owners near Indiana, shows a Telsa doing this when the right line of the gore is very washed out compared to the left. At 85/101 (the recent Tesla crash) the lines are mostly stronger but there is a 30-40 foot gap in the right line which perhaps could trick a car into entering and following the gore. The gore at 85/101 also is lacking the chevron “do not drive here” stripes often found at these gores. It is not good at stationary objects like the crumple barrier, but its warning stripes are something that should be in its classification database.

Once again, the Tesla is just a smart cruise control. It is going to make mistakes like this, which is why they tell you you have to keep watching. Perhaps crashes like this will make people do that.

The NTSB is angry that Tesla released any information. I was not aware they frowned on this. This may explain Uber’s silence during the NTSB investigation there.

The governor of Arizona has told Uber to “get an Uber” and stop testing in the state. With no instructions on how to come back.

Unlike the early positive statements from Tempe police, this letter is harsh and to the point. It’s even more bad news for Uber, and the bad news is not over. Uber has not released any log data that makes them look better, the longer they take to do that, the more it seems that the data don’t tell a good story for them.

In other news, both MobilEye and Velodyne have issued releases that their systems would have seen the pedestrian. Waymo has said the same, and I believe that all of them are correct. Waymo has a big press event scheduled for this week in New York, rumoured to announce some new shuttle operations there. I wonder how much consideration they gave to delaying it, because in spite of their superior performance, a lot of the questions they will get at the press conference won’t be about their new project.

There are more signs that Uber’s self-driving project may receive the “death penalty,” or at the very least a very long and major setback. A long and major setback in a field where Uber thought “second place is first loser” to quote Anthony Levandowski.

Source: Uber

Today I’m going to examine how you attain safety in a robocar, and outline a contradiction in the things that went wrong for Uber and their victim. Each thing that went wrong is both important and worthy of discussion, but at the same time unimportant. For almost every thing that went wrong Is something that we want to prevent going wrong, but it’s also something that we must expect will go wrong sometimes, and to plan for it.

In particular, I want to consider how things operate in spite of the fact that people will jaywalk, illegal or not, car systems will suffer failures and safety drivers will sometimes not be looking.

What’s new

First, an update on developments.

Uber has said it is cooperating fully, but we certainly haven’t heard anything more from them, or from the police. That’s because:

• Police have indicated that the accident has been referred for criminal investigation, and the NTSB is also present.
• The family (only a stepdaughter is identified) have retained counsel, and are demanding charges and considering legal action.

A new story in the New York Times is more damning for Uber. There we learn:

• Uber’s performance has been substandard in Arizona. They are needing an intervention after 13 miles of driving on average. Other top companies like Waymo go many thousands of miles.
• Uber just recently switched to having one safety driver instead of two, though it was still using two in the more difficult test situations. Almost all companies use two safety drivers, though Waymo has operations with just one, and quite famously, zero.
• Uber has safety drivers use a tablet to log interventions and other data, and there are reports of safety drivers doing this while in self-drive mode. Wow.
• Uber’s demos of this car have shown the typical “what the car sees” view to both passengers and the software operator. It shows a good 360 degree view as you would expect from this sort of sensor suite, including good renderings of pedestrians on the sidewalk. Look at this youtube example from two years ago.
• Given that when operating normally, Uber’s system has the expected pedestrian detection, it is probably not the case that Uber’s car just doesn’t handle this fairly basic situation. We need to learn what specifically went wrong.
• If Uber did have a general pedestrian detection failure at night, there should have been near misses long before this impact. Near misses are viewed as catastrophic by most teams, and get immediate attention, to avoid situations like this fatality.
• If equipment such as LIDARs or cameras or radars were to fail, this would generally be obvious to the software system, which should normally cause an immediate alert asking the safety driver to take over. In addition, most designs have some redundancies, so they can still function at some lower level with a failed sensor while they wait for the safety driver — or even attempt to get off the road.
• The NYT report indicates that new CEO Dara Khosrowshahi had been considering cancelling the whole project, and a big demo for him was pending. There was pressure on to make the demo look good.

In time, either in this investigation or via lawsuits, we should see:

• Logs of raw sensor data from the LIDAR, radar and camera arrays, along with tools to help laypeople interpret this data.
• If those logs clearly show the victim well in advance of impact, why did the vehicle not react? What sort of failure was it?
• Why was the safety driver staring at something down and to the right for so long? Why was there only one safety driver on this road at night at 40mph?

That the victim was jaywalking is both important and unimportant

The law seems to be clear that the Uber had the right of way. The victim was tragically unwise to cross there without looking. The vehicle code may find no fault with Uber. In addition, as I will detail later, crosswalk rules exist for a reason, and both human drivers and robocars will treat crosswalks differently from non-crosswalks.

Even so, people will jaywalk, and robocars need to be able to handle that. Nobody can handle somebody leaping quickly off the sidewalk into your lane, but a person crossing 3.5 lanes of open road is something even the most basic cars should be able to handle, and all cars should be able to perceive and stop for a pedestrian standing in their lane on straight non-freeway road. (More on this in a future article.)

The law says this as well. While the car has right of way, the law still puts a duty on the driver to do what they reasonably can to avoid hitting a jaywalker in the middle of the road.

That Uber’s system failed to detect the pedestrian is both important and unimportant

We are of course very concerned as to why the system failed. In particular, this sort of detect-and-stop is a very basic level of operation, expected of even the most simple early prototypes, and certainly of a vehicle from a well funded team that’s logged a million miles.

At the same time, cars must be expected to have failures, even failures as bad as this. In the early days of robocars, even at the best teams, major system failures happened. I’ve been in cars that suddenly tried to drive off the road. It happens, and you have to plan for it. The main fallback is the safety driver, though now that the industry is slightly more mature, it is also possible to use simpler automated systems (like ADAS “forward collision warning” and “lanekeeping” tools) to also guard against major failures.

We’re going to be very hard on Uber, and with justification, for having such a basic failure. “Spot a pedestrian in front of you and stop” have been moving into the “solved problem” category, particularly if you have a high-end LIDAR. But we should not forget there are lots of other things that can, and do go wrong that are far from solved, and we must expect them to happen. These are prototypes. They are on the public roads because we know no other way to make them better, to find and solve these problems.

That the safety driver wasn’t looking is both important and unimportant

She clearly was not doing her job. The accident would have been avoided if she had been vigilant. But we must understand that safety drivers will sometimes look away, and miss things, and make mistakes.

That’s true for all of us when we drive, with our own life and others at stake. Many of us do crazy things like send texts, but even the most diligent are sometimes not paying enough attention for short periods. We adjust controls, we look at passengers, we look behind us and (as we should) check blindspots. Yet the single largest cause of accidents is “not paying attention.” What that really means is that two things went wrong at once — something bad happened while we were looking somewhere else. For us the probability of an accident is highly related to the product of those two probabilities.

The same is true for robocars with safety drivers. The cars will make mistakes. Sometimes the driver will not catch it. When both happen, an accident is possible. If the total probability of that is within the acceptable range (which is to say, the range for good human drivers) then testing is not putting the public at any extraordinary risk.

This means a team should properly have a sense of the capabilities of its car. If it’s needing interventions very frequently, as Uber was reported to, it needs highly reliable safety driving. In most cases, the answer is to have two safety drivers, 2 sets of eyes potentially able to spot problems. Or even 1.3 sets of eyes, because the 2nd operator is, on most teams, including Uber, mostly looking at a screen and only sometimes at the road. Still better than just one pair.

At the same time, since the goal is to get to zero safety drivers, it is not inherently wrong to just have one. There has to be a point where a project graduates to needing only one. Uber’s fault is, possibly, graduating far, far too soon.

To top all this, safety drivers, if the company is not careful, are probably more likely to fatigue and look away from the road than ordinary drivers in their own cars. After all, it is actually safer to do so than it is to do in your own car. Tesla autopilot owners are also notoriously bad at this. Perversely, the lower the intervention rate, the more likely it is people will get tempted. Companies have to combat this.

If you’re a developer trying out some brand new and untrusted software, you safety drive with great care. You keep your hands near the wheel. Your feet near the pedals. Your eyes on the lookout. You don’t do it for very long, and you are “rewarded” by having to do an intervention often enough that you never tire. To consider the extreme view of that, think about driving adaptive cruise control. You still have to steer, so there’s no way you take your eyes off the road even though your feet can probably relax.

Once your system gets to a high level (like Tesla’s autopilot in simple situations or Waymo’s car) you need to find other ways to maintain that vigilance. Some options include gaze-tracking systems that make sure eyes are on the road. I have also suggested that systems routinely simulate a failure, but drifting out of their lane when it is safe to do so, but correcting it before it gets dangerous if for some reason the safety driver does not intervene. A safety driver who is grabbing the wheel 3 times an hour and scored on it is much less likely to miss the one time a week they actually have to grab it for real.

That it didn’t brake at all — that’s just not acceptable

While we don’t have final confirmation, reports suggest the vehicle did not slow at all. Even if study of the accident reveals a valid reason for not detecting the victim 1.4 seconds out (as needed to fully stop) there are just too many different technologies that are all, independently, able to detect her at a shorter distance which should have at least triggered some braking and reduced severity.

They key word is independently. As explained above, failures happen. A proper system is designed to still do the best it can in the event of failures of independent components. Failure of the entire system should be extremely unlikely, because the entire system should not be a monolith. Even if the main perception system of the car fails for some reason (as may have happened here) that should result in alarm bells going off to alert the safety driver, and it should also result in independent safety systems kicking in to fire those alarms or even hit the brakes. The Volvo comes with such a system, but that system is presumably disabled. Where possible, a system like that should be enabled, but used only to beep warnings at the safety driver. There should be a “reptile brain” at the low level of the car which, in the event of complete failure of all high level systems, knows enough to look at raw radar, LIDAR or camera data and sound alarms or trigger braking if the main system can’t.

All the classes of individual failures that happened to Uber could happen to a more sophisticated team in some fashion. In extreme bad luck they could even happen all at once. The system should be designed to make it very unlikely that they won’t all happen at once, and that the probability of that is less than the probability of a human having a crash.

More to come

So much to write here, so in the future look for thoughts on:

• How humans and robocars will act differently when approaching a crosswalk and not approaching one, and why
• More about how many safety drivers you have
• What are reasonable “good practices” that any robocar should have, and what are exceptions to them
• How do we deal with the fact that we probably have to overdrive our sensor range on high speed highways, as humans usually do?
• More about interventions and how often they happen and why they happen
• Does this mean the government should step in to stop bad actors like Uber? Or will the existing law (vehicle code, criminal law and tort law) punish them so severely — possibly with a “death penalty” for their project — that we can feel it’s working?

I have written a few times about the unusual nature of robocar accidents. Recently I was discussing this with a former student who is doing some research on the area. As a first step, she began looking at lists of all the reasons that humans cause accidents. (The majority of them, on police reports, are simply that one car was not in its proper right-of-way, which doesn’t reveal a lot.)

This led me, though to the following declaration that goes against most early intuitions.

Every human accident teaches us something about the way people have accidents. Every robocar accident teaches us about a way robocars will never have an accident again.

While this statement is not 100% true, it reveals the stark difference between the way people and robots drive. The whole field of actuarial science is devoted to understanding unusual events (with car accidents being the primary subject) and their patterns. When you notice a pattern, you can calculate probabilities that other people will do it, and they use that to price insurance and even set policy.

When a robocar team discovers their car has made any mistake, and certainly caused any incident, their immediate move is to find and fix the cause of the mistake, and update the software. That particular mistake will generally never happen again. We have learned very little about the pattern of robocar accidents, though the teams have definitely learned something to fix in their software. Since for now, and probably forever, the circumstances of robocar accidents will be a matter of public record, all teams and thus call cars will also learn portions of the same thing. (More on that below.)

The rule won’t be entirely true. There are some patterns. There are patterns of software bugs too — every programmer knows the risk of off-by-one errors and memory allocation mistakes. We actually build our compilers, tools and even processors to help detect and prevent all the known common programming mistakes we can, and there’s an active field of research to develop AI able to do that. But that does not teach us a lot about what type of car accidents this might generate. We know robocars will suffer general software crashes and any good system has to be designed to robustly handle that with redundancies. There is a broad class of errors known as perception false negatives, where a system fails to see or understand something on the road, but usually learning about one such error does not teach us much about the others of its class. It is their consequences which will be similar, not their causes.

Perception errors are the easiest to analogize to human activity. Humans also don’t see things before an accident. This, however, is due to “not paying attention,” or “not looking in that direction,” something that robots won’t ever be guilty of. A robot’s perception failure would be like a human’s mini-stroke temporarily knocking out part of our visual cortex (ie. not a real-world issue) or a flaw in the “design” of the human brain. In the robot, however, design flaws can usually be fixed.

There are some things that can’t be fixed, and thus teach us patterns. Some things are just plain hard, like seeing in fog or under snow, or seeing hidden vehicles/pedestrians. The fact that these things are hard can help you calculate probabilities of error.

This is much less true for the broad class of accidents where the vehicle perceives the world correctly, but decides the wrong thing to do. These are the mistakes that once done, will probably never be done badly again.

There actually have been very few accidents involving robocars that were the fault of the robocar system. In fact, the record is surprisingly good. I am not including things like Tesla autopilot crashes — the Tesla autopilot is designed to be an incomplete system and they document explicitly what it won’t do.

Indeed the only pattern I see from the few reported incidents is the obvious one — they happened in unusual driving situations. Merging with a bus when one wide lane is often used by 2 cars at once. Dealing with a lane splitting motorcycle after aborting an attempt at a lane change. (Note that police ruled against the motorcyclist in this case but it is being disputed in court.) Whatever faults occur here have been fixed by now.

Developers know that unusual situations are an area of risk, so they go out searching for them, and use simulators and test tracks to let them work extensively with them. You may not learn patterns, but you can come up with probability estimates to measure what fraction of everyday driving involves extraordinary situations. This can give you insurance-style confidence, to know that even if you aren’t sure you handle every unusual situation, the overall total risk is low.

One place this can be useful is in dealing with equipment failures. Today, a car driving only with vision and radar is not safe enough; a LIDAR is needed for the full safety level. If the LIDAR fails, however, the car is not blind, it is just a bit less safe. While you would not drive for miles with the LIDAR off, you might judge that the risk of driving to a safe spot to pull off on just camera and radar is acceptable, simply because the amount of driving in that mode will be very small. We do the same thing with physical hardware — driving with a blown out tire is riskier, but we can usually get off the road. We don’t insist every car have 5 wheels to handle that situation.

Learning from humans

It is also worth noting that the human race is capable of learning from accidents. In a sense, every traffic safety rule, most road signs, and many elements of road engineering are the result of learning from accidents and improving safety. While the fact that one human misjudges a left turn doesn’t stop other humans from doing so, if it causes a “no left turn” sign to go up, we do. Ironically, the robot does not need the no left turn sign — it will never misjudge the speed of oncoming vehicles and make the turn at the wrong time.

Car design also is guided a lot from lessons of past accidents. That’s everything from features like crumple zones which don’t affect human behaviour, to the blindspot warning system which does.

Pure neural network approaches

There are a few teams who hope to make a car drive with only a neural network. That is to say the neural network outputs steering controls and takes in sensor data. Such a system is more akin to humans in that a flaw found in that approach might be found again in other similarly designed vehicles. After an accident, such a car would have its network retrained so that it never made that precise mistake again (nor any of the other potential mistakes in its training library.) This might be a very narrow, retraining however.

This is one reason that only smaller teams are trying this approach. Larger teams like Waymo are making very extensive use of neural networks, but primarily in the area of improving perception, not in making driving decisions. If a perception error is discovered, the network retraining to fix it will ideally be quite extensive, to avoid related errors. Neural network perception errors also tend to be intermittent — ie. the network fails to see something in one frame, but sees it in a later frame. The QA effort is to make it see things sooner and more reliably.

Sharing crash reports

This raises the interesting question of sharing crash reports. Developers are now spending huge amounts of money racking up test miles on their cars. They want to encounter lots of strange situations and learn from them. Waymo’s 4 million miles of testing hardly came for free. This makes them and others highly reluctant to release to the public all that hard-won information.

The 2016 NHTSA car regulations, though highly flawed, included a requirement that vendors share all the sensor data on any significant incident. As you might expect, they resisted that, and the requirement was gone from the 2017 proposed regulations (along with almost all the others.)

Vendors are unlikely to want to share every incident, and they need incentives to do lots of test driving, but it might be reasonable to talk about sharing for any actual accident. It is not out of the question that this could be forced by law, though it is an open question on how far this should go. If an accident is the system’s fault, and there is a lawsuit, the details will certainly come out, but companies don’t want to air their dirty laundry. Most are so afraid of accidents that the additional burden of having their mistake broadcast to all won’t change the incentives.

Generally, companies all want as much test data as they can get their hands on. They might be convinced that while unpleasant, sharing all accident data helps the whole industry. Only a player who was so dominant that they had more data than everybody else combined (which is Waymo at this time) would not gain more from sharing than they lose.

In a shocker, it was announced that Uber and Waymo (Google/Alphabet) have settled their famous lawsuit for around $245 million in Uber stock. No cash, and Uber agrees it won’t use any Google hardware or software trade secrets — which it of course had always denied that it ever did.

I think this is a tremendous victory for Uber. Google had proposed a $1B settlement early on that was rejected. Waymo had not yet provided all the evidence necessary to show damages, but one has to presume they had more to come that made Uber feel it should settle. Of course, the cloud of a lawsuit and years of appeals over their programs and eventual IPO also were worth closing out.

What’s great for Uber is that it’s a stock deal. While the number is not certain, some estimates suggest that this amount of stock might not be much more than the shares of Uber lost by Anthony Levandowski when he was fired for not helping with the lawsuit. In other words, Uber fixes the problems triggered by Anthony’s actions by paying off Waymo with the stock they used to buy Otto from Anthony. They keep the team (which is really what they bought, since at 7 months of age, Otto had done some impressive work but nothing worth $700M) and they get clear of the lawsuit.

The truth is, Uber can’t be in a fight with Google. All Uber rides are booked through the platforms of Google and Apple. Without those platforms there is no Uber. I am not suggesting that Apple or Google would do illegal-monopoly tricks to fight Uber. They don’t have to, though there are some close-to-the-line tricks they could use that don’t violate anti-trust but make Uber’s life miserable. You simply don’t want to be in a war for your existence with the platform you depend on for that existence.

Instead, Alphabet now increases its stake in Uber. They are now more motivated to be positively inclined to it. There is still going to be a heavy competition between Waymo and Uber, but Waymo now has this incentive not to hurt Uber too much.

For a long time, it had seemed like there would be a fantastic synergy between the companies. Google had been an early investor in Uber. Waymo has the world’s #1 robocar technology. Uber has the world’s #1 brand in selling rides — the most important use of that technology. Together they would have ruled the world. That never happened, and is unlikely to happen now (though no longer impossible). Alphabet has instead invested in Lyft.

Absent working with Lyft or Uber, Waymo needs to create its own ride service on the scale that Uber has. Few companies could enter that market convincingly today, but Alphabet is one of those few. Yet they have never done this. You need to do more than a robot ride service. Robocars won’t take you from anywhere to anywhere for decades, and so you need a service that combines robocar rides on the popular routes, and does the long tail rides with human drivers. Uber and Lyft are very well poised to deliver that; Waymo is not.

Uber settles this dangerous lawsuit for “free” and turns Alphabet back from an enemy to a frenemy. They get to go ahead full steam, and if they botch their own self-drive efforts they still have the option of buying somebody else’s technology, even Waymo’s. With new management they are hoping to convince the public they aren’t chaotic neutral any more. I think they have come out of this pretty well.

For Waymo, what have they won? Well, they got some Uber stock, which is nice but it’s just money. Alphabet has immense piles of money that this barely dents. They stuck it to Anthony, and retarded Uber for a while. The hard reality is that many companies are developing long-range LIDAR like they alleged Anthony stole for Uber. When they built it, and Otto tried to build it, nobody had it for sale. Time has past and that’s just not as much of an advantage as it used to be. In addition, Waymo has put their focus (correctly) on urban driving, not the highway driving where long range LIDAR is so essential. While Anthony won’t use the knowledge he gained on the Waymo team to help Uber, several other former team members are there, and while they can’t use any trade secrets (and couldn’t before, really) their experience is not so restricted.

For the rest of the field, they can no longer chuckle at their rivals fighting. Not so great news for Lyft and other players.

Click on the CES 2018 Gallery to view it. Make sure to see the captions, which will either appear at the bottom of the screen, or if you clicke the “Info” button (“i” in circle) it will open up a side panel with the caption, and then you can go through the images with arrow keys or the arrow buttons.

In the gallery you will see commentary on 3 different flying car offerings, many LIDARs, 6 delivery robots and the silliest product of CES 2018.

In other news

It’s been reported that Pony.ai got a $112M series A which shows the valuation frenzy is continuing. Pony.ai was founded by veterans of Baido (and Google Chauffeur), but what is more surprising is that their plan is not very ambitious, at least for now — cars for restricted environments such as campuses and small towns. They will go after the Chinese market first.

The U.S. Dept. of Transport will make a 3rd round of robocar regulations this summer. The first round was much too detailed, the 2nd round fixed that but said almost nothing. The 3rd round will probably be a bit closer to the middle, and will also deal with trucks, which were left out of earlier rules.

GM revealed photos of what they say is the production form of their self-driving car based on the Chevy Bolt and Cruise software. They say it will be released next year, making it almost surely the first release from a major car company if they make it.

As reported in their press release and other sources their goals are ambitious. While Waymo is the clear leader, it has deployed in Phoenix, because that is probably the easiest big city for driving in the USA. Cruise/GM claims they have been working on the harder problem of a dense city like San Francisco.

What’s notable, though, about the Cruise picture is what’s not in it, namely much in the way of dashboard or controls. There is a small screen, and a few controls but little else. Likewise the Waymo 3rd generation “Firefly” car had almost no controls at all.

The car has climate controls and window controls and little else. Of course a touchscreen can control a lot of other things, especially when the “driver” does not have to watch the road.

Combine this with the concept Smart-car self-driving interior from Daimler below, and you see a change in the thinking of the car industry towards the thinking of the high-tech industry.

At most car conferences today, a large fraction of what you see is related to putting new things inside the car — fancier infotainment tools and other aspects of the “connected car.” These are the same companies who charge you $2,000 to put a navigation system into your car that you will turn off in 2 years because your phone does a better job.

It is not surprising that Google expects you to get your music, entertainment and connectivity from the device in your pocket — they make Android. It is a bigger step for GM and Daimler to realize this, and bodes well for them.

While the car is not finalized and the software certainly isn’t, GM feels it has settled on the hardware design of its version 1 car, and is going into production on it. I suspect they will make changes and tweaks as new sensors come down the pipeline, but traditionally car companies have always locked down the main elements of the hardware design a couple of years before a vehicle is available for sale.

One thing both of these cars need more of, which I know well from road tripping, is surfaces and places to put stuff. Cupholders and door pockets aren’t enough when a car is a work or relaxation station rather than something to drive.

What is not clear is if they have been bold enough to get rid of many of the other features not needed if you’re not driving, like fancy adjustable seats. The side-view mirrors are gone but sensors are in their place (it is widely anticipated that they will allow even human driven cars to replace the mirrors with cameras, since that’s better and lower drag.) Waymo’s firefly had the mirrors because the law still demands them.

GM is also already working with NHTSA to get this car an exception to the Federal Motor Vehicle Safety Standards which require the things like the steering wheel that they took out. The feds say they will work quickly on this so it seems likely. Several states are already preparing the legal regime necessary, and GM suggests it will deploy something in 2019.

Not too long ago, I would have ranked GM very far down on the list of carmakers likely to succeed in the robocar world. After they acquired Cruise they moved up the chart, but frankly I had been skeptical about how much a small startup, no matter how highly valued, could change a giant automaker. It now seems that intuition was wrong.

When it comes to robocars, new LIDAR products were the story of CES 2018. Far more companies showed off LIDAR products than can succeed, with a surprising variety of approaches. CES is now the 5th largest car show, with almost the entire north hall devoted to cars. In coming articles I will look at other sensors, software teams and non-car aspects of CES, but let’s begin with the LIDARs.

Velodyne

When it comes to robocar LIDAR, the pioneer was certainly Velodyne, who largely owned the market for close to a decade. Their $75,000 64-laser spinning drum has been the core product for many years, while most newer cars feature their 16 and 32 laser “puck” shaped unit. The price was recently cut in half on the pucks, and they showed off the new $100K 128 laser unit as well as a new more rectangular unit called the Veloray that uses a vibrating mirror to steer the beam for a forward view rather than a 360 view.

The Velodyne 64-laser unit has become such an icon that its physical properties have become a point of contention. The price has always been too much for any consumer car (and this is also true of the $100K unit of course) but teams doing development have wisely realized that they want to do R&D with the most high-end unit available, expecting those capabilities to be moderately priced when it’s time to go into production. The Velodyne is also large, heavy, and because it spins, quite distinctive. Many car companies, and LIDAR companies have decried these attributes in their efforts to be different from Waymo (which uses their own in-house LIDAR now) and Velodyne. Most products out there are either clones of the Velodyne, or much smaller units with a 90 to 120 degree field of view.

Quanergy

I helped Quanergy get going, so I have an interest and won’t comment much. Their recent big milestone is going into production on their 8 line solid state LIDAR. Automakers are very big on having few to no moving parts so many companies are trying to produce that. Quanergy’s specifications are well below the Velodyne and many other units, but being in production makes a huge difference to automakers. If they don’t stumble on the production schedule, they will do well. With lower resolution instruments with smaller fields of view, you will need multiple units, so their cost must be kept low.

Luminar and 1.5 micron

The hot rising star of the hour Luminar, with its high performance long range LIDAR. Luminar is part of the subset of LIDAR makers using infrared light in the 1.5 micron (1550nm) range. The eye does not focus this light into a spot on the retina, so you can emit a lot more power from your laser without being dangerous to the eye. (There are some who dispute this and think there may be more danger to the cornea than believed.)

The ability to use more power allows longer range, and longer range is important, particularly getting a clear view of obstacles over 200m away. At that range, more conventional infrared lid Ar in the 900nm band has limitations, particularly on dark objects like a black car or pedestrian in black clothing. Even if the limitations are reduced, as some 900nm vendors claim, that’s not good enough for most teams if they are trying to make a car that goes at highway speeds.

At 80mph on dry pavement, you can hard brake in 100m. On wet pavement it’s 220m to stop, so you should not be going 80mph but people do. But a system, like a human, needs time to decide if it needs to brake, and it also doesn’t want to brake really hard. It usually takes at least 100ms just to receive a frame from sensors, and more time to figure out what it all means. No system figures it out from just one frame, usually a few frames have to be analyzed.

On top of that, 200m out, you really only get a few spots on a LIDAR from a car, and one spot from a pedestrian. That first spot is a warning but not enough to trigger braking. You need a few frames and want to see more spots and get more data to know what you’re seeing. And a safety margin on top of that. So people are very interested in what 1.5 micron LIDAR can see — as well as radar, which also sees that far.

The problem with building 1.5 micron LIDAR is that light that red is not detected with silicon. You need more exotic stuff, like indium gallium arsenide. This is not really “exotic” but compared to silicon it is. Our world knows a lot about how to do things with silicon, and making things with it is super mature and cheap. Anything else is expensive in comparison. Made in the millions, other materials won’t be.

The new Velodyne has 128 lasers and 128 photodetectors and costs a lot. 128 lasers and detectors for 1.5 micron would be super costly today. That’s why Luminar’s design uses two modules, each with a single laser and detector. The beam is very powerful, and moved super fast. It is steered by moving a mirror to sweep out rasters (scan lines) both back and forth and up and down. The faster you move your beam the more power you can put into it — what matters is how much energy it puts into your eye as it is sweeping over it, and how many times it hits your eye every second.

The Luminar can do a super detailed sweep if you only want one sweep per second, and the point clouds (collections of spots with the distance and brightness measured, projected into 3D) look extremely detailed and nice. To drive, you need at least 10 sweeps per second, and so the resolution drops a lot, but is still good.

Another limit which may surprise you is the speed of light. To see something 250m away, the light takes 1.6 microseconds to go out and back. This limits how many points per second you can get from one laser. Speeding up light is not an option. There are also limits on how much power you can put through your laser before it overheats.

(To avoid overheating, you can concentrate your point budget on the regions of greatest interest, such as those along the road or those that have known targets. I describe this in This US patent. While I did not say it, several people reported to me that Luminar’s current units require a fairly hefty box to deliver power to their LIDAR and do the computational post processing to produce the pretty point clouds.

Luminar’s device is also very expensive (they won’t publish the price) but Toyota built a test vehicle with 4 of their double-laser units, one in each of the 4 directions. Some teams are expressing the desire to see over 200m in all directions, while others think it is really only necessary to see that far in the forward direction.

You do have to see far to the left and right when you are entering an intersection with cross traffic, and you also have to see far behind if you are changing lanes on a German autobahn and somebody might be coming up on your left 100km/h faster than you (it happens!) Many teams feel that radar is sufficient there, because the type of decision you need to make (do I go or don’t I) is not nearly so complex and needs less information.

As noted before, while most early LIDARS were in the 900nm bands, Google/Waymo built their own custom LIDAR with long range, and the effort of Uber to build the same is the subject of the very famous lawsuit between the two companies. Princeton Lightwave, another company making 1.5 micron LIDAR, was recently acquired by Ford/Argo AI — an organization run by Bryan Salesky, another Google car alumnus.

I saw a few other companies with 1.5 micron LIDARS, but not as far along as Luminar, but several pointing out they did not need the large box for power and computing, suggesting they only needed about 20 to 40 watts for the whole device. One was Innovusion, which did not have a booth, but showed me the device in their suite. However, in the suite it was not possible to test range claims.

Tetravue and new Time of Flight

Tetravue showed off their radically different time of flight technology. So far there have been only a few methods to measure how long it takes for a pulse of light to go out and come back, thus learning the distance.

The classic method is basic sub-nanosecond timing. To get 1cm accuracy, you need to measure the time at close to about 50 picosecond accuracy. Circuits are getting better than that. This needs to be done with both scanning pulses, where you send out a pulse and then look in precisely that direction for the return, or with “flash” LIDAR where you send out a wide, illuminating pulse and then have an array of detector/timers which count how long each pixels took to get back. This method works at almost any distance.

The second method is to use phase. You send out a continuous beam but you modulate it. When the return comes back, it will be out of phase with the outgoing signal. How much out of phase depends on how long it took to come back, so if you can measure the phase, you can measure the time and distance. This method is much cheaper but tends to only be useful out to about 10m.

Tetravue offers a new method. They send out a flash, and put a decaying (or opening) shutter in front of an ordinary return sensor. Depending on when the light arrives, it is attenuated by the shutter. The amount it is attenuated tells you when it arrived.

I am interested in this because I played with such designs myself back in 2011, instead proposing the technique for a new type of flash camera with even illumination but did not feel you could get enough range. Indeed, Tetravue only claims a maximum range of 80m, which is challenging — it’s not fast enough for highway driving or even local expressway driving, but could be useful for lower speed urban vehicles.

The big advantages of this method are cost — it uses mostly commodity hardware — and resolution. Their demo was a 1280×720 camera, and they said they were making a 4K camera. That’s actually too much resolution for most neural networks, but digital crops from within it could be very happy, and make for the best object recognition result to be found, at least on closer targets. This might be a great tool for recognizing things like pedestrian body language and more.

At present the Tetravue uses light in the 800nm bands. That is easier to receive with more efficiency on silicon, but there is more ambient light from the sun in this band to interfere.

The different ways to steer

In addition to differing ways to measure the pulse, there are also many ways to steer it. Some of those ways include:

• Mechanical spinning — this is what the Velodyne and other round lidars do. This allows the easy making of 360 degree view LIDARs and in the case of the Velodyne, it also stops rain from collecting on the instrument. One big issues is that people are afraid of the reliability of moving parts, especially on the grand scale.
• Moving, spinning or vibrating mirrors. These can be sealed inside a box and the movement can be fairly small.
• MEMS mirrors, which are microscopic mirrors on a chip Still moving, but effectively solid state. These are how DLP projectors work. Some new companies like Innovision featured LIDARs steered this way.
• Phased arrays — you can steer a beam by having several emitters and adjusting the phase so the resulting beam goes where you want it. This is entirely solid state.
• Spectral deflection — it is speculated that some LIDARS do vertical steering by tuning the frequency of the beam, and then using a prism so this adjusts the angle of the beam.
• Flash LIDAR, which does not steer at all, and has an array of detectors

There are companies using all these approaches, or combinations of them.

The range of 900nm LIDAR

The most common and cheapest LIDARS, are as noted, in the 900 nm wavelength band. This is a near infrared band, but it is far enough from visible that not a lot of ambient light interferes. At the same time, up here, it’s harder to get silicon to trigger on the photons, so it’s a trade-off.

Because it acts like visible light and is focused by the lens of the eye, keeping eye safe is a problem. At a bit beyond 100m, at the maximum radiation level that is eye safe, fewer and fewer photons reflect back to the detector from dark objects. Yet many vendors are claiming ranges of 200m or even 300m in this band, while others claim it is impossible. Only a hands-on analysis can tell how reliably these longer ranges can actually be delivered, but most feel that can’t be done at the level needed.

There are some tricks which can help, including increasing sensitivity, but there are physical limits. One technique that is being considered is dynamic adjustment of the pulse power, reducing it when the target is close to the laser. Right now, if you want to send out a beam that you will see back from 200m, it needs to be so powerful that it could hurt the eye of somebody close to the sensor. Most devices try for physical eye-safety; they don’t emit power that would be unsafe to anybody. The beam itself is at a dangerous level, but it moves so fast that the total radiation at any one spot is acceptable. They have interlocks so that the laser shuts down if it ever stops moving.

To see further, you would need to detect the presence of an object (such as the side of a person’s head) that is close to you, and reduce the power before the laser scanned over to their eyes, keeping it low until past the head, then boosting it immediately to see far away objects behind the head. This can work, but now a failure of the electronic power control circuits could turn the devices into a non-safe one, which people are less willing to risk.

The Price of LIDARs

LIDAR prices are all over the map, from the $100,000 Velodyne 128-line to solid state units forecast to drop close to $100. Who are the customers that will pay these prices?

High End

Developers working on prototypes often chose the very best (and thus most expensive) unit they can get their hands on. The cost is not very important on prototypes, and you don’t plan to release for a few years. These teams make the bet that high performance units will be much cheaper when it’s time to ship. You want to develop and test with the performance you can buy in the future.’

That’s why a large fraction of teams drive around with $75,000 Velodynes or more. That’s too much for production unit but they don’t care about that. It made people predict incorrectly that robocars are far in the future.

Middle End

Units in the $7,000 to $20,000 range are too expensive as an add-on feature for a personal car. There is no component of a modern car that is this expensive, except the battery pack in a Tesla. But for a taxi, it’s a different story. With so many people sharing it, the cost is not out of the question compared to the human driver which is taken out of the equation. In fact, some would argue the $100,000 LIDAR over 6 years is still cheaper than that.

In this case, cost is not the big issue, it’s performance. Everybody wants to be out on the market first, and if a better sensor can get you out a year sooner, you’ll pay it.

Low end

LIDARs that cost (or will cost) below $1,000, especially below $250, are the ones of interest to major automakers who still think the old way: Building cars for customers with a self-drive feature.

They don’t want to add a great deal to the bill of materials for their cars and are driving all the low end, typically solid state devices.

None

None of these LIDARS are available today in automotive quantities or quality levels. Thus you see companies like Tesla, who want to ship a car today, designing without LIDAR. Those who imagine LIDAR as expensive believe that lower cost methods, like computer vision with cameras, are the right choice. They are right in the very short run (because you can’t get a LIDAR) and in the very long run (when cost will become the main driving factor) but probably wrong in the time scales that matter.

Some LIDARs are CES

Here is a list of some of the other LIDAR companies I came across at CES. There were even more than this.

AEye — MEMS LIDAR and software fused with visible light camera

Cepton — MEMS-like steering, claims 200 to 300m range

Robosense RS-lidar-m1 pro — claims 200m MEMS, 20fps, .09 deb by .2 deg, 63 x 20

Surestar R-Fans (16 and 32 laser, puck style) to 20hz,

Leishen MX series — short range for robots, 8 to 48 line units, (puck style)

ETRI LaserEye — Korean research product

Benewake Flash Lidar, shorter range.
.
Infineon box LIDAR (Innoluce prototype)

Innoviz, mems mirror 900nm LIDAR with claimed longer range

Leddartech — older company from Quebec, now making a flash LIDAR

Ouster — $12,000 super light LIDAR.

To come: Thermal Sensors, Radars, and Computer Vision, better cameras

Credit:Waymo

Here are the biggest Robocar stories of 2017.

Waymo starts pilot with no safety driver behind the wheel

By far, the biggest milestone of 2017 was the announcement by Waymo of their Phoenix Pilot which will feature cars with no safety driver behind the wheel, and the hints at making this pilot open to the public.

The huge deal is that Waymo’s lawyers and top executives signed off on the risk of running cars with no safety driver to take over in emergencies. There is still an employee in the back who can do an emergency shutdown but they can’t grab the traditional controls. A common mistake in coverage of robocars is to not understand that it’s “easy” to make a car that can do a demo, but vastly harder to make one that has this level of reliability. That Waymo is declaring this level puts them very, very far ahead of other teams.

Many new LIDAR and other sensor companies enter the market

The key sensor for the first several years of robocars will almost surely be LIDAR. At some point in the future, vision may get good enough but that date is quite uncertain. Cost is not a big issue for the first few years, safety is. So almost everybody is gearing up to use LIDAR, and many big companies and startups have announced new LIDAR sensors and lower prices.

News includes Quanergy (I am an advisor) going into production on a $250 8-line solid state unit, several other similar units in development from many companies, and several new technologies including 1.5 micron LIDARs from Luminar and Princeton Lightwave, 128 plane LIDARs from Velodyne and radical alternate technologies from Oryx in Israel and others. In addition, several big players have acquired LIDAR companies, indicating they feel it is an important competitive advantage.

At the same time, Waymo (which created its own special long range LIDAR) has been involved in a giant lawsuit against Uber, alleging that the Otto team appropriated Waymo secrets to build their own.

Here is some coverage I had on LIDAR deals.

In more recent news, today Velodyne cut the price of their 16 laser puck to $4,000. 16 planes is on the low side as a solo sensor but this price is quite reasonable for anybody building a taxi.

Regulations get reversed.

In 2016 NHTSA published 116 pages of robocar regulations. Under the new administration, they reversed this and published some surprisingly light handed replacements. States have also been promoting local operations, with Arizona coming out as one of the new winners.

Intel buys MobilEye

There were many big acquisitions with huge numbers, including NuTonomy (by Delphi) but the biggest ever deal was the $16B purchase of MobilEye by Intel.
MobilEye of course has a large business in the ADAS world but Intel wants the self-driving car part and paid a multi billion dollar premium for it.

Uber orders 24,000 Volvos

It’s not a real order quite yet but this intent to buy $1B of cars to put Uber software on shows how serious things are getting, and should remove from people’s minds the idea that Uber doesn’t intend to own a fleet.

Flying cars get a tiny bit more real

They aren’t here yet, but there’s a lot more action on Flying Cars, or in particular, multirotor drone-style vehicles able to carry a person. It looks like these are going to happen, and they are the other big change in the works for personal transportation. It remains uncertain if society will tolerate noisy helicopters filling the skies over our cities, but they certainly will be used for police, ambulance, fire and other such purposes, as well as over water and out in the country.

A little more uncertain is the Hyperloop. While the science seems to work, the real question is one of engineering and cost. Can you actually do evacuated tubes reliably and at a cost that works?

Intel and Warner made a splash at the LA Auto Show announcing how Warner will develop entertainment for viewing while riding in robotaxis. It’s not just movies to watch, their hope is to produce something more like an amusement park ride to keep you engaged on your journey.

Like most partnership announcements around robocars, this one is mainly there for PR since they haven’t built anything yet. The idea is both interesting and hype.

I’ll start with the negative. I think people will carry their entertainment with them in their pockets, and not want it from their cars. Why would I want a different music system with a different interface when my own music and videos are already curated by me and stored in my phone? All I really want is a speaker and screen to display them on.

This is becoming very clear on planes, where I prefer to watch movies I have pre-downloaded on my phone than what is on the bigger screen of the in-flight entertainment system. There are several reasons for that:

• The UIs on most in-flight systems suck really, really badly. I mean it’s amazing how bad most of them are. (Turns out there is a reason for that.) Cars will probably do it better but the history is not promising.
• Your personal device is usually newer with more advanced technology because you replace it every 2 years. You have curated the content in it and know the interface.
• On airplanes in particular, they believe rules force them to pause your experience so that they can announce that duty free sales are now open in 3 languages. And 20 or more other interruptions, only a couple of which are actually important to hear for an experienced flyer.

So Warner is wise in putting a focus on doing something you can’t do with your personal gear, such as a VR experience, or immersive screens around the car. There is a unique opportunity to tune the VR experience to the actual motions of the car. In traffic, you can only tune to the needed motions. On the open road, you might actually be able to program a trip that deliberately slows or speeds up or turns when nobody else is around to make a cool VR experience.

While that might be nice, it would be mostly a gimmick, more like a ride you try once. I don’t think people will want to go everywhere in the batmobile. As such it will be more of a niche, or marketing trick.

More interesting is the ability to reduce carsickness with audio-visual techniques. Some people get pretty queasy if they look down for a long time at a book or laptop. Others are less bothered. A phone held in the hand seems to be easier to use for most than something heavier, perhaps because it moves with the motion of the car. For many years I have proposed that cars communicate their upcoming plans with subtle audio or visual cues so that people know when they are about to turn or slow down. Some experiments are now being reported on this and it will be interesting to see the results.

If you ride on a subway, bus or commuter train today, the scene is now always the same. A row of people, all staring at their phones.

Advertising

Some commenters have speculated that another goal here may be to present advertising to hapless taxi passengers. After all, ride a New York cab and many others and you will see an annoying video loop playing. Each time you have to go through the menus to mute the volume. With hailed taxis, you can’t shop, and so they can also get away with doing this — what are you going to do, get out of the cab and wait for the next one?

I hope that with mobile-phone hail, competition prevents this sort of attempt to monetize the time of the customer. I definitely want my peace and quiet, and the revenue from the advertising — typically well under a dollar an hour — can’t possibly offset that for me.

A small mystery from Robocar history was resolved recently, and revealed at the DARPA grand challenge reunion at CMU.

The story is detailed here at IEEE spectrum and I won’t repeat it all, but a brief summary goes like this.

In the 2nd grand challenge, CMU’s Highlander was a favourite and doing very well. Mid-race it started losing engine power and it stalled for long enough that Stanford’s Stanley beat it by 11 minutes.

It was discovered recently a small computerized fuel injector controller in the Hummer (one of only two) may have been damaged in a roll-over that Highlander had, and if you pressed on it, the engine would reduce power or fail.

People have wondered how the robocar world might be different if they had not had that flaw. Stanford’s victory was a great boost for their team, and Sebastian Thrun was hired to start Google’s car team — but Chris Urmson, lead on Highlander, was also hired to lead engineering, and Chris would end up staying on the project for much longer than Sebastian who got seduced by the idea of doing Udacity. Google was much more likely to have closer ties to Stanford people anyway, being where it is.

CMU’s fortunes might have ended up better, but they managed to be the main source of Uber’s first team.

There are many stories of small things making a big difference. Also well known is how Anthony Levandowski, who entered a motorcycle in the race, forgot to turn on a stabilizer. The motorcycle fell over 2 seconds after he released it, dashing all of his team’s work. Anthony of course did OK (as another leader on the Google team, and then to Uber) but of course has recently had some “trouble”.

Another famous incident came when Volvo was doing a demo for press of their collision avoidance system. You could not pick a worse time for a failure, and of course there is video of it.

They had tested the demo extensively the night before. In fact they tested it too much, and left a battery connected during the night, so that it was drained by the morning when they showed off to the press.

These stories remind people of all the ways things go wrong. More to the point, they remind us that we must design expecting things to go wrong, and have systems that are able to handle that. These early demos and prototypes didn’t have that, but cars that go on the road do and will.

Making systems resilient is the only answer when they get as complex as they are. Early car computers were pretty simple, but a self-driving system is so complex that it is never going to be formally verified or perfect. Instead, it must be expected that every part will fail, and the failure of every part — or even every combination of parts — should be tested in both simulation, and where possible in reality. What is tested is how the rest of the system handles the failure, and if it doesn’t handle it, that has to be fixed.

It does not need to handle it perfectly, though. For example, in many cases the answer to failure will be, “We’re at a reduced safety level. Let’s get off the road, and summon another car to help the passengers continue on their way.”

It might even be a severely reduced safety level. Possibly even, as hard as this number may be to accept, 100 times less safe! That’s because the car will never drive very far in that degraded condition. Consider a car that has one incident every million miles. In degraded condition, it might have an incident every 10,000 miles. You clearly won’t drive home in that condition, but the 1/4 mile of driving at degraded level is as risky as 25 miles of ordinary driving at full operational level, which is a risk taken every day. As long as vehicles do not drive more than a short distance at this degraded level, the overall safety record should still be satisfactory.

Of course, if the safety level degrades to a level that could be called “dangerous” rather than “less safe” that’s another story. That must never be allowed.

An example of this would be failure of the main sensors, such as a LIDAR. Without a LIDAR, a car would rely on cameras and radar. Companies like Tesla think they can make a car fully safe with just those two, and perhaps they will some day. But even though those are not yet safe enough, they are safe enough for a problem like getting off the road, or even getting to the next exit on a highway.

This is important because we will never get perfection. We will only get lower and lower levels of risk, and the risk will not be constant — it will be changing with road conditions, and due to system or mechanical failures. But we can still get the safety level we want — and get the technology on the road.

Uber and Volvo announced an agreement where Uber will buy, in time, up to 24,000 specially built Volvo XC90s which will run Uber’s self-driving software and, presumably, offer rides to Uber customers. While the rides are some time away, people have made note of this for several reasons.

• This is a pretty big order for Volvo — it’s $1B of cars at the retail price, and 1/3 of the total sales of XC90s in 2017.
• This is a big fleet — there are only 12,000 yellow cabs in New York City, for example, though thanks to Uber there are now far more hailable vehicles.
• In spite of Volvo’s fairly major software efforts, they will be entirely on the hardware side for this deal, and it is not exclusive for either party.

I’m not clear who originally said it — I first heard it from Marc Andreesen — but “the truest form of a partnership is called a purchase order.” In spite of the scores of announced partnerships and joint ventures announced to get PR in the robocar space, this is a big deal, but it’s a sign of the sort of deal car makers have been afraid of. Volvo will be primarily a contract manufacturer here, and Uber will own the special sauce that makes the vehicle work, and it will own the customer. You want to be Uber in this deal, But what company can refuse a $1B order?

It also represents a big shift for Uber. Uber is often the poster child for the company that replaced assets with software. It owns no cars and yet provides the most rides. Now, Uber is going to move to the capital intensive model of owning the cars, and not having to pay drivers. There will be much debate over whether it should make such a shift. As noted, it goes against everything Uber represented in the past, but there is not really much choice.

First of all, to do things the “Uber” way would require that a large number of independent parties bought and operated robocars and then contracted out to Uber to bring them riders when not being used by their owners. Like UberX without having to drive the car. The problem is, that world is still a long way away. Car companies have put their focus on cars that can’t drive unmanned — much or at all — because that’s OK for the private car buyer. They are also far behind companies like Waymo and Uber in producing taxi capable vehicles.

If Uber waited for the pool of available private cars to get large enough, it would miss the boat. Other companies would have moved into its territory and undercut it with cheaper and cooler robotaxi service.

Secondly, you really want to be very sure about the vehicles you deploy in your first round. You want to have tested them, and you need to certify their safety because you are going to be liable in accidents no matter what you do. You can get the private owners to sign a contract taking liability but you will get sued anyway as the deep pocket if you do. This means you want to control the whole experience.

The truth is, capital is pretty cheap for companies like Uber. Even cheaper for companies like Apple and Google that have the world’s largest pools of spare capital sitting around. The main risk is that these custom robocars may not have any resale value if you bet wrong on how to build them. Fortunately, taxis wear out in about 5 years of heavy use.

Uber continues to have no fear of telling the millions of drivers who work “for” them that they will be rid of them some day. Uber driver is an unusual job, and nobody thinks of it as a career, so they can get away with this.

Trolley problem gets scarier

Academic ethicists, when defending discussions of the Trolley Problem claim that while they understand the problems are not real, they are still valuable teaching tools to examine real questions.

The problem is the public doesn’t understand this, and is morbidly fascinated beyond all rationality with the idea of machines deciding who lives or dies. This has led Barack Obama to ask about it in his first statement on robocars, and many other declarations that we must figure out this nonsense question before we deploy robocars on the road. The now-revoked proposed NHTSA guidelines of 2016 included a theoretically voluntary requirement that vendors outline their solutions to this “problem.”

This almost got more real last week when a proposed UK bill would have demanded trolley solutions. The bill was amended at the last minute, and a bullet dodged that would have delayed the deployment of life saving technology while trying to resolve truly academic questions.

It is time for ethical ethicists to renounce the trolley problem. Even if, inside, they still think it’s got value, that value is far outweighed by the irrational fears and actions it triggers in public debate. Real people are dying every day on the roads, and we should not delay saving them to figure out how to do the “right” thing in hypothetical situations that are actually extremely rare to nonexistent. Figuring out the right thing is the wrong thing. Save solving trolley problems for version 4, and get to work on version 0.2.

There is real ethical work to be done, covering situations that happen every day. Real world safety tradeoffs and their morality. Driving on roads where breaking the vehicle code is the norm. Contrasting cost with safety. These are the places where ethical expertise can be valuable.

Simulators take off

For a long time I have promoted the idea of an open source simulator. Now two projects are underway.

The first is the project Apollo simulator from Baidu and a new entrant called Carla is also in the game.

This is good to see, but I hope the two simulators also work together. One real strength of an open platform simulator is that people all around the world can contribute scenarios to it, and then every car developer can test their system in those scenarios. We want every car tested in every scenario that anybody can think of.

Waymo has developed its own simulator, and fed it with every strange thing their cars have encountered in 5M kilometers of real world driving. It’s one of the things that gives them an edge. They’ve also loaded the simulator with everything their team members can think of. This way, their driving system has the experience of seeing and trying out every odd situation that will be encountered in many lifetimes of human driving, and eventually on every type of road.

That’s great, but no one company can really build it all. This is one of the great things to crowdsource. Let all the small developers, all the academics, and even all the hobbyists build simulations of dangerous scenarios. Let people record and build scenarios for driving in every city of the world, in every situation. No one company can do that but the crowd can. This can give us the confidence that any car has at least at some level encountered far more than any human driver ever could, and handled it well.

Unusual liability rule

Some auto vendors have proposed a liability rule for privately owned robocars that will protect them from some liability. The rule would declare that if you bought a robocar from them, and you didn’t maintain it according to the required maintenance schedule then the car vendor would not be liable for any accident it had.

It’s easy to see why automakers would want this rule. They are scared of liability and anything that can reduce it is a plus for them.

At the same time, this will often not make sense. Just because somebody didn’t change the oil or rotate the tires should not remove liability for a mistake by the driving system that had no relation to those factors.

What’s particularly odd here is that robocars should always be very well maintained. That’s because they will be full of sensors to measure everything that’s going on, and they will also be able to constantly test every system that can be tested.
Consider the brakes, for example. Every time a robocar brakes, it can measure that the braking is happening correctly. It can measure the temperature of the brake discs. It can listen to the sound or detect vibrations. It can even, when unmanned, find itself on an empty street and hit the brakes hard to see what happens.

In other words, unexpected brake failure should be close to impossible (particularly since robocars are being designed with 2 or 3 redundant braking systems.)

More to the point, a robocar will take itself in for service. When your car is not being used, it will run over for an oil change or any other maintenance it needs. You would have to deliberately stop it to prevent it from being maintained to schedule. Certainly no car in a taxi fleet will remain unmaintained except through deliberate negligence.

Robocar news is fast and furious these days. I certainly don’t cover it all, but will point to stories that have some significance. Plus, to tease you, here’s a clip from my 4K video of the new Apple car that you’ll find at the end of this post.

Lidar acquisitions

There are many startups in the Lidar space. Recently, Ford’s Argo division purchased Princeton Lightwave a small LIDAR firm which was developing 1.5 micron lidars. 1.5 micron lidars include Waymo’s own proprietary unit (subject of the lawsuit with Uber) as well as those from Luminar and a few others. Most other lidar units work in the 900nm band of near infrared.

Near infrared lasers and optics can be based on silicon, and silicon can be cheap because there is so much experience and capacity devoted to making it. 1.5 micron light is not picked up by silicon, but it’s also not focused by the lens of the eye. That means that you can send out a lot more power and still not hurt the eye, but your detectors are harder to make. That extra power lets you see to 300m, while 900nm band lidars have trouble with black objects beyond 100m.

100m is enough for urban driving, but is not a comfortable range for higher speeds. Radar senses far but has low resolution. Thus the desire for 1.5 micron units.

GM/Cruise also bought Strobe, a small lidar firm with a very different technology. Their technology is in the 900nm band, but they are working on ways to steer the beam without moving mirrors the way Velodyne and others do. (Quanergy, in which I have stock, also is developing solid state units, as are several others.) They have not published but there is speculation on how Strobe’s unit works.

What’s interesting is that these players have decided, like Waymo, Uber and others, that they should own their own lidar technology, rather than just buy it from suppliers. This means one of two things:

• They don’t think anybody out there can supply them with the LIDAR they need — which is what motivated Waymo to build their own, or
• They think their in-house unit will offer them a competitive advantage

On the surface, neither of these should be true. Suppliers are all working on making lidars because most people think they will be needed. And folks are working on both 900nm and 1.5 micron units, eager to sell. It’s less clear if any of these units will be significantly better than the ones the independent suppliers are building. That’s what is needed to get a competitive edge. The unit needs to be longer range, better resolution, better field of view or more reliable than supplier units. It’s not clear why that will be, but nobody has released solid specs.

What shouldn’t matter is that they can make it cheaper in-house, especially for those working on taxi service. First of all, it’s very rare you can get something cheaper by buying the entire company. Secondly, it’s not important to make it much cheaper for the first few years of production. Nobody is going to win or lose based on whether their taxi unit costs a few thousand more dollars to make.

So there must be something else that is not revealed driving these acquisitions.

Velodyne, which pioneered the lidar industry for self-driving cars, just announced their new
128 line lidar with twice the planes and 4x the resolution of the giant “KFC Bucket” unit found on most early self-driving car prototypes.

The $75,000 64-laser Velodyne kick-started the industry, but it’s big and expensive. This new one will surely also be expensive but is smaller. In a world where many are working with the 16 and 32 laser units, the main purpose of this unit, I think, will be for those who want to develop with the sensor of the future.

Doing your R&D with high-end gear is often a wise choice. In a few years, the high resolution gear will be cheaper and ready for production, and you want to be ready for that. At the same time, it’s not yet clear how much 128 lines gains over 64. It’s not easy to identify objects in lidar, but you don’t absolutely have to so most people have not worried too much about it.

Pioneer, the Japanese electronics maker, has also developed a new lidar. Instead of trying to steer a laser entirely with solid state techniques, theirs uses MEMS mirrors, similar to those in DLP projectors. This is effectively solid state even though the mirrors actually move. I’ve seen many lidar prototypes that use such mirrors but for some reason they have not gone into production. It is a reasonably mature technology, and can be quite low cost.

More acquisitions and investment

Delphi recently bought Nutonomy, the Singapore/MIT based self-driving car startup. I remember visiting them a few time s in Singapore and finding them to be not very far along compared to others. Far enough along to fetch $400M. Delphi is generally one of the better-thinking tier one automotive suppliers and now it can go full-stack with this purchase.

Of course, since most automakers have their own full stack efforts underway, just how many customers will the full-stack tier one suppliers sell to? They may also be betting that some automakers will fail in their projects, and need to come to Delphi, Bosch or others for rescue.

Another big investment is Baidu’s “Project Apollo.” This “moonshot” is going to invest around $1.5B in self-driving ventures, and support it with open source tools. They have lots of partners, so it’s something to watch.

Other players push forward

Navya was the first company to sell a self-driving car for consumer use. Now their new vehicle is out. In addition, yesterday in Las Vegas, they started a pilot and within 2 hours had a collision. Talk about bad luck — Navya has been running vehicles for years now without such problems. It was a truck that backed into the Navya vehicle, and the truck driver’s fault, but some are faulting it because all it did was stop dead when it saw the truck coming. It did not back out of the way, though it could have. Nobody was hurt.

Aurora, the startup created by Chris Urmson after he left Waymo, has shown off its test vehicles. No surprise, they look very much like the designs of Waymo’s early vehicles, a roof rack with a Velodyne 64 laser unit on top. The team at Aurora is top notch, so expect more.

Apple’s cars are finally out and about. Back in September I passed one and took a video of it.

You can see it’s loaded with sensors. No fewer than 12 of the Velodyne 16 laser pucks and many more to boot. Apple is surely following that philosophy of designing for future hardware.

Credit:Waymo

In a major milestone for robocars, Waymo has announced they will deploy in Phoenix with no human safety drivers behind the wheel. Until now, almost all robocars out there have only gone out on public streets with a trained human driver behind the wheel, ready to take over at any sign of trouble. Waymo and a few others have done short demonstrations with no safety driver, but now an actual pilot, providing service to beta-testing members of the public, will operate without human supervision.

https://youtube.com/watch?v=aaOB-ErYq6Y%3Frel%3D0

This is a big deal, and indicates Waymo’s internal testing is showing a very strong safety record. The last time they published numbers, they had gone 83,000 miles between “required interventions.” While in safety driver training, we are told to intervene at any sign of a problem, these interventions are tested in simulation to find out what would have happened if there had been no intervention. If the car would have done the right thing, it’s not a required intervention.

Waymo must have built their number up a great deal from there. People have an accident that is reported to insurance about ever 250,000 miles, and to police every 500,000 miles. Injury accidents happen every 1.2M miles, and fatalities every 80M miles. In Waymo’s testing, where they got hit a lot by other drivers, they discovered that there are “dings” about every 100,000 miles that don’t get reported to police or insurance.

People have argued about how good you have to be to put a robocar on the road. You need to be better than all those numbers. I will guess that Waymo has gotten the “ding” number up above 500,000 miles — which is close to a full human lifetime of driving. Since they have only driven 3.5M miles they can’t make real-world estimates of the frequency of injuries and certainly not of fatalities, but they can make predictions. And their numbers have convinced them, and the Alphabet management, that it’s time to deploy.

Congratulations to all the team.

They did this not just with real world testing, but building a sophisticated simulator to test zillions of different situations, and a real world test track where they could test 20,000 different scenarios. And for this pilot they are putting it out on the calm and easy streets of Phoenix, probably one of the easiest places to drive in the world. Together, that gives the confidence to put “civilians” in the cars with no human to catch an error. Nothing will be perfect, but this vehicle should outperform a human driver. The open question will be how the courts treat that when the first problem actually does happen. Their test record suggests that may be a while; let us hope it is.

Where do we go from here?
This pilot should give pause to those who have said that robocars are a decade or more away, but it also doesn’t mean they are full here today. Phoenix was chosen because it’s a much easier target than some places. Nice, wide streets in a regular grid. Flat terrain. Long blocks. Easy weather with no snow and little rain. Lower numbers of pedestrians and cyclists. Driving there does not let you drive the next day in Boston.

But neither does it mean it takes you decades to go from Phoenix to Boston, or even to Delhi. As Waymo proves things out in this pilot, first they will prove the safety and other technical issues. Then they will start proving out business models. Once they do that, prepare for a land rush as they leap to other cities to stake the first claim and the first-mover advantage (if there is one, of course.) And expect others to do the same, but later than Waymo, because as this demonstrates, Waymo is seriously far ahead of the other players. It took Waymo 8 years to get to this, with lots of money and probably the best team out there. But it’s always faster to do something the 2nd time. Soon another pilot from another company will arise, and when it proves itself, the land rush will really begin.

Many websites paint a very positive picture of the robocar future. And it is positive, but far from perfect. One problem I worry about in the short term is the way robocars are going to make traffic worse before they get a chance to make it better.

The goal of all robocars is to make car travel more pleasant and convenient, and eventually cheaper. You can’t make something better and cheaper without increasing demand for it, and that means more traffic.

This is particularly true for the early-generation pre-robocar vehicles in the plans of many major automakers. One of the first products these companies have released is sometimes called the “traffic jam assist.” This is a self-driving system that only works at low speed in a traffic jam.

Turns out that’s easy to do, effectively a solved problem. Low speed is inherently easier, and the highway is a simple driving environment without pedestrians, cyclists, intersections or cars going the other way. When you are boxed in with other cars in a jam, all you have to do is go with the flow. The other cars tell you where you need to go. Sometimes it can be complex when you get to whatever is blocking the road to cause the jam, but handoff to a human at low speeds is also fairly doable.

These products will be widely available soon, and they will make traffic jams much more pleasant. Which means there might be more of them.

I don’t have a 9 to 5 job, so I avoid travel in rush hour when I can. If somebody suggests we meet somewhere at 9am, I try to push it to 9:30 or 10. If I had a traffic jam assist car, I would be more willing to take the meeting at 9. When on the way, if I encountered a traffic jam, I would just think, “Ah, I can get some email done.”

After the traffic jam assist systems, come the highway systems which allow you to take your eyes off the road for an extended time. They arrive pretty soon, too. These will encourage slightly longer commutes. That means more traffic, and also changes to real estate values. The corporate-run commuter buses from Google, Yahoo and many other tech companies in the SF Bay Area have already done that, making people decide they want to live in San Francisco and work an hour’s bus ride away in Silicon Valley. The buses don’t make traffic worse, but those doing this in private cars will.

Is it all doom?

Fortunately, some factors will counter a general trend to worse traffic, particularly as full real robocars arrive, the ones that can come unmanned to pick you up and drop you off.

• As robocars reduce accident levels, that will reduce one of the major causes of traffic congestion.
• Robocars don’t need to slow down and stare at accidents or other unusual things on the road, which also causes congestion.
• Robocars won’t overcompensate on “sags” (dips) in the road. This overcompensation on sags is the cause of almost half the traffic congestion on Japanese highways
• Robocars look like they’ll be mainly electric. That doesn’t do much about traffic, but it does help with emissions.
• Short-haul “last mile” robocars can actually make the use of trains, buses and carpools vastly more convenient.
• Having only a few cars which drive more regularly, even something as simple as a good quality adaptive cruise control, actually does a lot to reduce congestion.
• The rise of single person half-width vehicles promises a capacity increase, since when two find one another on the road, they can share the lane.
• While it won’t happen in the early days, eventually robocars will follow the car in front of them with a shorter gap if they have a faster reaction time. This increases highway capacity.
• Early robocars won’t generate a lot of carpooling, but it will pick up fairly soon (see below.)

What not to worry about

There are a few nightmare situations people have talked about that probably won’t happen. Today, a lot of urban driving involves hunting for parking. If we do things right, robocars won’t ever hunt for parking. They (and you) will be able to make an online query for available space at the best price and go directly do it. But they’ll do that after they drop you off, and they don’t need to park super close to your destination the way you need to. To incorporate city spaces into this market, a technology upgrade will be needed, and that may take some time, but private spaces can get in the game quickly.

What also won’t happen is people telling their car to drive around rather than park, to save money. Operating a car today costs about $20/hour, which is vastly more than any hourly priced parking, so nobody is going to do that to save money unless there is literally no parking for many miles. (Yes, there are parking lots that cost more than $20, but that’s because they sell you many hours or a whole day and don’t want a lot of in and out traffic. Robocars will be the most polite parking customers around, hiding valet-style at the back of the lot and leaving when you tell them.)

Another common worry is that people will send their cars on long errands unmanned. That mom might take the car downtown, and send it all the way back for dad to do a later commute, then back to pickup the kids at school. While that’s not impossible, it’s actually not going to be the cheap or efficient thing to do. Thanks to robotaxis, we’re going to start thinking of cars as devices that wear out by the mile, not by the year, and all their costs will be by the mile except parking and $2 of financing per day. All this unmanned operation will almost double the cost of the car, and the use of robotic taxi services (Robocar Uber) will be a much better deal.

There will be empty car moves, of course. But it should not amount to more than 15% of total miles. In New York, taxis are vacant of a passenger for 38% of miles, but that’s because they cruise around all day looking for fares. When you only move when summoned, the rate is much better.

And then it gets better

After this “winter” of increased traffic congestion, the outlook gets better. Aside from the factors listed above, in the long term we get the potential for several big things to increase road capacity.

The earliest is dynamic carpooling, as you see with services like UberPool and LyftLines. After all, if you look at a rush-hour highway, you see that most of the seats going by are empty. Tools which can fill these seats can increase the capacity of the roads close to three times just with the cars that are moving today.

The next is advanced robocar transit. The ability to make an ad-hoc, on-demand transit system that combines vans and buses with last mile single person vehicles in theory allows almost arbitrary capacity on the roads. At peak hours, heavy use of vans and buses to carry people on the common segments of their routes could result in a 10-fold (or even more) increase in capacity, which is more than enough to handle our needs for decades to come.

Next after that is dynamic adaptation of roads. In a system where cities can change the direction of roads on demand, you can get more than a doubling of capacity when you combine it with repurposing of street parking. On key routes, street parking can be reserved only for robocars prior to rush hour, and then those cars can be told they must leave when rush hour begins. (Chances are they want to leave to serve passengers anyway.) Now your road has seriously increased capacity, and if it’s also converted to one-way in the peak direction, you could almost quadruple it.

The final step does not directly involve robocars, since all cars must have a smartphone and participate for it to work. This is the use of smart, internet based road metering. With complete metering, you never get more cars trying to use a road segment than it has capacity to handle, and so you very rarely get traffic congestion. You also don’t get induced demand that is greater than the capacity, solving the bane of transportation planners.

Recently Madrona Ventures, in partnership with Craig Mundie (former Microsoft CTO) released a white paper proposing an autonomous vehicle corridor between Seattle and Vancouver on I-5 and BC Highway 99. While there are some useful ideas in it, the basic concept contains some misconceptions about both traffic management, infrastructure planning, and robocars.

Carpool lanes are hard

The proposal starts with a call for allowing robocars in the carpool lanes, and then moving to having a robocar only lane. Eventually it moves to more lanes being robocar only, and finally the whole highway. Generally I have (mostly) avoided too much talk of the all-robocar road because there are so many barriers to this that it remains very far in the future. This proposal wants to make it happen sooner, which is not necessarily bad, but it sure is difficult.

Carpool lanes are poorly understood, even by some transportation planners. For optimum traffic flow, you want to keep every lane at near capacity, but not over it. If you have a carpool lane at half-capacity, you have a serious waste of resources, because the vast majority (around 90%) of the carpools are “natural carpools” that would exist regardless of the lane perk. They are things like couples or parents with children. A half-empty carpool lane makes traffic worse for everybody but the carpoolers, for whom the trip does improve.

That’s why carpool lanes will often let in electric cars, and why “high occupancy toll” lanes let in solo drivers willing to pay a price. In particular with the HOT lane, you can set the price so you get just enough cars in the carpool lane to make it efficient, but no more.

(It is not, of course, this simple, as sometimes carpool lanes jam up because people are scared of driving next to slow moving regular lanes, and merging is problematic. Putting a barrier in helps sometimes but can also hurt. An all-robocar lane would avoid these problems, and that is a big plus.)

Letting robocars into the carpool lane can be a good idea, if you have room. If you have to push electric cars out, that may not be the best public goal, but it is a decision a highway authority could make. (If the robocars are electric, which many will be, it’s OK.)

The transition, however, from “robocars allowed” to “robocars only” for the lane is very difficult. Because you do indeed have a decent number of carpools (even if only 10% are induced) you have to kick them out at some point to grow robocar capacity. You can’t have a switch day without causing more traffic congestion for some time after it. If you are willing to build a whole new lane (as is normal for carpool creation) you can do it, but only by wasting a lot of the new lane at first.

Robocar packing

Many are attracted to the idea that robocars can follow more closely behind another vehicle if they have faster reaction times. They also have the dream that the cars will be talking to one another, so they can form platoons that follow even more closely.) The inter car communication (V2V) creates too much computer security risk to be likely, though some still dream of a magic solution which will make it safe to have 1500kg robots exchanging complex messages with every car they randomly encounter on the road. Slightly closer following is still possible without it.

Platooning has a number of issues. It was at first popular as an idea because the lead car could be human driven. You didn’t have to solve the whole driving problem to make a platoon. Later experiments showed a number of problems, however.

• If not in a fully dedicated lane, other drivers keep trying to fit themselves into the gaps in a platoon, unless they are super-close
• When cars are close, they throw up stones from the road, constantly cracking windshields, destroying a car’s finish, and in some experiments, destroying the radiator!
• Any failure can be catastrophic, since multiple cars will be unable to avoid being in the accident.
• Fuel savings of workable following distances are around 10%. Nice, but not exciting.

To have platoons, you need cars designed with stone-shields or some other technique to stop stones from being thrown. You need a more secure (perhaps optical rather than radio) protocol for communication of only the simplest information, such as when brakes are being hit. And you must reach a safety level where the prospect of chain accidents is no longer frightening.

In any event, the benefits of packing are not binary. Rather, in a lane that is 90% robocars and 10% human, you get 90% of the benefit of a 100% robocar lane. There is no magic special benefit you get at 100% as far as packing is concerned. This is even true to some degree with the problems of erratic human drivers. Humans will brake for no good reason, and this causes traffic jams. Research shows that just a small fraction of robocars on the road who will react properly enough to this are enough to stop this from causing major traffic jams. There is actually a diminishing return from having more robocars. Traffic flow does need some gaps in it to absorb braking events, and while you could get away with fewer in an all robocar road, I am not sure that is wise. As long as you have a modest buffer, robocars trailing a human who brakes for no reason can absorb it and restore the flow as soon as the human speeds up again.

Going faster

There is a big benefit to all-robocar lanes if you are willing to allow the cars in that lane to drive much faster. That’s something that can’t happen in a mixed lane. The white paper makes only one brief mention of that benefit.

Other than this, the cars don’t get any great benefit from grouping. I mean, anybody would prefer to drive with robocars, which should drive more safely and more regularly. They won’t block the lane the way human drivers do. They will tailgate you (perhaps uncomfortably so) but they will only do so when it’s safe. They could cluster together to enjoy this benefit on their own, without any need for regulations.

The danger of robocar-only lanes

One of the biggest reasons to be wary of robocar only lanes is that while this proposal does not say it, most proposals have been put forward in the belief that robocars are not safe enough to mix with regular traffic. That is true today for the prototypes, but all teams plan to make vehicles which do meet that safety goal before they ship.

Many dedicated lane proposals have essentially called for robocar operation only in the dedicated lanes, and manual driving is required in other lanes. If you declare that the vehicles are not safe without a special lane, you turn them into vehicles with a very limited domain of operation. Since the creation of new dedicated lanes will be a very long (decades long) process, it’s an incredible damper on the deployment of the technology. “Keep those things in their own special lanes” means delay those things by decades.

The white paper does not advocate this. But there is a danger that the concept will be co-opted by those who do. As long as the benefits are minor, why take that risk?

Do we need it?

In general, any plan that calls for infrastructure change or political change is risky because of the time scales involved. It is quite common for governmental authorities to draft plans that take many years or decades to solve things software teams will solve in months or even, at the basic level, in hours. We want to be always sure that there is not a software solution before we start the long and high-momentum path of infrastructure change. Even change as simple as repainting.

Most of the benefits that come from all-robocar highway lanes arrive without mandating it. The ability for greater speed is the main one that doesn’t. All this happens everywhere, without planning, and political difficulty. Banning human drivers from lanes is going to be politically difficult. Banning them from the main artery would be even harder.

For great speed, I actually think that airplanes and potentially the hyperloop provide interesting answers, at least for trips of more than 150 miles. The white paper makes a very common poor assumption — that other technologies will stand still as we move to 2040. I know this is not true. I have big hopes for better aviation, including electric planes, robotic planes and most of all, better airports that create a seamless transfer from robocar to aircraft entirely unlike the nightmare we have built today.

On the ground, while I am not a fan of existing rail technology, new technologies like hyperloop are just starting to show some promise. If it can be built, hyperloop will be faster and more energy efficient, and through the use of smaller pods rather than long trains, offer travel without a schedule.

On the plus side, a plan for robocar only lanes is not a grand one. If you can sell it politically, you don’t need to build much infrastructure. It’s just some signs and new paint.

Some other users for all-robocar lanes

Once density is high enough, I think all-robocar lanes could be useful as barriers on a highway with dynamic lane assignment. To do this, you would just have a big wide stretch of pavement, and depending on traffic demand, allocate lanes to a direction. The problem is the interface lane. We may not want human drivers to drive at 75mph with other cars going the other way just 4 feet away. Robocars, however, could drive exclusively in the two border lanes, and do it safely. They would also drive a little off-center to create a larger buffer to avoid the wind-shake of passing close. No trucks in these lanes!

In an ideal situation, you would get a lot more capacity by paving over the shoulders and median to do this. With no median, though, you still have a risk of runaway cars (even robocars) crossing into oncoming traffic. A simpler solution would be to do this on existing highways. If you have a 6 lane highway, you could allocate 4 lanes one way and 2 the other, but insist that the two border lanes be robocars only, if we trust them. A breakdown by a robocar going in the counter-direction at high speed could still be an issue. Of course, this is how undivided highways are, but they have lower speeds and traffic flow.

General Motors announced this week that they would “take full responsibility” if a crash takes place during an autonomous driving trip. This follows a pledge to do the same made some time ago by Daimler, Google and Volvo and possibly others.

What’s interesting is that they don’t add the caveat “if the system is at fault.” Of course, if the system is not at fault, they can get payment from the other driver, and so it’s still OK to tell the passenger or owner that GM takes responsibility.

GM is moving on a rapid timetable with the technology they bought with Cruise not too long ago. In fact, rumours of a sooner than expected release actually shot their stock up a bit this week.

Even to this day I still see articles which ask the question, “who is liable in an accident?” and then don’t answer it as though the answer is unknown or hard to figure out. It never was. There was never any doubt that the creators of these vehicles would take responsibility for any accidents they cause. Even if they tried not to, the liability would fall to them in the court system. People have been slow to say it because lawyers always advise clients, “never say in advance that you will take liability for something!” Generally good advice, but pointless here, and the message of responsibility makes customers feel better. Would you get into a taxi if you knew you would be liable if the driver crashed?

Senate bill

In other news this week, a Senate panel passed its own version of the House bill deregulating robocars. Notable was the exclusion of trucks, at the request of the Teamsters. I have predicted since this all began that the Teamsters would eventually bring their influence to bear on automated trucking. They will slow things down, but it’s a battle they won’t win. Truck accidents kill 4,000 people every year, and truck driving is a grueling boring profession whose annual turnover sometimes exceeds 100%. At that rate, if they introduced all-automated truck fleets today, it would be a very long time before somebody who actually wanted a trucking job lost it to automation. Indeed, even in the mostly automated world there will still be routes and tasked best served by humans, and they will be served by those humans who want it.

Actually, this new-world trucking will be a much nicer job. It will be safer, and nobody will drive the long-haul cross-country routes that grind you with boredom, take you away from your home and family for a week or more while you eat bad food and sleep in cheap motels or the back of your rig.

Uber

Speaking of trucking, while I have not been commenting much on the Waymo/Uber lawsuit because of my inside knowledge, and the personalities don’t bear too much on the future of the technology, it certainly has been getting fast and furious.

You can read the due diligence report Uber had prepared before buying Otto, and a Wired article which starts with a silly headline but has some real information as well.

Other items

Luminar, the young 1.5 micron LIDAR startup, has announced that Toyota will use their LIDARs.

Lyft has added Ford, along with Google to its partner list. Since Lyft did a $500M investment deal with GM, it’s clear they don’t want to stick with just one player, even for that sum. Google may have larger sums — it does seem clear that the once happy partnership of Uber and Google is over.

Baidu announced a 10 billion Yuan investment fund for self-driving startups.

Rumours suggest Waymo may expand their Phoenix pilot to a real self-driving taxi service for the public sooner than expected.

NHTSA released their latest draft robocar regulations just a week after the U.S. House passed a new regulatory regime and the senate started working on its own. The proposed regulations preempt state regulation of vehicle design, and allow companies to apply for high volume exemptions from the standards that exist for human-driven cars.

It’s clear that the new approach will be quite different from the Obama-era one, much more hands-off. There are not a lot of things to like about the Trump administration but this could be one of them. The prior regulations reached 116 pages with much detail, though they were mostly listed as “voluntary.” I wrote a long critique of the regulations in a 4 part series which can be found in my NHTSA tag. They seem to have paid attention to that commentary and the similar commentary of others.

At 26 pages, the new report is much more modest, and actually says very little. Indeed, I could sum it up as follows:

• Do the stuff you’re already doing
• Pay attention to where and when your car can drive and document that
• Document your processes internally and for the public
• Go to the existing standards bodies (SAE, ISO etc.) for guidance
• Create a standard data format for your incident logs
• Don’t forget all the work on crash avoidance, survival and post-crash safety in modern cars that we worked very hard on
• Plans for how states and the feds will work together on regulating this

Goals vs. Approaches

The document does a better job at understanding the difference between goals — public goods that it is the government’s role to promote — and approaches to those goals, which should be entirely the province of industry.

The new document is much more explicit that the 12 “safety design elements” are voluntary. I continue to believe that there is a risk they may not be truly voluntary, as there will be great pressure to conform with them, and possible increased liability for those who don’t, but the new document tries to avoid that, and its requests are much milder.

The document understands the important realization that developers in this space will be creating new paths to safety and establishing new and different concepts of best practices. Existing standards have value, but they can at best encode conventional wisdom. Robocars will not be created using conventional wisdom. The new document takes the approach of more likely recommending that the existing standards be considered, which is a reasonable plan.

A lightweight regulatory philosophy

My own analysis is guided by a lightweight regulatory approach which has been the norm until now. The government’s role is to determine important public goals and interests, and to use regulations and enforcement when, and only when, it becomes clear that industry can’t be trusted to meet these goals on its own.

In particular, the government should very rarely regulate how something should be done, and focus instead on what needs to happen as the end result, and why. In the past, all automotive safety technologies were developed by vendors and deployed, sometimes for decades, before they were regulated. When they were regulated, it was more along the lines of “All cars should now have anti-lock brakes.” Only with the more mature technologies have the regulations had to go into detail on how to build them.

Worthwhile public goals include safety, of course, and the promotion of innovation. We want to encourage both competition and cooperation in the right places. We want to protect consumer rights and privacy. (The prior regulations proposed a mandatory sharing of incident data which is watered down greatly in these new regulations.)

I call this lightweight because others have called for a great deal more regulation. I don’t, however, view it is highly laissez-faire. Driving is already highly regulated, and the idea that regulators would need to write rules to prevent companies from doing things they have shown no evidence of doing seems odd to me. Particularly in a fast-changing field where regulators (and even developers) admit they have limited knowledge of what the technology’s final form will actually be.

Stating the obvious

While I laud the reduction of detail in these regulations, it’s worth pointing out that many of the remaining sections are stripped to the point of mostly outlining “motherhood” requirements — requirements which are obvious and that every developer has known for some time. You don’t have to say that the vehicle should follow the vehicle code and not hit other cars. Anybody who needs to be told that is not a robocar developer. The set of obvious goals belongs better in a non-governmental advice document (which this does in fact declare itself in part to be, though of course governmental) than in something considered regulatory.

Overstating the obvious and discouraging the “black box.”

Sometimes a statement can be both obvious but also possibly wrong in the light of new technology. The document has many requirements that vendors document their thinking and processes which may be very difficult to do with systems built with machine learning. Machine learning sometimes produces a “black box” that works, but there is minimal knowledge as to how it works. It may be that such systems will outperform other systems, leaving us with the dilemma of choosing between a superior system we can’t document and understand, and an inferior one we can.

There is a new research area known as “explainable AI” which hopes to bridge this gap and make it possible to document and understand why machine learning systems operate as they do. This is promising research but it may never be complete. In spite of this, EU regulations currently are already attempting to forbid unexplainable AI. This may cut off very productive avenues of development — we don’t know enough to be sure about this as yet.

Some minor notes

The name

The new report pushes a new term — Automated Driving Systems. It seems every iteration comes up with a new name. The field is really starting to need a name people agree on, since nobody seems to much like driverless cars, self-driving cars, autonomous vehicles, automated vehicles, robocars or any of the others. This one is just as unwieldy, and its acronym is an English word and thus hard to search for.

The levels

The SAE levels continue to be used. I have been critical of the levels before, recently in this satire. It is wrong to try to understand robocars primarily through the role of humans in their operation, and wrong to suggest there is a progression of levels based on that.

The 12 safety elements

As noted, most of the sections simply advise obvious policies which everybody is already doing, and advise that teams document what they are doing.

1. System Safety

This section is modest, and describes fairly common existing practices for high reliability software systems. (Almost to the point that there is no real need for the government to point them out.)

2. Operational Design Domain

The idea of defining the situations where the car can do certain things is a much better approach than imagining levels of human involvement. I would even suggest it replace the levels, and the human seen simply as one of the tools to be used to operate outside of certain domains. Still, I see minimal need for NHTSA to say this — everybody already knows that roads and their conditions are different and complex and need different classes of technology.

3. Object and Event Detection and Response, 4. Fallback, 5. Validation, 6. HMI

Again, this is fairly redundant. Vendors don’t need to be told that vehicles must obey the vehicle code and stay in their lane and not hit things. That’s already the law. They know that only with a fallback strategy can they approach the reliability needed.

7. Computer Security

While everything here is already on the minds of developers, I don’t fault the reminder here because traditional automakers have a history of having done security badly. The call for a central clearing house on attacks is good, though it should not necessarily be Auto-ISAC.

8. Occupant Protection

A great deal of the current FMVSS (Federal Motor Vehicle Safety Standards) are about this, and because many vehicles may use exemptions from FMVSS to get going, a reminder about this is in order.

10. Data Recording

The most interesting proposal in the prior document was a requirement for public sharing of incident and crash data so that all teams could learn from every problem any team encounters. This would speed up development and improve safety, but vendors don’t like the fact it removes a key competitive edge — their corpus of driving experience.

The new document calls for a standard data format, and makes general motherhood calls for storing data in a crash, something everybody already does.

The call for a standard is actually difficult. Every vehicle has a different sensor suite and its own tools to examine the sensor data. Trying to standardize that on a truly useful level is a serious task. I had expected this task to fall to outside testing companies, who would learn (possibly reverse engineering) the data formats of each car and try to put them in a standard format that was actually useful. I fear a standard agreed upon by major players (who don’t want to share their data) will be minimal and less useful.

State Roles

A large section of the document is about the bureaucratic distribution of roles between states and federal bodies. I will provide analysis of this later.

Conclusion

This document reflects a major change, almost a reversal, and largely a positive one. Going forward from here, I would encourage that the debate on regulation focus on

• What public goods does the government have an interest in protecting?
• Which ones are vendors showing they can’t be trusted to support voluntarily, both by present actions and past history?
• How can innovation be encouraged and facilitated, and good communication be made to the public about what’s going on

One of the key public goods missing from this document is privacy protection. This is one of the areas where vendors don’t have a great past history.
Another one is civil rights protection — for example what powers police will want over cars — where the government has a bad history.

Tesla Motors autopilot (photo:Tesla)

The NTSB (National Transportation Safety Board) has released a preliminary report on the fatal Tesla crash with the full report expected later this week. The report is much less favourable to autopilots than their earlier evaluation.

(This is a giant news day for Robocars. Today NHTSA also released their new draft robocar regulations which appear to be much simpler than the earlier 116 page document that I was very critical of last year. It’s a busy day, so I will be posting a more detailed evaluation of the new regulations — and the proposed new robocar laws from the House — later in the week.)

The earlier NTSB report indicated that though the autopilot had its flaws, overall the system was working. This is to say that though drivers were misusing the autopilot, the combined system including drivers not misusing the autopilot combined with those who did, was overall safer than drivers with no autopilot. The new report makes it clear that this does not excuse the autopilot being so easy to abuse. (By abuse, I mean ignore the warnings and treat it like a robocar, letting it driving you without actively monitoring the road, ready to take control.)

While the report mostly faults the truck driver for turning at the wrong time, it blames Tesla for not doing a good enough job to assure that the driver is not abusing the autopilot. Tesla makes you touch the wheel every so often, but NTSB notes that it is possible to touch the wheel without actually looking at the road. NTSB also is concerned that the autopilot can operate in this fashion even on roads it was not designed for. They note that Tesla has improved some of these things since the accident.

This means that “touch the wheel” systems will probably not be considered acceptable in the future, and there will have to be some means of assuring the driver is really paying attention. Some vendors have decided to put in cameras that watch the driver or in particular the driver’s eyes to check for attention. After the Tesla accident, I proposed a system which tested driver attention from time to time and punished them if they were not paying attention which could do the job without adding new hardware.

It also seems that autopilot cars will need to have maps of what roads they work on and which they don’t, and limit features based on the type of road you’re on.

These are very complex questions, and certainly most teams developing cars have not spent a lot of time on solutions to them at present. Indeed, I expect that these will not be solved issues until after the first significant pilot projects are deployed, because as long as robocars are a small fraction of the car population, they will not have that much effect on how things go. Some people who have given up car ownership for robocars — not that many in the early days — will possibly find themselves hunting for transportation the way other people who don’t own cars do today.

It’s a different story when, perhaps a decade from now, we get significant numbers of people who don’t own cars and rely on robocar transportation. That means people who don’t have any cars, not the larger number of people who have dropped from 2 cars to 1 thanks to robocar services.

I addressed a few of these questions before regarding Tsunamis and Earthquakes.

A few key questions should be addressed:

1. How will the car fleets deal with massively increased demand during evacuations and flight during an emergency?
2. How will the cars deal with shutdown and overload of the mobile data networks, if it happens?
3. How will cars deal with things like floods, storms, earthquakes and more which block roads or make travel unsafe on certain roads?

Most of these issues revolve around fleets. Privately owned robocars will tend to have steering wheels and be usable as regular cars, and so only improve the situation. If they encounter unsafe roads, they will ask their passengers for guidance, or full driving. (However, in a few decades, their passengers may no longer be very capable at driving but the car will handle the hard parts and leave them just to provide video-game style directions.)

Increased demand

An immediately positive thing is the potential ability for private robocars to, once they have taken their owners to safety, drive back into the evacuation zone as temporary fleet cars, and fetch other people, starting with those selected by the car’s owner, but also members of the public needing assistance. This should dramatically increase the ability of the car fleet to get people moved.

Nonetheless, it is often noted that in a robocar taxi world, there don’t need to be nearly as many cars in a city as we have today. With ideal efficiency, there would be exactly enough seats to handle the annual peak, but few more. We might drop to just 1/4 of the cars, and we might also have many of them be only 1 or 2 seater cars. There will be far fewer SUVs, pickup trucks, minivans and other large cars, because we don’t really need nearly as many as we have today.

To counter this, mandatory carpooling may become required. This will be fought because it means you don’t get to take all the physical stuff you want to bring in the event of something like flooding. Worse, we might see conflict between people wanting to bring pets (in carriers) which could take seats which might be used by people. In a very urgent situation, we could see an order coming down requiring pets to be left behind lest people be left behind. Once it’s clear all the people will make it out, people or rescue workers could go back for the pets, but that’s not very tenable.

One solution, if there is enough warning of the disaster (as there is for storms but not for other disasters) is for cars from outside the region to be pressed into service. There may be millions of cars within 1-2 hours drive of a disaster zone, allowing a major increase in capacity within a short time. This would include private cars as well as taxi fleets from other cities. Those cities would face a reduction in service and more carpooling. As I write this, Irma is heading to Florida but it is uncertain where. Fleets of cars could, in such a situation, be on their way from other states, distributing themselves along the probable path, and improving the position as the path is better known. They could be ready to do a very quick mass evacuation once the forecast is certain, and return home if nothing happens. For efficiency they could also drive themselves to be placed on car carriers and trains.

Disaster management agencies will need to build tools that calculate how many people need to be moved, and what capacity exists to move them. This will let them calculate how much excess capacity is there to move pets and household possessions.

If there are robotic transit vehicles they could help a lot. Cars might ferry people from homes to stations where robotic buses (including those from other cities, and human driven buses) could carry lots of people. Then they could return, with no risk to a driver.

The last hopeful item is the ability to do better traffic management. That’s an issue in any disaster, as people will resist following rules. If people get used to the idea of line direction reassignment, it can do a lot here. Major routes could be changed to have only one lane going towards the disaster area. That lane would be robocar or emergency vehicle only. The next lane would be going out of the disaster area, and it would be strictly robocar only. The robocars could safety drive in opposite directions at high speed without a barrier, and they could provide a buffer for the human driven cars in all the other lanes. One simple solution might be to have the inbound lanes converted to robocar only, with allocation of lanes based on traffic demand. The outbound lanes would remain outbound and have a mix of cars. The buses would use the robocar lanes for a congestion free quick trip out and in.

Saving cars

It is estimated that as many as a million cars were destroyed by flooding in Harvey. Fortunately, robocars could be told to wake up and leave, even if not pressed into service for evacuation. With good knowledge of the topography of the land, they can also calculate the depth of water by looking at the shape of a flood, and never drive where they could get stuck. Few cars would be destroyed by the flood.

Loss of data

We’ve seen data networks fail. Cars need the data networks to get orders to travel to destinations to pick people up. They also want updates on road conditions, closures, problems and reallocations.

This is one of the few places where DSRC could help, as it would not depend on the mobile data networks. But it’s not enough to justify this rare purpose, and mesh networking is not currently in its design. It is probably more effective to build tools to keep the mobile data networks up, such as a network of emergency cell towers mounted in robotic trucks (and boats and planes?) that could travel quickly to provide limited service, for use by vehicles and for emergency communications only. Keep people to text messages and the networks have tons of capacity.

Existing cell towers could also be hardened, to have at least enough backup power for an evacuation, if not for a long disaster.

Roads changed by disasters

You can probably imagine 1,000 strange things that could happen during a disaster. Flooded streets. Trees and powerlines down. Buildings collapsed. Cracks in the road. Washed out bridges. Approaching tsunamis. High winds. Crashed cars.

The good thing is, if you can imagine it, so can the teams building test systems for robocars. They are building simulators to play out every strange situation they can think of, or that they’ve ever encountered in many human lifetimes of real driving on the road. Every car then gets tested to see what it will do in those strange situations, and 1,000 variations of each situation.

Cars could know the lay of the land, so that they could predict how deep water is or where flooding will go. They could know where the high ground is and how to get there without going over the low ground. If the data networks are up, they could get information in real time on road problems and disaster situations. One car might run into trouble, but after that, no other car will go that way if they shouldn’t. This even applies to traffic, something we already do with tools like Waze.

War

War is one of the most difficult challenges. Roads will be blocked or changed. Certainly places will be extremely dangerous and must not be visited. Checkpoints will be created that you must stop for. Communications networks will be compromised. Parties may be attempting to break into your car to take it over and turn it into a weapon against you or others. Insurgents may be modifying robocars and even ordinary drive-by-wire cars to turn them into bomb delivery systems. Cars or streets may come under active attack from snipers, artillery, grenade throwers and more. In the most extreme case, a nuclear weapon or chemical weapon might be used.

The military wants autonomous vehicles. It wants them to move cargo in dangerous but not active war zones, and it wants them for battle. It will be dealing with all these problems, but there is no clear path from their plans to civilian vehicles. Most civilian developers will not consider war situations very heavily until they start wanting to sell cars for use in conflict zones. At first the primarily solution will be to have a steering wheel to allow manual control. The second approach will be what I call “video game mode” where you can drive the car with a video game controller. It will take charge of not hitting things, you will just tell it where to go — what turns to make, what side of an obstacle to drive on, and most scare of all, to override its own sensors which believe it can’t go forward because of an obstacle.

In a conflict zone, communications will become very suspect and unreliable. No operations can depend on communications, and all communications should be seen as possible vectors for attack. At the same time you need data about places to avoid — and I mean really avoid. This problem needs lots more thought, and for now, I don’t know of people thinking about robotaxi service in conflict zones.

Source: here.com

Almost all robocars use maps to drive. Not the basic maps you find in your phone navigation app, but more detailed maps that help them understand where they are on the road, and where they should go. These maps will include full details of all lane geometries, positions and meaning of all road signs and traffic signals, and also details like the texture of the road or the 3-D shape of objects around it. They may also include potholes, parking spaces and more.

The maps perform two functions. By holding a representation of the road texture or surrounding 3D objects, they let the car figure out exactly where it is on the map without much use of GPS. A car scans the world around it, and looks in the maps to find a location that matches that scan. GPS and other tools help it not have to search the whole world, making this quick and easy.

Google, for example, uses a 2D map of the texture of the road as seen by LIDAR. (The use of LIDAR means the image is the same night and day.) In this map you see the location of things like curbs and lane markers but also all the defects in those lane markers and the road surface itself. Every crack and repair is visible. Just as you, a human being, will know where you are by recognizing things around you, a robocar does the same thing.

Some providers measure things about the 3D world around them. By noting where poles, signs, trees, curbs, buildings and more are, you can also figure out where you are. Road texture is very accurate but fails if the road is covered with fresh snow. (3D objects also change shape in heavy snow.)

Once you find out where you are (the problem called “localization”) you want a map to tell you where the lanes are so you can drive them. That’s a more traditional computer map, though much more detailed than the typical navigation app map.

Some teams hope to get a car to drive without a map. That is possible for simpler tasks like following a road edge or a lane. There you just look for a generic idea of what lane markings or road edges should look like, find them and figure out what the lanes look like and how to stay in the one you want to drive in. This is a way to get a car up and running fast. It is what humans do, most of the time.

Driving without a map means making a map

Most teams try to do more than driving without a map because software good enough to do that is also software good enough to make a map. To drive without a map you must understand the geometry of the road and where you are on it. You must understand even more, like what to do at intersections or off-ramps.

Creating maps is effectively the act of saying, “I will remember what previous cars to drive on this road learned about it, and make use of that the next time a car drives it.”

Put this way it seems crazy not to build and use maps, even with the challenges listed below. Perhaps some day the technology will be so good that it can’t be helped by remembering, but that is not this day.

The big advantages of the map

There are many strong advantages of having the map:

• Human beings can review the maps built by software, and correct errors. You don’t need software that understands everything. You can drive a tricky road that software can’t figure out. (You want to keep this to a minimum to control costs and delays, but you don’t want to give it up entirely.)
• Even if software does all the map building, you can do it using arbitrary amounts of data and computer power in cloud servers. To drive without a map you can must process the data in real time with low computing resources.
• You can take advantage of multiple scans of the road from different lanes and vantage points. You can spot things that moved.
• You can make use of data from other sources such as the cities and road authorities themselves.
• You can cooperate with other players — even competitors — to make everybody’s understanding of the road better.

One intermediate goal might be to have cars that can drive with only a navigation map, but use more detailed maps in “problem” areas. This is pretty similar, except in database size, with automatic map generation with human input only on the problem areas. If your non-map driving is trustworthy, such that it knows not to try problem areas, you could follow the lower cost approach of “don’t map it until somebody’s car pulled over because it could not handle an area.”

Levels of maps

There are two or three components of the maps people are building, in order to perform the functions above. At the most basic level is something not too far above the navigation maps found in phones. That’s a vector map, except with lane level detail. Such maps know how many lanes there are, and usually what lanes connect to what lanes. For example, they will indicate that to turn right, you can use either of the right two lanes at some intersections.

Usually on top of that will be physical dimensions for these lanes, with their shape and width. The position information may be absolute (ie. GPS coordinates) but in most cases cars are more interested in the position of things relative to one another. It doesn’t matter that you drive exactly the path on the Earth that the lane is in, what matters is that you’re in the right lane relative to the edge of the road. That’s particularly true when you have to deal with re-striping.

Maps will have databases of interesting objects. The most interesting will be traffic signals. It is much easier to decode them if you know exactly where they are in advance. Cars also want to know the geometry of sidewalks and crosswalks to spot where pedestrians will be and what it means if they are there.

Somewhat independent of this are the databases of texture, objects or edges which the car uses to figure out exactly where it is on the map. A car’s main job is “stay in your lane” which means knowing the trajectory of the lane and where you are relative to the lane.

Even those who hope to “drive without a map” still want the basic navigation map, because driving involves not just staying in lanes, but deciding what to do at intersections. You still need to pick a route, just as humans use maps in tools like Waze. The human using Waze still often has the job of figuring out where the lanes are and which one to be in for turns, and how to make the turn, but a map still governs where you will be making turns.

The cost of maps

The main reason people seek to drive without a map is the cost of making maps. Because there is a cost, it means your map only covers the roads you paid to map. If you can only drive at full safety where you have a map, you have limited your driving area. You might say, “Sorry, I can’t go down that road, I don’t have a map.”

This is particularly true if mapping requires human labour. Companies like Google started by sending human driven cars out to drive roads multiple times to gather data for the map. Software builds the first version of the map, and humans review it. This has to be repeated if the roads change.

Maps also are fairly large, so a lot of data must be moved, but storage is cheap, and almost all of it can be moved when cars are parked next to wifi.

To bring down this cost, many companies hope to have ordinary “civilian” drivers go out and gather the sensor data, and to reduce the amount of human labour needed to verify and test the maps.

When the road changes

The second big challenge with maps is the fact that roads get modified. The map no longer matches the road. Fortunately, if the map is detailed enough, that’s quite obvious to the car’s software. The bigger challenge is what to do.

This means that even cars that drive on maps must have some ability to drive when the map is wrong, and even absent. The question is, how much ability?

A surprise change in the road should actually be rare. They happen every day of course as construction crews go out on jobs, but it’s only a surprise to the first car to encounter the change. That very first car will immediately log in the databases that there is a change. If it still drives the road, it will also upload sensor data about the new state of the road. We all see construction zones every day, but how often are the first car even to see that zone?

Most construction zones are scheduled and should not be a surprise even to the first car. Construction crews are far from perfect, so there will still be surprises. In the future, as crews all carry smartphones and have strict instructions to log construction activity with that phone before starting, surprises should become even more rare. In addition, in the interests of safety, the presence of such zones is likely to be shared, even among competitors.

Once a problem zone is spotted, all other cars will know about it. Unmanned cars will probably take a very simple strategy and avoid that section of road if they can, until the map is updated. Why take any risk you don’t need to? Cars with a capable human driver in them may decide they can continue through such zones with the guidance of the passenger. (This does not necessarily mean the passenger taking the controls, but instead just helping the car if it gets confused about things like two sets of lane markings, or unusual cones, or a construction flag crew.)

Nissan has also built a system where the car can ask a remote operations center for such advice, if there is data service at the construction zone. Unmanned cars will probably avoid routes where there could be surprise construction in a place with no data service.

As noted above, several teams are trying to make cars that drive without maps, even in construction zones. Even the cars with maps can still make use of such ability. Even if the car is not quite as safe as it is with a correct map, this will be so rare that the overall safety level can still be acceptable. (Real driving today consists of driving a mix of safer and more dangerous roads after all.) The worst case, which should be very rare, would be a car pulling over, deciding it can’t figure out the road and can’t get help from anybody. A crew in another car would come out to fetch it quickly.

The many players in mapping

This long introduction is there to help understand all the different types of efforts that are going on in mapping and localization. There is lots of variation.

Google/Waymo

The biggest and first player, Google’s car team was founded by people who had worked on Google Streetview. For them the idea of getting cars to scan every road in a region was not daunting — they had done it several times before. Their approach uses detailed texture maps of all roads and surrounding areas. Google is really the world’s #1 map company, so this was a perfect match for them.

Waymo’s maps are not based on Google Maps information, they are much more detailed. Unlike Google Maps, which they give out free to everybody to build on top of, the Waymo maps are proprietary, at least for now.

Navteq/Here

The company with the silly name of “Here” was originally a mapping company named Navteq. It was purchased by Nokia, renamed to “Here” and then sold to a consortium of German automakers. They will thus share their mapping efforts, and also sell the data to other automakers. In addition, the company gets to gather data from a giant fleet of cars from its owners and customers.

Here’s product is called “HD Maps” and it has some similarity to Google’s efforts in scope, but they took a lower cost approach to building them. They build a 3D map of the world using LIDAR. You can find an article about their approach at Here.

TomTom

The Dutch navigation company was already feeling the hurt from the move to phone-based navigation, and with my encouragement, entered the space of self-driving car maps. They have decided to take a “smaller data” approach. Their maps measure the width, not just of the road, but of the space around the road. The width is the distance from what you see looking left and looking right. Sensors will measure the presence of trees, poles, buildings and more to build a profile of the width. That’s enough to figure out where you are on the map, and also where you are on the road.

I’m not sure this is the right approach. It can work, but I don’t think there is a lot of merit in keeping the map small. That’s like betting that bandwidth and storage and computing will be expensive in the future. That’s always been the wrong bet.

MobilEye

MobilEye (now a unit of Intel) has cameras in a lot of cars. They provide the ADAS functions (like adaptive cruise control, and emergency braking) for a lot of OEMs, and they are trying to take a lead in self-driving cars. That’s what pushed their value up to $16B when Intel bought them.

MobilEye wants to leverage all those cars by having them scan the world as they drive, and looks for differences from ME’s compact maps. The maps are very small — just 3D locations of man-made objects around the highway. The location of these objects can be determined by a camera using motion parallax — how things like signs and poles move against the background.

ME believes they can get this data compact enough so that every car with their gear can be uploading updates to maps over the cell network. That way any changes to the road will be reflected in their database quickly, and before they get dramatic, and before a self-driving car gets there.

This is a good plan and the company that does this with the most cars sending data will have an advantage. Like TomTom this makes the bad bet that taking low bandwidth will be an important edge. A more interesting question is how strong the value is in live updates. A fleet that is 10x bigger will discover a change to the road sooner, but is there a big advantage to discovering it in 1 minute vs. 10 minutes?

Tesla

Tesla is one of the few companies hoping to drive without a map, or with a very limited map. A very limited map is more like a phone navigation map — it is not used to localize or plan, but does provide information on specific locations, such as details about off-ramps, or locations of potholes.

Tesla also has an interesting edge because they have many cars out there in production with their autopilot system. That gives them huge volumes of new data every day, though it is the limited data of their cameras. By having customers gather data about the roads, that’s given them a jump up.

Civil Maps

Civil Maps is a VC funded mapping startup. Their plan is to use neural network AI techniques to do a better job at turning image and sensor data into maps. That’s a good, but fairly obvious idea. The real challenge will be just how well they can do it. When it comes to the question of turning the map into a map of lane paths that guide where the vehicle drives, errors can’t be tolerated. If the AI software can’t figure out where the lane is, the software in the car isn’t going to do it either. If successful, the key will be to reduce the amount of human QA done on the maps, not to eliminate it.

Civil Maps publishes technical articles on their web site about their approaches — kudos to them.

DeepMap

DeepMap is another VC funded startup trying to generate a whole map ecosystem. They have not said a lot about their approach, other than they want to use data from production cars rather than having survey fleets do the mapping and re-mapping. That’s hardly a big leap — everybody will use that data if they can get it, and the battle there will partly depend on who has access to the data streams from cars that are out driving with good sensors. We’ll see in the future what other special sauce they want to provide.

Others

Almost every team has some mapping effort. Most teams do not roam a large enough set of roads to have encountered the cost and scaling problems of mapping yet. Only those attempting production cars (like Tesla and Audi) that allow driving without constant supervision had truly needed to deal with a very wide road network. In fact, those planning taxi fleets will not have to cover a wide road network for a number of years.

Most players expect to buy from a provider if they can. While all teams seek competitive edges, this is one sector where the edge is less and the value of cooperation is high. Indeed, the big question in mapping as an industry, is will it become cooperative — as in the case of 3 German automakers co-owning “Here Inc.” or will it become a competitive advantage, with one player making a better product because they have better or cheaper maps?

Infrastructure providers

It seems like a natural for the folks who build and maintain infrastructure to map it. A few things stand in the way of that. Because teams will be trusting the safety of their vehicles to their maps, they need to be very sure about the QA. That means either doing it themselves, or working with a provider whose QA process they can certify.

Working with the thousands of agencies who maintain and build roads is another story. Making all their data consistent and safety critical is a big challenge. Providers will certainly make use of data that infrastructure providers offer, but they will need to do expensive work on it in some cases.

Infrastructure providers can and should work to make sure that “surprises” are very rare. They will never be totally eliminated, but things can be improved. One simple step would be the creation of standardized databases for data on roads and road work. Authorities can pass laws saying that changes to the road can’t be done until they are logged in a smartphone app. This is not a big burden — everybody has smartphones, and those phones know where they are. In fact, smartphones used by contractors can even get smarts to notice that the contractor might be doing work without logging it. Old cheap phones could be stuck in every piece of road maintenance equipment. Those phones would say, “Hmm, I seem to suddenly be parked on a road but there is no construction logged for this area” and alert the workers or a control center.

All new road signs could be also logged by a smartphone app. A law could be made to say, “A road sign is not actually legally in effect until it is logged.” In addition, contractors can face financial penalties for changing roads without logging them. “Fire up the app when you start and end work or you don’t get paid” — that will make it standard pretty quickly.

This week’s news is preliminary, but a U.S. house committee panel passed some new federal regulations which suggest sweeping change in the US regulatory approach to robocars.

Today, all cars sold must comply with the Federal Motor Vehicles Safety Standards (FMVSS). This is a huge set of standards, and it’s full of things written with human driven cars in mind, and making a radically different vehicle, like the Zoox, or the Waymo Firefly, or a delivery robot, is simply not going to happen under those standards. There is a provision where NHTSA can offer exemptions but it’s in small volumes, for prototype and testing vehicles mostly. The new rules would allow a vendor to get an exemption to make 100,000 vehicles per year, which should be enough for the early years of robocar deployment.

Secondly, these and other new regulations would preempt state regulations. Most players (except some states) have pushed for this. Many states don’t want the burden of regulating robocar design, since they don’t have the resources to do so, and most vendors don’t want what they call a “patchwork” of 50 regulations in the USA. My take is different. I agree the cost of a patchwork is not to be ignored, but the benefits of having jurisdictional competition may be much greater. When California proposed to ban vehicles like the Google Firefly, Texas immediately said, “Come to Texas, we won’t get in your way.” That pushed California to rethink. Having one regulation is good — but it has to be the right regulation, and we’re much too early in the game to know what the right regulation is.

This is just a committee in the house, and there is lots more distance to go, including the Senate and all the other usual hurdles. Whatever people thought about how much regulation there should be, everybody has known that the FMVSS needs a difficult and complex revision to work in the world of robocars, and a temporary exemption can be a solution to that.

In San Francisco, I’m just back from the annual Automated Vehicle Symposium, co-hosted by the AUVSI (a commercial unmanned vehicle organization) and the Transportation Research Board, a government/academic research organization. It’s an odd mix of business and research, but also the oldest self-driving car conference. I’ve been at every one, from the tiny one with perhaps 100-200 people to this one with 1,400 that fills a large ballroom.

Toyota Research VC Fund

Tuesday morning did not offer too many surprises. The first was an announcement by Toyota Research Institute of a $100M venture fund. Toyota committed $1B to this group a couple of years ago, but surprisingly Gil Pratt (who ran the DARPA Robotics Challenge for humanoid-like robots) has been somewhat a man of mixed views, with less optimistic forecasts.

Different about this VC fund will be the use of DARPA like “calls.” The fund will declare, “Toyota would really like to see startups solving problem X” and then startups will apply, and a couple will be funded. It will be interesting to see how that pans out.

Nissan’s control room is close to live

At CES, Nissan showed off their plan to have a remote control room to help robocars get out of sticky situations they can’t understand like unusual construction zones or police directing traffic. Here, they showed it as further along and suggested it will go into operation soon.

This idea has been around for a while (Nissan based it on some NASA research) and at Starship, it has always been our plan for our delivery robots. Others are building such centers as well. The key question is how often robocars need to use the human assistance, and how you make sure that unmanned vehicles stay in regions where they can get a data connection through which to get help. As long as interventions are rare, the cost is quite reasonable for a larger fleet.

This answers the question that Rod Brooks (of Rethink Robotics and iRobot) recently asked, pondering how robocars will handle his street in Cambridge, where strange things like trucks blocking the road to do deliveries, are frequently found.

It’s a pretty good bet that almost all our urban spaces will have data connectivity in the 2020s. If any street doesn’t have solid data, and has frequent bizarre problems of any type, yet is really important for traversal by unmanned vehicles — an unlikely trifecta — it’s quite reasonable for vehicle operators to install local connectivity (with wifi, for example) on that street if they can’t wait for the mobile data companies to do it. Otherwise, don’t go down such streets in empty cars unless you are doing a pickup/drop-off on the street.

Switching Cities

Karl Iagenemma of nuTonomy told the story of moving their cars from Singapore, where driving is very regulated and done on the left, to Boston where it is chaotic and done on the right.

The short summary is that the switch was harder than they expected. At the same time, I feel that if a small company like nuTonomy can do it, it is not a big burden globally. That’s important because it reflects on the question of whether we need one single set of regulations across the United States or Europe, or if it’s better to have a patchwork with jurisdictional competition allowing innovation in how vehicles are regulated.

How much testing do vehicles need?

Nidi Kalra of Rand spoke about their research suggesting that testing robocars is an almost impossible task, because it would take hundreds of millions to a billion miles of driving to prove that a robocar is 10% better than human drivers.

This paper was published last year but I didn’t comment on it. I will post a more detailed commentary on it (and the reaction to it) shortly.

Security

Jonathan Petit presented interesting results previously at this meeting about his attacks on LIDARS. Tired of holding breakout workshops on security and nothing happening, he decided instead to just challenge the audience to E-mail him and others about their security concerns and plans. With 1,400 attendees, he got 4 responses. This crowd at least, is not taking security properly. Of course, only some portion of the room are actual developers of robocars. Most are researchers and academics and non-engineers. Still, the result is disappointing.

Accidents

The breakout on “what happens after an accident” day #1 was off-the-record, but a few general observations:

The police representative didn’t think there would be major changes in police investigations. They don’t seem to think the full 3-D recordings of the accidents in the cars will be easy to get their hands on so they’ll go about things the same way as before.
Trial lawyers argued about whether the standard of strict liability — pay if you caused the accident — rather than payment only when negligence is found, might become the norm.
Automakers are torn on this issue. On the one hand, who wants to pay if you weren’t negligent? On the other hand, it is only with negligence findings that high liability can be found. The cost of discovering the reason for a robocar’s error will be very high, with detailed code examination, and deposition of all programmers and expert witnesses. It may be simpler to pay every time than to have complex and costly lawsuits half the time, if you seriously reduce the number of accidents.

On the other hand, many states have liability caps on accidents which would preclude cases getting very expensive. If the max payout is $300,000 you aren’t going to spend a million trying to get it, and you have no reason to refuse a settlement near that cap number.

Infrastructure

The AVS has a lot of governmental people, and they’re all very keen to imagine their role, which they see as making the infrastructure “ready” for robocars. There was a whole long session on the topic, and many people who imagine there is a lot to do.

This is the wrong impression. Robocars are being designed to handle the infrastructure we already have, and only low-skill robocar makers are suggesting we need to make significant changes to the infrastructure to enable these vehicles.

For example, some automakers, making very basic camera based lanekeeping systems which find the lanemarkers using various algorithms have complained that they are poor quality on many roads. But Google, who actually got cars on the road first, designed their system to not require any quality from the lane markers. In fact, Google’s cars only need lane markers to be sure the humans know where to drive, and to know where to put the lanes in their internal maps. (They do want lane markers if the road has had new construction and has changed from the maps.)

Google’s algorithms actually prefer badly painted lane markers, because they find their location by matching the texture of the road, which includes holes in lane markers, road repairs and many other factors. It’s not a human way of driving, of course, but it doesn’t expect the road to change to suit the car.

For almost any proposal I have seen for how we might make infrastructure “robocar ready” there is a far cheaper and faster-to-develop solution that involves having the cars get smarter. Infrastructure change is only needed if there is a compelling case for why a fix in software can’t be found, or a case for why it can’t be done in virtual infrastructure.

Indeed, almost all the activity of infrastructure maintainers should focus on maintaining the virtual infrastructure instead. They should work to make sure roads are changed without logging it in a database, that road signs are all logged in databases and new ones don’t go into force until logged in the databases. Such logging isn’t hard — it’s as simple as a mobile app on the phones of the crews who install new signs or make other changes, and strong rules requiring use of the app. For example, severe financial penalties for not logging changes in the app.

I continue to advance the proposition that “you don’t change the roads to suit your cars, you improve your cars to deal with the roads we have.” At least for the near future.

Sharks

It’s no surprise my favourite session was one I spoke at, the Shark Tank. We saw 4 proposals on how robocars will change the world, and we sharks got to debate the issues around them with the audience. I didn’t just like the session because of my own participation as a shark. Unlike many sessions it also had a lot of audience involvement. The 4 propositions we tackled were:

• Congestion will go away
• Transportation agencies will shrink, and so will transit agencies
• Trucking will be quickly revolutionized
• Car ownership will end

Surprisingly, the proposal from the libertarian Reason magazine representative for the withering of these agencies got almost no opposition. While I have felt this is likely myself, I did not expect a room of others to agree. There was much more dispute around congestion. More were skeptical of my proposals that we might meter trips with smartphones to reduce congestion than I had hoped.

International

The event closed with a summary of various international efforts. This matched my impression from the recent conference in Germany — in most of the rest of the world, government involvement is quite high, but also highly non-productive. The budget size of many of the EU and Japanese funded projects for example, far exceeded the budget of Google’s early efforts, yet Google produced an impressive car while the EU projects produced only minor results.

Particularly popular all over the whole world these days is the forming of consortia and alliances which sound impressive but accomplish very little. I can’t escape the feeling when I see the announcement of a new alliance or partnership which does not actually say what concrete thing will be done by the alliance that it’s mostly for show, and not for real work.

Going forward

This conference began as the only self-driving conference and has grown. The problem with the growth is that most of the audience is new to the conference and the field. This pushes the sessions to be “dumbed down” with too much introductory background. While I am more informed than the average attendee, and will never get the perfect conference for me, I would like to see the sessions focus more on truly new things, things that are surprising. Companies who present have been told not to do marketing pitches or old news, the same has to apply to academics. This is challenging because academics spend a lot of time doing rigourous verification of things that are obvious. That’s a worthwhile task, but not right for the main stage of a joint industry/academic event. That’s why I liked the shark tank — it had a focus on issues about which informed people disagreed. That guarantees that much of the audience will be surprised by what they learn.

Coming up this week: Testing durations, and a new satire on the NHTSA levels and other regulation.

I’ve written a few times that perhaps the biggest unsolved problem in robocars is how to know we have made them safe enough. While most people think of that in terms of government certification, the truth is that the teams building the cars are very focused on this, and know more about it than any regulator, but they still don’t know enough. The challenge is going to be convincing your board of directors that the car is safe enough to release, for if it is not, it could ruin the company that releases it, at least if it’s a big company with a reputation.

We don’t even have a good definition of what “safe enough” is though most people are roughly taking that as “a safety record superior to the average human.” Some think it should be much more, few think it should be less. Tesla, now with the backing of the NTSB, has noted that their autopilot system — combined with a mix of mostly attentive but some inattentive humans, may have a record superior to the average human, for example, even though with the inattentive humans it is worse.

Last week I attended a conference in Stuttgart devoted to robocar safety testing, part of a larger auto show including an auto testing show. It was interesting to see the main auto testing show — scores of expensive and specialized machines and tools that subject cars to wear and tear, slamming doors thousands of times, baking the surfaces, rattling and vibrating everything. And testing the electronics, too.

In Europe, the focus of testing is very strongly on making sure you are compliant with standards and regulations. That’s true in the USA but not quite as much. It was in Europe some time ago that I learned the word “homologation” which names this process.

There is a lot to be learned from the previous regimes of testing. They have built a lot of tools and learned techniques. But robocars are different beasts, and will fail in different ways. They will definitely not fail the way human drivers do, where usually small things are always going wrong, and an accident happens when 2 or 3 things go wrong at once. The conference included a lot of people working on simulation, which I have been promoting for many years. The one good thing in the NHTSA regulations — the open public database of all incidents — may vanish in the new rules, and it would have made for a great simulator. The companies making the simulators (and the academic world) would have put every incident into a shared simulator so every new car could test itself in every known problem situation.

Still, we will see lots of simulators full of scenarios, and also ways to parameterize them. That means that instead of just testing how a car behaves if somebody cuts it off, you test what it does if it gets cut off with a gap of 1cm, or 10cm, or 1m, or 2m, and by different types of vehicles, and by two at once etc. etc. etc. The nice thing about computers is you can test just about every variation you can think of, and test it in every road situation and every type of weather, at least if your simulator is good enough,

Yoav Hollander, who I met when he came as a student to the program at Singularity U, wrote a report on the approaches to testing he saw at the conference that contains useful insights, particularly on this question of new and old thinking, and what regulations drive vs. liability and fear of the public. He puts it well — traditional and certification oriented testing has a focus on assuring you don’t have “expected bugs” but is poor at finding unexpected ones. Other testing is about finding unexpected bugs. Expected bugs are of the “we’ve seen this sort of thing before, we want to be sure you don’t suffer from it” kind. Unexpected bugs are “something goes wrong that we didn’t know to look for.”

Avoiding old thinking

I believe that we are far from done on the robocar safety question. I think there are startups who have not yet been founded who, in the future, will come up with new techniques both for promoting safety and testing it that nobody has yet thought of. As such, I strongly advise against thinking that we know very much about how to do it yet.

A classic example of things going wrong is the movement towards “explainable AI.” Here, people are concerned that we don’t really know how “black box” neural network tools make the decisions they do. Car regulations in Europe are moving towards banning software that can’t be explained in cars. In the USA, the draft NHTSA regulations also suggest the same thing, though not as strongly.

We may find ourselves in a situation where we take to systems for robocars, one explainable and the other not. We put them through the best testing we can, both in simulator and most importantly in the real world. We find the explainable system has a “safety incident” every 100,000 miles, and the unexplainable system has an incident every 150,000 miles. To me it seems obvious that it would be insane to make a law that demands the former system which, when deployed, will hurt more people. We’ll know why it hurt them. We might be better at fixing the problems, but we also might not — with the unexplainable system we’ll be able to make sure that particular error does not happen again, but we won’t be sure that others very close it it are eliminated.

Testing in sim is a challenge here. In theory, every car should get no errors in sim, because any error found in sim will be fixed or judged as not really an error or so rare as to be unworthy of fixing. Even trained machine learning systems will be retrained until they get no errors in sim. The only way to do this sort of testing in sim will be to have teams generate brand new scenarios in sim that the cars have never seen, and see how they do. We will do this, but it’s hard. Particularly because as the sims get better, there will be fewer and fewer real world situations they don’t contain. At best, the test suite will offer some new highly unusual situations, which may not be the best way to really judge the quality of the cars. In addition, teams will be willing to pay simulator companies well for new and dangerous scenarios in sim for their testing — more than the government agencies will pay for such scenarios. And of course, once a new scenario displays a problem, every customer will fix it and it will become much less valuable. Eventually, as government regulations become more prevalent, homologation companies will charge to test your compliance rate on their test suites, but again, they will need to generate a new suite every time since everybody will want the data to fix any failure. This is not like emissions testing, where they tell you that you went over the emissions limit, and it’s worth testing the same thing again.

The testing was interesting, but my other main focus was on the connected car and security sessions. More on that to come.

While very few details have come out, Reuters reports that new proposed congressional bills on self-driving cars will reverse many of the provisions I critiqued in the NHTSA regulations last year.

One big change is a reversal of the new idea of pre-market regulation. Today, new car technologies are not regulated before they are deployed, but NHTSA proposed giving itself the power to regulate technologies even before they exist. Currently, most car technologies like adaptive cruise control, autopilots, forward collision avoidance, lane keeping and the like remain unregulated after a decade or more of deployment with few, if any, problems.

This is important because the old doctrine of “we don’t regulate until we see a problem the industry won’t fix on its own” is a much better one for innovation, and the speed of innovation is key in deciding which countries and companies lead this technology. The opposite approach of “we try to imagine what might go wrong and ban it ahead of time” may seem safer, but it’s definitely an impediment to innovation and may actually result in far more deaths through the delay of life-saving technologies.

Harder to judge is the preemption of state rules. While states are also attempting to pre-regulate, having a laboratory of 50 different competing states can also be good for innovation on the legal side. There is not one answer, and while it’s more complex to deal with 50 sets of regulations instead of one, it’s not that much more complex.

One of the few interesting and good ideas in the NHTSA regs may also vanish. NHTSA wanted all vendors to make available all sensor logs from all incidents. As I predicted, companies pushed back on this — their testing logs and the resulting test suites are very important competitive assets. The company with the best test suite is the furthest on the path to the safety needed for deployment. On the other hand, sharing this data would let everybody get further on that path, faster.

There has been lots of other news during the long road-trip I am on in Europe. This includes more entrants in the race, the retirement of Google’s 3rd generation “koala” car, more at Uber. I will report from the Autonomous Car Testing and Development conference in Stuttgart starting Tuesday.

Source: Waymo

Waymo (Google) has announced a pilot project in Phoenix offering a full ride service in their new minivans. Members of the public can sign up — the link is sure to be overwhelmed with applicants, but it has videos and more details — and some families are already participating. There’s also a Waymo Blog post. I was in Phoenix this morning as it turns out, but to tell real estate developers about robocars; not for this.

There are several things notable about Waymo’s pilot:

1. They are attempting to cover a large area — they claim twice the size of San Francisco or 90 square miles. That’s a lot. It’s enough to cover the vast majority of trips for some pilot users. In other words, this is the first pilot which can test what it’s like to offer a “car replacement.”
2. They are pushing at families, which means children, including those not of driving age. The mother in the video expects to use it to send some children to activities. While I am sure there will be safety drivers watching over things, trusting children to the vehicles is a big milestone. Google’s safety record, with safety drivers, suggests this is actually a very safe choice for the parents, but there is emotion over trusting children to robots (other than the ones that go up and down shafts in buildings.)
3. In the videos, they are acting like there are no safety drivers but there surely are, for legal reasons as well as safety.
4. They are using the Pacifia minivans. The Firefly bubble cars are too slow for anything but neighbourhood operation. The minivans feature motorized doors, a feature that, although minor and commonplace, meets the image of what you want from a self-driving car.

Apple is in the game

There has been speculation recently because of some departures from Apple’s car team that they had given up. In fact, last week they applied for self-driving car test plates for California. I never thought they had left the game.

The big step is that once they test in California on public streets they will lose a lot of their stealth. Too many people in California (especially around Apple) know what to look for and we’ll see photos of these cars in the wild before too long. Which is good. I have never understood Apple’s deep culture of secrecy.

Waymo has reportedly bought a large test track in Nevada to expand their testing. Others are announcing pilot projects including Baidu, fitting surprisingly well with the schedule I have been telling people for the last few years. Namely that by late 2010 — pilots, until tech, regulatory and business kinks are worked out. Then by 2020 — first commercial deployments of taxi services. Mid-2020 — land rush as companies with working products rush to be the first to conquer individual city markets.

Uber/Waymo suit heats up

The Uber/Waymo lawsuit has become a soap opera. Because I have friends on both sides of the fight, I have not been commenting. Recent efforts include an attempt by Waymo to entirely shut down the Uber/Otto project or to force Anthony Levandowski to leave. This would be bad for Uber, especially the full shutdown, but as I wrote earlier Uber has a special advantage. Even if their project is shut down, they still have the number 1 brand and position for selling rides. Uber doesn’t own or control their cars now. If they have to buy somebody else’s robocars it is not nearly as strong a position as they might like, but it is not a failure.

Flying cars take off

I have been writing recently more about flying cars. Yesterday, my friends Larry Page and Sebastian Thrun revealed Kitty Hawk — a prototype flying car. You can see the cool video of it:

This is yet another piece of evidence that making a personal flying drone is certainly doable and going to happen. I even think the air traffic control problems can be solved. Four big questions remain:

1. How will we get enough energy in the vehicle to get a decent range? Not with today’s lithium-ion.
2. How will we get it quiet enough that you will tolerate regular takeoff and landing near you? (I am quoted in the NYT article on Kitty Hawk on this point.)
3. How will we get it energy efficient enough to not make our transportation less sustainable? Not with pure multirotors, though perhaps with ones that convert to fixed-wing flight. Or perhaps with electricity that’s so clean, we don’t care.
4. Will we tolerate skies packed full of (possibly noisy) flying drones?

Tesla settles suit

Telsa had been suing the startup founded by Stirling Anderson, who had been key in their autopilot project, and Chris Urmson, who for most of its existence ran Google’s project (and who I worked for while there.) The two are a powerhouse so their startup will be one to watch and the settlement will clear funding hurdles for them, even if it ended up being costly — but we don’t know many details.

Luminar, a bay area startup, has revealed details on their new LIDAR. Unlike all other commercial offerings, this is a LIDAR using 1.5 micron infrared light. They hope to sell it for $1,000.

Source: Luminar

1.5 micron LIDAR has some very special benefits. The lens of your eye does not focus medium depth infrared light like this. Ordinary light, including the 0.9 micron infrared light of the lasers in most commercial LIDARS is focused to a point by the lens. That limits the amount of power you can put in the laser beam, because you must not create any risk to people’s eyes.

Because of this, you can put a lot more power into the 1.5 micron laser beam. That, in turn, means you can see further, and collect more points. You can easily get out to 250 meters, while regular lidars are limited to about 100m and are petering out there.

What doesn’t everybody use 1.5 micron? The problem is silicon sensors don’t react to this type of light. Silicon is the basis of all mass market electronics. To detect 1.5 micron light, you need different materials, which are not themselves that hard to find, but they are not available cheap and off the shelf. So far, this makes units like this harder to build and more expensive.

If Luminar can do this, it will be valuable.

Why do you need to see 250m? Well, you don’t for city driving, though it’s nice. For highway driving, you can get by with 100m as well, and you use radar to help you perceive, at very low resolution, what’s going on beyond that. Still, there are things that radar can’t tell you. Rare things, but still important. So you need a sensor that sees further to spot things like stalled cars under bridges. Radar sees those, but can’t tell them from the bridge.

To this point, Google has been the only company to say they have a long range LIDAR, but it has not been for sale. And as we all know, there is a famous lawsuit underway accusing Uber/Otto of copying Google’s LIDAR designs.

The Luminar point clouds are impressive. This will be a company to watch. (In the interests of disclosure, I am an advisor to Quanergy, another LIDAR startup.)

Article first appeared on robocars.com

A new report from Navigant Research includes the chart shown below, ranking various teams on the race to robocar deployment. It’s causing lots of press headlines about how Ford is the top company and companies like Google and Uber are far behind. I elected not to buy the $3,800 report, but based on the summary I believe their conclusions are ill founded to say the least.

This ordering smacks of old-world car industry thinking. Saying that Ford and GM are ahead of Waymo/Google is like saying that Foxconn is ahead of Google or Apple in the smartphone market. Foxconn makes the iPhone of course, and makes lots of money at a modest profit margin of a few percent. Apple and Google don’t make their phones, they design them and the software platform.

Ford and GM might feel good reading this report, but they should not. I do actually like Ford’s plan quite a bit — especially their declaration that they will not sell their robocar to end-users. I also like Daimler’s declaration that they want to have a taxi style service called “Car2Come” after their Car2Go one-way on-demand car rental service. (Americans giggle at the name, Germans are never bothered by such things. :-)

GM does not belong on the list, other than for its partnership with Lyft. They were wise to acquire a company like Cruise — and I know the folks at Cruise, this is not a criticism of them — but it’s not enough to catapult you to the front of the list.

A similar article came out a few months ago, declaring that Silicon Valley was sure to lose to Detroit, because Detroit knows how to make cars, and Silicon Valley doesn’t. The report went further and declared that Google was falling behind because they had said they did not plan to make a car. The author had mistakenly thought Google had plans to make a car — Google never said anything like that — and so decided that the announcement that they would not make one was a big retreat on their part.

Companies like Google, Apple and Uber have never stated they wished to make cars, or felt they were any good at it. If they want to make cars, they have the cash to go buy a car company, but there is no need to do that. There are a couple of dozen companies around the world who are already very good at making cars, and if you come to them with an order for 100,000 cars to your specification, they will jump to say “yes, sir!” Some of the companies, the big leaders like Toyota or BMW, might well refuse that order, not wanting to be the supplier for a threat to their existence. But it won’t help them. Somebody will be that supplier. If not a German, Japanese or U.S. company, then a Korean company, or failing that a Chinese company. In fact, Foxconn has said it is interested in making cars, and Apple is designing them, so the Apple-Foxconn relationship may be far more than a metaphor for this situation.

When you summon an Uber, you don’t care what nameplate is on the car. When you summon UberSelect, you don’t care if it’s Lexus, or Mercedes or BMW. Uber was your brand, and you aren’t buying the car for 15 years, you are buying it for 15 minutes. Brand plays a completely different role.

Companies like Waymo, Apple, Uber, Zoox and others would be foolish to manufacture cars, unless they want a car so radically different that nobody knows how to make it. (Then, they might decide to be the first to figure it out.) The car manufacturers would be foolish to turn down the giant purchase order, or partnerships with whoever has the best technology.

The winner of the transportation game of the future will be the company that thinks outside the car. That doesn’t mean the big car companies can’t do that. It’s just harder for them to do.

The chart’s not entirely wrong. Honda is pretty far behind — but PSA is even further behind. BMW, Daimler and Ford are among the best of the car companies, but Tesla and Volvo deserve higher ranking. Hyundai is not ahead of Toyota, and Tesla, while not ahead of Waymo, is in a pretty good place. Bosch is a surprising absentee from the list. FCA should be on it, just very low on the chart, along with the smaller Japanese vendors and many Chinese vendors.

But be clear. Making the car is essential, but it’s also old and a commodity. The value will lie in those building the self-driving software systems and sensors, and those putting services together around the technologies. The big automaker’s advantages — nameplate and reputation, reliability, manufacturing skill and capacity, retail channel experience — these are all less valuable or commoditized. They have to act fast to move to new business models that will make it in the future. Of course, one plan is to own the important components I name above, and several companies like BMW, Daimler, Ford, Nissan and Volvo are trying to do that. But they’re behind Waymo by a fair distance.

First ride: Encountering a school bus on real city streets in Austin, Texas. Credit: Waymo/Google

Recently we’ve seen a series of startups arise hoping to make robocars with just computer vision, along with radar. That includes recently unstealthed AutoX, the off-again, on again efforts of comma.ai and at the non-startup end, the dedication of Tesla to not use LIDAR because it wants to sell cars today before LIDARs can be bought at automotive quantities and prices.

Their optimism is based on the huge progress being made in the use of machine learning, most notably convolutional neural networks, at solving the problems of computer vision. Milestones are dropping quickly in AI and particularly pattern matching and computer vision. (The CNNs can also be applied to radar and LIDAR data.)

There are reasons pushing some teams this way. First of all, the big boys, including Google, already have made tons of progress with LIDAR. There right niche for a startup can be the place that the big boys are ignoring. It might not work, but if it does, the payoff is huge. I fully understand the VCs investing in companies of this sort, that’s how VCs work. There is also the cost, and for Tesla and some others, the non-availability of LIDAR. The highest capability LIDARs today come from Velodyne, but they are expensive and in short supply — they can’t make them to keep up with the demand just from research teams!

Note, for more detailed analysis on this, read my article on cameras vs. lasers.

For the three key technologies, these trends seem assured:

1. LIDAR will improve price/performance, eventually costing just hundreds of dollars for high-resolution units, and less for low-res units.
2. Computer vision will improve until it reaches the needed levels of reliability, and the high-end processors for it will drop in cost and electrical power requirements.
3. Radar will drop in cost to tens of dollars and software to analyse radar returns will improve.

In addition, there are some more speculative technologies whose trends are harder to predict, such as long-range LWIR LIDAR, new types of radar, and even a claimed lidar alternative that treats the photons like radio waves.

These trends are very likely. As a result, the likely winner continues to be a combination of all these technologies, and the question becomes which combination.

LIDAR’s problem is that it’s low resolution, medium in range and expensive today. Computer Vision (CV)’s problem is that it’s insufficiently reliable, depends on external lighting and needs expensive computers today. Radar’s problem is super low resolution.

Option one — high-end LIDAR with computer vision assist

High end LIDARs, like the 32 and 64 laser units favoured by the vast majority of teams, are extremely reliable at detecting potential obstacles on the road. They never fail (within their range) to differentiate something on the road from the background. But they often can’t tell you just what it is, especially at a distance. It won’t know a car from a pickup truck or 2 pedestrians from 3. It won’t read facial expressions or body language. It can read signs but only when they are close. It can’t see colours, such as traffic signals.

The fusion of the depth map of LIDAR with the scene understanding of neural net based vision systems is powerful. The LIDAR can pull the pedestrian image away from the background, and then make it much easier for the computer vision to reliably figure out what it is. The CV is not 100% reliable, but it doesn’t have to be. Instead, it can ideally just improve the result. LIDAR alone is good enough if you take the very simple approach of “If there’s something in the way, don’t hit it.” But that’s a pretty primitive result that may brake too much for things you should not brake for.

Consider a bird on the road or a blowing trash bag. It’s a lot harder for the LIDAR system to reliably identify those things. On the other hand, the visions systems will do a very good job at recognizing the birds. A vision system that makes errors 1 time every 10,000 is not adequate for driving. That’s too high an error rate as you encounter thousands of obstacles every hour. But missing 1 bird out of 10,000 means that you brake unnecessarily for a bird perhaps once every year or two, which is quite acceptable.

Option two — lower end LIDAR with more dependence on vision

Low end lidars, with just 4 or so scanning planes, cost a lot less. Today’s LIDAR designs basically need to have an independent laser, lens and sensor for each plane, and so the more planes, the more cost. But that’s not enough to identify a lot of objects, and will be pretty deficient on things low to the ground or high up, or very small objects.

The interesting question is, can the flaws of current computer vision systems be made up for by a lower-end, lower cost LIDAR. Those flaws, of course, include not always discerning things in their field. They also include needing illumination at night. This is a particular issue when you want a 360-degree view — one can project headlights forward and see as far as they see, but you can’t project headlights backwards or to the side without distracting drivers.

It’s possible one could use infrared headlights in the other directions (or forward for that matter.) After all, the LIDAR sends out infrared laser beams. There are eye safety limits (your iris does not contract and you don’t blink to IR light) but the heat output is also not very high.

Once again, the low end lidar will eliminate most of the highly feared false negatives (when the sensor doesn’t see something that’s there) but may generate additional false positives (ghosts that make the vehicle brake for nothing.) False negatives are almost entirely unacceptable. False positives can be tolerated but if there are too many, the system does not satisfy the customer.

This option is cheaper but still demands computer vision even better than we have today. But not much better, which makes it interesting.

Other options

Tesla has said they are researching what they can do with radar to supplement cameras. Radar is good for obstacles in front of you, especially moving ones. Better radar is coming that does better with stationary objects and pulls out more resolution. Advanced tricks (including with neural networks) can look at radar signals over time to identify things like walking pedestrians.

Radar sees cars very well (especially licence plates) but is not great on pedestrians. On the other hand, for close objects like pedestrians, stereo vision can help the computer vision systems a lot. You mostly need long range for higher speeds, such as the highways, where vehicles are your only concern.

Who wins?

Cost will eventually be a driver of robocar choices, but not today. Today, safety is the only driver. Get it safe, before your competitors do, at almost any cost. Later make it cheap. That’s why most teams have chosen the use of higher end LIDAR and are supplementing in with vision.

There is an easy mistake to make, though, and sometimes the press and perhaps some teams are making it. It’s “easy” on the grand scale to make a car that can do basic driving and have a nice demo. You can do it with just LIDAR or just vision. The hard part is the last 1%, which takes 99% of the time, if not more. Google had a car drive 1,000 miles of different roads and 100,000 total roads in the first 2 years of their project back in 2010, and even in 2017 with by far the largest and most skilled team, they do not feel their car is ready. It gets easier every day, as tech advances, to get the demo working, but that should not be mistaken for the real success that is required.

This post originally appeared on robocars.com.

Caltrain at Palo Alto, CA. Source: Wikipedia Commons

[Editor’s note: According to a recent article on SFGate, Transportation Secretary Elaine Chao has put the brakes on $647 million for Caltrain to go electric. The delay may kill the project entirely.]

Caltrain is the commuter rail line of the San Francisco peninsula. It’s not particularly good; California is, after all, the land of the car commuter, but a plan is underway to convert it from diesel to electric. This made news at the same time as California Republican house members announced they want to put a stop to both this project and the larger California High-Speed Rail that hopes to open in 2030.

For various reasons, they may be right about the high-speed rail — but stop the electric trains? Electric trains are better than diesel; they are cleaner, faster and quieter. But one number stands out in the plan — to electrify 51 miles of track, and do other related improvements, it is forecast to cost over $1.5 billion dollars. That’s around $30M per mile.

So I started to ask, what other technology could we purchase with $1.5 billion plus a private right-of-way through the most populated areas of silicon valley and the peninsula? Caltrain carries about 60,000 passengers/weekday (30,000 each way.) That’s about $50,000 per rider. In particular, what about a robotic transit line, using self-driving cars, vans and buses?

Paving over the tracks is relatively inexpensive. In fact, if we didn’t have buses, you could get by with fairly meager pavement since no heavy vehicles would travel the line. You could leave the rails intact in the pavement, though that makes the paving job harder. You want pavement because you want stations to become “offline” — vehicles depart the main route when they stop so that express vehicles can pass them by. That’s possible with rail, but in spite of the virtues of rail, there are other reasons to go to tires.

Fortunately, due to the addition of express trains many years ago, some stations already are 4 tracks wide, making it easy to convert stations to an express route with space by the side for vehicles to stop and let passengers on/off. Many other stations have parking lots or other land next to them allowing reasonably easy conversion. A few stations would present some issues.

Making robocars for a dedicated track is easy; we could have built that decades ago. In fact, with their much shorter stopping distance they could be safer than trains on rails. Perhaps we had to wait to today to convince people that one could get the same safety off of rails. Another thing that only arrived recently was the presence of smartphones in the hands of almost all the passengers, and low cost computing to make kiosks for the rest. That’s because the key to a robotic transit line would be coordination on the desires of passengers. A robotic transit line would know just who was going from station A to station J, and attempt to allocate a vehicle just for them. This vehicle would stop only at those two stations, providing a nonstop trip for most passengers. The lack of stops is also more energy efficient, but the real win is that it’s more pleasant and faster. With private ROW, it can easily beat a private car on the highways, especially at rush hour.

Another big energy win is sizing the vehicles to the load. If there are only 8 passengers going from B to K, then a van is the right choice, not a bus. This is particularly true off-peak, where vast amounts of energy are wasted moving big trains with just a few people. Caltrain’s last train to San Francisco never has more than 100 people on it. Smaller vehicles also allow for more frequent service in an efficient manner, and late night service as well. (Most commuter trains shut down well before midnight.) Knowing you can get back is a big factor in whether you take a transit line at night.

An over-done service with a 40 passenger bus every 2 seconds would move 72,000 people (but really 30,000) in one hour in one direction to Caltrain’s 30,000 in a day. So of course we would not build that, and there would only be a few buses, mainly for rush hour. Even a fleet of just 4,000 9 passenger minvans (3 rows of 3) could move around 16,000 per hour (but really 8,000) in each direction. Even if each van was $50,000 each, we’ve spent only $200M of our $1.5B, though they might wear out too fast at that price, so we could bump the price and give them a much longer lifetime.

Energy

These vans and cars could be electric. This could be done entirely with batteries and a very impressive battery swap system, or you could have short sections of track which are electrified — with overhead rails or even third rails. The electric lines would be used to recharge batteries and supercapacitors, and would only be present on parts of the track. Unlike old 3rd rail technology, which requires full grade separation, modern technology could produce rails that are only electrified when an authorized vehicle is in contact. Even so, you would not have electrification at any grade crossing.

Other alternatives would be things like natural gas (which is cheap and much cleaner than liquid fuels, though still emits CO2) because it can be refilled quickly. Or hydrogen fuel cell vehicles could work here — hydrogen can be refilled quickly and can be zero emissions. Regular fossil fuel is also an option for peak times. For example the rush hour buses might make more sense running on CNG or even gasoline. The lack of starts and stops can make this pretty efficient.

Getting off the track

Caltrain’s station in San Francisco is quite far from most of the destinations people want to go to. It’s one of the big reasons people don’t ride it. Vans on tires, however, have the option of keeping going once they get to the station. Employers could sponsor vehicles that arrive at the station and keep driving to their office tower. Vans could also continue to BART or more directly to underground Muni, long before the planned subway is ready. Likewise on the peninsula, vans and buses would travel from stations to corporate HQ. Google, Yahoo, Apple and many other companies already run transit fleets to bring employees in — you can bet that given the option they would gladly have those vans drive the old rail line at express speeds. On day one, they could have a driver who only drives the section back and forth between the station and the corporate office. In the not too distant future, the van or bus would of course drive itself. It’s not even out of the question that one of the passengers in a van, after having taken a special driving test, could drive that last mile, though you may need to assure somebody drives it back.

At-grade crossings

I noted above that capacity would be slightly less than half of full. That’s because Caltrain has a fair number of at-grade crossings in the mid-peninsula. Today, these have railway gates. They would be converted to more normal traffic signals. The robotic vehicles would coordinate their trips to travel in bunches, leaving gaps where the cross-street’s light can be turned green. If any car was detected trying to run the red, the signal could be uploaded to allow all the robotic vans to slow or even brake hard. Unlike trains, they could brake in reasonable amounts of time if somebody stalls on the old track. You would also detect people attempting to drive on the path or walk on it. Today’s cameras and cheap LIDARs can make that affordable.

Over time, there is also the option in some places to build special crossings. Because the vans and cars would all be not very high, much less expensive underpasses could be created under some of the roads for use only by the smaller vehicles. Larger vehicles would still need to bunch themselves together to leave gaps for the cross-traffic. One could also create overpasses rated only for lightweight vehicles at much lower cost, though those would still need to be high enough for trucks to go underneath. In addition, while cars can handle much, much steeper grades than trains, it could get disconcerting to handle too much up and down at 100mph. And yes, in time, they would go 100mph or even faster. And in time, some would even draft one another to both increase capacity and save energy — creating virtual trains where there used to be physical ones.

And then, obsolete

This robotic transit line would be much better than the train. But it would also be obsolete in just a couple of decades! As the rest of the world moves to more robocars, the transit line would switch to being just another path for the robocars. It would be superior, because it would allow only robocars and never have traffic congestion. You would have to pay extra to use it at rush hour, but many vehicles would, and large vehicles would get preference. The stations would largely vanish as all vehicles are able to go door to door. Most of the infrastructure would get re-used after the transit line shuts down.

It might seem crazy to build such a system if it will be obsolete in a short time, but it’s even crazier to spend billions on shoring up 19th century train.

What about the first law?

I’ve often said the first law of robocars is you don’t change the infrastructure. In particular, I am in general against ideas like this which create special roads just for robocars, because it’s essential that we not imagine robocars are only good on special roads. It’s only when huge amounts of money are already earmarked for infrastructure that this makes sense. Now we are well on the way to making general robocars good for ordinary streets. As such, special cars only for the former rail line run less risk of making people believe that robocars are only safe on dedicated paths. In fact, the funded development would almost surely lead to vehicles that work off the path as well, and allow high volume manufacturing of robotic transit vehicles for the future.

Could this actually happen?

I do fear that our urban and transit planners are unlikely to be so forward looking as to abandon a decades old plan for a centuries old technology overnight. But the advantages are huge:

• It should be cheaper
• Many companies could do it, and many would want to, to fund development of other technology
• It would almost surely be technology from the Bay Area, not foreign technology, though vehicle manufacturing would come from outside
• They could also get money for the existing rolling stock and steel in the rails to fund this
• The service level would be vastly better. Wait times of mere minutes. Non-stop service. Higher speeds.
• The energy use would be far lower and greener, especially if electric, CNG or hydrogen vehicles are used

The main downside is risk. This doesn’t exist yet. If you pave the road to retain the rails embedded in them, you would not need to shut down the rail line at first. In fact, you could keep it running as long as there were places that the vans could drive around trains that are slowing or stopping in the stations. Otherwise, you do need to switch one day.

This shows a special advantage that Uber has over other players here. Their own project is very active, but unlike others, it doesn’t cripple Uber if it fails. Uber’s business is selling rides, and it will continue to be. If Uber can’t do it with its own cars, it can buy somebody else’s. Uber does not have the intention to make cars (neither does Google and that’s probably true of most other non-car companies.) There are many companies who will make cars to order for you. But if Google’s self-drive software (and hardware) project fails, they are left with very little. If Uber’s fails, they are still very much in business, but not as much in control of the underlying vehicles. As long as there are multiple suppliers for Uber to choose from, they are good.

One nightmare for the car companies is the reduction in value of their brands. If you summon “UberSelect” (the luxury Uber) you don’t care if it is a Lexus or Mercedes that shows up. As long as it’s a decent luxury car, you are good, because you are not buying the car, you are using it for 20 minutes. Uber is the brand you are trusting — and car companies fear that. I presume one thing that Daimler wants from this announcement is to remind people that they are a leader and may well be the supplier of cars to companies like Uber. But will they be in charge of the relationship? I doubt it.

Lyft should have the same advantage — but it took a $500M investment from GM which strongly pressures it to use whatever solution GM creates. Of course, if GM’s project fails, Lyft still has the freedom to use another, including Mercedes.

Other News

A lawsuit from Tesla against former Tesla autopilot team leader Sterling Anderson and former head of Google Chauffeur (now Waymo) Chris Urmson reveals little, other than the two have a company which will get a lot of attention in the space. But that’s enough. Google’s project is the most advanced one in the world. I was there and worked for Chris in its early days. Tesla’s is not necessarily the most advanced technologically — it has no LIDAR development — but it’s way ahead of others in terms of getting out there and deploying to gain experience, which has given it a headstart, especially in camera/radar-based systems. The leaders of the two projects together will cause a stir in the auto business.

California published its summary of all the reports submitted by vendors testing robocars in the state. You can read the individual reports. They are interesting, but several other outlines have created summaries of the reports calculating things like the number of interventions per mile. On these numbers, Google’s lead is extreme. Of over 600,000 autonomous miles driven by the various teams, Google/Waymo was 97% of them — in other words, 30 times as much as everybody else put together.

Beyond that, their rate of miles between disengagements (around 5,000 — a 4x improvement over 2015) is one or two orders of magnitude better than the others, and in fact for most of the others, they have so few miles that you can’t even produce a meaningful number. Only Cruise, Nissan and Delphi can claim enough miles to really tell.

Tesla is a notable entry. In 2015 they reported driving zero miles, and in 2016 they did report a very small number of miles with tons of disengagements from software failures (one very 3 miles.) That’s because Tesla’s autopilot is not a robocar system, and so miles driven by it are not counted. Tesla’s numbers must come from small scale tests of a more experimental vehicle. This is very much not in line with Tesla’s claim that it will release full autonomy features for their cars fairly soon, and that they already have all the hardware needed for that to happen.

Unfortunately, you can’t easily compare these numbers:

• Some companies are doing most of their testing on test tracks, and they do not need to report what happens there.
• Companies have taken different interpretations of what needs to be reported. Most of Cruise’s disengagements are listed as “planned” but in theory those should not be listed in these reports. But they also don’t list the unplanned ones which should be there.
• Delphi lists real causes and Nissan is very detailed as well. Others are less so.
• Many teams test outside California, or even do most of their testing there. Waymo/Google actually tests a bunch outside California, making their numbers even bigger.
• Cars drive all sorts of different roads. Urban streets with pedestrians are much harder than highway miles. The reports do list something about conditions but it takes a lot to compare apples to apples. (Apple is not one of the companies filing a report, BTW.)

One complication is that typically safety drivers are told to disengage if they have any doubts. It thus varies from driver to driver and company to company what “doubts” are and how to deal with them.

Google has said their approach is to test any disengagement in simulator, to find out what probably would have happened if the driver did not disengage. If there would have been a “contact” (accident) then Google considers that a real incident and those are rarer than is reported here. Many of the disengagements are when software detects faults with software or sensors. There, we do indeed have a problem, but like human beings who zone out, not all such failures will cause accidents or even safety issues. You want to get rid of all of them, to be sure, but if you are are trying to compare the safety of the systems to humans, it’s not easy to do.

It’s hard to figure out a good way to get comparable numbers from all teams. The new federal guidelines, while mostly terrible, contain an interesting rule that teams must provide their sensor logs for any incident. This will allow independent parties to compare incidents in a meaningful way, and possibly even run them all in simulator at some level.

It would be worthwhile for every team to be required to report incidents that would have caused accidents. That requires a good simulator, however, and it’s hard for the law to demand this of everybody.

Earlier I posted my gallery of CES gadgets and included a photo of the eHang 184 from China, a “personal drone” able, in theory, to carry a person up to 100kg.

Whether the eHang is real or not, some version of the personal automated flying vehicle is coming, and it’s not that far away. When I talk about robocars, I am often asked “what about flying cars?” and there will indeed be competition between them. There are a variety of factors that will affect that competition, and many other social effects not yet much discussed.

The VTOL Multirotor

There are two visions of the flying car. The most common is VTOL — vertical takeoff and landing — something that may have no wheels at all because it’s more a helicopter than a car or airplane. The recent revolution in automation and stability for multirotor helicopters — better known as drones — is making people wonder when we’ll get one able to carry a person. Multirotors almost exclusively use electric motors because you must adjust speed very quickly to get stability and control. You also want the redundancy of multiple motors and power systems, so you can lose a rotor or a battery and still fly.

This creates a problem because electric batteries are heavy. It takes a lot of power to fly this way. Carrying more batteries means more weight — and thus more power needed to carry the batteries. There are diminishing returns, and you can’t get much speed, power or range before the batteries are dead. OK in a 3 kilo drone, not OK in a 150 kilo one.

Lots of people are experimenting with combining multirotor for takeoff and landing, and traditional “fixed wing” (standard airplane) designs to travel any distance. This is a great deal more efficient, but even so, still a challenge to do with batteries for long distance flight. Other ideas including using liquid fuels some way. Those include just using a regular liquid fuel motor to run a generator (not very efficient) or combining direct drive of a master propeller with fine-control electric drive of smaller propellers for the dynamic control needed.

Another interesting option is the autogyro, which looks like a helicopter but needs a small runway for takeoff.

The traditional aircraft

Some “flying car” efforts have made airplanes whose wings fold up so they can drive on the road. These have never “taken off” — they usually end up a compromise that is not a very good car or a very good plane. They need airports but you can keep driving from the airport. They are not autonomous.

Robocars offer an interesting alternative. You can build a system where a robocar takes you from home to the best local short airstrip, taking you right out to an autonomous aircraft that is sitting waiting. You transfer, and it immediately takes off and flies you to another short airstrip, where another robocar awaits you. This allows you to travel in a car that’s a car and a plane that’s a plane, with no compromise.

The big challenges

Automating the intense level of safety and equipment reliability

In general, planes today are not fast modes of travel for their pilots. A typical small aircraft owner going out to fly has to drive to an airport that’s not very convenient, park and get their plane. (If they planned ahead, the hangar crew has taken their plane out and done the basics on it.) Even with the prep, there is a fairly long pre-flight check to do, assuring everything is just so, checking fuel levels with your eyes as well as instruments and more. Then you go through a dance with the control tower, taxi around (possibly in line behind others) and eventually get to take off and start climbing. Only then are you on your way. At the other end, you do it all in reverse, tie down and hangar your plane, and find your way to a rental car or ground transportation. For trips of under 100 miles, it’s not usually worth it.

Autonomous flying cars require more than just well built and superbly safe flying systems. (Flying itself is actually a pretty easy robotics problem.) It’s all the other stuff that will be the challenge. Because failures of equipment while up in the air can be so dangerous, vehicles must be maintained and checked to a level that is orders of magnitude greater than what we do with cars. If your car engine conks out, you pull off to the side of the road. If your brakes go out, it’s bad, but you apply the emergency brake and call a tow truck.

We’ll demand fail-safe operation for all parts of the flying car. It will have to be able to lose any major component and get you down safely.

Noise

Problem number one for VTOL is noise. Helicopters are not anywhere near silent. You might crave one for yourself, but no way you’ll accept your neighbours constantly flying helicopters in and out of their backyard, next to yours, at all hours. Not compared to the silence of the electric car.

Even if we have VTOL cars, we might still limit their operations (especially at night) to special landing yards. Your robotaxi could get you to the landing yard so it’s not as much of a burden, but using your own yard (unless you have a large estate, or live in a high-rise building with a heliport on top) is going to be difficult.

Energy

Right now, multirotor aircraft use a lot of energy to fly. Ground cars can be much more efficient. Society as a whole is seeking to greatly improve the efficiency of our transportation, not make it worse. Unless we make the flying car super efficient, it will be relegated only to speciality use, where the ground car just won’t work.

Fixed wing aircraft can be more efficient. Jets are very wasteful but lower speed aircraft can be efficient after takeoff.

Crowded skies

If personal flight becomes very popular, we would face the prospect of a sky seriously crowded with the vehicles in urban spaces. Computer systems could probably handle management of the traffic, since in 3 dimensions you get extra room, though you want much longer headways than cars use. In addition to being a visible blight and a noise source, there will be some safety concerns. Even a tiny number of these vehicles falling out of the sky and hitting things (or people) on the ground will cause more concern than cars do, even though they depart the road and hit people. This would be added to the large traffic in cargo drones.

The traffic management is non-trivial, but I believe it can be solved. There are still issues even after it’s solved.

Tourism

One of the places we might see radical change quickly is in tourism. If it’s cheap and easy, tourists will want to see everything from a flying car, especially one that can hover. Every amazing view, every scene, every architectural wonder, every city, will probably be best viewed from the air, or certainly desirable to view from the air as well as the ground. Every hiking trail you’ve not taken to some interesting sight will become a potential place people would like to go in their flying car.

Outside the cities, the problems of the flying cars are less present. The flights will be short and slow. You can travel to special locations for takeoff and landing, and make noise there. The territory will be rural or parkland in many cases, with more modest crowds and nobody to fall on in the event of rare safety failures.

I predict this could be so popular that it would have to be restricted. The crowds of tourists in buses at many locations are already overwhelming them. Now offer everybody something like the eHang. You might forbid this in national parks, but legally it would be difficult to forbid it for the disabled. The disabled can’t get up those trails, can’t get access to those amazing viewpoints. And by the disabled, I mean all the elderly, who can claim that their physical condition prevents their access to a site. The issue, of course, is that a large fraction of tourists are the elderly, armed with free time and extra money. Tourism in flying cars would be a dream come true.

There are places in the world (such as above Mauna Loa’s active volcano) that already have so much traffic from helicopter tours (at $200/person or more) that they need to have controlled airspace and specific flight patterns. Autonomous flying vehicles will be able to handle that concept, but still, the volume of traffic will be huge when anybody can take the ride for $10.

Public transport

Since we can’t make a multirotor for a single person, talking about group vehicles is even more premature, but we already have lots of public transit aviation today. Right now it’s done at airports, and never used for short distances because you spend far more time going through the airport than in the air. As I describe in A Robocar Airport it’s possible to make a much more efficient airport even for traditional planes. It would be great to go further and imagine the “flying bus” — an automated vehicle for a small group which is less like an airplane and more like one of the vans I describe in The Future of Transit. There, travel is coordinated and 10-20 single person robocars would converge within a minute of one another next to the autonomous flying bus. They would quickly get in — no security for something this small and fast — and within one minute be taking off down the runway.

Such a service might be better than things like high-speed rail for travel in big cities. Because it can go from any airport to any airport — or with VTOL from any landing yard to any landing yard — such vehicles would offer superior travel times, free from congestion. If a flying bus service took you from Silicon Valley to San Francisco’s ferry terminal in 15 minutes at a decent price, it would be quite popular and displace car traffic.

Specialty uses

If we don’t let everybody fly all the time, who will be the special cases we let fly? Will it simply be the rich who pay a high fee for the opportunity? (The fee can’t be so high as to match the cost of a helicopter today.)

• The flying ambulance is an obvious win — though we’re not at the level of building electric multirotors that could fly something that heavy. The lack of emergency vehicles on the regular roads will also improve traffic for others.
• Some delivery will go by drone, though perhaps only the light and urgent packages.
• There could be a lottery or other allocation, letting people fly some days, but not most.
• Government officials will certainly want to claim they have the importance to justify this. In some cases (like VIPs so big they close roads for their motorcades) this is a win for all.
• The police will clearly do this, as will some portion of fire crews (those not carrying heavy gear.) Anybody who uses helicopters today.
• People who live or work in remote country locations who can make what noise they want at their home, and mostly fly over uninhabited country.
• People populating mountainsides in crowded cities, though possibly only to transfer to a robotaxi in the flatlands.
• People living on islands in seaside cities, though possibly only to transfer to a robotaxi on shore.
• Flying carpooling (above and beyond the transit described below.) This requires multi-person flying cars.

If we don’t allow anybody who can afford it to fly, who else might get the privilege? Answer in the comments.

Tesla Model S autopilot-software. Source: Tesla

NHTSA released the report from their Office of Defects Investigation on the fatal Tesla crash in Florida last spring. It’s a report that is surprisingly favorable to Tesla. So much so that even I am surprised.

While I did not think Tesla would be found defective, this report seems to come from a different agency than the one that recently warned comma.ai that:

It is insufficient to assert, as you do, that the product does not remove any of the driver’s responsibilities” and “there is a high likelihood that some drivers will use your product in a manner that exceeds its intended purpose.

The ODI report rules that Tesla properly considered driver distraction risks in its design of the product. It goes even further, noting that drivers using Tesla Autopilot (including those monitoring it properly and those who did not) still had a decently lower accident rate for mile than drivers of ordinary cars without autopilot. In other words, while the Autopilot without supervision is not good enough to drive on its own, the Autopilot even with occasionally lapsed supervision that is known to happen is still overall a safer system than not having the Autopilot at all.

This will provide powerful support for companies developing autopilot style systems, and companies designing robocars who wish to use customer supervised driving as a means to build up test miles and verification data. They are not putting their customers at risk as long as they do it as well as Tesla. This is interesting (and the report notes that evaluation of Autopilot distraction is not a settled question) because it seems probable that people using the Autopilot and ignoring the road to do e-Mail or watch movies are not safer than regular drivers. But the overall collection of distracted and watchful drivers is still a win.

This might change as companies introduce technologies which watch drivers and keep them out of the more dangerous inattentive style of use. As the autopilots get better, it will become more and more tempting, after all.

Tesla stock did not seem to be moved by this report. But it was also not moved by the accident or other investigations — it actually went on a broadly upward course for 2 months following the announcement of the fatality.

The ODI’s job is to judge if a vehicle is defective. That is different from saying it’s not perfect. Perfection is not expected, especially from ADAS and similar systems. The discussion about the finer points of whether drivers might over-trust the system are not firmly settled here. That can still be true without the car being defective and failing to perform as designed or being designed negligently.

CES is the big event for major car makers to show off robocar technology. Most of the north hall, and a giant parking lot next to it, were devoted to car technology and self-driving demos.

Gallery of CES comments

Earlier I posted about many of the pre-CES announcements and it turns out there were not too many extra events during the show. I went to visit many of the booths and demos and prepared some photo galleries. The first is my gallery on cars. In this gallery, each picture has a caption so you need to page through them to see the actual commentary at the bottom under the photo. Just 3 of many of the photos are in this post.

BMW’s concept car starts to express the idea of an ultimate non-driving machine. Inside you see that the back seat has a bookshelf in it. Chances are you will just use your eReader, but this expresses and important message — that the car of the future will be more like a living, playing or working space than a transportation space.

Nissan

The main announcement during the show was from Nissan, which outlined their plans and revealed some concept cars you will see in the gallery. The primary demo they showed involved integration of some technology worked on by Nissan’s silicon valley lab leader, Maarten Sierhuis in his prior role at NASA. Nissan is located close to NASA Ames (I myself work at Singularity University on the NASA grounds) and did testing there.

Their demo showed an ability to ask a remote control center to assist a car with a situation it doesn’t understand. When the car sees something it can’t handle, it stops or pulls over, and people in the remote call center can draw a path on their console to tell the car where to go instead. For example, it can be drawn how to get around an obstacle, or take a detour, or obey somebody directing traffic. If the same problem happens again, and it is approved, the next car can use the same path if it remains clear.

I have seen this technology a number of places before, including of course the Mars rovers, and we use something like it at Starship Technologies for our delivery robots. This is the first deployment by a major automaker.

Nissan also committed to deployment in early 2020 as they have before — but now it’s closer.

You can also see Nissan’s more unusual concepts, with tiny sensor pods instead of side-view mirrors, and steering wheels that fold up.

Startups

Several startups were present. One is AIMotive, from Hungary. They gave me a demo ride in their test car. They are building a complete software suite, primarily using cameras and radar but also able to use LIDAR. They are working to sell it to automotive OEMs and already work with Volvo on DriveMe. The system uses neural networks for perception, but more traditional coding for path planning and other functions. It wasn’t too fond of Las Vegas roads because the lane markers are not painted there — lanes are divided only with Bott’s Dots. But it was still able to drive by finding the edge of the road. They claim they now have 120 engineers working on self-driving systems in Hungary.

Startup NAuto showed their dashcam system, currently sold for monitoring drivers. They plan to use the data gathered by it to help people train self-driving systems.

RobotTuner in the Netherlands has built a simulated environment for testing and development of cars. They offered to let people race it like a video game. The sample car was a neural network model — you can’t really train them in simulation or you will train on the artifacts of simulation — but you can use them to test out your simulation.

Civil Maps, only recently funded, was out pushing their new mapping database, built by using neural network techniques and people’s laser and video scans of the road.

Navya continues to push and deploy in more places in the shuttle market. They gave rides as well.

I didn’t take most of the rides. Turns out the rides are getting dull. If the system works well, the ride is boring. I see the wheel move and I see a screen where the car is drawing boxes around the cars and other obstacles I see. If they have designed the demo well it is not making too many mistakes.

Faraday Future finally showed a production car. Press on it has been mixed — it’s not shipping yet, and it has impressive specs but they don’t seem to be that much better than the Tesla. One cute feature was a pop-up 360 degree LIDAR in the hood. Frankly, the hood is not a great place to put such a LIDAR, it only sees forward and to the sides, but it looks sleek — as though looking sleek is important in first generation cars.

FF has talked about some interesting self-driving plans, but so far that has taken a backseat (!) to their push as a high-end EV vendor to compete with Tesla. Let’s hope we hear more in the future of Faraday Future.

There were several people promoting new LIDARs at low prices. These LIDARs use MEMs mirrors — the same tool used in DLP projectors — and will thus have no large moving parts. These vendors claimed prices and capabilities to compete with Quanergy’s solid state units. Two of the vendors claimed 200m of range. That’s either a revolutionary technology or a lie, however. LIDARs in the near-infrared just can’t emit enough light to be eye safe and have it bounce off a black car 200m away and come back to the sensor bright enough to be visible against sunlight. You would need a whole new type of sensitivity — and today’s LIDARs use single photon detectors — or a way to tune out noise. Which perhaps they have, but it would be big news. You can see a white car at 200m, which may be what they are promoting, but a LIDAR that only sees white cars is not really good enough.

But the world does need to see 200m. Today’s LIDARs don’t do that, and highway speeds demand able to see stalled objects (which radar is not good at) at 200m out.

Delphi

I was impressed with Delphi’s work and attitudes. Their new test cars include even more sensors, including MobilEye. The OEMs that fail to produce their own internal self-driving systems will be going to companies like Delphi to get such systems.

Delphi believes, as I do, that the big market will be in cars for robotaxi (mobility on demand) services, rather than high-end luxury cars for individual owners that feature self-drive systems. Certainly, with services like Uber being as hot as they are, robotaxi services will be accessible to a much larger segment of the population. Most car companies focus on making cars with added self-driving at the high end, and such cars will sell — but the robotaxis will drive more miles and thus sell more.

Concepts

There were a lot of concept cars. As per usual, most of them were silly — a lot of money spent to make the company look cool with something that will never actually ship.

Popular choices were very sparse and open interiors. Also shown by several were steering wheels that could fold away or hide. I think that’s a good thing to make, though it won’t be fancy multi-motor pop outs like we see here. Once you stop driving except in special situations, you might be fine with a much simpler driving interface like handlebars (which can pop out of a dashboard more easily) or even a wheel you pull out of a shelf and plug in. Eventually, it won’t have any mechanical linkage, it will have a drive-by-wire linkage that can bypass the self-drive systems in the event of total system failure.

Also super popular — see the BMW — are panels to hide the wheels. This can reduce drag but there are a lot of problems with doing it, especially on real roads. It is done to look futuristic.

Many concepts showed small sticks with sensors instead of side-view mirrors. Today, the law mandates side-view mirrors, but soon it will allow the use of cameras which show on a screen inside the car. Car makers want to do this because it reduces drag and they have to meet their fuel efficiency targets. Everybody also loves unusual doors, which either open upwards or are in the suicide door configuration.

I suspect all these fancy add-ons may appear in high-end cars, but not in robotaxis. There you want low costs. The convenience of electric doors (like in minivans today) may be popular for taxis, so we’ll probably see that. And no need for individual doors. One person vehicles will be meant for one person, and 4 person vehicles will be meant for 3 or 4, who all get in at the same time.

For more photos and commentary, check out my gallery on cars and look to a later posting of my gallery on the hype over “smart” and “connected” devices in the internet-o-thingies.

You may have seen a lot of press around a dashcam video of a car accident in the Netherlands. It shows a Tesla in AutoPilot hitting the brakes around 1.4 seconds before a red car crashes hard into a black SUV that isn’t visible from the viewpoint of the dashcam. Many press have reported that the Tesla predicted that the two cars would hit, and because of the imminent accident, it hit the brakes to protect its occupants.

The accident is brutal but apparently nobody was hurt.

https://www.youtube.com/watch?v=lqqN5iRrAiM

The press speculation is incorrect. It got some fuel because Elon Musk himself retweeted the report linked to, but Telsa has in fact confirmed the alternate and more probable story which does not involve any prediction of the future accident. In fact, the red car plays little to no role in what took place.

Tesla’s autopilot uses radar as a key sensor. One great thing about radar is that it tells you how fast every radar target is going, as well as how far away it is. Radar for cars doesn’t tell you very accurately where the target is (roughly it can tell you what lane a target is in.) Radar beams bounce off many things, including the road. That means a radar beam can bounce off the road under a car that is in front of you, and then hit a car in front of it, even if you can’t see the car. Because the radar tells you “I see something in your lane 40m ahead going 20mph and something else 30m ahead going 60mph” you know it’s two different things.

The Tesla radar saw just that — the black SUV was hitting the brakes (possibly for a dirt patch that appears to show on the video) and the red car wasn’t. Regardless of the red car being there, the autopilot knew that if another car ahead was braking hard, it should also brake hard, and it did. Yes, it’s possible that it could also calculate that the red car, if it keeps going, will hit the black car, but that’s not entirely relevant — it’s clear that the Tesla should stop, regardless of what the red car is going to do. Tesla reported in their blog about how they were doing more with the radar, including tracking hidden vehicles with it. The ability of automotive radar to do this has been known for some time, and I have always presumed most teams have taken advantage of it. You don’t always get returns from hidden cars, but it’s worth using them if you do.

In the future, we will see robocar systems predicting accidents, but I am not aware of this being announced by any team. All robocars are tracking all objects ahead of them, for position and velocity, and they are extrapolating their velocity and predicting where they will go. Those predictions would also include detecting that vehicles might hit (if they continue their current course) and also if they could not avoid hitting at a certain point. If an imminent accident is predicted, it would make sense to know that and also react to it in advance. A car might even be able to predict a bit of what will happen after the accident, though that is chaotic.

A system like that would outperform the autopilot or any automatic emergency braking system. Presently, those systems largely track objects in their lane. They don’t brake because cars are stopped in adjacent lanes, because that would mean they could not work in traffic jams or carpool lanes when there are lanes at different speeds, and they could not deal with stalled cars on the side of the road.

However, if you saw what the Tesla saw from the lane to the right, it would still be a very smart thing to brake. Tesla has not commented on this, but I presume its system would not have braked if it had been in that lane, at least not braked before the accident. It might brake because other cars like the red car immediately moved into the right lane.

If you liked this post on robocars, you’ll also enjoy these articles:

• What if the city ran Waze and you had to obey it? Could this cure congestion?
• What are the right disability rules for robotaxis?
• Google car is now Waymo
• Comma.ai cancels comma-one add-on box after threats from NHTSA
• Robocar news: Comma One goes open source, creating simulations for robocars in New Zealand earthquakes

See all the latest robotics news on Robohub, or sign up for our weekly newsletter

BMW says it will deploy a fleet of 40 cars in late 2017

Bumping up the timetables, BMW has declared it will have a fleet of 40 self-driving series 7 cars, using BMW’s technology combined with MobilEye and Intel. Intel has recently been making a push to catch up to Nvidia as a chipmaker supplier to automakers for self-driving. It hasn’t quite said what the cars will do, but they will be trying lots of different roads. So far BMW has mostly been developing its own tech. More interesting has been their announcement of plans to sell rides via their DriveNow service. This was spoken of a year ago but not much more has been said.

https://www.youtube.com/watch?v=sqRkYemvjr8

Intel also bought 15% of “HERE” the company formerly known as Navteq and Nokia. Last year, the German automakers banded together to buy HERE from Nokia and the focus has been on “HD” self-driving maps.

Hyundai, Delphi show off cars

There are demo cars out there from Delphi and a Hyundai Ioniq. Delphi’s car has been working for a while (it’s an Audi SUV) but recently they have also added a bunch of MobilEye sensors to it. Reports about the car are good, and they hope to have it ready by 2019, showing up in 2020 or 2021 cars on dealer lots.

Toyota sticks to concepts

Toyota’s main announcement is the Concept-i meant to show off some UI design ideas. It’s cute but still very much a car, though with all the silly hallmarks of a concept — hidden wheels, strangely opening doors and more.

Quanergy announces manufacturing plans for $250 solid state LIDAR

Quanergy (Note: I am on their advisory board) announced it will begin manufacturing this year of automotive grade $250 solid state LIDARs. Perhaps this will stop all the constant articles about how LIDAR is super-expensive and means that robocars must be super-expensive too. The first model is only a taste of what’s to come in the next couple of years as well.

New Ford has a sleeker design

Ford has become the US carmaker to watch (in addition to Tesla) with their announcement last year that they don’t plan to sell their robocars, only use them to offer ride service in fleets. They are the first and only carmaker to say this is their exclusive plan. Just prior to CES, Ford showed off a new test model featuring smaller Velodyne pucks and a more deliberate design.

I have personally never understood the desire to design robocars to “look like regular cars.” I strongly believe that, just like the Prius, riders in the early robocars will want them to look distinctive, so they can show off how they are in a car of the future. Ford’s carm based on the Fusion hybrid, is a nice compromise — clearly a robocar with its sensors, but also one of sleek and deliberate design.

Nvidia keeps its push

Nvidia has a new test car they have called BB8. (Do they have to licence that name?) It looks fairly basic, and they show a demo of it taking somebody for a ride with voice control, handling a lot of environments. It’s notable that at the end, the driver has to take over to get to the destination, so it doesn’t have everything, nor would we expect it. NVIDIA is pushing their multi-GPU board as the answer to how to get a lot of computing power to run neural networks in the car.

What’s coming

Announcements are due tomorrow from Nissan and probably others. I’ll report Friday from the show floor. See you there!

If you liked this post on robocars, you’ll also enjoy these articles:

• What if the city ran Waze and you had to obey it? Could this cure congestion?
• What are the right disability rules for robotaxis?
• Google car is now Waymo
• Comma.ai cancels comma-one add-on box after threats from NHTSA
• Robocar news: Comma One goes open source, creating simulations for robocars in New Zealand earthquakes

See all the latest robotics news on Robohub, or sign up for our weekly newsletter

Image: Kenguru

Robocars are broadly going to be a huge boon for many people with disabilities, especially disabilities that make it difficult to drive or those that make it hard to get in and out of vehicles. Existing disability regulations and policies were written without robocars in mind, and there are probably some improvements that need to be made.

While I was at Google, I helped slightly with the project to show the first non-employee getting to use the car to run errands. The subject we selected was 95% blind, and of course he can’t drive, and even using transit is a burden. It was obvious to him immediately how life-changing the technology will be.

Some background on disabled transport

There are two rough policy approaches to making things more accessible. One requires that we make everything accessible. The other uses special accommodations for the disabled.

Making everything accessible is broadly preferred by advocates. Wheelchair ramps on all public buildngs etc. Doing less than this runs a risk of “separate but equal” which quickly becomes separate and inferior. It’s also hugely expensive, and while that cost is borne by people like building owners and society, there is not unlimited budget, and there are arguments that there may be more efficient ways to spend the resources that are available. There are also lots of very different disabilities, and you need very different methods to deal with impairments in sight, mobility, hearing, cognition and the rest.

Over 50 million people in the USA have some sort of disability, so this is no minor matter.

In transportation, there is a general goal to make public transit accessible. To supplement that, or where that is not done, there are the paratransit rules. Paratransit offers people who meet certain tests an alternate ride (usually in a door to door van) for themselves and a helper for no more than twice the cost of a regular bus ticket. That sounds great until you learn you also have to schedule it a day in advance, and have a one-hour pickup window (which the disabled hate) and it’s hugely expensive, with an average cost per ride of over $30, which cities hate. (In the worst towns, it is $60/ride.) In some cities it approaches half the transit budget. Some cities, looking at that huge cost, let some disabled customers just use taxis for short trips, which provide much better service and cost much less. (Though to avoid over-use they put limitations on this.)

There are Americans with Disabilities Act rules for taxis. Regular sedan taxis are not directly regulated though there can be no discrimination of disabled customers who are capable of riding in a sedan. Any new van of up to 8 seats has to been accessible, which often means things like wheelchair lifts. In addition, once a taxi fleet has accessible vans, it has to offer “equivalent service” levels. This might mean that if it has 200 sedans, it can’t buy just one van because there would be much longer wait times to get that van. To get around this, a lot of companies use a loophole and purchase only used vans. The law only covers the use of new vans. Companies like Uber and Lyft don’t own vehicles at all, and so are not governed in the same way by fleet requirements, though they do offer accessible vehicle services in some cities.

When Uber and similar companies move to offering robotaxi service with vehicles they own, these laws would apply to them. Unlike some companies, the used van loophole will also be difficult since most robotaxis will be custom built new.

New Types of Vehicles

Robotaxi service offers the promise of a vehicle on demand, and it offers the potential of a vehicle well fitted to the trip. Mostly I talk about things like the ability to use a small and inexpensive one person vehicle for solo urban trips (which are 80% of trips, so this is a big deal) but it also means sending an SUV when 3 people want to go skiing, or a pickup-truck for a work run, or a van designed for socializing when a group of people want to travel together.

It also offers the ability to create vehicles just for people with certain disabilities. One example I find quite interesting is the Kenguru — a small, single person vehicle which is hollow, and allows a user in a wheelchair to just roll in the back and steer it with hand controls. For wheelchair users with working arms, this is hugely superior to designs that require you to get out of your chair into a car seat, or which involve the time delays of using a wheelchair lift. Especially with nobody to assist. Roll-in, roll-out can match the convenience of the able-bodied. The current Kenguru is to be steered, but a self-driving vehicle like this could handle even those in power chairs, and offer a fold-down bench for an able-bodied companion.

Being computerized, these vehicles will also offer accessible user interfaces. Indeed, they may mostly rely on the user’s phone, which will already be customized to their needs.

Custom-designed to meet particular disabilities, these vehicles will both serve the disabled better and frankly be not that useful for others. As such, regimes that require adapting all vehicles to handle both types of customers may have the right spirit, but provide inferior service.

Another key benefit of robotaxi service for the disabled will be the low price. Reduced job prospects drive many with disabilities into poverty. Service that is naturally low in price will be enabling.

Equivalent service or Separate but Superior

Providing “equivalent” service is difficult with traditional taxis, particularly for smaller fleets. Robotaxis, which don’t mind waiting around because no human driver is waiting, make this easier to do. The service level of a robotaxi service is based on the density of currently unused vehicles in your area. Increase fleet size with the same demand, and service level goes up. As long as fleet size is not way overblown, so that vehicles still wear out by the mile rather than by the year, increasing fleet size is not nearly as expensive as it is for regular cars or human driven taxis.

This means you can, fairly readily, offer equivalent or even superior service at a pretty reasonable cost. As long as disabled-designed vehicles are made in decent quantities to keep their costs low, the cost should be close to the cost of regular vehicles. In the public interest, regular vehicle customers might subsidize the slightly higher cost of these lower volume vehicles.

With increased fleets, service levels would generally be superior to the regular fleets, but not always. The law generally allows this, but the disabled community will need to understand a few unequal things that probably will happen:

• Slightly more advanced notice of rides will often make it possible to provide service at lower cost. Regular vehicles will naturally be present on every block. Disabled vehicles might be present with less density during high use times, but the ability to reposition lets even slight advance notice do a lot.
• For those in groups, it may not be easy to carry a person in a wheelchair along with several non-wheelchair passengers. This might mean the wheelchair passenger goes in their own vehicle (with videoconference link.) This is not as good, but is much more cost effective than requiring every van to have a wheelchair lift.
• To increase service levels, it is likely competing companies would cooperate on serving the disabled, and pool fleets. Until the disabled become a profitable market rather than one done to meet goals of public good, companies will prefer to work together. As such if you call for an Uber, you might often get a Lyft or other small fleet car.
• Low cost disabled transport may mean that accessible public transit and paratransit slowly fade. Public transit which has its own tracks will continue to be accessible as it offers a speed advantage which may not be met on the roads, but otherwise it may be much cheaper to offer private robotaxis than to make all transit accessible. This would mean a group of people might not be able to ride transit together if it’s not accessible.
• Small electric vehicles may be allowed to enter buildings, dropping passengers right at elevator lobbies or other destinations.

The biggest trade-off will be the loss of social group experiences. There certainly will be buses and vans with lifts which allow groups of mixed-ability passengers to travel together, but it is unlikely these would be so common as to offer the same service level as ordinary vans. With advance notice of just 10 minutes, they could probably be available.

If you liked this post on robocars, you’ll also enjoy these articles:

• What if the city ran Waze and you had to obey it? Could this cure congestion?
• Google car is now Waymo
• Comma.ai cancels comma-one add-on box after threats from NHTSA
• Robocar news: Comma One goes open source, creating simulations for robocars in New Zealand earthquakes

See all the latest robotics news on Robohub, or sign up for our weekly newsletter

Source: Waymo

Google’s car project (known as “Chauffeur”) kickstarted the entire robocar revolution, and Google has put in more work, for longer, than anybody. The car was also the first project of what became Google “X” (or just “X” today under Alphabet. Inside X, a lab devoted to big audacious “moonshot” projects that affect the physical world as well as the digital, they have promoted the idea that projects should eventually “graduate,” moving from being research to real commercial efforts.

Alphabet has announced that the project will be its own subsidiary company with the new name “Waymo.” The name is not the news; what’s important is the move away from being a unit of a mega-company like Google or Alphabet. The freedoms to act that come with being a start-up (though a fairly large and well funded one) are greater than units in large corporations have. Contrast what Uber was able to do, skirting and even violating the law until it got the law changed, with what big corporations need to do.

Google also released information about how in 2015 they took Steve Mahan — the blind man who was also the first non-employee to try out a car for running errands — for the first non-employee (blind or otherwise) fully self-driving ride on public streets, in a vehicle with no steering wheel and no backup safety driver in the vehicle. (This may be an effort to counter large amount of press about public ride offerings by nuTonomy in Singapore and Uber in Pittsburgh, as well as truck deliveries by Uber/Otto in 2016.)

It took Google/Alphabet 6 years to let somebody ride on public streets in part because it is a big company. It’s an interesting contrast with how Otto did a demonstration video after just a few months of life of a truck driving a Nevada highway with nobody behind the wheel (but Otto employees inside and around it.) That’s the sort of radical step that startups do.

Waymo declared their next goal is to “let people use our vehicles to do everyday things like run errands, commute to work, or get safely home after a night on the town.” This is the brass ring, a “Mobility on Demand” service, which is able to pick people up (ie. run unmanned) and even carry a drunk person.

The last point is important. To carry a drunk is a particular challenge. In terms of improving road safety, it’s one of the most worthwhile things we could do with self-driving cars since drunks have so many of the accidents. To carry a drunk, you can’t let the human take control even if they want to. Unlike unmanned operation, you must travel at the speed impatient humans demand, and you must protect the precious cargo. To make things worse, in some legal jurisdictions, they still want to consider the person inside the car the “driver,” which could mean that since the “driver” is impaired, operation is illegal.

Waymo as leader

The importance of this project is hard to overstate. While most car companies had small backburner projects related to self-driving going back many years, and a number of worthwhile research milestones were conquered in the 90s and even earlier, the Google/Waymo project, which sprang from the Darpa Grand Challenge, energized everybody. Tiny projects at car companies all got internal funding because car companies couldn’t tolerate the press and the world thinking and writing that true future of the car was coming from a non-car company, a search engine company. Now the car companies have divisions with thousands of engineers and it’s thanks to Google. The Google/Waymo team was accomplishing tasks 5 years ago that most projects are only now just getting to, especially in non-highway driving. They were rejecting avenues (like driving with a human on standby ready to take the wheel on short notice) in 2013 that many projects are still trying to figure out.

Indeed, even in 2010, when I first joined the project and it had just over a dozen people, it had already accomplished more complex tasks that most projects, even the Tesla Autopilot that some people think is in the lead, have yet to accomplish.

Let’s see where Waymo goes.

I believe we have the potential to eliminate a major fraction of traffic congestion in the near future, using technology that exists today which will be cheap in the future. The method has been outlined by myself and others in the past, but here I offer an alternate way to
explain it which may help crystallize it in people’s minds.

Today many people drive almost all the time guided by their smartphone, using navigation
apps like Google Maps, Apple Maps or Waze (now owned by Google.) Many have come to drive as though they were a robot under the command of the app, trusting and obeying it
at every turn. Tools like these apps are even causing controversy, because in the hunt
for the quickest trip, they are often finding creative routes that bypass congested
major roads for local streets that used to be lightly used.

Put simply, the answer to traffic congestion might be, “What if you, by law, had to
obey your navigation app at rush hour?” To be more specific, what if the cities and towns that own the streets handed out reservations for routes on those streets to you via those apps, and your navigation app directed you down them? And what if the cities made sure there were never more cars put on a piece of road than it had capacity to handle?

The value is huge. Estimates suggest congestion costs around $160B per year and 42 hours of time for every driver, or around another $160B.

Road metering works

This approach would exploit one principle in road management that’s been most effective
in reducing congestion, namely road metering. The majority of traffic congestion is caused,
no surprise, by excess traffic — more cars trying to use a stretch of road than it has the capacity to handle. There are other things that cause congestion — accidents, gridlock and irrational driver behaviour, but even these only cause traffic jams when the road is near or over capacity.

Today, in many cities, highway metering is keeping the highways flowing far better than they used to. When highways stall, the metering lights stop cars from entering the freeway as fast as they want. You get frustrated waiting at the metering light but the reward is you
eventually get on a freeway that’s not as badly overloaded.

Another type of metering is called congestion pricing. Pioneered in Singapore, these
systems place a toll on driving in the most congested areas, typically the downtown cores
at rush hour. They are also used in London, Milan, Stockholm and some smaller towns, but have never caught on in many other areas for political reasons. Congestion charging can easily be viewed as allocating the roads to the rich when they were paid for by everybody’s taxes.

A third successful metering system is the High-occupancy toll lane. HOT lanes take
carpool lanes that are being underutilized, and let drivers pay a market-based price to use them solo. The price is set to bring in just enough solo drivers to avoid wasting the spare
capacity of the lane without overloading it. Taking those solo drivers out of the other
lanes improves their flow as well. While not every city will admit it, carpool lanes themselves have not been a success. 90% of the carpools in them are families or others who would have carpooled anyway. The 10% “induced” carpools are great, but if the carpool lane only runs at 50% capacity, it ends up causing more congestion than it saves.

HOT is a metering system that fixes that problem…

Click here to cotinue reading on robocars.com!

If you liked this article, you may also want to read these articles:

• Comma.ai cancels comma-one add-on box after threats from NHTSA
• Robocar news: Comma One goes open source, creating simulations for robocars in New Zealand earthquakes
• The delicate balance between brilliance, safety and arrogance
• Oxbotica’s autonomous vehicle software learns its environment and how it changes over time
• Robocar parking is incredibly cheap

See all the latest robotics news on Robohub, or sign up for our weekly newsletter

Source: commaai

There have been few postings this month, as I took the time to enjoy a holiday in New Zealand around speaking at the SingularityU New Zealand summit in Christchurch. The night before the summit, we enjoyed a 7.8 earthquake not so far from Christchurch, whose downtown was 2/3 demolished after quakes in 2010 and 2011. On the 11th floor of the hotel it was a disturbing nailbiter of swaying back and forth for over 2 minutes — but of course, swaying is what the building is supposed to do, that means it’s working. The shocks were rolling, not violent, and in fact, we got more violent jolts from aftershocks a week later when we went to Picton.

While driving around that region, we encountered this classic earthquake scene on the road:

There were many like this and in fact the main highway of the South Island was destroyed long-term not too far away, cutting off several towns. A scene like this makes you wonder just what a robocar would do in these situations. I already answered this question in a blog post on how to handle a tsunami. Fortunately, there was only a mild tsunami for this quake. A tsunami will result in a warning in the rich world and the car will know the elevation map of the roads and know how to get to high ground. In some places, like Japan, there is also an advanced earthquake warning system that tells you quakes are coming well before they hit since electrons go much faster than seismic waves. With such a system, robocars should receive a warning and come to a stop unless they need to evacuate a tsunami zone. Without such a warning, we still could imagine the road cracking and collapsing in front of you as might have happened on this road. Of course, the cones and signs that warned me days later would not be present.

The answer again lies in the fact that pictures (like mine) will be used to create situations like this in a simulator and car developers will be able to test their systems with simulated quake damage. I’ve spoken since 2010 on the value of a shared simulator environment and I think if government agencies like NHTSA want to really help development, providing funding and tools for such an environment would be a good step. NHTSA’s proposal that all developers share their logs of all incidents would clearly make such a simulator better, but there is pushback because of the proprietary value of those logs. When it comes to strange situations like earthquakes, I doubt there would be much pushback on having an open and shared simulator environment.

New Zealand’s government is taking a very welcoming approach to robocars. They are not regulating for a while and have invited developers to come and test. They have even said it’s OK to test unmanned vehicles under some fairly simple rules. NZ does not have any auto industry and it’s quite remote, but we’ll see if they can attract developers to come test. Their roads feature something you don’t see much in the USA — tons and tons of one-lane bridges and other one-lane stretches of highway. Turns out that robocars, with a little bit of communication, can make efficient use of one-lane two-way roads, so it might be worth exploring.

[Editor’s note: Christchurch will host New Zealand’s first trial of a fully autonomous electric vehicle next year. The trials will largely be conducted on the Christchurch Airport campus, starting on private roads with no public present, with the long-term aim of moving to public roads once the safety case has been made and all regulatory approvals are in place.]

Open Source Comma One box

Speaking of Open, today Comma.ai, which previously had declared they were giving up on their neural network autopilot due to NHTSA threats today announced they have open sourced their software, along with hardware designs and case designs. NHTSA did not want them making an autopilot and said they could not simply rely on the fact that drivers were told they must be diligent, It will be very interesting to see how NHTSA reacts to the release of open designs that anybody can then install on their car.

The automotive industry has had a long history of valuing the tinkerer. All the big car companies had their beginnings with small tinkerers and inventors. Some even died in the very machines they were inventing. These beginnings have allowed people to do all sorts of playing around in their garages with new car ideas, without government oversight, in spite of the risk to themselves and even others on the road. If a mechanic wants to charge you for working on your car they must be licenced, but you are free to work on it yourself with no licence, and even build experimental cars. You just can’t sell them. And even those rights have been eroded.

Clearly far fewer people will have the inclination to build an autopilot using the comma.ai tools by themselves. But it won’t be that hard to do, and they can make it easier with time, too. One could even imagine a car which already had the necessary hardware so that you only needed to download software to make it happen.

In recent times, there has been a strong effort to prevent people with tinkering with their cars, even in software. One common area of controversy has been around engine tuning. Engine tuning is regulated by the EPA to keep emissions low. Car vendors have to show they have done this — and they can’t program their car to give good emissions only on the test while getting better performance off the test as VW did. But owners have been known to want to make such modifications. Now we will see modifications that affect not just emissions but safety. Car companies don’t want to be responsible if you modify the code in your car and there is an accident involving both their code and yours. As such, they will try to secure their car systems so you can’t change them, and the government may help them or even insist on it. When you add computer security risks to the mix — who can certify the modified car can’t be taken over and used as a weapon? — it will get even more fun.

I will also point out that I suspect that comma’s approach would not know what to do about the collapsed road because it would never have been trained in that situation. It might, however, simply sound an alert and kick out, not being able to find the lane anymore.

Regulatory pushback

Regular readers will have seen my strong critique of the NHTSA rules. The other major news during my break was the pushback from major players in the public comment on the regulations. In some ways, the regulations didn’t do enough to give vendors the certainty they need to make their plans. At the same time they were criticized for not giving enough flexibility to vendors. In addition, as expected, they resist giving up their proprietary data in the proposed forced sharing. I predict continued ambivalence on the regulations. Big players actually like having lots of regulations, because big players know how to deal with that and small players don’t.

This post originally appeared on robocars.com.

Source: commaai

Comma.ai, the brash startup attempting to make a self-driving system entirely from a neural network has announced it will cancel the “comma one” add-on box it has planned to sell to owners of certain Honda vehicles. The box stuck on the rear-view mirror and used the car’s own bus commands to provide an autopilot similar to those offered by car makers, with lane-keeping and adaptive cruise control.

Of particular importance is the letter from NHTSA to comma.ai which I suggest you read. This letter creates several big issues:

1. There are many elements of this letter that would also apply to Tesla and other automakers which have built supervised autopilot functions.
2. Of particular interest is the paragraph that says: “it is insufficient to assert, as you do, that the product does not remove any of the driver’s responsibilities” and “there is a high likelihood that some drivers will use your product in a manner that exceeds its intended purpose.” That must be very scary for Tesla.
3. I noted before that the new NHTSA regulations appear to forbid the use of “black box” neural network approaches to the car’s path planning and decision making. I wondered if this made illegal the approach being done by Comma, NVIDIA and many other labs and players. This may suggest that.
4. We now have a taste of the new regulatory regime and had it existed before systems like Tesla’s Autopilot, Mercedes Traffic Jam Assist, and Cruise’s original aftermarket autopilot would never have been able to get off the ground.
5. George Hotz of comma declares, “Would much rather spend my life building amazing tech than dealing with regulators and lawyers. It isn’t worth it. The comma one is cancelled. comma.ai will be exploring other products and markets. Hello from Shenzhen, China.”

To be clear, comma is a tiny company taking a radical approach, so it is not a given that what NHTSA has applied to them would have been or will be unanswerable by the big guys. Because Tesla’s Autopilot is not a pure machine learning system, they can answer many of the questions in the NHTSA letter that comma can’t. They can do much more extensive testing that a tiny startup can’t. But even so, a letter like this sends a huge chill through the industry.

It should also be noted that in Comma’s photos the box replaced the rear-view mirror, and NHTSA had reason to ask about that.

Source: commaai

George’s declaration that he’s in Shenzen gives us the first sign of the new regulatory regime pushing innovation away from the United States and California. I will presume the regulators will say, “We only want to scare away dangerous innovation” but the hard truth is that is a very difficult thing to judge. All innovation in this space is going to be a bit dangerous. It’s all there trying to take the car — the 2nd most dangerous legal consumer product — and make it safer, but it starts from a place of danger. We are not going to get to safety without taking risks along the way.

I sometimes ask, “Why do we let 16-year-olds drive?” They are clearly a major danger to themselves and others. Driver testing is grossly inadequate. They are not adults so they don’t have the legal rights of adults. We let them drive because they are going to start out dangerous and then get better. It is the only practical way for them to get better, and we all went through it. Today’s early companies are teenagers. They are going to take risks. But this is the fastest and only practical way to let them get better and save millions.

“…some drivers will use your product in a manner that exceeds its intended purpose”

This sentence, though in the cover letter and not the actual legal demand, looks at the question asked so much after the Tesla fatal crash. The question which caused Consumer Reports to ask Tesla to turn off the feature. The question which caused MobilEye, they say, to sever their relationship with Tesla.

The paradox of the Autopilot is this: The better it gets, the more likely it is to make drivers over-depend on it. The more likely they will get complacent and look away from the road. And thus, the more likely you will see a horrible crash like the Tesla fatality. How do you deal with a system which adds more danger the better you make it? Customers don’t want annoying countermeasures. This may be another reason that “Level 2,” as I wrote yesterday is not really a meaningful thing.

NHTSA has put a line in the sand. It is no longer going to be enough to say that drivers are told to still pay attention.

Black box

Comma is not the only company trying to build a system with pure neural networks doing the actual steering decisions (known as “path planning”.) NVIDIA’s teams have been actively working on this, as have several others. They plan to make commentary to NHTSA about these elements of the regulations, which should not be forbidding this approach until we know it to be dangerous.

It is challenging. It’s hard to do QA on neural networks. You can examine any single state of them, but not really understand them. You can fix the errors they make, but not know how you fixed it or whether your fix is going to work in other cases. On their own that sounds too scary, but the problem is they are outperforming other algorithms at many of the problems they are being applied to.

If you have two systems:

1. A conventionally coded system which you fully understand, that has one safety incident per 100,000 miles,
2. A black box machine learning system that has one safety incident per 150,000 miles, but you have minimal understanding of.

Which is the one that is better to put on the road? It’s not a no-brainer.

Scaring people away

George Hotz is about as brash as they come, so some might discount his bluster about moving his projects to China. But this is a real thing. When regulations come, they tend to favour the big players. The big players have large QA teams and compliance departments. They can afford them, and they move conservatively because of them. Big players help write the regulations to favour their way of doing things — sometimes selfishly but sometimes just because it is the only thing they know.

The USA and California led the way in robocars in part because it was unregulated. In the USA, everything is permitted unless it was explicitly forbidden and nobody thought to write “no robots” in the laws. Progress in other countries where everything is forbidden unless it is permitted was much slower. The USA is moving in the wrong direction.

Countries like China are another animal altogether. There, everything is forbidden unless a smart autocrat allows it but once the smart (or stupid) ruling official makes it happen, the other red tape can vanish. It is not the best system overall, but it can win at things they put their minds to.

Tesla

Has Tesla received such a letter? What would they have done if they received it in their early days? While Tesla and Elon Musk love the Autopilot project, the hard truth is Tesla’s biggest problem is going to be making enough Model 3s to meet demand and doing it profitably. Taking out Autopilot isn’t going to cut their sales at all — they have too many sales and not enough cars. Only their forward vision and guts will keep them going if NHTSA does the same thing to them. I do believe Tesla can answer questions like those in the demand letter more easily than comma.ai. But not trivially, and they almost surely will need to bump the countermeasures.

Some people have wondered about my forecast in the spreadsheet on robotaxi economics about the very low parking costs I have predicted. I wrote about most of the reasons for this in my 2007 essay on Robocar Parking, but let me expand and add some modern notes here.

The glut of parking

Today, researchers estimate there are between 3 and 8 parking spots for every car in the USA. The number 8 includes lots of barely used parking (all the shoulders of all the rural roads, for example) but the value of 3 is not unreasonable. Almost all working cars have a spot at their home base, and a spot at their common destination (the workplace.) There are then lots of other places (streets, retail lots, etc.) to find that 3rd spot. It’s probably an underestimate.

We can’t use all of these at once, but we’re going to get a great deal more efficient at it. Today, people must park within a short walk of their destination. Nobody wants to park a mile away. Parking lots, however, need to be sized for peak demand. Shopping malls are surrounded by parking that is only ever used during the Christmas shopping season. Robocars will “load balance” so that if one lot is full, a spot in an empty lot too far away is just fine.

Small size and Valet Density

When robocars need to park, they’ll do it like the best parking valets you’ve ever seen. They don’t even need to leave space for the valet to open the door to get out. (The best ones get close by getting out the window!) Because the cars can move in concert, a car at the back can get out almost as quickly as one at the front. No fancy communications network is needed; all you need is a simple rule that if you boxed somebody in, and they turn on their lights and move an inch towards you, you move an inch yourself (and so on with those who boxed you in) to clear a path. Already, you’ve got 1.5x to 2x the density of an ordinary lot.

I forecast that many robotaxis will be small, meant for 1-2 people. A car like that, 4’ by 12’ would occupy under 50 square feet of space. Today’s parking lots tend to allocate about 300 square feet per car. With these small cars you’re talking 4 to 6 times as many cars in the same space. You do need some spare space for moving around, but less than humans need.

When we’re talking about robotaxis, we’re talking about sharing. Much of the time robotaxis won’t park at all, they will be off to pick up their next passenger. A smaller fraction of them will be waiting/parked at any given time. My conservative prediction is that one robotaxi will replace 4 cars (some estimate up to 10 but they’re overdoing it.) So at a rough guess we replace 1,000 cars, 900 of which are parked, with 250 cars, only 150 of which are parked at slow times. (Almost none are parked during the busy times.)

Many more spaces available for use

Robocars don’t park, they “stand.” Which means we can let them wait all sorts of places we don’t let you park. In front of hydrants. In front of driveways. In driveways. A car in front of a hydrant will be gone at the first notification of a fire or sound of a siren. A car in front of your driveway will be gone the minute your garage opens or, if your phone signals your approach, before you get close to your house. Ideally, you won’t even know its there. You can also explicitly rent out your driveway space for money if you wish it. You could rent your garage too, but the rate will be so low you will prefer to use it to add a new room to your house unless you still own a car.

In addition, at off-peak times (when less road capacity is needed) robocars can double park or triple park along the sides of roads. (Human cars would need to use only the curb spots, but the moment they put on their turn signal, a hole can clear through the robocars to let them out.)

So if we consider just these numbers — only 1/6 of the time spent parking and either 4 times the density in parking lots or 2-3 times the volume of non-lot parking (due to the 2 spots per car and loads of extra spots) we’re talking about a huge, massive, whopping glut of parking. Such a large glut that in time, a lot of this parking space will be converted to other uses, in time reducing the glut.

Ability to move in response to demand

To add to this glut, robocars will be the best parking customers you could ever imagine. If you own a parking lot, you will have sold the space at the back or top of your lot to the robocars — they will park in the unpopular more remote sections for a discount. The human driver customers will take those spots by the entrance. As your lot fills up, you can ask the robocars to leave, or pay more. If a high paying human driver appears at the entrance, you can tell the robocars you want their space, and off they will go to make room. Or they can look around on the market and discover they should just pay you more to keep the space. The lot owner is always making the most they can.

If robocars are electric, they will also be excellent visitors by making little noise and not emitting soot to dirty your walls. They will leave a tiny amount of rubber and that’s about it.

The “spot” market

All of this will be driven by what I give the ironic name of the “spot” market in parking. Such markets are already being built by start-ups for human drivers. In this market, space in lots will be offered and bid for like any other market. Duration will be negotiated, too. Cars will evaluate potential waiting places based on price and the time it will take to get there and park, as well as the time to get to their likely next pickup. A privately owned car might drive a few miles to a super cheap lot to wait 7 hours, but when it’s closer to quitting time, pay a premium (in competition with many others of course) to be close to their master.

Ideally, driveway owners and cities would put all the spots they control into the spot market too. Free parking may vanish and be replaced by very cheap parking. Human drivers with smartphones will also buy their parking in the spot market, ideally. In some cases, free parking will appear there with conditions. A retailer may offer free parking to human drivers (who probably want to shop) or frictionless validation systems could be built. However, parking may get so cheap that all of this becomes unimportant.

The zero incremental cost

Parking is perhaps the lowest incremental cost product there is. If you own a parking spot, your costs don’t change if it’s occupied or not — not in a world of automatic electronic payments. If you leave a spot empty when somebody would pay for it — pay almost anything — you are leaving money on the table. There are many products roughly of this class, but few as strongly in it as a parking spot. Especially when we’re talking about an electric robocar that drips no fluids, emits no soot, pays online, and will leave if you have a higher paying customer. There’s not even opportunity cost in that case. About the only cost is extremely minor pavement wear, deposit of rubber and traffic going in and out. Perhaps a slight increment in security (though cars will monitor their own.)

What happens if we have a glut of spaces, and leaving a space empty leaves money on the table? It means the base price of parking is very, very cheap. People will accept offers of 10 cents/hour because they will get nothing if they refuse.

In some markets, players band together (though this is technically illegal) and put an artificial floor on prices. This could happen here — it could even be made legal to “save the industry” which will indeed be in trouble. But it only takes one person with an empty lot within a few miles to break ranks. What cities do with their on-street parking will also have a big effect.

Street parking is, as noted a special case. At peak times, most cars and all robotaxis are in service, and park/stand only briefly. On high-demand routes, street parking will be forbidden, and robotaxis will hear and obey an order to leave. Off-peak, more cars need to park, but road capacity needs are also less, and cities can open street parking. They can even open 2 lanes per side of double-parked street parking on the wider streets of lower usage. In addition, any driveways which held a car that went out on commute become available, as do the spaces in front of all driveways. Cities will face pressure from parking lots not to drop prices too low on this parking. Street parking will be more valuable, because it takes zero time to get in and out of a street space compared to a parking lot (especially a multi-floor one.)

At night, there is tons of long term capacity; all the office and closed retail lots, the sides of most streets, and the driveways of homes who have given up car ownership. The owned cars will be living there (after visiting the charging station for a few hours.)

Parking and charging

When electric robocars need to charge, they can travel to the electricity. You don’t need to put charging stations at every home and in every parking lot, though you will have some there. The best places for charging stations will be power grid transformer substations, where many megawatts can be tapped at low infrastructure cost. This may be some of the most expensive parking to be found and lots around these substations will be the few to go up in value.

People already make robotic fueling stations for gasoline, so robotic plug-in seems pretty likely here as well. You will pay for the electricity, but the car will also pay for time taking up a spot so cars will want to charge fast. Fortunately charging is getting pretty fast, and the 10kwh battery of a single person robotaxi won’t take long. Today, there is a trade-off: The faster you charge, the more risk you have of reducing the life of the battery, so there will be a sweet spot charge rate for cars not in a hurry. Robocars will rarely be in a hurry.

When the glut ends

At a rate of only dimes per hour, nobody is likely to build more parking, and many people will want to convert their parking lots to new purposes. That will take time — in fact in many cities the zoning laws actually demand that buildings have adequate parking. Open parking lots are easy to repurpose, but big parking structures might be difficult to convert and require demolition. Some parking structures are in basements or make up the central core of buildings, spaces without light that are not useful for offices.

As the spaces disappear, the prices will go up, but at the same time, there will be more and more robocars and robotaxis. Outside of the downtown cores, land is not free but bare paved land is actually not that expensive. An acre of land can hold over 700 single person robotaxis parked densely. Even at $100,000 per acre that’s somewhat tolerable, and in most places land is cheaper than that. If only holding lightweight vehicles, you can pave at a much lower cost, too.

In super-dense downtown cores, parking will still cost money, but cars will have the option to get a couple of miles out of town. For robotaxi users, if there is not enough capacity to bring in all the cars at 5pm for their passengers, vans and buses could arise to take people in bulk in roughly the right direction to outside the congestion zone, where their personal ride will be waiting right where the van stops.

So will you circle for parking?

I have frequently seen people say they imagine robocar owners would tell their cars to just drive around rather than pay for parking. That’s never been very likely. Today’s cars cost about $20/hour to run, and that’s more than today’s parking. Tomorrow’s cars may cost as little as $2-3/hour to run but that will be more than tomorrow’s parking. (It’s also unlikely you would send you car 15 miles to your home to wait. The parking will cost less, and robotaxi service for those still at home will be cheap as well.)

It is my hope that cities will take the opportunity to convert some of the no longer needed parking lots to interesting new purposes, including of course parks of the natural kind. Yes, some will turn into condo towers and office blocks, but it’s a rare opportunity to gain a bonanza of land in the cities.

The long awaited list of recommendations and potential regulations for robocars has just been released by the National Highway Traffic Safety Administration (NHTSA), the federal agency that regulates car safety and safety issues in car manufacture. Normally, NHTSA does not regulate car technology before it is released into the market, and the agency, while it says it is wary of slowing down this safety-increasing technology, has decided to do the unprecedented — and at a whopping 115 pages.

Broadly, this is very much the wrong direction. Nobody — not Google, Uber, Ford, GM or certainly NHTSA — knows the precise form these cars will have when deployed. Almost surely something will change from our existing knowledge today. They know this but still wish to move. Some of the larger players have pushed for regulation. Big companies like certainty. They want to know what the rules will be before they invest. Startups thrive better in the chaos, making up rules as they go along.

NHTSA hopes to define “best practices” but the best anybody can do in 2016 is lay down existing practices and conventional wisdom. The entirely new methods of providing safety that are yet to be invented won’t be in such a definition.

The document is very detailed, so it will generate several blog posts of analysis. Here I present just initial reactions. Those reactions are broadly negative. This document is too detailed by an order of magnitude. Its regulations begin today, but fortunately, they are also accepting public comment. The scope of the document is so large, however, that it seems extremely unlikely that they would scale back this document to the level it should be at. As such, the progress of robocar development in the USA may be seriously negatively affected.

Vehicle performance guidelines

The first part of the regulations is a proposed 15 point safety standard. It must be certified (by the vendor) that the car meets these standards. NHTSA wants the power, according to an Op-Ed by no less than President Obama, to be able to pull cars from the road that don’t meet these safety promises.

Framework for vehicle performance guidance. Credit: NHTSA, Federal Automated Vehicles Policy

• Data Recording and Sharing
• Privacy
• System Safety
• Vehicle Cybersecurity
• Human Machine Interface
• Crashworthiness
• Consumer Education and Training
• Registration and Certification
• Post-Crash Behavior
• Federal, State and Local Laws
• Operational Design Domain
• Object and Event Detection and Response
• Fall Back (Minimal Risk Condition)
• Validation Methods
• Ethical considerations

As you might guess, the most disturbing is the last one. As I have written many times, the issue of ethical “trolley problems” where cars must decide between killing one person are another are a philosophy class tool, not a guide to real-world situations. Developers should spend as close to zero effort on these problems as possible since they are not common enough to warrant special attention. If not for our morbid fascination with machines making life or death decisions in hypothetical situations. Let the policymakers answer these questions if they want to; programmers and vendors don’t.

For the past couple of years, this has been a game that’s kept people entertained and ethicists employed. The idea that government regulations might demand solutions to these problems before these cars can go on the road is appalling. If these regulations are written this way, we will delay saving lots of real lives in the interest of debating which highly hypothetical lives will be saved or harmed in ridiculously rare situations.

NHTSA’s rules demand that ethical decisions be “made consciously and intentionally.” Algorithms must be “transparent” and based on input from regulators, drivers, passengers and road users. While the section makes mention of machine learning techniques, it seems in the same breath to forbid them.

Most of the other rules are more innocuous. Of course, all vendors will know and have little trouble listing what roads their car works on, and they will have extensive testing
data on the car’s perception system and how it handles every sort of failure. However, the requirement to keep the government constantly updated will be burdensome. Some vehicles will be adding streets to their route map literally ever day.

While I have been a professional privacy advocate, and I do care about just how the privacy of car users is protected, I am frankly not that concerned during the pilot project phase about how well this is done. I do want a good regime — and even the ability to do anonymous taxi — so it’s perhaps not too bad to think about these things now, but I suspect these regulations will be fairly meaningless unless written in consultation with independent privacy advocates. The hard reality is that during the test phase, even a privacy advocate has to admit that the cars will need to make very extensive recordings of everything they can, so that any problems encountered can be studied and fixed and placed into the test suite.

50 state laws

NHTSA’s plan has been partially endorsed by the self-driving coalition for safer streets (whose members include big players Ford, Google, Volvo, Uber and Lyft.) They like the fact that it has guidance for states on how to write their regulations, fearing that regulations may differ too much state to state. I have written that having 50 sets of rules may not be that bad an idea because jurisdictional competition can allow legal innovation and having software load new parameters as you drive over a border is not that hard.

In this document, NHTSA asks the states to yield to the DOT on regulating robocar operation and performance. States should stick to registering cars, rules of the road, safety inspections and insurance. States will regulate human drivers as before, but the feds will regulate computer drivers.

States will still regulate testing, in theory, but the test cars must comply with the federal regulations.

New authorities

A large part of the document just lists the legal justifications for NHTSA to regulate in this fashion and is primarily for policy wonks. Section 4, however, lists new authorities NHTSA is going to seek in order to do more regulation.

Some of the authorities they may see include:

• Pre-market safety assurance: Defining testing tools and methods to be used before selling
• Pre-market approval authority: Vendors would need approval from NHTSA before selling, rather than self-certifying compliance with the regulations
• Hybrid approaches of pre-market approval and self-certification
• Cease and desist authority: The ability to demand cars be taken off the road
• Exemption authority: An ability to grant rue exemptions for testing
• Post-sale authority to regulate software changes
• Much more

Other quick notes:

• NHTSA has abandoned their levels in favour of the SAE’s. The SAE’s were almost identical of course, with the addition of a “level 5” which is meaningless because it requires a vehicle that can drive literally everywhere, and there is not really a commercial reason to make a car at present that can do that.
• NHTSA is now pushing the acronym “HAV” (highly automated vehicle) as yet another contender in the large sea of names people use for this technology. (Self-driving car, driverless car, autonomous vehicle, automated vehicle, robocar etc.)

More analysis is on the way.

Tesla Model S autopilot-software. Source: Tesla

Tesla’s spat with MobilEye reached a new pitch this week with Tesla announcing a new release of their autopilot and plans. As reported earlier, MobilEye announced during the summer that they would not be supplying the new and better versions of their EyeQ system to Tesla. Since that system was/is central to the operation of the Tesla Autopilot, they may have been surprised that MBLY stock took a big hit after the announcement (though it recovered for a while, and is now back down) and TSLA did not.

Statements and documents now show a nastier battle with MobilEye saying they were worried about Tesla using their tool in an unsafe way, invoking all the debate about the fatality and other crashes allegedly caused by people who were lulled into not bothering to supervise the Autopilot. Tesla says that, instead, they have been developing their own advanced vision tools and that MobilEye was afraid and told Tesla if they wanted more EyeQ chips, they would need to halt the competing project and commit to ME. That’s a nasty spat.

Tesla’s own efforts represent a threat to MobilEye from the growing revolution in neural network pattern matchers. Computer vision is going through a big revolution. MobilEye is a big player in that revolution because their ASICs do both standard machine vision functions and can do neural networks. An ASIC will beat a general purpose processor when it comes to cost, speed, and power, but only if the ASIC’s abilities were designed to solve those particular problems. Since it takes years to bring an ASIC to production, you have to aim right. MobilEye aimed pretty well, but at the same time, lots of research out there is trying to aim even better or do things with more general purpose chips, like GPUs. Soon we will see ASICs aimed directly at neural network computations. To solve the problem with neural networks, you need the computing horsepower, and you need well designed deep network architectures, plus, you need the right training data and lots of it. Tesla and ME both are gaining lots of training data. Many companies, including Nvidia, Intel and others are working on the hardware for neural networks. Most people would point to Google as the company with the best skills in architecting the networks, though there are many doing interesting work there. (Google’s DeepMind built the tools that beat humans at the seemingly impossible game of Go, for example.) It’s definitely a competitive race.

Using radar

While Tesla works on their vision systems, they also announced a plan to make much more use of radar. That’s an interesting plan. Radar has been the poor 3rd-class sensor of the robocar, after LIDAR and vision. Everybody uses it — you would be crazy not to unless you need to be very low cost. Radar sees further than the other systems, and it tells you immediately how fast any radar target you see is moving relative to you. It sees through fog and other weather, and it can even see under and around big cars in front of you as it bounces off the road and other objects. It’s really good at licence plates, as well.

What radar doesn’t have is high resolution. Today’s automotive radars have gotten good enough to tell you what lane an object like another car is in, but they are not designed to have any vertical resolution — you will get radar returns from a stalled car ahead of you on the road and a sign above that lane, and not be sure of the difference. You need your car to avoid a stalled car in your lane, but you can’t have a car that hits the brakes every time it sees a road sign or bridge!

mage source: EDN Network

Real world radar is messy. Your antennas send out and receive from a very broad cone with potential signals from other directions and from side lobes. Reflections are coming from vehicles and road users but also from the ground, hills, trees, fences, signs, bushes and bridges. It’s work to get reliable information from it. Early automotive radars found the best solution was to use the doppler speed information, and discard all returns from anything that wasn’t moving towards or away from you — including stalled cars and cross traffic.

One thing that can help (imperfectly) is a map. You can know where the bridges and signs are so you don’t brake for them. Now you can brake for the stalled cars and the cross traffic the Tesla failed to see. You still have an issue with a stalled car under a bridge or sign, but you’re doing a lot better.

There’s a lot of room for improvement in radar and I will presume — Tesla has not said — that Tesla plans to work on this. The automotive radars everybody buys (from companies like Bosch) were made for the ADAS market — adaptive cruise control, emergency braking etc. It is possible to design new radars with more resolution (particularly in the vertical) and other approaches. You can also try for more resolution, particularly by splitting the transmitter and receiver to produce a synthetic larger aperture. You can go into different bands and get more bandwidth and get more resolution in general. You can play more software tricks, and particularly, you can learn by examining not just single radar returns, but rather the pattern of returns over time. (After all, humans don’t navigate from still frames, we depend on our visual system’s deep evolved ability to use motion and other clues to understand the world.)

The neural networks are making strides here. For example, while pedestrians produce basic radar returns, it turns out that their walking stride has a particular pattern of changes that can be identified by neural networks. People are doing research now on how examining the moving and dynamic pattern of radar returns can help you get more resolution and also identify shapes and motion patterns of objects and figure out what they are.

I will also speculate that it might be possible to return to a successor of the “sweeped” radars of old, the ones we are used to seeing in old war movies. Modern car radars don’t scan like that, but I have to wonder if with new techniques, like phased arrays to steer virtual beams (already the norm in military radar) and modern high-speed electronics, that we might produce radars that get a better sense of where their target is. We’re also getting better at sensor fusion — identifying a radar target in an image or LIDAR return to help learn more about it.

The one best way to improve radar resolution would be to use more bandwidth. There have been experiments in using ultrawideband signals in the very high frequencies which may offer promise. As the name suggests, UWB uses a very wide band, and it distributes its energy over that very wide band, which means it doesn’t put too much energy into any one band and has less chance of interfering in those bands. It’s also possible that the FCC, seeing the tremendous public value that reliable robocars offer, might consider opening up more spectrum for use in radar applications using modern techniques and thus increase the resolution.

In other words, Tesla is wise to work on getting more from radar. With the loss of all MobilEye’s vision tools, they will have to work hard to duplicate and surpass that. For now, Tesla is committed to using parts that are for sale for existing production cars, costing hundreds of dollars. That has taken LIDAR “off their radar” even though almost all research teams depend on LIDAR and expect LIDAR to be cheap in a couple of years. (Including the LIDAR from Quanergy, a company I advise.)

Comma announces a $1,000 autopilot box

I wrote earlier about comma.ai and their efforts to drive with just vision, radar and neural networks. They now plan to offer a box for $1,000 to give you some basic autopilot functionality as an add-on.

To do this, they are working with only some specific car models, namely some Honda vehicles that already have advanced ADAS in them. Using the car’s internal bus, they can talk to the sensors in these cars (such as the radar and possibly the camera) and also send control signals to actuate the steering, brakes and throttle. Then their neural networks can take the sensor information, and output the steering and speed commands to keep you in the lane. (Details are scant so I don’t know if the Comma One box uses its own camera or depends on access to the car’s.)

When I rode in Comma’s prototype it certainly wasn’t up to the level of the Tesla autopilot or some others, but it has been several months so I can’t judge it now. Like the Tesla autopilot, the Comma will not be safe enough to drive the car on its own, and you will need to supervise and be ready to intervene at any time. If you get complacent, as some Tesla drivers have, you could get injured or killed. I have yet to learn what measures Comma will take to make sure people keep their eyes on the road.

Generally, I feel that autopilots are not very exciting products when you have to watch them all the time — as you do — and that bolt-on products are not particularly exciting. Cruise’s initial plan (after they abandoned valet parking) was a bolt-on autopilot, but they soon switched to trying to build a real vehicle, and that got them the huge $700M sale to General Motors.

But for Comma, there is a worthwhile angle. Users of this bolt-on box will be helping to provide training data to improve their systems. In fact, they will be paying for the privilege of testing the system and training it. Something that companies like Google did the old-fashioned way, paying a staff of professionals to drive the cars and gather data. For a tiny, young startup it’s a worthwhile approach.

If you liked this article, you may also want to read:

• Robocar recap: Uber buys Otto, folks leave Google, Ford goes big, Tesla drops MobilEye
• Will robocars be heaven or hell for our cities?
• Robocar platooning, or just carpool?
• Understanding the massive gulf between the Tesla Autopilot and a real robocar, in light of the crash

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Source: Wikipedia Commons

The vision of many of us for robocars is a world of less private car ownership and more use of robotaxis — on-demand ride service in a robocar. That’s what companies like Uber clearly are pushing for, and probably Google, but several of the big car companies including Mercedes, Ford and BMW among others have also said they want to get there — in the case of Ford, without first making private robocars for their traditional customers.

In this world, what does it cost to operate these cars? How much might competitive services charge for rides? How much money will they make? What factors, including price, will they compete on, and how will that alter the landscape?

Here are some basic models of cost. I compare a low-cost 1-2 person robotaxi, a higher-end 1-2 person robotaxi, a 4-person traditional sedan robotaxi and the costs of ownership for a private car, the Toyota Prius 2, as calculated by Edmunds. An important difference is that the taxis are forecast to drive 50,000 miles/year (as taxis do) and wear out fully in 5 years. The private car is forecast to drive 15,000 miles/year (higher than the average for new cars, which is 12,000) and still have many years and miles of life left in it. As such, the taxis fully depreciate in this 5-year timeline, the private car only partly.

Some numbers are speculative. I am predicting that the robotaxis will have an insurance cost well below today’s cars, which cost about 6 cents/mile for liability insurance. The taxis will actually be self-insured, meaning this is the expected cost of any incidents. In the early days, this will not be true — the taxis will be safer, but the incidents will cost more until things settle down. As such the insurance prices are for the future.

Fuel is electric for the taxis, gasoline/hybrid for the Prius. The light vehicle is very efficient.

Maintenance is also speculative. Today’s cars spend about 6 cents/mile, including 1 cent/mile for the tires. Electric cars are expected to have lower maintenance costs, but the totals here are higher because the car is going 250,000 miles not 75,000 miles like the Prius. With this high level of maintenance and such smooth driving, I forecast low repair cost.

Parking is cheaper for the taxis for several reasons. First, they can freely move around looking for the cheapest place to wait, which will often be free city parking or the cheapest advertised parking on the auction “spot” market. They do not need to park right where the passenger is going, as the private car does. They will park valet style, and so the small cars will use less space and pay less too. Parking may actually be much cheaper than this, even free in many cases. Of course, many private car owners do not pay for parking overtly so this varies a lot from city to city.

(You can view the spreadsheet directly on Google docs and download it to your own tool to play around with the model.)

The Prius has one of the lowest costs of ownership of any regular car (take out the parking and it’s only 38 cents/mile) but its price is massively undercut by the electric robotaxi, especially my estimates for the half-width electric city car. (I have not even included the tax credits that apply to electric cars today.) For the taxis, I add 15% vacant miles to come up with the final cost.

The price of the Prius is the retail cost (on which you must also pay tax) but a taxi fleet operator would pay a wholesale, or even manufacturer’s cost. Of course, they now have the costs of running a fleet of self-driving cars. That includes all the virtual stuff (software, maps and apps) with websites and all the other staff of a big service company ranging from lawyers to marketing departments. This is hard to estimate because if the company gets big, this cost will not be based on miles, and even so, it will not add many cents per mile. The costs of the Prius for fuel, repair, maintenance and the rest are also all retail. The taxi operator wants a margin, and a big margin at first, though with competition this margin would settle to that of other service businesses.

So, what will the price be of your robocar ride? Today, an Uber ride tends to be a mix of a “per ride” fee (which hopefully covers the cost of getting to you) and about $1 to $1.50 per mile and about $13/hour in the USA. (In Peru it’s $3.60 per hour and just 40 cents/mile.) Depending on your speed, USA UberX will cost you $2 to $2.50 per mile. As such, $1/mile would be a very attractive, but also very profitable introductory price.

Competition should drive prices down to more normal margins. Typical gross profits in transportation are 50%. This suggests from 20 to 25 cents/mile for the urban solo ride up to 50 cents/mile for the 4 passenger sedan (or just 12 cents/mile/person if you fill it up or carpool in it.) This means the small car at 25 cents/mile has revenues of only $35/day (around $12,500 per year) with COGS of $16/day. I have not factored in per-ride fees or trip minimums, which are normal in Taxi/Uber services. (The Uber $5 trip minimum covers a whopping 20 miles in this car.) I expect trip minimal to drop to compete with $2.50 transit tickets, especially when pooling.

Google runs a gross margin that’s very high (about 67%) but on the other end consider that Uber (though not profitable yet) takes 20% of its gross cost of a ride (and pays its own expenses from that.)

What are these small cars?

I describe a number of ways a 1-2 person robotaxi might be very cheap in this article. Let’s go into more detail on why it might get manufactured for around $10K (retail would be higher.) Note that there are already many low-end 4 person gasoline cars which retail around $14,000 such as the Nissan Versa and Chevy Spark. Outside the USA, there are even cheaper cars, like the $8,000 Renault Twizzy. Some of these cars are full-width and come with dashboards full of stuff, pedals, wheels, doors, hatchbacks and more. They meet crash-safety standards and the rest of the FMVSS.

The basic commuter car will be:

• Just over half the width and under 2/3 the length of these cars
• Weigh perhaps half of those cars, using far less materials
• Have no dashboard, no radio, no pedals, no wheel
• Have only one door instead of 3 or 4, and no trunk (just an internal cargo area)
• Have only 1-2 seats (with associated airbags and components)
• Have vastly fewer parts
• In the urban version, not go over 45mph, allowing a low-cost motor

What it will have is a battery pack. At 100 watt-hours/mile, a 10kwh battery would give it 100 miles of range, more than enough with recharging breaks during the day to cover the average of 140 miles/day it will do as an urban taxi. The forecast cost of $150/kwh puts that pack at $1,500. Expensive, but when combined with the much simpler electric motor powertrain, a reasonable cost.

It will also have about $1,000 of sensors and computers in it. These prices get this low because these vehicles are made in the many hundreds of thousands, and eventually millions. It also has a few things not found in cheap cars, like a single power door lock.

The higher-end vehicle at $16,000 is pretty similar in design, but the extra money goes into making it more luxurious:

• Better seats and interior materials, replaced more often
• Better suspension for smoother ride
• Better sound padding
• A screen and sound system, or better ones than in the base vehicle
• A distinctive exterior with better styling and fancier paint, to let people see you are in the higher end vehicle.
• Power windows. (Both cars have power locks, and neither car need seat position adjustments.)
• Some fancier interior furniture (tables, drink holders, laptop holders etc.)
• Possibly a motorized door for easy entry (one of the more expensive add-ons)
• Possibly a sunroof (but there will also be open-top cars for order if you want them and the weather is good.)

When I have examined the lists of luxury features on high-end cars, I have been surprised at how few of them are needed in the luxury robocar. Driver assist functions, 10-way motorized seats, motorized mirrors, telescoping steering wheels, fancy wheels, fancy lights and many other features play no role in the robotaxi.

My numbers are aggressive — so if you doubt them, increase them. The result is still a robotaxi at a low per-mile operating and capital cost, which means in a competitive market, it will be available at a price that easily competes with private car ownership, and even subsidized bus rides.

Source: Otto/YouTube

Brad Templeton, from Robocars.com, discusses the latest robocar related news.

The past few weeks has seen big robocar news. Real news — not the constant “X is partnering with Y” press releases that fill the airwaves sometimes.

Uber has made a deal to purchase Otto, a self-driving truck company I wrote about earlier, founded by several friends of mine from Google. The rumoured terms of the deal are astronomical — possibly 1% of Uber’s highly valued stock (which means almost $700M) and other performance rewards. I have no other information yet on the terms, but it’s safe to say Otto was just getting started with ambitious goals and would not have sold for less than an impressive amount. For a company only six months old, the rumoured terms surpass even the amazing valuation stories of Cruise and Zoox.

While Otto has been working on self-driving technology for trucks, any such technology can also move into cars. Uber already has an active lab in Pittsburgh, but up to now has not been involved in long-haul trucking (it does do local deliveries in some places). There are many startups calling themselves the “Uber for Trucks” and Otto has revealed it was also working on shipping management platform tools, so this will strike some fear into those startups. Because of my friendship with Otto’s team, I will do more commentary when details become public.

In other Uber news, Uber has announced it will sell randomly assigned Uber rides in their self-driving vehicles in Pittsburgh. If your ride request is picked at random (and because it’s in the right place) Uber will send one of their own cars to drive you on your ride, and will make the ride free to boot. Of course, there will be an Uber safety driver in the vehicle monitoring it, ready to take over in any problem or complex situation. So the rides are a gimmick to some extent, but if they were not free, it would be a sign of another way to get customers to pay for the cost of testing and verifying self-driving cars. The free rides, however, will probably cause more people to take Uber rides hoping they will win the lottery and get not simply the free ride, but the self-driving ride.

GM announced a similar program for Lyft — but not until next year.

Ford also goes all-in, but with a later date

Ford announced it wants to commit to making unmanned capable taxi vehicles, the same thing Uber, Google, Cruise/GM, Zoox and most non-car companies want to make. For many years I have outlined the difference between the usual car company approaches, which are evolutionary and involve taking cars and improving their computers and the approaches of the non-car companies which bypass all legacy thinking (mostly around ADAS) to go directly to the final target. I call that “taking a computer and putting wheels on it.” It’s a big and bold move for Ford to switch to the other camp and a good sign for them. They have said they will have a fleet of such vehicles as soon as 2021.

Other companies have announced plans to go for unmanned capable taxi vehicles, particularly Daimler and BMW, but they still maintain active “evolutionary” projects which hope to build on their prior ADAS work.

Velodyne

The other big news is that Ford, together with Baidu, have made a $150M investment in Velodyne. Velodyne has been for a decade the primary supplier of LIDARs for robocar research projects, but their LIDARs are research instruments: high performance at high cost. In the meantime, other companies (such as Quanergy, for whom I am an advisor) have been pushing to make better-priced LIDARs for real production. The Velodynes have been the right choice because it is smart to research with what’s expensive today, knowing it will get cheap in the future. Baidu’s test cars have had Velodyne’s on them, but there were recent rumours of Baidu experimenting with alternatives to LIDAR, but this investment may counter those rumours.

This investment will move Velodyne to a lower cost production unit, though at present at a higher cost than the solid-state plans of Quanergy and some others.

Google departures

Otto included a few people from Google’s car team. It was also recently revealed that Chris Urmson, who had run the team for most of its existence and was moved to “CTO” after the arrival of John Krafcik, has departed. I worked for Chris, but he has not told me his reason for departure. In general Google’s car team has gotten huge, and the recently departed people are mostly folks who were among the first 20 on the team. This may simply be a sign of massive expansion, I have not heard much to credit the idea that there is some sort of internal chaos. In addition, deals like the Otto deal will certainly cause people to consider if they want to work for a big corporation or break out on their own.

MobilEye

A few weeks ago, MobilEye issued a press release that they would not be supplying the next generation of their EyeQ chips to Tesla. The EyeQ4 is a great deal more sophisticated than the EyeQ3 that Telsa uses and Tesla even uses an older version with a more limited set of features — including, most famously, the inability to detect things like cross traffic. Since MobilEye announced it, it seems like it was their decision (you don’t usually issue a press release to say you have been dumped by a smaller customer.) It caused their stock to take a quick big hit (disclaimer, I own a small amount) on the day of the announcement, but it recovered reasonably well. Tesla’s stock was not affected.

Tesla folks have always claimed that they only made minimal use of ME’s functionality and that the real special sauce was theirs. ME folks have claimed the opposite. This break-up will settle that question, and presumably, Tesla has been working on alternatives for a while as they don’t seem to be in much panic about it. Tesla has made announcements of plans for their new hardware and those plans look very similar to the EyeQ4, so they must have had their own parallel research underway.

If you liked this article, you may also want to read:

• Robocar news around the globe: Tesla crash, Declaration of Amsterdam, and automaker services
• 50 different state regulations for robocars is not a bad idea
• Will robocars be heaven or hell for our cities?
• Robocar platooning, or just carpool?

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Source: US DOT

At the recent AUVSI/TRB conference in San Francisco, there was talk of upcoming regulation, particularly from NHTSA. Secretary of Transportation Foxx and his NHTSA staff spoke with just vague hints about what might come in the proposals due this fall. Generally, they said good things, namely that they are wary of slowing down the development of the technology. But they said things that suggest other directions.

Secretary Foxx began by agreeing that the past history of automotive driving systems was quite different. Regulations have typically been written years or decades after technologies have been deployed. And written regulations tend to involve standards which the vendors self-certify their compliance with. What this means is that there is not a government test center that confirms a car complies with the rules in the safety standards. Instead, the vendor certifies they are following the rules. If they certify falsely, that can get them in trouble later with regulators, and more importantly, in lawsuits. It’s by far the best approach unless the vendors have shown that they can’t be trusted in spite of the fear of these actions.

But Foxx said that they were going to go against that history and consider “pre-market regulation.” Regular readers will know I think that’s an unwise idea, and so do many regulators, who admit that we don’t know enough about the final form of the technology to regulate yet.

Fortunately, it was also suggested that NHTSA’s new documents would be more in the form of “guidance” for states. Many states ask NHTSA to help them write self-driving car regulations. Which gets us to a statement that was echoed by several speakers to justify federal regulation, “Nobody wants 50 different regulations” on these cars.

At first, that seems obvious. I mean, who would want it to be that complex? Clearly, it’s simpler to have to deal with only one set of regulations. But while that’s true, it doesn’t mean it’s the best idea. They are overestimating the work involved in dealing with different regulations, and underestimating the value of having the ability for states to experiment with new ideas in regulation, and the value of having states compete on who can write the best regulations.

If regulations differed so much between states as to require different hardware, that makes a stronger case. But most probable, we are talking about rules that affect the software. That can be annoying, but it’s just annoying. A car can switch what rules it follows in software when it crosses a border with no trouble. It already has to, just because of the different rules of the road found in every state, and indeed, every city and even every street! Having a few different policies state-by-state is no big addition.

Jurisdictional competition is a good thing though, particularly with emerging technologies. Let some states do it wrong, and others do it better, at least at the start. Let them compete to bring the technology first to their region, and invent new ideas on how to regulate something the world has never seen. Over time these regulations can be normalized. By the time people are making 10s of millions of robocars, that normalization will make more sense. But most vendors only plan to deploy in just a few states to begin with anyway. If a state feels its regulations are making it harder for the cars to spread to its cities, it can copy the rules of the other state it likes best.

The competition assures any mistake is localized — and probably eventually fixed. If California follows through with banning unmanned operation, as they have proposed, Texas has said it won’t.

I noted that if the hardware has to change, that’s more of an issue. It’s still not that much of an issue, because cars that operate as taxi services will probably never leave their base state. Most of them will have limited operational zones, and except in cities that straddle state borders, they won’t even leave town, let alone leave the state. Some day, the cars might do interstate trips, but even then you can solve this by having one car drive you to the border and then transfer to a car for the other state. Annoying, but only slightly, and not a deal-breaker on the service. A car you own and take on road trips is a different story.

The one way having different state regulations would be a burden would be if there were 50 different complex certification processes to go through. Today, the federal government regulates how cars are made and the safety standards for that. The states regulate how cars operate on the roads. Robocars do blur that line, because how they are made controls how they drive.

For now, I still believe the tort system — even though it differs in all 50 states — is the best approach to regulation. It already has all developers highly paranoid about safety. When the day comes for certification, a unified process could make sense, but that day is still very far away. But for the regulations of just how these cars will operate, it might make sense to keep that with the states, even though it’s now part of the design of the car rather than the intentions of a human driver.

In time, unified regulations will indeed be desired by all, once we’ve had the time to figure out what the right regulations should be. But today? It’s too soon. Innovation requires variety.

One lone taxi cab surrounded by quite a few self-driving cars in congestion.

Robin Chase wrote an article wondering if robocars will improve or ruin our cities and asked for my comment on it. It’s a long article, and I have lots of comment, since I have been considering these issues for a while. On this site, I spend most of my time on the potential positive future, though I have written various articles on downsides and there are yet more to write about.

Robin’s question has been a popular one of late, in part a reaction by urban planners who are finally starting to think more deeply on the topic and reacting to the utopian visions sometimes presented. I am guilty of such visions, though not as guilty as some. We are all seduced in part by excitement of what’s possible in a world where most or all cars are robocars — a world that is not coming for several decades, if in our lifetimes at all. It’s fair to look at the topic from both sides, as no technology is 100% good.

When I first met Robin, she was, like most people, a robocar skeptic. She’s done pioneering work in new transportation ideas but the pace of improvement has surprised even the optimists. I agree with many of the potential negatives directions that she and others paint; in fact, I’ve said them myself. Nonetheless, my core position is that we can and probably will get tremendous good out of this. While I want city planners to understand these trends, I think it’s too early for them to actually attempt to guide them. Even the developers of the technology don’t quite know the final form it will take when it starts taking over the transport world in the 2020s. Long-term planning is simply impossible at this stage — it must be done not with the knowledge of 2016 but with the knowledge of 2023. That approach — the norm in the high tech world, where we expect the world to constantly change underneath us — is anathema to governments and planners. When Marc A. said that software was eating the world, he was telling the world that it will need to start learning the rules of innovation that come from the high tech, internet and computer worlds.

Instead, today’s knowledge can at least guide planners in what not to do. Not to put big investments in things likely to become obsolete. Not to be too clever in thinking they understand the “smart city” of 2025. They need to be like the builders of the internet, who made the infrastructure as simple and stupid as they could, moving innovation away from the infrastructure and into the edges where it could flourish in a way that astounded humanity.

Congestion

We will get more congestion in the start. Not because of empty vehicles cruising around — most research suggests that will be around 15% of miles, and then only after everybody switches. We’ll get more congestion from two factors:

• The early cars, especially the big car company offerings, will make traffic jams more tolerable. As such, people will not work as hard to avoid them.
• Car travel will be come much better and much cheaper; far more people will be able to use it, so they’ll travel more miles in cars than they do today.
• For some, longer commutes will be more tolerable so they will live further from work. That won’t increase congestion in the central areas (they would still have driven those roads if they lived closer to work) but will increase it in the more remote places.
• The tolerance for longer commutes may increase “sprawl.”

The good news is that the era of the ubiquitous smartphone brings us the potential for a traffic “miracle” — the ability to entirely eliminate traffic congestion. I first made that remarkable claim in 2008 in my article on congestion. I have a new article in the works which expands on this and makes it easier to understand. The plan is a rare one for me, because the city is heavily involved, but mostly in virtual infrastructure rather than physical. Virtual infrastructure needs to be the new buzzword of the city planner, because only virtual infrastructure is flexible enough to adapt to a changing world.

While this, and other plans to eliminate congestion won’t actually arise very quickly, the reason is not technological, it’s political. So the rise in congestion for the reasons cited above has a silver lining — it will push the public to be more accepting of entirely new ways of managing traffic.

The other way we can attack congestion is through the potential to make vastly superior group transit. Today’s transit sucks. It uses more energy than cars, provides slow and limited service from station to station (not door to door) in limited areas. When it does work efficiently, at rush hour, people travel standing, packed like sardines. People hate it so much that they spend over $8,000/year on vastly more expensive car ownership, the 2nd largest expense in most households. Robocars offer the potential for very appealing group transit which takes people efficiently from door-to-door in luxury vans on their schedule and along fast routes. Truly appealing transit might greatly increase ridership at congested times.

Robin suggests her Prius could drive around for $1.50/hour rather than park and that will make things worse. Perhaps if people make the same mistake it could, but when you look at it, you realize it costs closer to $20/hour to have a car drive around, and the fuel is just part of that. (Most auto web sites rate the Prius as costing 50 cents/mile, and at 25mph that’s only $12.50 per hour but in reality urban miles tend to cost more than highway miles so I like hourly rates. The Prius is rare though in that it uses less fuel in city miles.) Certainly no rational actor would do this. In addition, as more cars are shared, parking will become plentiful, particularly since a car no longer needs to park right where it dropped you off, but can instead request price bids on the “spot market” and find space going spare not too far away, which will certainly be available for well under $1.50/hour.

Unemployment

Fewer people will drive for a living. At the same time, there are more bank tellers today than in 1970. They just don’t cash your cheques and give out withdrawls much anymore. This topic deserves a great deal more verbiage, of course, but the kicker is this: These professional drivers are killing several thousand Americans every year while doing their jobs. Only doctors kill more. While the economic disruption is not an illusion, there is no way you can justify artificially preserving a job that is killing so many people. It’s a bit like arguing everybody should smoke so that tobacco farmers don’t lose their jobs.

Shared cars & parking

This will be huge, at least the part about sharing rides. Sharing cars for solo rides does not reduce miles driven or the number of cars made, but it does vastly reduce the amount of parking needed. Sharing rides reduces everything. I go much further in my vision to bring ride sharing to the level of dynamically allocated self-driving vans which replace today’s mass transit with something much more desired by the public and much more efficient at the same time.

I do hope the city parking lots are turned into parks mostly. The privately owned lots will get other uses, though downtown multi-floor lots are a bit harder to change.

Energy grid

It’s true that a major move to electric cars might require more electric capacity. Though they will charge mostly at night when power is cheap (though not solar.) One thing that many people don’t realize we won’t need is charging infrastructure. The great thing about robocars is they go where the energy is. The robocar will drive to the transformer substation which is packed with charging points — you don’t need to put charging stations in parking lots or houses.

However, at least today, electric cars are not cheaper than gasoline ones. The electricity is dirt cheap — under 3 cents/mile. The problem is at today’s battery prices, the battery depreciation is 20 to 40 cents per mile, much more than gasoline. Fortunately, there are optimistic signs about cheaper batteries and longer lasting batteries which could fix this.

But as robocars shrink — especially to one person vehicles for solo riders — they will become much cheaper than today’s cars, and also much more efficient. More efficient than the cars, but also all US transit systems. At a cost of around 30 cents/mile, car transportation will be available to billions more than can afford it today, and certainly to almost all Americans. That has its congestion downsides.

What should cities do?

As noted above, it’s more about what they should not do. I am rebuilding my recommendations here, but my current list includes this:

• Avoid regulation until you know what players can’t be trusted to do, and then fix only that.
• No more light rail or other single-use right-of-way. Stick to plain, bare pavement which can handle everything.
• Create “transfer points” for carpools, robotaxi and robovan services to quickly — really quickly — transfer passengers between vehicles. These are useful for robocars, smartphone carpooling and even today’s transit.
• Don’t require new buildings to put in tons of parking if they don’t want to.
• Make as much of your infrastructure virtual as you can. Encourage lots of data networks in the town, with the newest (5G and later) protocols in 2020.
• If installing dedicated ROW for transit, make sure it can be converted to use by robocars in the future so the capacity isn’t wasted most of the time. If making tunnels, make sure stations are “offline” so that other vehicles can pass stopped vehicles, and make ramps for access by approved vehicles from the street.

Trucks platooning on the motorway. Photo: Dan Boman

At the recent AUVSI/TRB symposium, a popular research topic was platooning for robocars and trucks. Platooning is perhaps the oldest practical proposal when it comes to car automation because you can have the lead vehicle driven by a human, even a specially trained one, and thus, resolve all the problems that come from road situations too complex for software to easily handle.

Some early experiments indicate fuel savings, though relatively modest ones. At practical distances, you can see about 10% saving for following vehicles and 5% for the lead vehicle. Unfortunately, a few big negatives showed up. It’s hard to arrange platoons, errors can become catastrophic multi-car pile-ups, other drivers keep inserting themselves into the gap unless it’s dangerously small, and the surprising deal-breaker that comes from the stone chips which are thrown up by lead vehicles which destroy the finish — and in some cases the radiator or windshield — of following cars. They can also create a congestion problem and highway exit problem the way existing convoys of trucks sometimes do that.

One local company named Peloton is making progress with a very simple platooning problem. They platoon two (and only two) trucks on rural highways. The trucks find one another over the regular data networks, and when they get close they establish a local radio connection (using the DSRC protocol that many mistakenly hope will be the standard for vehicle to vehicle communications.) Both drivers keep driving, but the rear driver goes feet-off-the-pedals like a cruise control. The system keeps the vehicles a fixed distance to save fuel. The trucks don’t mind the stone chips too much. Some day, the rear driver might be allowed to go in the back and sleep, which would allow cargo to move 22 hours/day at a lower cost, probably similar to the cost of today’s team driving (about 50 cents/mile) but with two loads instead of one.

Trucks are an easy win, but I also saw a lot of proposals for car platoons. Car platoons are meant to save fuel, but also to increase road capacity. But after looking at all the research, a stronger realization came to me. If you have robocars, why would you platoon when you can carpool?. To carpool, you need to find two cars who are going to share a long segment of the trip together. Once you have found that, however, you get far more savings in fuel and road usage if the cars can quickly pause together and the passengers from one transfer into the other. Then the empty car can go and move other commuters. This presumes, of course, that the cars are like almost all cars out there today, with many empty seats. When the groups of passengers come to where their path diverts, the vehicle would need to stop at a transfer point, and some passengers would move into waiting robotaxis to take them the rest of the way.

Source: peloton-tech

All of this is not as convenient as platooning, which in theory can happen without slowing down and finding a transfer point. This is one reason that the carpool transfer stations I wrote about last month could be a very useful thing. Such stations would add only 1-2 minutes of delay, and that’s well worth it if you consider that compared to platooning, this carpooling means a vastly greater fuel saving (almost 50%) and a much greater increase in road capacity, with none of the other downsides of platooning.

If you’re thinking ahead, however, you will connect this idea to my proposed plan for the future of group transit. The real win is to have the computers of the transport service providers notice the common routes of passengers early, before they even get into a vehicle, and thus pool them together with minimal need to stop and switch cars.

Some folks have imagined designing cars that can physically couple, which would produce very efficient platoons and not add a delay. The problem (aside from the difficulties in doing this safely) is that this requires a physical standard, and physical standards are much harder to get working than software ones. It requires you find a platooning partner who has the same hardware you do, rather than software platooning, which can work with any style of car. Automated matching and carpooling makes no requirements on the individual robocars and their design, which gives it the best path to success.

It is possible (though a bit frightening) to imagine a special bus which could dock to robocars to allow transfer of passengers at speed. Some of you may have seen that a Chinese company has built the formerly hypothetical straddling bus (actually a train) that has cars drive under it. If you were assured a perfectly smooth road one could imagine a docking extension which could surround a car door of a perfectly synced robocar and allow a transfer. I suspect that’s all pretty far in the future.

Beyond the carpool

In a robocar world, we should see a move to having vehicles with fewer empty seats. This happens if more people use single person vehicles for their solo trips, and as carpooling and other technologies make sure that the four-seater vehicles end up with more people. Indeed, if the carpooling works, that happens naturally. At that point one might say, “now’s the time to platoon.” There is merit to that, but it comes later, rather than sooner. At this later date, we can be more comfortable with the safety, and have a greater density of vehicles making it more likely to find others vehicles ready to platoon. Of course, we’ll also have more vans and buses on the road who can combine even larger groups, if you find groups with a lot of journey in common. Platooning is practical even for a few miles, while carpooling tends to need a longer amount of shared journey to make it worth the switch.

At that point in the technology, you can do much more serious platoons, with larger groups of cars, and distances which are short enough for even greater benefit, and quick enough to strongly discourage people trying to insert themselves in the middle of the platoon.

So platoons will come and give us even more road capacity. Carpooling, though, is already happening, with 50% of Uber requests in San Francisco being done in UberPool mode. It is the more likely early answer.

Tesla and Powerwall. Credit: Tesla

Today, I want to look at some implications of Tesla’s Master Plan, Part Deux which caused some buzz this week. There was other news, of course, including the AUVSI/TRB meeting which I attended, and will report on, forecast dates from Volvo, BMW and others, hints from Baidu, Faraday Future and Apple, and more.

In Musk’s blog post he laid out these elements of Tesla’s plan:

• Integrating generation and storage (with SolarCity and the Powerwall and your car.)
• Expand into trucks and minibuses
• More autonomy in Tesla cars
• Hiring out your Tesla as a robotaxi, when not using it

Except for the first one, all of these are ideas I have covered extensively. It is good to see an automaker start work in these directions. As such, while I will mostly agree with what Tesla is saying, there are a few issues to discuss.

Electric (self-driving) minibus and trucks

In my article earlier this year on the future of transit I laid out why transit should mostly be done with smaller (van sized) vehicles, taking ad-hoc trips on dynamic paths, rather than the big vehicle, fixed-route, fixed-schedule approach that is taken today. The automation is what makes this happen (especially when you add the ability of single person robocars to do first and last miles.) Making the bus electric can make it greener, though, making it run full almost all the time is far more important for that.

The same is true for trucks, but both trucks and buses have huge power needs which presents problems for having them be electric. Electric’s biggest problem here is the long recharge time, and that puts your valuable asset out of service. For trucks, the big win of having a robotruck is that it can drive 24 hours/day; you don’t want to take that away by making it electric. This means you want to look into things like battery swap, or perhaps more simply, tractor swap. In that case, a truck would pull into a charging station and disconnect from its trailer and another tractor that recharged would grab on and keep it going.

You can also do that with a bus but passengers don’t want to swap, so, you may swap them out between runs. Here, the autonomous ability of the bus is key because it can drive to where the super-powerful recharging is. To recharge a bus you want a lot of power, an amount not available in many locations, and it’s costly to bring it in. If the bus comes to where the electricity already is (like transformer substations) it solves that problem. The bus is still down during recharging, of course. The bigger you make the battery, the longer you can run, but unfortunately, that also means it costs more, takes longer to recharge and weighs more, costing efficiency and making it more dangerous.

More autonomy

Well, Tesla continues to push autonomy without LIDAR. As the cost of LIDAR drops (Disclosure: I have an equity stake in this business) it will not make sense for anybody to make their vehicle less safe just to save a few hundred dollars. Still, the more competition the merrier, so Tesla’s push will at the very least teach the world some things.

Once Tesla has a door-to-door vehicle, Musk suggests that it will offer the scenario I first wrote about back in 2007 in A week in robocars — that when you are not using your car, it might hire itself out as a taxi. Tesla even plans to create a brokering service for this, competing with the Ubers of the world.

Since I first began thinking this way, I have actually become less enamoured of the idea, though I do think it will happen. I felt you might do it to help you afford a more expensive car. UberX is in effect your car making money for you, but you have to do the annoying work of driving it.

The problem stems from an over importance I and many others attributed to the problem that cars sit idle most of the time. In the robotaxi world, a larger and larger fraction of cars won’t do that, and as rates become competitive, people will think of this as a pure capital business — you make a capital investment in a car, and it pays a return charging fees to take people around. The more efficient that market gets, the purer it gets, and it becomes dominated by the professionals, not the amateurs. We have seen this happen with AirBnB, which began mostly with people renting out spare rooms and time in their apartments, and has migrated to having a lot more units that are full-time on AirBnB. If you hire out your car a lot, you start losing the values that made you want to own it in the first place — the ability to keep your stuff in it, to always have it ready for you, the ability to let it get messy, and the ability to impress people with ownership of it. Each transition between users also has a cost. We’ve seen some turmoil in existing peer-to-peer carshare companies. Turo changed its name from Relayrides and also changed to a more long-term rental model. Our SU students who started Getaround.com are still going with pure P2P though.

The counter force comes from people who are paying way more than they should for car ownership who thus have “spare” capacity they might as well sell. That’s true, but its value is only the value of the capital. Today, capital is cheap. So cheap that the interest on a $30,000 loan at 4% as you might have for a car is around $3/day or 14 cents/hour. That’s the cost of letting your asset sit around doing nothing, and it’s not that large. In fact, it might be that parking space costs more than that and is the actual incentive to get it out working.

Another big issue with electrics is that if somebody hires your car, and then you want it, it may be discharged and you can’t use it. With gasoline, you can always quickly refill.

Cars that are hired out will wear out by the mile, not the year. As such, every time it is hired out it’s depreciating, and smart people will see that.

In the future, we’ll see a mix — people who own, people who own and hire out, people who drop the number of cars they own due to robotaxi services, and people who give up car ownership altogether because of them. I foresee many of each category, but for now, the people who hire out are the smallest category. Of course, in the early days, when there are not many robotaxis around, the Teslas will be popular.

Generation and storage

The one part of the plan that I have trouble with is the idea of combining solar generation with battery storage. Today, that’s just a bad idea. Solar panels generate electricity whenever the sun shines on them. If you connect them to batteries, they will charge the batteries, but soon the batteries will get more and more full, and be unable to take the power from the panels. As they get closer to full, you will be simply throwing away all the nice power the panels are generating. And people like their batteries to be full. When you do the economics of a solar panel, it depends on using all the power it makes. If you throw away half the power from your panels, you can quickly turn them from economical and green to silly.

That’s why you need to tie solar panels to the grid. There, every watt they generate is used, and it’s used offsetting the dirty power generation of the grid.

Of course, you can have solar panels and batteries connected to the grid. You can use the batteries to store power when it’s cheap, and use it when it’s more expensive. Unfortunately, solar panels’ best attribute is they generate their power in the day, when power is expensive. The most expensive time is mid-to-late afternoon and in an ideal world you want to sell all your power to the hungry grid, not store it. (It is not an ideal world, however, and not all utilities will buy the power from you at the real market price.) Before 1pm could be a good time to store some solar power in local batteries. (Most solar panels get aimed south and peak at noon, but smarter panels aim a bit west and peak later in the day if they are selling power back to the grid.)

Tesla’s Powerwall does aim at people who have bad utility contracts. For them, it does make sense to store any surplus from a solar panel system during the day. But that’s not a sustainable plan. In a real grid, that would not make sense. What could make sense is buying power at night when it’s cheap, and using it by day. In a proper market, you should be able to sell your solar surplus for far more than it costs to refill at night. As such, solar generation and storage are really two different things, and Elon Musk’s declaration that they have to be connected does not make sense to me — except where the market is broken.

In time, solar generation is going to go up — not just at homes, but utilities. When that happens, we could get a surplus of solar mid-day, and a shortage as the sun goes down. In that case, batteries at the grid or house level can make sense.

I should note it is possible to be green with non-grid connected solar if you can be sure you will use every watt-hour your panels generate in your home and car. This means you must underprovision, and then use the grid for the rest. ie. you are still on the grid, but never feeding it from your renewables or batteries.

The cell phone ride hail apps like Uber and Lyft are now reporting great success with actual ride-sharing, under the names UberPool, LyftLines and Lyft Carpool. In addition, a whole new raft of apps to enable semi-planned and planned carpooling are out making changes.

The success of carpool apps

The most remarkable number I have seen has Uber stating that 50% of rides in San Francisco are now UberPool. With UberPool, the system tries to find people with overlapping ride segments and quotes you a flat price for your ride. When you get it, there may already be somebody there, or your car may travel a small bit out of your way to pick up or drop somebody off. It’s particularly good for airports, but is also working in cities. The prices are often extremely good. During a surge, it might be a more affordable alternative.

It’s often been observed that as you watch any road, you see a huge volume of empty seats go down it. Even partial filling all those empty seats would make our roads vastly more efficient and higher capacity, as well as greener. Indeed, the entire volume of most transit systems could probably be easily absorbed, and a great deal more, if those empty seats were filled.

The strongest approach to date has been the hope that carpool lanes would encourage people to carpool. Sadly, this doesn’t happen very much. Estimates suggest that only 10% of the cars in the carpool lane are “induced” carpools — the rest are people like couples who already would have gone together. As such, many carpool lanes actually increase congestion rather than reducing it, because they create few induced carpools and take away road capacity. That’s why many cities are switching to HOT lanes where solo drivers can pay to get access to excess carpool lane capacity, or allowing electric/PHEV vehicles into the carpool lane.

Most carpool apps today have a focus on people who are employees of the same company. Companies have tools to organize carpools for ages, and this works modestly well, but typically the carpools are semi-permanent — the same group rides in together each day, sometimes trading off who drives. The companies provide incentives like cash and special parking.

The new generation of carpool apps (outside Uber) tend to focus on people at the same company, and as such they mostly work with big companies. There they can add the magic of dynamic carpooling, which means allowing people to be flexible about when they come and go, and matching them up with different cars of other employees. This makes sense as an early business for many reasons:

• People inherently trust their co-workers;
• Co-workers naturally share the same workplace, so you only have to find one who live within a reasonable distance;
• Companies will subsidize the carpooling for many reasons, including saving them parking.

The subsidies can often include a very important one, the guaranteed ride back. Some of these apps say that when you want to leave, if they can’t find a carpool going near your house, they will provide alternate transportation, such as transit tickets or a Taxi/Uber style ride. This gives people the confidence to carpool in with one dynamically assigned group, knowing they will never be stuck at the office with no way home. Independent carpool services can also offer such a guarantee by adding a cost to every ride, but it’s easier for a company to do it. In fact, companies will often pay for the cost of the apps that do this, so that all the employees see is the car operating cost being shared among the poolers.

Multi-leg carpools

What has not happened much today is the potential of the multi-leg carpool, where you ride in one car for part of the trip, and another car (or another mode) for another part. Of course changing cars or modes is annoying compared to door-to-door transportation, though it’s the norm for transit riders.

Today, must carpool apps will have the driver go slightly off their route — often off the highway — to pick up a rider or return one home. (Normally the morning destination is a commercial building, usually the same building.)

A multi-leg service has some similarities to the concepts of multi-leg robocar transit I outlined previously. In one vision, the actual carpool sticks to highways and arterial roads, and never deviates from the expected route of the driver or any of the poolers. Poolers get to the carpool by using some other means — including a private Uber style ride — and then join it for the highway portion. If they are not going to the same place as other poolers, they can also use such a ride at the other end, though having two transfers reduces the appeal a fair bit.

This “last mile” leg can be something like Uber, or transit, or a bicycle (including one-way bicycle systems) or a “kiss and ride” drop-off by a spouse, or even another carpool. The difference is to make it dynamic, with live tracking of all parties involved, to reduce waits at the transfer points to very short times. (With robocars and vans, the waits will be measured in seconds, but human drivers won’t be that reliable.)

In spite of the inconvenience of having to do a transfer, if the wait is short, it’s better than the downsides of the driver or other poolers having to go far off the highway to handle a fellow pooler, and there can even be financial incentives to make things smooth.

Transfer points on arterials

The main barrier in the way of a truly frictionless transfer is the absence of good and easy places to do the transfer in many locations. This might be something that highway planners should consider in building or modifying future roads. The benefits can happen today, well before robocars, so it can get on the radar of the planners today. When the robocar transit arrives, tremendous benefits are possible.

Today, there is something a bit like this. In many cities, there are bus lines that run on highways. In some cases, bus stops have been built embedded in the highway, allowing the bus to stop without fully leaving the highway. A common example can be found on intersections which have a private on-ramp/off-ramp lane which stops mergers from interfering with primary traffic. Sometimes these are just off to the side of the regular highway, but in all cases the bus pulls off the highway and then into the bus stop. Riders have some safe path from the non-highway world, including bus stops on regular streets and arterials.

In the fast-transfer world, you want something like this, though you don’t necessarily need a path to other roads. A rider brought in an Uber can be dropped off there, and in interchanges with a private collector lane, the car that drops the rider off can easily get back onto the regular road in the opposite direction.

In the map is an intersection that already has all the ingredients needed for carpool transfer points — collector lanes, long ramps and lots of spare space. Most intersections are not as adaptable as this one, but new and reconstructed intersections can be adapted in much less space. In addition, transfer points may be possible in the center median, if there is room, under bridges, through the installation of a staircase from the bridge. (If there is no elevator, the disabled can be brought to the transfer point through a longer route that goes on the highway.) This is a common layout for transit lines which run down the median.

Full cloverleaf is better for the placement of transfer points, though there are other places they can go in other intersection designs. (It’s become popular of late to replace full cloverleaf intersections with the parclo design that comes from my hometown of Toronto. This change is mostly done to avoid the complex merge and tight turns of a full cloverleaf, though robocars can handle the full clover just fine. You can easily put some transfer points in a parclo, you just have an extra minute or two spent by the stopping carpool.

Transfer points are dirt cheap infrastructure, pretty much identical to bus stops, though ideally they would use angled parking so vehicles can come and go without blocking others. You do want space for a van or even a bus to come when you have found a super-carpool synergy, as will probably be the case at the peak of rush hour. Of course, if the volume of poolers grows very high, it justifies making larger transfer points and more of them. For super peak times, it’s OK to use transfer points that are just off the highways (where parking lots to do this are plentiful) because with high volume, pools are making just one stop to pick up passengers and can handle a small detour.

Transfer with parking

Of course, today the easiest way to do these carpools is with “carpool lots” not too far from the highway — places with spare parking which allow carpool riders to drive to the lot to meet their carpool driver. Indeed, carpoolers should be those who own cars because the first goal is to take a car off the road that otherwise have driven, and the second goal is to fill the empty seat with somebody who would otherwise have been on transit.

It can be difficult to get lots of parking convenient to the highway. One carpool lot I use has room for only about 50 cars. Nice that it’s there, but it takes no more than 50 cars off the road. At scale, one could imagine it being worthwhile to have shuttles from parking lots to on-highway transfer points, though nobody likes having to do 3 or 4 legs for a trip unless it’s zero wait time. If Robocars were not coming, one could imagine designing future highways with transfer points connected to parking lots. The people of the past did not imagine robocars or cell phone coordination of carpooling.

Tesla car. Source: Tesla

Brad Templeton describes Tesla’s Autopilot as a ‘distant cousin of a real robocar’ that primarily uses a MobilEye EyeQ3 camera combined with radars and ultrasonic sensors. Unlike robocar sensors, Tesla doesn’t have a lidar or use a map to help it understand the road and environment.

It’s not surprising there is much debate about the fatal Tesla Autopilot crash revealed to us last week. The big surprise to me is that Tesla and MobilEye stock seem entirely unaffected. For many years, one of the most common refrains I would hear in discussions about robocars was, “This is all great, but the first fatality and it’s all over.” I never believed it would all be over, but I didn’t think there would barely be a blip.

There has been lots of blips in the press and online, of course, but most have had some pretty wrong assumptions. Tesla’s Autopilot is a distant cousin of a real robocar, and that would explain why the fatality is no big deal for the field, but the press shows that people don’t know that.

Tesla’s Autopilot is really a fancy cruise control. It combines several key features from the ADAS (Advance Driver Assist) world, such as adaptive cruise control, lane-keeping and forward collision avoidance, among others. All these features have been in cars for years, and they are also combined in similar products in other cars, both commercial offerings and demonstrated prototypes. In fact, Honda promoted such a function over 10 years ago!

Tesla’s Autopilot primarily uses the MobilEye EyeQ3 camera, combined with radars and some ultrasonic sensors. It doesn’t have a lidar (the gold standard in robocar sensors) and it doesn’t use a map to help it understand the road and environment.

Most importantly, it is far from complete. There is tons of stuff it’s not able to handle. Some of those things it can’t do are known, some are unknown. Because of this, it is designed to only work under constant supervision by a driver. Tesla drivers get this explained in detail in their manual and when they turn on the Autopilot.

ADAS cars are declared not to be self-driving cars in many state laws

This is nothing new — lots of cars have lots of features to help drive (including the components used like cruise controls, each available on their own) which are not good enough to drive the car, and only are supposed to augment an alert driver, not replace one. Because car companies have been selling things like this for years, when the first robocar laws were drafted, they made sure there was a carve-out in the laws so that their systems would not be subject to the robocar regulations companies like Google wanted.

The Florida law, similar to other laws, says:

“The term [Autonomous Vehicle] excludes a motor vehicle enabled with active safety systems or driver assistance systems, including, without limitation, a system to provide electronic blind spot assistance, crash avoidance, emergency braking, parking assistance, adaptive cruise control, lane keep assistance, lane departure warning, or traffic jam and queuing assistant, unless any such system alone or in combination with other systems
enables the vehicle on which the technology is installed to drive without the active control or monitoring by a human operator.”

The Tesla’s failure to see the truck was not surprising

There’s been a lot of writing (and I did some of it) about the particulars of the failure of Tesla’s technology, and what might be done to fix it. That’s an interesting topic, but it misses a very key point. Tesla’s system did not fail. It operated within its design parameters, and according to the way Tesla describes it, in its manuals and warnings. The Tesla system, not being a robocar system, has tons of stuff it does not properly detect. A truck crossing the road is just one of those things. It’s also poor on stopped vehicles and many other situations.

Tesla could (and in time, will) fix the system’s problem with cross traffic. (MobilEye itself has that planned for its EyeQ4 chip coming out in 2018, and freely admits that the EyeQ3 Tesla uses does not detect cross traffic well.) But fixing that problem would not change what the system is, and not change the need for constant monitoring that Tesla has always declared it to have.

People must understand that the state of the art in camera systems is not today anywhere near the level needed for a robocar. That’s why most advanced robocar research projects all use LIDAR. There are those (including Tesla and MobilEye) who hope that the day might come soon when a camera can do it, but that day is not yet here. As such, any camera based car is going to make mistakes like these. Fix this one and there will be another. While the Tesla Autopilot failed to see the truck, that was an error, a tragic one, but not a failure of the Autopilot. It was an expected limitation, one of many. The system performed as specified, and as described in the Tesla user manual and the warnings to drivers. I will say this again, the Tesla Autopilot did not fail, it made an error expected under its design parameters.

The problem of getting too good — and punishing that

No, the big issue here is not what the Tesla Autopilot can’t handle, but the opposite. The issue is that it’s gotten good enough that people are mistaking it for a self-driving car, and they are taking their eyes off the road. Many stories abound of people doing e-mail, and there is an allegation that Brown, the deceased driver of this Tesla, might have been watching a movie. Movie or not Brown was not paying attention, as he could have easily seen and braked in time to avoid hitting that truck.

Tesla warns explicitly not to do that. Brown was a fairly skilled guy and he also should have known not to do that (if he did.) But as Tesla has gotten better, there is no question that people are not heeding the warning and getting a bit reckless. And Tesla knows this, of course.

This brings up all sorts of issues. Does it matter that Tesla knows that people are ignoring the warnings so long as it gives them sternly? Is there a duty of care to warn people that there is a danger you will ignore the warnings? Is there a duty of care to make sure (with technology) that they don’t ignore them?

Perhaps you see the problem here — the better the system gets, the more likely it is that it will make people complacent. People are stupid. They get away with something a few times and they imagine they will always get away with it. Older supervised products like cruise control needed your attention fairly often; there was no way you could watch a movie when using them. Cruise control needs your intervention to steer a few times a minute. Early autopilot-like systems need your intervention every few minutes. But Tesla got good enough that on open highways, it might not need your intervention for hours, or the entire highway part of a trip. Eventually it will get to not needing it for days, or even a month. Who would not be tempted to look away for a little while, and then a lot, if it gets there — or gets to needing only one intervention a year?

Our minds are bad at statistics. We, human drivers, have a tiny accident perhaps once every 10 years on average and one that is reported to insurance about every 25 years. We have a fatal accident about every 8,000 years of average driving, closer to 20,000 years on the highway. A rate of just one serious intervention every year sounds amazingly trustworthy. It seems like you took a bigger risk just getting on the road. But in fact, it’s very poor performance. Most people agree to be a true robocar, we’re going to need a car that makes an accident-causing mistake less often as a human, perhaps even twice as good or more. And the Tesla isn’t even remotely close. Even Google, which is much, much closer, isn’t there yet.

The incremental method

But we want systems to get better. It seems wrong to have to say that the better a system gets, the more dangerous it is. That a company should face extra liability for making the system better than the others. That’s not the direction we want to go. It’s definitely not the way that all the car companies want to go. They want to build their self-driving car in an evolutionary incremental way. They want to put supervised autopilots out on the road, and keep improving them until one day they wake up and say, “Hey, our system hasn’t needed an intervention on this class of roads for a million miles!” That day they can consider making it a real robocar for those roads. That’s different from companies like Google, Uber, Apple, Zoox and other non-car companies who want to aim directly for the final target, and are doing so with different development and testing methods.

Other views on the complacency issue

It should be noted that most other automakers who have made similar products have been much more keen on using tools to stop drivers from getting complacent and not being diligent in supervising. Some make you touch the wheel every few seconds. Some have experimented with cameras to watch the driver’s eyes. GM has announced a “super cruise” product for higher end Cadillacs for several years, but every year pulled back on shipping it, not feeling they have sufficient “countermeasures” to stop accidents like the Tesla one.

Google famously did testing with their own employees of a car that was quite a bit superior to the Tesla back in 2012-2013. Even though Google was working on a car that would not need supervision on certain routes, they required their employees (regular employees, not those on the car team) to nonetheless pay attention. They found that in spite of their warnings, about a week into commuting, some of these employees would do things that made the car team uncomfortable. This is what led Google to decide to make a prototype with no steering wheel, setting the bar higher for the team, requiring the car to handle every situation and not depend on an unreliable human.

Putting people at risk

Tesla drivers are ignoring warnings and getting complacent, and putting themselves and others at risk. But cars are full of features that cause this. The Tesla, and many other sports cars, can accelerate like crazy. It can drive 150mph. And every car maker knows that people take their muscle cars and speed with them, sometimes excessively and recklessly. We don’t blame the car makers for making cars capable of this, or for knowing this will happen. Car makers put in radios, knowing folks will get distracted fiddling with them. Car makers know many of their customers will drive their cars drunk, and kill lots of people. Generally, the law and society does not want to blame car makers for not having a breath alcohol test. People drive sleepy and half-blind from cataracts, and we always blame the driver for being reckless.

There is some difference between enabling a driver to take risks, and encouraging complacency. Is that enough of a difference to change how we think about this? If we change how we think, how much will we slow down the development of technology that in the long term will prevent many accidents and save many lives?

In an earlier post I have suggested Tesla might use more “countermeasures” to assure drivers are being diligent. Other automakers have deployed or experimented with such systems, or even held back on releasing their similar products because they want more countermeasures. At the same time, many of the countermeasures are annoying and people worry they might discourage use of the autopilots. Indeed, my proposal, which states that if you fail to pay attention, your autopilot is disabled for the rest of the day, and eventually permanently, would frighten customers. They love their autopilot and would value it less if they had to worry about losing it. But such people are also the sort of people who are making the mistake of thinking a car that needs intervention once a year is essentially the same as a robocar.

The unwritten rules of the road

I have often mused on the fact that real driving involves breaking the rules of the road all the time, and this accident might be such a situation. While the final details are not in, it seems this intersection might not be a very safe one. It is possible, by the math of some writers, that the truck began its turn unable to see the Tesla, because there is a small crest in the road 1200’ out. In addition, some allege the Tesla might have been speeding as fast as 90mph — which is not hard to believe since the driver had many speeding tickets.

Normally, if you make an unprotected left turn, your duty is to yield right-of-way to oncoming traffic. Normally, the truck driver would be at fault. But perhaps the road was clear when he started turning, and the Tesla only appeared once he was in the turn? Perhaps the super wide median offers an opportunity the truck driver didn’t take, to pause after starting the turn to check again, and the truck driver remains at fault for not double checking with such a big rig.

Ordinarily, if a truck did turn and not yield to oncoming traffic, no accident would happen. That’s because the oncoming vehicle would see the truck, and the road has a lot of space for that vehicle to brake. Sure, the truck is in the wrong, but the driver of an oncoming vehicle facing a truck would have to be insane to proudly assert their right-of-way and plow into the truck. No sane and sober human driver would ever do that. As such, the intersection is actually safe, even without sufficient sightlines for a slow truck and fast car.

Because of that, real world driving involves stealing right-of-way all the time. People cut people off, nudge into lanes and get away with it because rational drivers yield the ROW that belongs to them. Sometimes this is even necessary to make the roads work.

With the Tesla driver inattentive, the worst happened. The Tesla’s sensors were not good enough to detect the truck and brake for it (and the human wasn’t insane but wasn’t looking at the road) so he didn’t do the job of compensating for the autopilot’s inadequacy. The result was death.

Measuring the safety of the Autopilot

Tesla touted a misleading number, saying they had gone 130 million miles in Autopilot before this first fatality. The reality is that on limited access freeways, the USA number for human drivers is one fatality per 180 million miles, not the 90 million they cited. The Autopilot is used mostly on freeways where it functions best.

Tesla will perhaps eventually publish numbers to help judge the performance of the Autopilot. Companies participating in the California self-driving car registration program are required to publish their numbers. Tesla participates but publishes “zero” all the time because the Autopilot does not qualify as a self-driving car.

Here are numbers we might like to see, perhaps broken down by classes or road, weather conditions, lighting conditions and more:

• Number of “safety” disengagements per mile, where a driver had to take the controls for a safety reason (as opposed to just coming to a traffic light or turn.)
• Number of safety disengagements which, after further analysis in simulator, would have resulted in an accident if the driver had not intervened.
• Number of “late” safety disengagements, or other indications of driver inattention.
• Number of disengagements triggered by the system rather than the driver.

The problem is Tesla doesn’t really have a source for many of these numbers. Professional safety drivers always log why they disengage, and detailed car logs allow the building of more data. Tesla would need to record more data to report this.

Tesla with LIDAR

Folks in Silicon Valley have spotted what appear to be official Tesla test cars with LIDARs on them (one research LIDAR and one looking more like a production unit.) This would be a positive sign for them on the path to a full robocar. Tesla’s problem is that until about 2018, there is not a cost-effective LIDAR they can ship in a car, and so their method of gathering data by observing customers drive doesn’t work. They could equip a subset of special cars with LIDARs and offer them to a special exclusive set of customers — Tesla drivers are such rabid fans they might even pay for the privilege of getting the special test car and trying it out. They would still need to supervise, though, and that means finding a way around the complacency issue.

If you enjoyed this article, you may also enjoy:

• What to expect from autonomous cars
• Should Tesla disable your Autopilot if you’re not diligent? Plus, a survey of robocar validation
• Man dies while driven by Tesla Autopilot
• Robocar news around the globe: Tesla crash, Declaration of Amsterdam, and automaker services

At Starship, we announced our first pilot projects for robotic delivery which will begin operating this summer. We’ll be working with a London food delivery startup Pronto as well as German parcel company Hermes and the Metro Group of retailers, plus Just Eat restaurant food delivery to trial on-your-schedule delivery of packages, groceries and meals to people’s homes.

(It’s a nice break from Tesla news — and besides, our little robots weigh so little and move so slowly that even if something went horribly wrong and they hit you, injury is quite unlikely.)

Hermes, which does traditional package delivery is very interested in what I think is one of the core values of robot delivery — namely delivery on the recipient’s schedule. Today, delivery is done on the schedule of delivery trucks, and you may or may not be home when it arrives. With a personal delivery robot, it will only come when you’re home, reducing the risk of theft and lost packages. Robots don’t mind waiting for you.

The last mile is a huge part of the logistics world. Starship robots will get you packages with less cost, energy, time, traffic, congestion and emissions than you going to the store to get it yourself. They use a combination of autonomous driving with human control centers able to remotely fix any problems the robots can’t figure out. Robots don’t mind pausing if they have a problem and our robots can stop in under 30cm. As we progress, operation will reach near full autonomy and super low cost.

((More details at the Starship press release. I am on the advisory team at Starship.))

If you enjoyed this article, you may also enjoy reading:

• Robohub Digest 11/15: Service and consumer robots catching on
• Going the last mile with robots
• Calculating why we should make a lot more robocars
• Delivery robots from Starship Technologies
• Self-driving trucks are coming — what will that mean?

Tesla Model S autopilot-software. Source: Tesla

In this article, Brad Templeton provides a rundown of different approaches for validation of self-driving and driver assist systems, a recommendation to Tesla and others to have countermeasures to detect drivers not watching the road, and permanently disable their Autopilot if they show a pattern of inattention.

The recent fatality for a man driven by the Tesla “autopilot” system has ignited debate on whether it was appropriate for Tesla to allow their system to be used as it was.

Tesla’s autopilot is a driver assist system, and Tesla tells customers it must always be supervised by an alert driver ready to take the controls at any time. The autopilot is not a working self-driving car system, and it’s not rated for all sorts of driving conditions, and there are huge numbers of situations that it is not designed to handle and can’t handle. Tesla knows that. But the public, press, and Tesla customers forget that and there are many Tesla users who are treating the autopilot like a real self-driving car system, and who are not paying attention to the road — and Tesla is aware of that as well. The press made this mistake as well, regularly writing fanciful stories about how Tesla was ahead of Google and other teams.

Brown, the driver killed in the crash, was very likely one of those people, and if so, he paid for it with his life. In spite of all the warnings Tesla may give about the system, some users
do get a sense of false security. There is an ongoing debate if that means driver assist systems are a bad idea.

There have been partial self-driving systems that require supervision since the arrival of the cruise control. Adaptive cruise control is even better, and other car companies have released autopilot like systems which combine adaptive cruise control with lane-keeping and forward collision avoidance, which hits the brakes if you’re about to rear-end another car. Mercedes has sold a “traffic jam assist” like the Telsa autopilot since 2014 that only runs at low speeds in the USA. You can even go back to a Honda demo in 2005 of an autopilot like system.

With cruise control, you might relax a bit but you know you have to pay attention. You’re steering and for a long time even adaptive cruise controls did not slow down for stopped cars. The problem with Tesla’s autopilot is that it was more comprehensive and better performing than earlier systems, and even though it had tons of things it could not handle, people have started to trust it with their lives.

Tesla’s plan can be viewed in several ways. One view is that Tesla was using customers as
“beta testers,” as guinea pigs for a primitive self-drive system which is not production ready — and that this is too much of a risk. Another is that Tesla built (and tested) a superior driver assist system with known and warned limitations and customers should have listened to those warnings.

Neither is quite right. While Tesla has been clear about the latter stance, with the knowledge that people will over-trust it, we must face the fact that it is not only the daring drivers who are putting themselves at risk, it’s also others on the road who are put at risk by the over-trusting drivers — or perhaps by Tesla. What if the errant car had not gone under a truck, but instead hit another car, or even plowed into a pedestrian when it careened off the road after the crash?

At the same time, Tesla’s early deployment approach is a powerful tool for the development and quality assurance of self-drive systems. I have written before about how testing is the big unsolved problem in self-driving cars. Companies like Google have spent millions with a staff of paid drivers testing their cars for 1.6 million miles. This is massively expensive and time consuming, and even Google’s money can’t easily generate the billions of miles of testing that some feel might be needed. Human drivers will have about 12 fatalities in a billion miles, and we want our self-driving cars to do much better. Just how we’ll get enough verification and testing done to bring this technology to the world is not a solved problem.

While we don’t want a repeat of the things that led to the death of this driver, there may be greater value to the field by having fleets of volunteers help verify these vehicles and if it can be done safely, the world probably needs it. But even if there is no way to do volunteer testing of systems that are still being refined — and they will still be in the being refined state for many years to come — there is still the question of whether you can have systems that need human supervision and intervention that get so good that humans ignore warnings and stop bothering to supervise them.

Here’s a look at a variety of approaches to testing, verifying, refining and learning:

In Simulator

The only way many billions of miles will be done is in a simulator, especially since you want to re-test all new software builds as much as you can before deploying. In a simulator you can test all sorts of risky situations you could never readily test in the real world. That’s why many years ago I laid out a plan for an open source simulator because I think this would be a great thing for everybody to contribute to. An open source simulator would soon feature every crazy situation anybody has even seen or thought of, and it would make all cars safer.

But simulation is just simulation, and it only features what people have thought of. You need more.

On test tracks

The most conservative, but most expensive and limited approach, is a private test track. And almost everybody starts on a closed testing ground because you need to when things are just getting going. They provide the confidence to eventually go out on the road — and go out on the road you must. You can really only do a fraction of a percent of your testing on test tracks, even with robot slave cars to drive along with. You just won’t encounter enough variation in terrain and driving situations, and you can only test a few cars at a time.

On the plus side, on test tracks you can use unmanned cars, reducing some costs, and you can run 24 hours/day.

There is a hybrid of simulator and test track called “Vehicle in the Loop” which puts the car on a dyno in a building while robots whiz around to provide relative motion. That requires less real estate and is safer, but automated test track operation is probably better at this point.

On the road with trained professional safety drivers

All companies eventually need to go out on real roads and deal with real situations. They do it by having paid professionals, usually with special training, who sit behind the wheel watching the vehicle and ready to take the controls at any sign of doubt. It is also common to have a second person who is monitoring the computer systems and sensor scans to make sure all is operating properly, and to tell the safety driver to take over if something is wrong. In fact, safety drivers are required for testing self-driving cars (but not autopilots/cruise software) by most of the state laws about self-driving cars.

Companies like Google tell their safety drivers to take control at any sign of doubt. They don’t want them to let the car keep driving to see if it would have braked properly for the children crossing the road. Later, they can use the simulator to replay what the car would have done if the safety driver had not intervened. If the car would have done something wrong, it’s a bug to be fixed at high priority.

Still, even trained safety drivers can make mistakes. Google has reported one accident where their systems made a mistake. The system decided that a bus it was sharing the lane with would let it in to traffic, but the bus did not. The safety driver didn’t intervene because he also judged the bus would let the car in.

In addition, when the systems are very good, safety drivers can also become complacent and not intervene as quickly as they should.

On the road with regular drivers but limited function

You could test a self-driving system by having it do only some of the driving. For example, you could make a system which steers but does not control the brakes. A driver who knows that she has to hit the brakes when approaching something is not going to stop paying attention. With this system you can note when the human hit the brakes, and see if the system also would have, and who was right. Chances are any surprise braking would be combined with taking the wheel. Of course, you would also learn when the driver grabbed the wheel.

Tesla Motors autopilot (photo:Tesla)

A Tesla blog post describes the first fatality involving a self drive system. A Tesla was driving on autopilot down a divided highway. For unknown reasons, a truck crossed the highway (something may have been wrong for that to happen.) A white truck body against a bright sky is not something the camera system in the Tesla perceives well, and a truck crossing perpendicular to you on the highway is also an unusual situation.

The truck was also high, so when the Tesla did not stop, it went “under” it, so that the windshield was the first part of the Tesla to hit the truck body, with fatal consequences for the “driver.” Tesla points out that the autopilot system has driven 130 million miles, while human drivers in the USA have a fatality about every 94 million miles (though it’s a longer interval on the highway.) The Tesla is a “supervised” system where the driver is required to agree they are monitoring the system and will take control in the event of any problem, but this driver, a major Tesla fan named Joshua Brown, did not hit the brakes. As such, the fault for this accident will presumably reside with Brown, or whatever caused the truck to be crossing a highway lane in this fashion. (Tesla also notes that had the front of the car hit the truck, the crumple zones and other safety systems would probably have saved the driver — hitting a high target is the worst case situation.)

Our condolences for the tragic loss https://t.co/zI2100zEGL

— Elon Musk (@elonmusk) June 30, 2016

It is worth noting that Brown was a major Tesla fan, and in fact, he is the person in a video that circulated in April claiming that the Autopilot saved him from a collision with a smaller truck that cut into a lane.

Any commentary here is preliminary until more facts are established, but here are my initial impressions:

• There has been much speculation of whether Tesla was taking too much risk by releasing autopilot so early, and this will be boosted after this.
• In particular, a core issue is that the autopilot works too well, and I have seen reports from many Tesla drivers of them trusting it far more than they should. The autopilot is fine if used as Tesla directs, but the better it gets, the more it encourages people to over-trust it.
• Both Tesla stock and MobilEye stock were up today, with a bit of downturn after-hours. The market may not have absorbed this. The MobilEye is the vision sensor used by the Tesla to power the autopilot, and the failure to detect the truck in this situation is a not-unexpected result for the sensor.
• For years, I have frequently heard it said that “the first fatality with this technology will end it all, or set the industry back many years.” My estimation is that this will not happen.
• One report suggests the truck was making a left turn, which is a more expected situation, though if a truck turned with oncoming traffic it would be at fault.

Camera vs. Lidar, and maps.

I have often written about the big question of cameras vs. LIDAR. Elon Musk is famously on record as being against LIDAR, when almost all robocar projects in the world rely on LIDAR. Current LIDARs are too expensive for production automobiles, but many companies, including Quanergy (where I am an advisor) are promising very low cost LIDARs for future generations of vehicles.

Here there is a clear situation where LIDAR would have detected the truck. A white truck against the sky would be no issue at all for the LIDAR, it would see it very well. In fact, a big white target like that would be detected beyond the normal range of a typical LIDAR. That range is an issue here — most LIDARs would only detect other cars about 100m out, but a big white truck would be detected a fair bit further. Either way, that’s not quite far enough to stop in time for an obstacle like this at highway speeds, however, the car would brake to make the impact vastly less, and a clever car might even have had time to swerve or deliberately turn to hit the wheels of the truck rather than slide underneath the body.

Another sensor that is problematic here is radar. Radar would have seen this truck no problem, but since it was perpendicular to the travel of the car, it would not be moving away from or towards the car, and thus have the doppler speed signature of a stopped object. Radar is great because it tracks the speed of obstacles, but because there are so many stationary objects, most radars have to just disregard such signals — they can’t tell a stalled vehicle from a sign, bridge or berm. To help with that, a map of where all the radar reflectors are located can help. If you get a sudden bright radar return from a truck or car somewhere that the map says a big object is not present, that’s an immediate sign of trouble. (At the same time, it means that you don’t easily detect a stalled vehicle next to a bridge or sign.)

One solution to this is longer range LIDAR or higher resolution radar. Google has said it has developed longer range LIDAR. It is likely in this case that even regular range LIDAR, or radar and a good map, might have noticed the truck.

Mercedes Benz autonomous concept car at the IAA 2015. Photo: VanderWolf Images/Bigstockphoto

What does that car of the future look like? There is no one answer; in this world, the car that is sent to pick you up can be tailored for your trip. The more people traveling, the bigger the car. If your trip does not involve a highway, it may not be a car capable for the highway.

With Mobility on Demand, you don’t buy a car, you buy rides. That’s certainly Uber’s plan, and is a plan that makes sense for Google, Apple and other no-car companies. But even Daimler, with Car2Go/Car2Come, BMW with DriveNow and GM with Lyft plan to sell you a ride rather than a car because it’s the more lucrative thing to do.

So, what does that car of the future look like? There is no one answer; in this world, the car that is sent to pick you up is tailored for your trip. The more people traveling, the bigger the car. If your trip does not involve a highway, it may not be a car capable for the highway. If your trip is up to a mountain cabin, it’s more like an SUV, but maybe not using an SUV to get a bottle of milk, like we do today. If it’s for a cruise to the beach on a sunny day, the roof may have been removed at the depot. If it’s for an overnight trip to a country home, it might be just beds!

I’ve outlined many of these changes in this article on design changes in cars but today I will focus on the incredibly cheap and simple design of what should become the most common vehicle made, namely the car designed for a short urban trip by one person. That’s 80% of trips and around 45% of miles, so this should be a large fraction of the fleet. I predict a lot of these cars will be made every year — more than all the cars made today, even though they are used as taxis and shared among many passengers.

What does it look like?

Small

Toyota i-Road concept car. Source: Wikipedia Commons

A car for 1-2 people will be small. It will probably be around 1.5m wide, narrow enough that you can fit two in a lane, and have it park efficiently when it needs to wait. If it’s for just one person, it won’t be very long either. For two people, there will be a “face to face” configuration which is longer and a “tandem” configuration, which is a bit shorter. The 2 person vehicles aren’t a lot bigger or heavier than the one person, so they might be the most common cars since you can serve a solo rider fairly efficiently with one, even if not perfectly efficient.

A car that is so narrow cannot corner quickly. A wide stance is much more stable. There are a few solutions to that, including combinations of these:

• The wheels bank independently, allowing the vehicle to lean like a motorcycle when in corners. This is the best solution, but it costs some money.
• Alternately, it’s a two-wheeler, which is also able to lean, but has other tricks like the LIT Motors C-1 to stay upright.
• It’s electric and has all the batteries in the floor, giving it a very low center of gravity. (One extreme example of this is the Tango, which uses lead batteries deliberately to give it that stability.)
• It never goes on fast roads, so it doesn’t need to corner very fast, and its precision robot driving assures it never corners so fast as to becomes unstable and also plans its route accordingly.

Lit C1 Gyroscopically Balanced Electric Motorcycle Source: Flickr/DeBold

Not super aerodynamic

The car already has a big win when it comes to aerodynamic drag by only being half-width. The non-highway version probably gives back a bit of that because you don’t need to worry as much about that if you are not going fast. Energy lost to drag goes up with the square of velocity. So, a 30mph car has 1/4 the drag of a 60mph car, and 1/8th the drag of a similar car of full width. The highway car needs to be shaped as close to a “teardrop” as you can, but the city car can get away with being a bit taller for more comfortable seating and entry/exit.

When I give talks on robocars the most common question asked is the one known as the “trolley problem” question. That is: “what will the car do if it has to choose between killing one person or another” or other related dilemmas. I have written frequently about how this is a low priority question in reality and is much more interesting to philosophy classes. It is a super-rare event and there are more important everyday ethical questions that self-driving car developers have to solve long before they will tackle this one.

In spite of this, the question persists in the public mind. We are fascinated and afraid of the idea of machines making life or death decisions. The tiny number of humans faced with such dilemmas don’t have a detailed ethical debate in their minds, they would go with their gut or very simple and quick reasoning. We are troubled because machines don’t have a difference between instant and carefully pondered reactions. The one time in a billion miles(*) that a machine faces such a question it would presumably make a calculated decision based on its programming. That’s foreign to our nature, and indeed, not a task desired by programmers or vendors of robocars.

There have been calls to come up with “ethical calculus” algorithms and put them in the cars. As a programmer, I could imagine coding such an algorithm, but I certainly would not want to, nor would I want to be held accountable for what it does, because, by definition, it’s going to do something bad. The programmer’s job is to make driving safer. On their own, I think most builders of robocars would try to punt the decision if they can. The simplest way to punt the decision is to program the car to follow the law, which generally means to stay in its right-of-way. Yes, that means running over people who ran into the road, as opposed to, veering onto the sidewalk to run over Hitler, or veering onto oncoming traffic to hit an unmanned car — possibly killing the veering car’s passenger. Staying in our lane is what the law says to do, it also strongly forbids going onto the sidewalk or another lane to deliberately hit something.

We might not like the law, but we do have the ability to change it.

Thus, I propose the following: Driving regulators should create a special panel which can rule on driving ethics questions. If a robocar developer sees a question which requires some sort of ethical calculation whose answer is unclear, they can submit that question to the panel. The panel can deliberate and provide an answer. If the developer conforms to the ruling, they are absolved of responsibility. They did the right thing.

Source: xkcd.com

The panel would, of course, have people with technical skill, to make sure rulings are reasonable and can be implemented. Petitioners could also appeal rulings that would impede development, though they would probably suggest answers and describe their difficulty to the panel in any petition.

The panel would not simply be presented with questions like, “How do you choose between hitting two adults, or one child?” It might make more sense to propose formulae for evaluating multiple different situations. In the end, it would need to be reduced to something you can do with code.

Reports from Tesla suggest they are gathering massive amounts of driving data from logs in their cars — 780 million miles of driving, and as much as 100 million miles in autopilot mode. This contrasts with the 1.6 million miles of test operations at Google. Huge numbers, but what do they mean now, and in the future?

As I’ve written before, testing is one of the largest remaining challenges in robocar development — how do you prove to yourself, and then to others, that you’ve reached the desired safety goals? Logging tons of miles are an important component. If car companies can get their customer to do the testing for them, that can be a significant advantage. (As I wrote last week, another group which can get others to do testing are companies like Uber and even operators of large commercial and taxi fleets.) Lots of miles mean lots of testing, lots of learning, and lots of data.

Does Tesla’s quick acquisition of so many miles mean they have lapped Google? The short answer is no, but it suggests a significant threat since Google is, for now, limited to testing with its small fleet and team of professional testing drivers.

Tesla is collecting a lot less data from its cars than Google, orders of magnitude less. First of all, Tesla has a lot fewer sensors and no LIDAR, and to the best of my knowledge from various sources I have spoken to, Tesla is only collecting a fraction of what their sensors gather. To collect all that they gather would be a huge data volume, not one you would send over the cell network, and even over the wifi at home, it would be very noticeable. Instead, reports suggest Tesla is gathering only data on incidents and road features the car did not expect or did not handle well. However, nothing stops them in the future from logging more, though they might want to get approval from owners to use all that bandwidth.

Tesla intends to make a car for people to buy today. As such, it has no LIDAR, because a car today, and even the autopilot, can be done without LIDAR. Tomorrow’s LIDARs will be cheap but today’s production LIDARs for cars are simple and/or expensive. So while the real production door-to-door self-driving car almost certainly uses LIDAR, Tesla is unable and unwilling to test and develop with it. (Of course, they can also argue that in a few years, neural networks will be good enough to eliminate the need for LIDAR. That’s not impossible, but it’s a risky bet. The first cars must be built in a safety-obsessed way, and you’re not going to release the car less safe than you could have made it just to save what will be only a few hundred dollars of cost.)

As noted, Google has been doing their driving with professional safety drivers, who are also recording a lot of data from the human perspective that ordinary drivers never will. That isn’t 100 times better, but it’s pretty significant.

Tesla is also taking a risk, and this has shown up in a few crashes. Their customers are beta testing a product that’s not yet entirely safe. In fact, it was a pretty bold move to do this, and it’s less likely that the big car companies would have turned their customers into beta testers — at least no until forced by Tesla. If they do, then the big automakers have even more customers than Tesla, and they can rack up even more miles of testing and data gathering.

When it comes to training neural networks, ordinary drivers can provide a lot of useful data. That’s why Commma.ai, who I wrote about earlier, is even asking volunteers to put a smartphone on their dash facing out to get them more training data. At present, this app does not do much, but it will not be hard to make one that offers things like forward collision warning and lane departure warning for free, paid for by the data it gathers.

Watch me on Dateline NBC: On Assignment

On Sunday, June 5, at 7 pm (Eastern and Pacific) the news show Dateline: NBC conducted a segment on self-driving cars featuring Sebastian Thrun, Jay Leno and myself. I sat down for several hours with Harry Smith, but who knows how much actual airtime will be shown. Here is the promo for the episode and another more specific one.

An image of some connected autonomous cars

Can our emotional fear of machines, and the call for premature regulation, be mollified by a temporary increase in liability which takes the place of specific regulations to keep people safe?

So far, most new automotive technologies, especially ones that control driving such as autopilot, forward collision avoidance, lane keeping, anti-lock brakes, stability control and adaptive cruise control, have not been covered by specific regulations. They were developed and released by vendors, sold for years or decades, and when (and if) they got specific regulations, those often took the form of ‘electronic stability control is so useful, we will now require all cars to have it.’ It has worked reasonably well.

Just because there are no specific regulations for these things does not mean they are unregulated. There are rafts of general safety regulations on cars, and the biggest deterrent to the deployment of unsafe technology is the liability system and the huge cost of recalls. As a result, while there are exceptions, most carmakers are safety paranoid to a rather high degree — just because of liability. At the same time, they are free to experiment and develop new technologies. Specific regulations tend to come into play when it becomes clear that automakers are doing something dangerous, and that they won’t stop doing it because of the liability. In part, this is because today it’s easy to assign blame for accidents to drivers, and often harder to assign it to a manufacturing defect, or to a deliberate design decision.

The exceptions, like GM’s famous ignition switch problem, arise because of the huge cost of doing a recall for a defect that will have rare effects. Companies are afraid of having to replace parts in every car they made when they know they will fail — even fatally — just one time in a million. The one person killed or injured does not feel like one in a million, and our system pushes the car maker (and thus all customers) to bear that cost.

I wrote an article on regulating Robocar Safety in 2015, and this post expands on some of those ideas.

Robocars change some of this equation. First of all, in robocar accidents, the car manufacturer (or driving system) is going to be liable by default. Nobody else really makes sense, and indeed, some companies, like Volvo, Mercedes and Google, have already accepted that. Some governments are talking about declaring it, but frankly, it could never be any other way. Making the owner or passenger liable is technically possible, however, do you want to ride in an Uber where you have to pay if it crashes for reasons having nothing to do with you?

Due to this, the fear of liability is even stronger for robocar makers.

Robocar failures will almost all be software issues. As such, once fixed, they can be deployed for free. The logistics of the “recall” will cost nothing. GM would have no reason not to send out a software update once they found a problem; they would be crazy not to. Instead, there is the difficult question of what to do between the time a problem is discovered and a fix has been declared safe to deploy. Shutting down the whole fleet is not a workable answer, it would kill deployment of robocars if several times a year all robocars stopped working.

In spite of all this history and the prospect of it getting even better, a number of people — including government regulators — think they need to start writing robocar safety regulations today, rather than 10-20 years after the cars are on the road as has been traditional. This desire is well-meaning and understandable, but it’s actually dangerous because it will significantly slow down the deployment of safety technologies that will save many lives by making the world’s 2nd most dangerous consumer product safer. Regulations and standards generally codify existing practice and conventional wisdom. They are bad ideas with emerging technologies, where developers are coming up with entirely new ways to do things and entirely new ways to be safe. The last thing you want is to tell vendors is that they must apply old-world thinking when they can come up with much better ways of thinking.

Sadly, there are groups who love old-world thinking, namely the established players. Big companies start out hating regulation but eventually come to crave it, because it mandates the way they do things and understand the law. This stops upstarts from figuring out how to do it better, and established players love that.

The fear of machines is strong, so it may be that something else needs to be done to satisfy all desires: The desire of the public to feel the government is working to keep these scary new robots from being unsafe and the need for unconstrained innovation. I don’t desire to satisfy the need to protect old ways of doing things.

One option would be to propose a temporary rule: For accidents caused by robocar systems, the liability, if the system should be at fault, shall be double that if a similar accident were caused by driver error. (Punitive damages for willful negligence would not be governed by this rule.) We know the cost of accidents caused by humans. We all pay for it with our insurance premiums, at an average rate of about 6 cents/mile. This would double that cost, pushing vendors to make their systems at least twice as safe as the average human in order to match that insurance cost.

Victims of these accidents (including hapless passengers in the vehicles) would now be doubly compensated. Sometimes no compensation is enough, but for better or worse, we have set on values and doubling them is not a bad deal. Creators of systems would have a higher bar to reach, and the public would know it.

While doubling the cost is a high price, I think most system creators would accept this as part of the risk of a bold new venture. You expect those to cost extra as they get started. You invest to make the system sustainable.

Over time, the liability multiplier would reduce, and the rule would go away entirely. I suspect that might take about a decade. The multiplier does present a barrier to entry for small players and we don’t want something like that around for too long.

WEpods: The first autonomous vehicle on Dutch public roads. Source: True Form/YouTube

We have the first report of a real Tesla autopilot crash. To be fair to Tesla, their owner warnings specify very clearly that the autopilot could crash in just this situation. In the video, there is a stalled car partly in the lane and the car in front swerves around, revealing little time for the driver, or the autopilot, to react.

The deeper issue is the way that the improving quality of the Tesla Autopilot and systems like it are lulling drivers into a false sense of security. I have heard reports of people who now are trusting the Tesla system enough to work while being driven, and indeed, most people will get away with this. And as people get away with it more and more, we will see people driving like this driver, not really prepared to react. This is one of the reasons Google decided not to make a system which requires driver takeover ever. As the system gets better, does it get more dangerous?

Declaration of Amsterdam

Last month, various EU officials gathered in Amsterdam and signed the Declaration of Amsterdam that outlines a plan for normalizing EU laws around self-driving cars. The meeting included a truck automation demo in the Netherlands and a self-drive transit shuttle demonstration. It’s a fairly bland document, more an expression of the times, and it sadly spends a lot of time on the red herring of “connected” vehicles and V2V/V2I, which governments seem to love, and self-driving car developers care very little about.

Let’s hope the regulatory touch is light. The reality is that even the people building these vehicles can’t make firm pronouncements on their final form or development needs, so governments certainly can’t do that, we must be careful that attempts to “help” but may hinder. We already have a number of examples of that happening in draft and real regulations and we’ve barely gotten started. For now, government statements should be limited to, “let’s get out of the way until people start figuring out how this will actually work, unless we see somebody doing something demonstrably dangerous that can’t be stopped except through regulations.” Sadly, too many regulators and commentators imagine it should be, “let’s use our limited current knowledge to imagine what might go wrong and write rules to ban it before it happens.”

Speech from the throne

It was a sign of the times when her Majesty the Queen, giving the speech from the throne in the UK parliament, laid out elements of self-driving car plans. The Queen drove jeeps during her military days, and so routinely drives herself at her country estates, otherwise she would be among the set of people most use to never driving.

The UK has 4 pilot projects in planning. Milton Keynes is underway, and later this year, a variation of the Ultra PRT pods in use at T5 of Heathrow airport — they run on private tracks to the car park — will go out on the open road in Greenwich. They are already signing up people for rides.

 Car companies thinking differently

In deciding which car companies are going to survive the transition to robocars, one thing I look for is willingness to stop thinking like a traditional car company which makes cars and sells them to customers. Most car company CEOs have said they don’t plan to keep thinking that way, but what they do is more important than what they say.

In the past we’ve seen Daimler say it will use their Car2Go service (with the name Car2Come probably likely to cause giggles in the USA) as a way to sell rides rather than cars. BMW has said the same about DriveNow. (And now GM has said this about its partnership with Lyft.) Daimler is also promoting their moovel app which tries to combine different forms of mobility, and BMW is re-launching DriveNow in Seattle as ReachNow which adds peer-to-peer carsharing and other modes of transportation to the mix.

Of course, these are tiny efforts for these big companies, but it scores big over companies still thinking only in the old ways. I’m looking at you, most car companies.

BWM also announced the iNext electric flagship sedan will offer self-drive in 2021.

Florida tells cities — think about self-driving

In 2010, I put out a call for urban planners to starting thinking about robocars and mostly it fell on deaf ears. Times are changing and this month Florida told its cities they should include consideration of this and other future transit in their updated long-term plans.

It is a tough call. Nobody’s predictions about the future here are good enough to make a firm plan and commit billions of dollars. At the same time, we are starting to learn that certain plans — especially status quo plans — are almost certainly seriously wrong. We may not have a perfect idea on what to spend city money on, but we can start to learn what not to do.

Fortunately, transportation is becoming a digital technology, which means you can change your plans much faster than physical infrastructure plans. Robocars like bare pavement, as stupid as can be. The ‘smarts’ go in the cars, not in the roads or the cities. So if you change your mind, you just have to reprogram your cars, not rebuild your city. Which is good, because you can’t rebuild your city.

China gets in the game

China is the world’s number one car manufacturing company. A lot of people don’t know that. Last year I visited the Shanghai auto show and it was a strange trip to walk through giant hall after giant hall of automakers you have never heard of before.

Self-driving car action in China has been slow. Right now everybody has a focus on just making regular cars for the rising middle class that is buying them as fast as they can. Wealthier Chinese usually buy foreign brands, although those cars are often made in China even though they have a VW or Buick nameplate.

Baidu has been working on cars for a couple of years and promises a pilot project in 2018. Recently, Chinese automaker Changan did an autopilot demo driving 1,000 miles. China has had its own annual academic version of the Darpa Grand Challenge for several years as well.

There is a strong chance that companies like Uber, Apple or Google, when they want to get their cars made, could go to Chinese manufacturers. Now that they’re getting practice at making their own technology. China is also a major source of electric vehicles, with future robotaxis likely to be small and electric. The manufacturers and suppliers with the most experience at making such vehicles are likely to be the winners.

Apple buys into Didi

Speaking of which, Apple just put a billion dollar investment into Didi. You may not know Didi, but it is the dominant phone-hail service in China with a much larger market share than Uber. It’s one of the few places Uber has lost in the market, but of course it’s China. As the auto industry moves to being about selling rides rather than cars, it’s an interesting move by Apple, which rarely does outside investments like this.