Source: Waymo

Waymo (Google) has announced a pilot project in Phoenix offering a full ride service in their new minivans. Members of the public can sign up — the link is sure to be overwhelmed with applicants, but it has videos and more details — and some families are already participating. There’s also a Waymo Blog post. I was in Phoenix this morning as it turns out, but to tell real estate developers about robocars; not for this.

There are several things notable about Waymo’s pilot:

1. They are attempting to cover a large area — they claim twice the size of San Francisco or 90 square miles. That’s a lot. It’s enough to cover the vast majority of trips for some pilot users. In other words, this is the first pilot which can test what it’s like to offer a “car replacement.”
2. They are pushing at families, which means children, including those not of driving age. The mother in the video expects to use it to send some children to activities. While I am sure there will be safety drivers watching over things, trusting children to the vehicles is a big milestone. Google’s safety record, with safety drivers, suggests this is actually a very safe choice for the parents, but there is emotion over trusting children to robots (other than the ones that go up and down shafts in buildings.)
3. In the videos, they are acting like there are no safety drivers but there surely are, for legal reasons as well as safety.
4. They are using the Pacifia minivans. The Firefly bubble cars are too slow for anything but neighbourhood operation. The minivans feature motorized doors, a feature that, although minor and commonplace, meets the image of what you want from a self-driving car.

Apple is in the game

There has been speculation recently because of some departures from Apple’s car team that they had given up. In fact, last week they applied for self-driving car test plates for California. I never thought they had left the game.

The big step is that once they test in California on public streets they will lose a lot of their stealth. Too many people in California (especially around Apple) know what to look for and we’ll see photos of these cars in the wild before too long. Which is good. I have never understood Apple’s deep culture of secrecy.

Waymo has reportedly bought a large test track in Nevada to expand their testing. Others are announcing pilot projects including Baidu, fitting surprisingly well with the schedule I have been telling people for the last few years. Namely that by late 2010 — pilots, until tech, regulatory and business kinks are worked out. Then by 2020 — first commercial deployments of taxi services. Mid-2020 — land rush as companies with working products rush to be the first to conquer individual city markets.

Uber/Waymo suit heats up

The Uber/Waymo lawsuit has become a soap opera. Because I have friends on both sides of the fight, I have not been commenting. Recent efforts include an attempt by Waymo to entirely shut down the Uber/Otto project or to force Anthony Levandowski to leave. This would be bad for Uber, especially the full shutdown, but as I wrote earlier Uber has a special advantage. Even if their project is shut down, they still have the number 1 brand and position for selling rides. Uber doesn’t own or control their cars now. If they have to buy somebody else’s robocars it is not nearly as strong a position as they might like, but it is not a failure.

Flying cars take off

I have been writing recently more about flying cars. Yesterday, my friends Larry Page and Sebastian Thrun revealed Kitty Hawk — a prototype flying car. You can see the cool video of it:

This is yet another piece of evidence that making a personal flying drone is certainly doable and going to happen. I even think the air traffic control problems can be solved. Four big questions remain:

1. How will we get enough energy in the vehicle to get a decent range? Not with today’s lithium-ion.
2. How will we get it quiet enough that you will tolerate regular takeoff and landing near you? (I am quoted in the NYT article on Kitty Hawk on this point.)
3. How will we get it energy efficient enough to not make our transportation less sustainable? Not with pure multirotors, though perhaps with ones that convert to fixed-wing flight. Or perhaps with electricity that’s so clean, we don’t care.
4. Will we tolerate skies packed full of (possibly noisy) flying drones?

Tesla settles suit

Telsa had been suing the startup founded by Stirling Anderson, who had been key in their autopilot project, and Chris Urmson, who for most of its existence ran Google’s project (and who I worked for while there.) The two are a powerhouse so their startup will be one to watch and the settlement will clear funding hurdles for them, even if it ended up being costly — but we don’t know many details.

Tesla Model S autopilot-software. Source: Tesla

NHTSA released the report from their Office of Defects Investigation on the fatal Tesla crash in Florida last spring. It’s a report that is surprisingly favorable to Tesla. So much so that even I am surprised.

While I did not think Tesla would be found defective, this report seems to come from a different agency than the one that recently warned comma.ai that:

It is insufficient to assert, as you do, that the product does not remove any of the driver’s responsibilities” and “there is a high likelihood that some drivers will use your product in a manner that exceeds its intended purpose.

The ODI report rules that Tesla properly considered driver distraction risks in its design of the product. It goes even further, noting that drivers using Tesla Autopilot (including those monitoring it properly and those who did not) still had a decently lower accident rate for mile than drivers of ordinary cars without autopilot. In other words, while the Autopilot without supervision is not good enough to drive on its own, the Autopilot even with occasionally lapsed supervision that is known to happen is still overall a safer system than not having the Autopilot at all.

This will provide powerful support for companies developing autopilot style systems, and companies designing robocars who wish to use customer supervised driving as a means to build up test miles and verification data. They are not putting their customers at risk as long as they do it as well as Tesla. This is interesting (and the report notes that evaluation of Autopilot distraction is not a settled question) because it seems probable that people using the Autopilot and ignoring the road to do e-Mail or watch movies are not safer than regular drivers. But the overall collection of distracted and watchful drivers is still a win.

This might change as companies introduce technologies which watch drivers and keep them out of the more dangerous inattentive style of use. As the autopilots get better, it will become more and more tempting, after all.

Tesla stock did not seem to be moved by this report. But it was also not moved by the accident or other investigations — it actually went on a broadly upward course for 2 months following the announcement of the fatality.

The ODI’s job is to judge if a vehicle is defective. That is different from saying it’s not perfect. Perfection is not expected, especially from ADAS and similar systems. The discussion about the finer points of whether drivers might over-trust the system are not firmly settled here. That can still be true without the car being defective and failing to perform as designed or being designed negligently.

You may have seen a lot of press around a dashcam video of a car accident in the Netherlands. It shows a Tesla in AutoPilot hitting the brakes around 1.4 seconds before a red car crashes hard into a black SUV that isn’t visible from the viewpoint of the dashcam. Many press have reported that the Tesla predicted that the two cars would hit, and because of the imminent accident, it hit the brakes to protect its occupants.

The accident is brutal but apparently nobody was hurt.

https://www.youtube.com/watch?v=lqqN5iRrAiM

The press speculation is incorrect. It got some fuel because Elon Musk himself retweeted the report linked to, but Telsa has in fact confirmed the alternate and more probable story which does not involve any prediction of the future accident. In fact, the red car plays little to no role in what took place.

Tesla’s autopilot uses radar as a key sensor. One great thing about radar is that it tells you how fast every radar target is going, as well as how far away it is. Radar for cars doesn’t tell you very accurately where the target is (roughly it can tell you what lane a target is in.) Radar beams bounce off many things, including the road. That means a radar beam can bounce off the road under a car that is in front of you, and then hit a car in front of it, even if you can’t see the car. Because the radar tells you “I see something in your lane 40m ahead going 20mph and something else 30m ahead going 60mph” you know it’s two different things.

The Tesla radar saw just that — the black SUV was hitting the brakes (possibly for a dirt patch that appears to show on the video) and the red car wasn’t. Regardless of the red car being there, the autopilot knew that if another car ahead was braking hard, it should also brake hard, and it did. Yes, it’s possible that it could also calculate that the red car, if it keeps going, will hit the black car, but that’s not entirely relevant — it’s clear that the Tesla should stop, regardless of what the red car is going to do. Tesla reported in their blog about how they were doing more with the radar, including tracking hidden vehicles with it. The ability of automotive radar to do this has been known for some time, and I have always presumed most teams have taken advantage of it. You don’t always get returns from hidden cars, but it’s worth using them if you do.

In the future, we will see robocar systems predicting accidents, but I am not aware of this being announced by any team. All robocars are tracking all objects ahead of them, for position and velocity, and they are extrapolating their velocity and predicting where they will go. Those predictions would also include detecting that vehicles might hit (if they continue their current course) and also if they could not avoid hitting at a certain point. If an imminent accident is predicted, it would make sense to know that and also react to it in advance. A car might even be able to predict a bit of what will happen after the accident, though that is chaotic.

A system like that would outperform the autopilot or any automatic emergency braking system. Presently, those systems largely track objects in their lane. They don’t brake because cars are stopped in adjacent lanes, because that would mean they could not work in traffic jams or carpool lanes when there are lanes at different speeds, and they could not deal with stalled cars on the side of the road.

However, if you saw what the Tesla saw from the lane to the right, it would still be a very smart thing to brake. Tesla has not commented on this, but I presume its system would not have braked if it had been in that lane, at least not braked before the accident. It might brake because other cars like the red car immediately moved into the right lane.

If you liked this post on robocars, you’ll also enjoy these articles:

• What if the city ran Waze and you had to obey it? Could this cure congestion?
• What are the right disability rules for robotaxis?
• Google car is now Waymo
• Comma.ai cancels comma-one add-on box after threats from NHTSA
• Robocar news: Comma One goes open source, creating simulations for robocars in New Zealand earthquakes

See all the latest robotics news on Robohub, or sign up for our weekly newsletter

Source: commaai

Comma.ai, the brash startup attempting to make a self-driving system entirely from a neural network has announced it will cancel the “comma one” add-on box it has planned to sell to owners of certain Honda vehicles. The box stuck on the rear-view mirror and used the car’s own bus commands to provide an autopilot similar to those offered by car makers, with lane-keeping and adaptive cruise control.

Of particular importance is the letter from NHTSA to comma.ai which I suggest you read. This letter creates several big issues:

1. There are many elements of this letter that would also apply to Tesla and other automakers which have built supervised autopilot functions.
2. Of particular interest is the paragraph that says: “it is insufficient to assert, as you do, that the product does not remove any of the driver’s responsibilities” and “there is a high likelihood that some drivers will use your product in a manner that exceeds its intended purpose.” That must be very scary for Tesla.
3. I noted before that the new NHTSA regulations appear to forbid the use of “black box” neural network approaches to the car’s path planning and decision making. I wondered if this made illegal the approach being done by Comma, NVIDIA and many other labs and players. This may suggest that.
4. We now have a taste of the new regulatory regime and had it existed before systems like Tesla’s Autopilot, Mercedes Traffic Jam Assist, and Cruise’s original aftermarket autopilot would never have been able to get off the ground.
5. George Hotz of comma declares, “Would much rather spend my life building amazing tech than dealing with regulators and lawyers. It isn’t worth it. The comma one is cancelled. comma.ai will be exploring other products and markets. Hello from Shenzhen, China.”

To be clear, comma is a tiny company taking a radical approach, so it is not a given that what NHTSA has applied to them would have been or will be unanswerable by the big guys. Because Tesla’s Autopilot is not a pure machine learning system, they can answer many of the questions in the NHTSA letter that comma can’t. They can do much more extensive testing that a tiny startup can’t. But even so, a letter like this sends a huge chill through the industry.

It should also be noted that in Comma’s photos the box replaced the rear-view mirror, and NHTSA had reason to ask about that.

Source: commaai

George’s declaration that he’s in Shenzen gives us the first sign of the new regulatory regime pushing innovation away from the United States and California. I will presume the regulators will say, “We only want to scare away dangerous innovation” but the hard truth is that is a very difficult thing to judge. All innovation in this space is going to be a bit dangerous. It’s all there trying to take the car — the 2nd most dangerous legal consumer product — and make it safer, but it starts from a place of danger. We are not going to get to safety without taking risks along the way.

I sometimes ask, “Why do we let 16-year-olds drive?” They are clearly a major danger to themselves and others. Driver testing is grossly inadequate. They are not adults so they don’t have the legal rights of adults. We let them drive because they are going to start out dangerous and then get better. It is the only practical way for them to get better, and we all went through it. Today’s early companies are teenagers. They are going to take risks. But this is the fastest and only practical way to let them get better and save millions.

“…some drivers will use your product in a manner that exceeds its intended purpose”

This sentence, though in the cover letter and not the actual legal demand, looks at the question asked so much after the Tesla fatal crash. The question which caused Consumer Reports to ask Tesla to turn off the feature. The question which caused MobilEye, they say, to sever their relationship with Tesla.

The paradox of the Autopilot is this: The better it gets, the more likely it is to make drivers over-depend on it. The more likely they will get complacent and look away from the road. And thus, the more likely you will see a horrible crash like the Tesla fatality. How do you deal with a system which adds more danger the better you make it? Customers don’t want annoying countermeasures. This may be another reason that “Level 2,” as I wrote yesterday is not really a meaningful thing.

NHTSA has put a line in the sand. It is no longer going to be enough to say that drivers are told to still pay attention.

Black box

Comma is not the only company trying to build a system with pure neural networks doing the actual steering decisions (known as “path planning”.) NVIDIA’s teams have been actively working on this, as have several others. They plan to make commentary to NHTSA about these elements of the regulations, which should not be forbidding this approach until we know it to be dangerous.

It is challenging. It’s hard to do QA on neural networks. You can examine any single state of them, but not really understand them. You can fix the errors they make, but not know how you fixed it or whether your fix is going to work in other cases. On their own that sounds too scary, but the problem is they are outperforming other algorithms at many of the problems they are being applied to.

If you have two systems:

1. A conventionally coded system which you fully understand, that has one safety incident per 100,000 miles,
2. A black box machine learning system that has one safety incident per 150,000 miles, but you have minimal understanding of.

Which is the one that is better to put on the road? It’s not a no-brainer.

Scaring people away

George Hotz is about as brash as they come, so some might discount his bluster about moving his projects to China. But this is a real thing. When regulations come, they tend to favour the big players. The big players have large QA teams and compliance departments. They can afford them, and they move conservatively because of them. Big players help write the regulations to favour their way of doing things — sometimes selfishly but sometimes just because it is the only thing they know.

The USA and California led the way in robocars in part because it was unregulated. In the USA, everything is permitted unless it was explicitly forbidden and nobody thought to write “no robots” in the laws. Progress in other countries where everything is forbidden unless it is permitted was much slower. The USA is moving in the wrong direction.

Countries like China are another animal altogether. There, everything is forbidden unless a smart autocrat allows it but once the smart (or stupid) ruling official makes it happen, the other red tape can vanish. It is not the best system overall, but it can win at things they put their minds to.

Tesla

Has Tesla received such a letter? What would they have done if they received it in their early days? While Tesla and Elon Musk love the Autopilot project, the hard truth is Tesla’s biggest problem is going to be making enough Model 3s to meet demand and doing it profitably. Taking out Autopilot isn’t going to cut their sales at all — they have too many sales and not enough cars. Only their forward vision and guts will keep them going if NHTSA does the same thing to them. I do believe Tesla can answer questions like those in the demand letter more easily than comma.ai. But not trivially, and they almost surely will need to bump the countermeasures.

Tesla Model S autopilot-software. Source: Tesla

Tesla’s spat with MobilEye reached a new pitch this week with Tesla announcing a new release of their autopilot and plans. As reported earlier, MobilEye announced during the summer that they would not be supplying the new and better versions of their EyeQ system to Tesla. Since that system was/is central to the operation of the Tesla Autopilot, they may have been surprised that MBLY stock took a big hit after the announcement (though it recovered for a while, and is now back down) and TSLA did not.

Statements and documents now show a nastier battle with MobilEye saying they were worried about Tesla using their tool in an unsafe way, invoking all the debate about the fatality and other crashes allegedly caused by people who were lulled into not bothering to supervise the Autopilot. Tesla says that, instead, they have been developing their own advanced vision tools and that MobilEye was afraid and told Tesla if they wanted more EyeQ chips, they would need to halt the competing project and commit to ME. That’s a nasty spat.

Tesla’s own efforts represent a threat to MobilEye from the growing revolution in neural network pattern matchers. Computer vision is going through a big revolution. MobilEye is a big player in that revolution because their ASICs do both standard machine vision functions and can do neural networks. An ASIC will beat a general purpose processor when it comes to cost, speed, and power, but only if the ASIC’s abilities were designed to solve those particular problems. Since it takes years to bring an ASIC to production, you have to aim right. MobilEye aimed pretty well, but at the same time, lots of research out there is trying to aim even better or do things with more general purpose chips, like GPUs. Soon we will see ASICs aimed directly at neural network computations. To solve the problem with neural networks, you need the computing horsepower, and you need well designed deep network architectures, plus, you need the right training data and lots of it. Tesla and ME both are gaining lots of training data. Many companies, including Nvidia, Intel and others are working on the hardware for neural networks. Most people would point to Google as the company with the best skills in architecting the networks, though there are many doing interesting work there. (Google’s DeepMind built the tools that beat humans at the seemingly impossible game of Go, for example.) It’s definitely a competitive race.

Using radar

While Tesla works on their vision systems, they also announced a plan to make much more use of radar. That’s an interesting plan. Radar has been the poor 3rd-class sensor of the robocar, after LIDAR and vision. Everybody uses it — you would be crazy not to unless you need to be very low cost. Radar sees further than the other systems, and it tells you immediately how fast any radar target you see is moving relative to you. It sees through fog and other weather, and it can even see under and around big cars in front of you as it bounces off the road and other objects. It’s really good at licence plates, as well.

What radar doesn’t have is high resolution. Today’s automotive radars have gotten good enough to tell you what lane an object like another car is in, but they are not designed to have any vertical resolution — you will get radar returns from a stalled car ahead of you on the road and a sign above that lane, and not be sure of the difference. You need your car to avoid a stalled car in your lane, but you can’t have a car that hits the brakes every time it sees a road sign or bridge!

mage source: EDN Network

Real world radar is messy. Your antennas send out and receive from a very broad cone with potential signals from other directions and from side lobes. Reflections are coming from vehicles and road users but also from the ground, hills, trees, fences, signs, bushes and bridges. It’s work to get reliable information from it. Early automotive radars found the best solution was to use the doppler speed information, and discard all returns from anything that wasn’t moving towards or away from you — including stalled cars and cross traffic.

One thing that can help (imperfectly) is a map. You can know where the bridges and signs are so you don’t brake for them. Now you can brake for the stalled cars and the cross traffic the Tesla failed to see. You still have an issue with a stalled car under a bridge or sign, but you’re doing a lot better.

There’s a lot of room for improvement in radar and I will presume — Tesla has not said — that Tesla plans to work on this. The automotive radars everybody buys (from companies like Bosch) were made for the ADAS market — adaptive cruise control, emergency braking etc. It is possible to design new radars with more resolution (particularly in the vertical) and other approaches. You can also try for more resolution, particularly by splitting the transmitter and receiver to produce a synthetic larger aperture. You can go into different bands and get more bandwidth and get more resolution in general. You can play more software tricks, and particularly, you can learn by examining not just single radar returns, but rather the pattern of returns over time. (After all, humans don’t navigate from still frames, we depend on our visual system’s deep evolved ability to use motion and other clues to understand the world.)

The neural networks are making strides here. For example, while pedestrians produce basic radar returns, it turns out that their walking stride has a particular pattern of changes that can be identified by neural networks. People are doing research now on how examining the moving and dynamic pattern of radar returns can help you get more resolution and also identify shapes and motion patterns of objects and figure out what they are.

I will also speculate that it might be possible to return to a successor of the “sweeped” radars of old, the ones we are used to seeing in old war movies. Modern car radars don’t scan like that, but I have to wonder if with new techniques, like phased arrays to steer virtual beams (already the norm in military radar) and modern high-speed electronics, that we might produce radars that get a better sense of where their target is. We’re also getting better at sensor fusion — identifying a radar target in an image or LIDAR return to help learn more about it.

The one best way to improve radar resolution would be to use more bandwidth. There have been experiments in using ultrawideband signals in the very high frequencies which may offer promise. As the name suggests, UWB uses a very wide band, and it distributes its energy over that very wide band, which means it doesn’t put too much energy into any one band and has less chance of interfering in those bands. It’s also possible that the FCC, seeing the tremendous public value that reliable robocars offer, might consider opening up more spectrum for use in radar applications using modern techniques and thus increase the resolution.

In other words, Tesla is wise to work on getting more from radar. With the loss of all MobilEye’s vision tools, they will have to work hard to duplicate and surpass that. For now, Tesla is committed to using parts that are for sale for existing production cars, costing hundreds of dollars. That has taken LIDAR “off their radar” even though almost all research teams depend on LIDAR and expect LIDAR to be cheap in a couple of years. (Including the LIDAR from Quanergy, a company I advise.)

Comma announces a $1,000 autopilot box

I wrote earlier about comma.ai and their efforts to drive with just vision, radar and neural networks. They now plan to offer a box for $1,000 to give you some basic autopilot functionality as an add-on.

To do this, they are working with only some specific car models, namely some Honda vehicles that already have advanced ADAS in them. Using the car’s internal bus, they can talk to the sensors in these cars (such as the radar and possibly the camera) and also send control signals to actuate the steering, brakes and throttle. Then their neural networks can take the sensor information, and output the steering and speed commands to keep you in the lane. (Details are scant so I don’t know if the Comma One box uses its own camera or depends on access to the car’s.)

When I rode in Comma’s prototype it certainly wasn’t up to the level of the Tesla autopilot or some others, but it has been several months so I can’t judge it now. Like the Tesla autopilot, the Comma will not be safe enough to drive the car on its own, and you will need to supervise and be ready to intervene at any time. If you get complacent, as some Tesla drivers have, you could get injured or killed. I have yet to learn what measures Comma will take to make sure people keep their eyes on the road.

Generally, I feel that autopilots are not very exciting products when you have to watch them all the time — as you do — and that bolt-on products are not particularly exciting. Cruise’s initial plan (after they abandoned valet parking) was a bolt-on autopilot, but they soon switched to trying to build a real vehicle, and that got them the huge $700M sale to General Motors.

But for Comma, there is a worthwhile angle. Users of this bolt-on box will be helping to provide training data to improve their systems. In fact, they will be paying for the privilege of testing the system and training it. Something that companies like Google did the old-fashioned way, paying a staff of professionals to drive the cars and gather data. For a tiny, young startup it’s a worthwhile approach.

If you liked this article, you may also want to read:

• Robocar recap: Uber buys Otto, folks leave Google, Ford goes big, Tesla drops MobilEye
• Will robocars be heaven or hell for our cities?
• Robocar platooning, or just carpool?
• Understanding the massive gulf between the Tesla Autopilot and a real robocar, in light of the crash

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Source: Otto/YouTube

Brad Templeton, from Robocars.com, discusses the latest robocar related news.

The past few weeks has seen big robocar news. Real news — not the constant “X is partnering with Y” press releases that fill the airwaves sometimes.

Uber has made a deal to purchase Otto, a self-driving truck company I wrote about earlier, founded by several friends of mine from Google. The rumoured terms of the deal are astronomical — possibly 1% of Uber’s highly valued stock (which means almost $700M) and other performance rewards. I have no other information yet on the terms, but it’s safe to say Otto was just getting started with ambitious goals and would not have sold for less than an impressive amount. For a company only six months old, the rumoured terms surpass even the amazing valuation stories of Cruise and Zoox.

While Otto has been working on self-driving technology for trucks, any such technology can also move into cars. Uber already has an active lab in Pittsburgh, but up to now has not been involved in long-haul trucking (it does do local deliveries in some places). There are many startups calling themselves the “Uber for Trucks” and Otto has revealed it was also working on shipping management platform tools, so this will strike some fear into those startups. Because of my friendship with Otto’s team, I will do more commentary when details become public.

In other Uber news, Uber has announced it will sell randomly assigned Uber rides in their self-driving vehicles in Pittsburgh. If your ride request is picked at random (and because it’s in the right place) Uber will send one of their own cars to drive you on your ride, and will make the ride free to boot. Of course, there will be an Uber safety driver in the vehicle monitoring it, ready to take over in any problem or complex situation. So the rides are a gimmick to some extent, but if they were not free, it would be a sign of another way to get customers to pay for the cost of testing and verifying self-driving cars. The free rides, however, will probably cause more people to take Uber rides hoping they will win the lottery and get not simply the free ride, but the self-driving ride.

GM announced a similar program for Lyft — but not until next year.

Ford also goes all-in, but with a later date

Ford announced it wants to commit to making unmanned capable taxi vehicles, the same thing Uber, Google, Cruise/GM, Zoox and most non-car companies want to make. For many years I have outlined the difference between the usual car company approaches, which are evolutionary and involve taking cars and improving their computers and the approaches of the non-car companies which bypass all legacy thinking (mostly around ADAS) to go directly to the final target. I call that “taking a computer and putting wheels on it.” It’s a big and bold move for Ford to switch to the other camp and a good sign for them. They have said they will have a fleet of such vehicles as soon as 2021.

Other companies have announced plans to go for unmanned capable taxi vehicles, particularly Daimler and BMW, but they still maintain active “evolutionary” projects which hope to build on their prior ADAS work.

Velodyne

The other big news is that Ford, together with Baidu, have made a $150M investment in Velodyne. Velodyne has been for a decade the primary supplier of LIDARs for robocar research projects, but their LIDARs are research instruments: high performance at high cost. In the meantime, other companies (such as Quanergy, for whom I am an advisor) have been pushing to make better-priced LIDARs for real production. The Velodynes have been the right choice because it is smart to research with what’s expensive today, knowing it will get cheap in the future. Baidu’s test cars have had Velodyne’s on them, but there were recent rumours of Baidu experimenting with alternatives to LIDAR, but this investment may counter those rumours.

This investment will move Velodyne to a lower cost production unit, though at present at a higher cost than the solid-state plans of Quanergy and some others.

Google departures

Otto included a few people from Google’s car team. It was also recently revealed that Chris Urmson, who had run the team for most of its existence and was moved to “CTO” after the arrival of John Krafcik, has departed. I worked for Chris, but he has not told me his reason for departure. In general Google’s car team has gotten huge, and the recently departed people are mostly folks who were among the first 20 on the team. This may simply be a sign of massive expansion, I have not heard much to credit the idea that there is some sort of internal chaos. In addition, deals like the Otto deal will certainly cause people to consider if they want to work for a big corporation or break out on their own.

MobilEye

A few weeks ago, MobilEye issued a press release that they would not be supplying the next generation of their EyeQ chips to Tesla. The EyeQ4 is a great deal more sophisticated than the EyeQ3 that Telsa uses and Tesla even uses an older version with a more limited set of features — including, most famously, the inability to detect things like cross traffic. Since MobilEye announced it, it seems like it was their decision (you don’t usually issue a press release to say you have been dumped by a smaller customer.) It caused their stock to take a quick big hit (disclaimer, I own a small amount) on the day of the announcement, but it recovered reasonably well. Tesla’s stock was not affected.

Tesla folks have always claimed that they only made minimal use of ME’s functionality and that the real special sauce was theirs. ME folks have claimed the opposite. This break-up will settle that question, and presumably, Tesla has been working on alternatives for a while as they don’t seem to be in much panic about it. Tesla has made announcements of plans for their new hardware and those plans look very similar to the EyeQ4, so they must have had their own parallel research underway.

If you liked this article, you may also want to read:

• Robocar news around the globe: Tesla crash, Declaration of Amsterdam, and automaker services
• 50 different state regulations for robocars is not a bad idea
• Will robocars be heaven or hell for our cities?
• Robocar platooning, or just carpool?

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Tesla and Powerwall. Credit: Tesla

Today, I want to look at some implications of Tesla’s Master Plan, Part Deux which caused some buzz this week. There was other news, of course, including the AUVSI/TRB meeting which I attended, and will report on, forecast dates from Volvo, BMW and others, hints from Baidu, Faraday Future and Apple, and more.

In Musk’s blog post he laid out these elements of Tesla’s plan:

• Integrating generation and storage (with SolarCity and the Powerwall and your car.)
• Expand into trucks and minibuses
• More autonomy in Tesla cars
• Hiring out your Tesla as a robotaxi, when not using it

Except for the first one, all of these are ideas I have covered extensively. It is good to see an automaker start work in these directions. As such, while I will mostly agree with what Tesla is saying, there are a few issues to discuss.

Electric (self-driving) minibus and trucks

In my article earlier this year on the future of transit I laid out why transit should mostly be done with smaller (van sized) vehicles, taking ad-hoc trips on dynamic paths, rather than the big vehicle, fixed-route, fixed-schedule approach that is taken today. The automation is what makes this happen (especially when you add the ability of single person robocars to do first and last miles.) Making the bus electric can make it greener, though, making it run full almost all the time is far more important for that.

The same is true for trucks, but both trucks and buses have huge power needs which presents problems for having them be electric. Electric’s biggest problem here is the long recharge time, and that puts your valuable asset out of service. For trucks, the big win of having a robotruck is that it can drive 24 hours/day; you don’t want to take that away by making it electric. This means you want to look into things like battery swap, or perhaps more simply, tractor swap. In that case, a truck would pull into a charging station and disconnect from its trailer and another tractor that recharged would grab on and keep it going.

You can also do that with a bus but passengers don’t want to swap, so, you may swap them out between runs. Here, the autonomous ability of the bus is key because it can drive to where the super-powerful recharging is. To recharge a bus you want a lot of power, an amount not available in many locations, and it’s costly to bring it in. If the bus comes to where the electricity already is (like transformer substations) it solves that problem. The bus is still down during recharging, of course. The bigger you make the battery, the longer you can run, but unfortunately, that also means it costs more, takes longer to recharge and weighs more, costing efficiency and making it more dangerous.

More autonomy

Well, Tesla continues to push autonomy without LIDAR. As the cost of LIDAR drops (Disclosure: I have an equity stake in this business) it will not make sense for anybody to make their vehicle less safe just to save a few hundred dollars. Still, the more competition the merrier, so Tesla’s push will at the very least teach the world some things.

Once Tesla has a door-to-door vehicle, Musk suggests that it will offer the scenario I first wrote about back in 2007 in A week in robocars — that when you are not using your car, it might hire itself out as a taxi. Tesla even plans to create a brokering service for this, competing with the Ubers of the world.

Since I first began thinking this way, I have actually become less enamoured of the idea, though I do think it will happen. I felt you might do it to help you afford a more expensive car. UberX is in effect your car making money for you, but you have to do the annoying work of driving it.

The problem stems from an over importance I and many others attributed to the problem that cars sit idle most of the time. In the robotaxi world, a larger and larger fraction of cars won’t do that, and as rates become competitive, people will think of this as a pure capital business — you make a capital investment in a car, and it pays a return charging fees to take people around. The more efficient that market gets, the purer it gets, and it becomes dominated by the professionals, not the amateurs. We have seen this happen with AirBnB, which began mostly with people renting out spare rooms and time in their apartments, and has migrated to having a lot more units that are full-time on AirBnB. If you hire out your car a lot, you start losing the values that made you want to own it in the first place — the ability to keep your stuff in it, to always have it ready for you, the ability to let it get messy, and the ability to impress people with ownership of it. Each transition between users also has a cost. We’ve seen some turmoil in existing peer-to-peer carshare companies. Turo changed its name from Relayrides and also changed to a more long-term rental model. Our SU students who started Getaround.com are still going with pure P2P though.

The counter force comes from people who are paying way more than they should for car ownership who thus have “spare” capacity they might as well sell. That’s true, but its value is only the value of the capital. Today, capital is cheap. So cheap that the interest on a $30,000 loan at 4% as you might have for a car is around $3/day or 14 cents/hour. That’s the cost of letting your asset sit around doing nothing, and it’s not that large. In fact, it might be that parking space costs more than that and is the actual incentive to get it out working.

Another big issue with electrics is that if somebody hires your car, and then you want it, it may be discharged and you can’t use it. With gasoline, you can always quickly refill.

Cars that are hired out will wear out by the mile, not the year. As such, every time it is hired out it’s depreciating, and smart people will see that.

In the future, we’ll see a mix — people who own, people who own and hire out, people who drop the number of cars they own due to robotaxi services, and people who give up car ownership altogether because of them. I foresee many of each category, but for now, the people who hire out are the smallest category. Of course, in the early days, when there are not many robotaxis around, the Teslas will be popular.

Generation and storage

The one part of the plan that I have trouble with is the idea of combining solar generation with battery storage. Today, that’s just a bad idea. Solar panels generate electricity whenever the sun shines on them. If you connect them to batteries, they will charge the batteries, but soon the batteries will get more and more full, and be unable to take the power from the panels. As they get closer to full, you will be simply throwing away all the nice power the panels are generating. And people like their batteries to be full. When you do the economics of a solar panel, it depends on using all the power it makes. If you throw away half the power from your panels, you can quickly turn them from economical and green to silly.

That’s why you need to tie solar panels to the grid. There, every watt they generate is used, and it’s used offsetting the dirty power generation of the grid.

Of course, you can have solar panels and batteries connected to the grid. You can use the batteries to store power when it’s cheap, and use it when it’s more expensive. Unfortunately, solar panels’ best attribute is they generate their power in the day, when power is expensive. The most expensive time is mid-to-late afternoon and in an ideal world you want to sell all your power to the hungry grid, not store it. (It is not an ideal world, however, and not all utilities will buy the power from you at the real market price.) Before 1pm could be a good time to store some solar power in local batteries. (Most solar panels get aimed south and peak at noon, but smarter panels aim a bit west and peak later in the day if they are selling power back to the grid.)

Tesla’s Powerwall does aim at people who have bad utility contracts. For them, it does make sense to store any surplus from a solar panel system during the day. But that’s not a sustainable plan. In a real grid, that would not make sense. What could make sense is buying power at night when it’s cheap, and using it by day. In a proper market, you should be able to sell your solar surplus for far more than it costs to refill at night. As such, solar generation and storage are really two different things, and Elon Musk’s declaration that they have to be connected does not make sense to me — except where the market is broken.

In time, solar generation is going to go up — not just at homes, but utilities. When that happens, we could get a surplus of solar mid-day, and a shortage as the sun goes down. In that case, batteries at the grid or house level can make sense.

I should note it is possible to be green with non-grid connected solar if you can be sure you will use every watt-hour your panels generate in your home and car. This means you must underprovision, and then use the grid for the rest. ie. you are still on the grid, but never feeding it from your renewables or batteries.

Tesla car. Source: Tesla

Brad Templeton describes Tesla’s Autopilot as a ‘distant cousin of a real robocar’ that primarily uses a MobilEye EyeQ3 camera combined with radars and ultrasonic sensors. Unlike robocar sensors, Tesla doesn’t have a lidar or use a map to help it understand the road and environment.

It’s not surprising there is much debate about the fatal Tesla Autopilot crash revealed to us last week. The big surprise to me is that Tesla and MobilEye stock seem entirely unaffected. For many years, one of the most common refrains I would hear in discussions about robocars was, “This is all great, but the first fatality and it’s all over.” I never believed it would all be over, but I didn’t think there would barely be a blip.

There has been lots of blips in the press and online, of course, but most have had some pretty wrong assumptions. Tesla’s Autopilot is a distant cousin of a real robocar, and that would explain why the fatality is no big deal for the field, but the press shows that people don’t know that.

Tesla’s Autopilot is really a fancy cruise control. It combines several key features from the ADAS (Advance Driver Assist) world, such as adaptive cruise control, lane-keeping and forward collision avoidance, among others. All these features have been in cars for years, and they are also combined in similar products in other cars, both commercial offerings and demonstrated prototypes. In fact, Honda promoted such a function over 10 years ago!

Tesla’s Autopilot primarily uses the MobilEye EyeQ3 camera, combined with radars and some ultrasonic sensors. It doesn’t have a lidar (the gold standard in robocar sensors) and it doesn’t use a map to help it understand the road and environment.

Most importantly, it is far from complete. There is tons of stuff it’s not able to handle. Some of those things it can’t do are known, some are unknown. Because of this, it is designed to only work under constant supervision by a driver. Tesla drivers get this explained in detail in their manual and when they turn on the Autopilot.

ADAS cars are declared not to be self-driving cars in many state laws

This is nothing new — lots of cars have lots of features to help drive (including the components used like cruise controls, each available on their own) which are not good enough to drive the car, and only are supposed to augment an alert driver, not replace one. Because car companies have been selling things like this for years, when the first robocar laws were drafted, they made sure there was a carve-out in the laws so that their systems would not be subject to the robocar regulations companies like Google wanted.

The Florida law, similar to other laws, says:

“The term [Autonomous Vehicle] excludes a motor vehicle enabled with active safety systems or driver assistance systems, including, without limitation, a system to provide electronic blind spot assistance, crash avoidance, emergency braking, parking assistance, adaptive cruise control, lane keep assistance, lane departure warning, or traffic jam and queuing assistant, unless any such system alone or in combination with other systems
enables the vehicle on which the technology is installed to drive without the active control or monitoring by a human operator.”

The Tesla’s failure to see the truck was not surprising

There’s been a lot of writing (and I did some of it) about the particulars of the failure of Tesla’s technology, and what might be done to fix it. That’s an interesting topic, but it misses a very key point. Tesla’s system did not fail. It operated within its design parameters, and according to the way Tesla describes it, in its manuals and warnings. The Tesla system, not being a robocar system, has tons of stuff it does not properly detect. A truck crossing the road is just one of those things. It’s also poor on stopped vehicles and many other situations.

Tesla could (and in time, will) fix the system’s problem with cross traffic. (MobilEye itself has that planned for its EyeQ4 chip coming out in 2018, and freely admits that the EyeQ3 Tesla uses does not detect cross traffic well.) But fixing that problem would not change what the system is, and not change the need for constant monitoring that Tesla has always declared it to have.

People must understand that the state of the art in camera systems is not today anywhere near the level needed for a robocar. That’s why most advanced robocar research projects all use LIDAR. There are those (including Tesla and MobilEye) who hope that the day might come soon when a camera can do it, but that day is not yet here. As such, any camera based car is going to make mistakes like these. Fix this one and there will be another. While the Tesla Autopilot failed to see the truck, that was an error, a tragic one, but not a failure of the Autopilot. It was an expected limitation, one of many. The system performed as specified, and as described in the Tesla user manual and the warnings to drivers. I will say this again, the Tesla Autopilot did not fail, it made an error expected under its design parameters.

The problem of getting too good — and punishing that

No, the big issue here is not what the Tesla Autopilot can’t handle, but the opposite. The issue is that it’s gotten good enough that people are mistaking it for a self-driving car, and they are taking their eyes off the road. Many stories abound of people doing e-mail, and there is an allegation that Brown, the deceased driver of this Tesla, might have been watching a movie. Movie or not Brown was not paying attention, as he could have easily seen and braked in time to avoid hitting that truck.

Tesla warns explicitly not to do that. Brown was a fairly skilled guy and he also should have known not to do that (if he did.) But as Tesla has gotten better, there is no question that people are not heeding the warning and getting a bit reckless. And Tesla knows this, of course.

This brings up all sorts of issues. Does it matter that Tesla knows that people are ignoring the warnings so long as it gives them sternly? Is there a duty of care to warn people that there is a danger you will ignore the warnings? Is there a duty of care to make sure (with technology) that they don’t ignore them?

Perhaps you see the problem here — the better the system gets, the more likely it is that it will make people complacent. People are stupid. They get away with something a few times and they imagine they will always get away with it. Older supervised products like cruise control needed your attention fairly often; there was no way you could watch a movie when using them. Cruise control needs your intervention to steer a few times a minute. Early autopilot-like systems need your intervention every few minutes. But Tesla got good enough that on open highways, it might not need your intervention for hours, or the entire highway part of a trip. Eventually it will get to not needing it for days, or even a month. Who would not be tempted to look away for a little while, and then a lot, if it gets there — or gets to needing only one intervention a year?

Our minds are bad at statistics. We, human drivers, have a tiny accident perhaps once every 10 years on average and one that is reported to insurance about every 25 years. We have a fatal accident about every 8,000 years of average driving, closer to 20,000 years on the highway. A rate of just one serious intervention every year sounds amazingly trustworthy. It seems like you took a bigger risk just getting on the road. But in fact, it’s very poor performance. Most people agree to be a true robocar, we’re going to need a car that makes an accident-causing mistake less often as a human, perhaps even twice as good or more. And the Tesla isn’t even remotely close. Even Google, which is much, much closer, isn’t there yet.

The incremental method

But we want systems to get better. It seems wrong to have to say that the better a system gets, the more dangerous it is. That a company should face extra liability for making the system better than the others. That’s not the direction we want to go. It’s definitely not the way that all the car companies want to go. They want to build their self-driving car in an evolutionary incremental way. They want to put supervised autopilots out on the road, and keep improving them until one day they wake up and say, “Hey, our system hasn’t needed an intervention on this class of roads for a million miles!” That day they can consider making it a real robocar for those roads. That’s different from companies like Google, Uber, Apple, Zoox and other non-car companies who want to aim directly for the final target, and are doing so with different development and testing methods.

Other views on the complacency issue

It should be noted that most other automakers who have made similar products have been much more keen on using tools to stop drivers from getting complacent and not being diligent in supervising. Some make you touch the wheel every few seconds. Some have experimented with cameras to watch the driver’s eyes. GM has announced a “super cruise” product for higher end Cadillacs for several years, but every year pulled back on shipping it, not feeling they have sufficient “countermeasures” to stop accidents like the Tesla one.

Google famously did testing with their own employees of a car that was quite a bit superior to the Tesla back in 2012-2013. Even though Google was working on a car that would not need supervision on certain routes, they required their employees (regular employees, not those on the car team) to nonetheless pay attention. They found that in spite of their warnings, about a week into commuting, some of these employees would do things that made the car team uncomfortable. This is what led Google to decide to make a prototype with no steering wheel, setting the bar higher for the team, requiring the car to handle every situation and not depend on an unreliable human.

Putting people at risk

Tesla drivers are ignoring warnings and getting complacent, and putting themselves and others at risk. But cars are full of features that cause this. The Tesla, and many other sports cars, can accelerate like crazy. It can drive 150mph. And every car maker knows that people take their muscle cars and speed with them, sometimes excessively and recklessly. We don’t blame the car makers for making cars capable of this, or for knowing this will happen. Car makers put in radios, knowing folks will get distracted fiddling with them. Car makers know many of their customers will drive their cars drunk, and kill lots of people. Generally, the law and society does not want to blame car makers for not having a breath alcohol test. People drive sleepy and half-blind from cataracts, and we always blame the driver for being reckless.

There is some difference between enabling a driver to take risks, and encouraging complacency. Is that enough of a difference to change how we think about this? If we change how we think, how much will we slow down the development of technology that in the long term will prevent many accidents and save many lives?

In an earlier post I have suggested Tesla might use more “countermeasures” to assure drivers are being diligent. Other automakers have deployed or experimented with such systems, or even held back on releasing their similar products because they want more countermeasures. At the same time, many of the countermeasures are annoying and people worry they might discourage use of the autopilots. Indeed, my proposal, which states that if you fail to pay attention, your autopilot is disabled for the rest of the day, and eventually permanently, would frighten customers. They love their autopilot and would value it less if they had to worry about losing it. But such people are also the sort of people who are making the mistake of thinking a car that needs intervention once a year is essentially the same as a robocar.

The unwritten rules of the road

I have often mused on the fact that real driving involves breaking the rules of the road all the time, and this accident might be such a situation. While the final details are not in, it seems this intersection might not be a very safe one. It is possible, by the math of some writers, that the truck began its turn unable to see the Tesla, because there is a small crest in the road 1200’ out. In addition, some allege the Tesla might have been speeding as fast as 90mph — which is not hard to believe since the driver had many speeding tickets.

Normally, if you make an unprotected left turn, your duty is to yield right-of-way to oncoming traffic. Normally, the truck driver would be at fault. But perhaps the road was clear when he started turning, and the Tesla only appeared once he was in the turn? Perhaps the super wide median offers an opportunity the truck driver didn’t take, to pause after starting the turn to check again, and the truck driver remains at fault for not double checking with such a big rig.

Ordinarily, if a truck did turn and not yield to oncoming traffic, no accident would happen. That’s because the oncoming vehicle would see the truck, and the road has a lot of space for that vehicle to brake. Sure, the truck is in the wrong, but the driver of an oncoming vehicle facing a truck would have to be insane to proudly assert their right-of-way and plow into the truck. No sane and sober human driver would ever do that. As such, the intersection is actually safe, even without sufficient sightlines for a slow truck and fast car.

Because of that, real world driving involves stealing right-of-way all the time. People cut people off, nudge into lanes and get away with it because rational drivers yield the ROW that belongs to them. Sometimes this is even necessary to make the roads work.

With the Tesla driver inattentive, the worst happened. The Tesla’s sensors were not good enough to detect the truck and brake for it (and the human wasn’t insane but wasn’t looking at the road) so he didn’t do the job of compensating for the autopilot’s inadequacy. The result was death.

Measuring the safety of the Autopilot

Tesla touted a misleading number, saying they had gone 130 million miles in Autopilot before this first fatality. The reality is that on limited access freeways, the USA number for human drivers is one fatality per 180 million miles, not the 90 million they cited. The Autopilot is used mostly on freeways where it functions best.

Tesla will perhaps eventually publish numbers to help judge the performance of the Autopilot. Companies participating in the California self-driving car registration program are required to publish their numbers. Tesla participates but publishes “zero” all the time because the Autopilot does not qualify as a self-driving car.

Here are numbers we might like to see, perhaps broken down by classes or road, weather conditions, lighting conditions and more:

• Number of “safety” disengagements per mile, where a driver had to take the controls for a safety reason (as opposed to just coming to a traffic light or turn.)
• Number of safety disengagements which, after further analysis in simulator, would have resulted in an accident if the driver had not intervened.
• Number of “late” safety disengagements, or other indications of driver inattention.
• Number of disengagements triggered by the system rather than the driver.

The problem is Tesla doesn’t really have a source for many of these numbers. Professional safety drivers always log why they disengage, and detailed car logs allow the building of more data. Tesla would need to record more data to report this.

Tesla with LIDAR

Folks in Silicon Valley have spotted what appear to be official Tesla test cars with LIDARs on them (one research LIDAR and one looking more like a production unit.) This would be a positive sign for them on the path to a full robocar. Tesla’s problem is that until about 2018, there is not a cost-effective LIDAR they can ship in a car, and so their method of gathering data by observing customers drive doesn’t work. They could equip a subset of special cars with LIDARs and offer them to a special exclusive set of customers — Tesla drivers are such rabid fans they might even pay for the privilege of getting the special test car and trying it out. They would still need to supervise, though, and that means finding a way around the complacency issue.

If you enjoyed this article, you may also enjoy:

• What to expect from autonomous cars
• Should Tesla disable your Autopilot if you’re not diligent? Plus, a survey of robocar validation
• Man dies while driven by Tesla Autopilot
• Robocar news around the globe: Tesla crash, Declaration of Amsterdam, and automaker services

Tesla Model S autopilot-software. Source: Tesla

In this article, Brad Templeton provides a rundown of different approaches for validation of self-driving and driver assist systems, a recommendation to Tesla and others to have countermeasures to detect drivers not watching the road, and permanently disable their Autopilot if they show a pattern of inattention.

The recent fatality for a man driven by the Tesla “autopilot” system has ignited debate on whether it was appropriate for Tesla to allow their system to be used as it was.

Tesla’s autopilot is a driver assist system, and Tesla tells customers it must always be supervised by an alert driver ready to take the controls at any time. The autopilot is not a working self-driving car system, and it’s not rated for all sorts of driving conditions, and there are huge numbers of situations that it is not designed to handle and can’t handle. Tesla knows that. But the public, press, and Tesla customers forget that and there are many Tesla users who are treating the autopilot like a real self-driving car system, and who are not paying attention to the road — and Tesla is aware of that as well. The press made this mistake as well, regularly writing fanciful stories about how Tesla was ahead of Google and other teams.

Brown, the driver killed in the crash, was very likely one of those people, and if so, he paid for it with his life. In spite of all the warnings Tesla may give about the system, some users
do get a sense of false security. There is an ongoing debate if that means driver assist systems are a bad idea.

There have been partial self-driving systems that require supervision since the arrival of the cruise control. Adaptive cruise control is even better, and other car companies have released autopilot like systems which combine adaptive cruise control with lane-keeping and forward collision avoidance, which hits the brakes if you’re about to rear-end another car. Mercedes has sold a “traffic jam assist” like the Telsa autopilot since 2014 that only runs at low speeds in the USA. You can even go back to a Honda demo in 2005 of an autopilot like system.

With cruise control, you might relax a bit but you know you have to pay attention. You’re steering and for a long time even adaptive cruise controls did not slow down for stopped cars. The problem with Tesla’s autopilot is that it was more comprehensive and better performing than earlier systems, and even though it had tons of things it could not handle, people have started to trust it with their lives.

Tesla’s plan can be viewed in several ways. One view is that Tesla was using customers as
“beta testers,” as guinea pigs for a primitive self-drive system which is not production ready — and that this is too much of a risk. Another is that Tesla built (and tested) a superior driver assist system with known and warned limitations and customers should have listened to those warnings.

Neither is quite right. While Tesla has been clear about the latter stance, with the knowledge that people will over-trust it, we must face the fact that it is not only the daring drivers who are putting themselves at risk, it’s also others on the road who are put at risk by the over-trusting drivers — or perhaps by Tesla. What if the errant car had not gone under a truck, but instead hit another car, or even plowed into a pedestrian when it careened off the road after the crash?

At the same time, Tesla’s early deployment approach is a powerful tool for the development and quality assurance of self-drive systems. I have written before about how testing is the big unsolved problem in self-driving cars. Companies like Google have spent millions with a staff of paid drivers testing their cars for 1.6 million miles. This is massively expensive and time consuming, and even Google’s money can’t easily generate the billions of miles of testing that some feel might be needed. Human drivers will have about 12 fatalities in a billion miles, and we want our self-driving cars to do much better. Just how we’ll get enough verification and testing done to bring this technology to the world is not a solved problem.

While we don’t want a repeat of the things that led to the death of this driver, there may be greater value to the field by having fleets of volunteers help verify these vehicles and if it can be done safely, the world probably needs it. But even if there is no way to do volunteer testing of systems that are still being refined — and they will still be in the being refined state for many years to come — there is still the question of whether you can have systems that need human supervision and intervention that get so good that humans ignore warnings and stop bothering to supervise them.

Here’s a look at a variety of approaches to testing, verifying, refining and learning:

In Simulator

The only way many billions of miles will be done is in a simulator, especially since you want to re-test all new software builds as much as you can before deploying. In a simulator you can test all sorts of risky situations you could never readily test in the real world. That’s why many years ago I laid out a plan for an open source simulator because I think this would be a great thing for everybody to contribute to. An open source simulator would soon feature every crazy situation anybody has even seen or thought of, and it would make all cars safer.

But simulation is just simulation, and it only features what people have thought of. You need more.

On test tracks

The most conservative, but most expensive and limited approach, is a private test track. And almost everybody starts on a closed testing ground because you need to when things are just getting going. They provide the confidence to eventually go out on the road — and go out on the road you must. You can really only do a fraction of a percent of your testing on test tracks, even with robot slave cars to drive along with. You just won’t encounter enough variation in terrain and driving situations, and you can only test a few cars at a time.

On the plus side, on test tracks you can use unmanned cars, reducing some costs, and you can run 24 hours/day.

There is a hybrid of simulator and test track called “Vehicle in the Loop” which puts the car on a dyno in a building while robots whiz around to provide relative motion. That requires less real estate and is safer, but automated test track operation is probably better at this point.

On the road with trained professional safety drivers

All companies eventually need to go out on real roads and deal with real situations. They do it by having paid professionals, usually with special training, who sit behind the wheel watching the vehicle and ready to take the controls at any sign of doubt. It is also common to have a second person who is monitoring the computer systems and sensor scans to make sure all is operating properly, and to tell the safety driver to take over if something is wrong. In fact, safety drivers are required for testing self-driving cars (but not autopilots/cruise software) by most of the state laws about self-driving cars.

Companies like Google tell their safety drivers to take control at any sign of doubt. They don’t want them to let the car keep driving to see if it would have braked properly for the children crossing the road. Later, they can use the simulator to replay what the car would have done if the safety driver had not intervened. If the car would have done something wrong, it’s a bug to be fixed at high priority.

Still, even trained safety drivers can make mistakes. Google has reported one accident where their systems made a mistake. The system decided that a bus it was sharing the lane with would let it in to traffic, but the bus did not. The safety driver didn’t intervene because he also judged the bus would let the car in.

In addition, when the systems are very good, safety drivers can also become complacent and not intervene as quickly as they should.

On the road with regular drivers but limited function

You could test a self-driving system by having it do only some of the driving. For example, you could make a system which steers but does not control the brakes. A driver who knows that she has to hit the brakes when approaching something is not going to stop paying attention. With this system you can note when the human hit the brakes, and see if the system also would have, and who was right. Chances are any surprise braking would be combined with taking the wheel. Of course, you would also learn when the driver grabbed the wheel.

Reports from Tesla suggest they are gathering massive amounts of driving data from logs in their cars — 780 million miles of driving, and as much as 100 million miles in autopilot mode. This contrasts with the 1.6 million miles of test operations at Google. Huge numbers, but what do they mean now, and in the future?

As I’ve written before, testing is one of the largest remaining challenges in robocar development — how do you prove to yourself, and then to others, that you’ve reached the desired safety goals? Logging tons of miles are an important component. If car companies can get their customer to do the testing for them, that can be a significant advantage. (As I wrote last week, another group which can get others to do testing are companies like Uber and even operators of large commercial and taxi fleets.) Lots of miles mean lots of testing, lots of learning, and lots of data.

Does Tesla’s quick acquisition of so many miles mean they have lapped Google? The short answer is no, but it suggests a significant threat since Google is, for now, limited to testing with its small fleet and team of professional testing drivers.

Tesla is collecting a lot less data from its cars than Google, orders of magnitude less. First of all, Tesla has a lot fewer sensors and no LIDAR, and to the best of my knowledge from various sources I have spoken to, Tesla is only collecting a fraction of what their sensors gather. To collect all that they gather would be a huge data volume, not one you would send over the cell network, and even over the wifi at home, it would be very noticeable. Instead, reports suggest Tesla is gathering only data on incidents and road features the car did not expect or did not handle well. However, nothing stops them in the future from logging more, though they might want to get approval from owners to use all that bandwidth.

Tesla intends to make a car for people to buy today. As such, it has no LIDAR, because a car today, and even the autopilot, can be done without LIDAR. Tomorrow’s LIDARs will be cheap but today’s production LIDARs for cars are simple and/or expensive. So while the real production door-to-door self-driving car almost certainly uses LIDAR, Tesla is unable and unwilling to test and develop with it. (Of course, they can also argue that in a few years, neural networks will be good enough to eliminate the need for LIDAR. That’s not impossible, but it’s a risky bet. The first cars must be built in a safety-obsessed way, and you’re not going to release the car less safe than you could have made it just to save what will be only a few hundred dollars of cost.)

As noted, Google has been doing their driving with professional safety drivers, who are also recording a lot of data from the human perspective that ordinary drivers never will. That isn’t 100 times better, but it’s pretty significant.

Tesla is also taking a risk, and this has shown up in a few crashes. Their customers are beta testing a product that’s not yet entirely safe. In fact, it was a pretty bold move to do this, and it’s less likely that the big car companies would have turned their customers into beta testers — at least no until forced by Tesla. If they do, then the big automakers have even more customers than Tesla, and they can rack up even more miles of testing and data gathering.

When it comes to training neural networks, ordinary drivers can provide a lot of useful data. That’s why Commma.ai, who I wrote about earlier, is even asking volunteers to put a smartphone on their dash facing out to get them more training data. At present, this app does not do much, but it will not be hard to make one that offers things like forward collision warning and lane departure warning for free, paid for by the data it gathers.

Watch me on Dateline NBC: On Assignment

On Sunday, June 5, at 7 pm (Eastern and Pacific) the news show Dateline: NBC conducted a segment on self-driving cars featuring Sebastian Thrun, Jay Leno and myself. I sat down for several hours with Harry Smith, but who knows how much actual airtime will be shown. Here is the promo for the episode and another more specific one.

Source: Tesla

Though it’s been anticipated for a few months now, Tesla announced yesterday the release of their highway cruise update for Model S cars from the last year.

Tesla’s offering is not too different from the “highway cruise” other automakers have shown — essentially, a combination of lanekeeping and adaptive cruise control (ACC), both of which have been around for a while. The vehicle reportedly insists you keep your hands touching the wheel, but some owners report you can take them off. Of particular note is the addition of lane-changing, which you can do by flicking the turn signal. You must check behind you, since if you attempt this when the next lane is moving fast and somebody is coming up on you quickly, not knowing this fact could cause a real problem. The vehicle won’t do the lane change if its blind spot detector sees an adjacent car, but it won’t stop you from cutting off somebody who is not in that zone and gaining on you.

I am curious about the claim that the system will “overtake” another car — I have not tried it out yet, but I presume since it should not change lanes on its own, user input will be required to command driving around another car. While you can’t safely make a lane change into a lane where you see no cars, you actually can make the change after passing a car because you know where that car is and that nothing is going to rush at you in the lane.

The release of this and similar products will test a supposition I made earlier, that these products may not be as exciting as hoped. Worse, some drivers may find it a bit frightening to trust giving control to a vehicle knowing that from time to time they will need to grab the wheel. I myself have felt that apprehension when driving cars with adaptive cruise control. For example, since I’m able to see things like a stopped car ahead before the radar or camera system does, I have wondered if the automatic cruise control (ACC) will stop for it.

On the other hand, many people have reported that even though they must supervise, highway cruise can make the trip more relaxing, just as basic cruise control does. The trick is to get your brain into putting focus on its new sole task — to supervise — so that the rest of your brain can relax. With cruise control, you are reasonably able to have one part of your brain worry about steering, and relax the part that was going to worry about speed. So this may happen here.

More of a concern are the people who will trust it too much. Many of us already do crazy things, like texting or playing with things on our phones when we are doing fully manual driving, so it’s a given this will continue to happen in cars with highway cruise. But while it will be safer to take your eyes from the road for a longer period than you should with a manual car, people may magnify that. If you take your eyes off the road and the car ahead of you stops suddenly, the car will very probably brake for you — as will any car with forward collision avoidance — but not 100% of the time.

I may have more to say after taking a ride in one of these. I think the first really interesting product won’t be this, but a more full-auto traffic jam assist, that will drive for you in a traffic jam and allow you to take your attention off the road entirely to read or work on your phone or computer.

At the low speeds of a traffic jam, boxed in by other cars, the driving problem is much simpler. You don’t need to see the lanes, you just need to follow the cars. If the car in front of you zooms ahead, the traffic jam is over, and the driver needs to manually speed the car up. Done at low speeds, that transition can be fairly safe, and in addition, if the driver does nothing and the car slows to a halt, that is not unsafe — just annoying — in a breaking up traffic jam.

The main remaining problem in traffic jams is what to do when the cars in front of you change lanes. Are they turning just to go into another lane, or is it because all the lanes are turning due to restriping (common in jams) or an obstacle. You need to get that right, and let the driver know about it, but you can’t buzz the driver every time somebody does a lane change or the product is not useful.

Several car vendors (and probably Tesla) have been working on this, and could release it quite soon if they get the guts and legal approval to do so.

This post originally appeared on robocars.com.

Tesla did a lot of work on building cooling systems for standard cylinder Lithium-Ion cells and was able to make a high performance vehicle. The Model S also by default charges to only 80% of capacity because battery life is hurt by charging all the way to full. In fact, charging to 3.92 volts (about 60%) capacity is the sweet spot. Some of the other things that reduce battery life include:

• Discharging too close to empty
• Getting too warm while discharging
• Getting too warm while charging, and in particular causing thermal expansion, which creates physical damage
• Even ordinary warmth, where the vehicle is stored for long periods, particularly at high charge, is dangerous. The closer to freezing the better, and even above 25 degrees centigrade causes some loss.

The important, but little reported statistic for a battery is the total watt-hours you will be able to get out of it during its usable lifetime. This tells you the lifetime of the battery in miles, and the cost tells you the cost per mile. How important is this? If the Tesla $40,000 battery lasts you 150,000 miles and sells for $10,000 when done, the straight-line cost per mile is 20 cents/mile — more than the cost of gasoline in most cars, and much more than the 3 cent/mile or less cost of electricity.

Humans will drive as humans want to drive, and it’s hard to change that. They will accelerate for both fun and to get ahead of other cars. They will take mixes of short trips and long trips. They don’t know how long their trips will be and demand a flexible vehicle that is always ready for anything.

Electric robotaxis change that game. They will drive predictably, rarely ever demanding quick acceleration. A driver likes zippy fun, a passenger wants a gentle ride. They can go even further, and set their driving pattern based on the temperature of their batteries. Are we making the batteries too warm? Then “cool off,” literally. This applies both to fast starts and also slowing down. Regenerative braking conserves energy and increases range, but doing it too hard heats the batteries. Start slowing down sooner — especially if you have data on what traffic lights and traffic are doing and it can make a big difference.

Robotaxis can always use the sweet spot of the battery charge duty cycle

• You will rarely be sent a robotaxi that, in order to get you, needs to dig deep into its maximum range.
• Often demand is predictable, so if need be, vehicles can be charged above 60% only when such demand is expected or is arising.
• While robotaxis will prefer to charge at night when power is cheapest, they can charge any time to get back up to the optimal level.
• As I’ve noted before, battery swap doesn’t work well for humans, but robots don’t mind making an appointment or driving out of their way for a swap. This makes it easy to use batteries only in the sweet spot, and to charge them only at night on cheap power.
• If battery swap is not an option, there are many options to supplement range during peak demand. Vehicles can go to depots to pick up trunk batteries, battery trailers, or even slot-in units with small motorcycle engines and liquid fuel tanks. If this is cheaper than the alternatives, it’s an option.
• When it gets hot, robotaxis can seek out the shade, or even places with cooling, to keep the batteries from being too warm.

Robotaxis don’t mind the loss of range all that much

As a battery ages, its capacity drops. Humans hate that — having bought a car with a 100 mile range they won’t accept it can now only do 60. For a human, that means time to replace the battery. For a robotaxi, that just means you have a shorter range, and you don’t get sent on long range trips. Or you may decide that while before, you only charged to 60% to get maximum battery life, now you charge more, knowing it will eat the remaining life, but getting the most out of the battery.

Of course, as the range drops, now you run into another problem. You’re carrying around the extra weight of battery for half the range, and it’s costing you energy to do that, especially in an ultralight car where the battery is the biggest component of the weight. (This also enters into the math of whether it makes sense to charge only to 60%.) Eventually the time comes that the battery is not practical. This is the time to sell it. Tesla and others are working to produce a home and grid storage market for used car batteries. In those applications, the weight doesn’t matter, just the cost for the remaining lifetime watt-hours. You care about the capacity, but you pay a market price for it.

Eventually, even this is not practical and you scrap to recycle the materials.

Typical predictions for Lithium-Ion run from 500 to 1,000 cycles. Tesla’s techniques seem to be beating that. With robotaxis, who knows just how many lifetime kwh we’ll be able to get out of these batteries, or perhaps even other chemistries. Turns out that human drivers like a chemistry that keeps its life as long as possible then falls off a cliff. Slow decline is harder to sell — but slow decline chemistries, like Lithium Iron Phosphate and others could make more sense for the robots that don’t care.

Grid storage?

It’s often suggested that electric cars could be used as grid storage. Problem is, with car batteries today, it costs around 15 cents to put a kwh into a battery and get it out. That means to be grid storage, you need to have the spot price on the grid be the price you bought at, plus 15 cents, plus a margin to make it worth this. Night power can get as low as 6 cents, so this does happen, but not as much as one might hope. The problem is that the grid’s peak demand is around 4 to 7pm, which is also a peak time for driving. That’s the last time most car owners will want to drain off their batteries to make a bit of money on the power. You will only do that if you know you won’t be using the car. For a robotaxi fleet, that might be the case. Of course, selling power to the grid you will do it only at a rate that does not harm your battery or warm it up too much.

When the grid gets to a super peak, the price can really spike to attractive numbers. That’s because building extra power plant capacity just for those rare days is expensive, and so almost any price is better. Here we could talk about cars as storage, when we know their batteries are not going to be used. That’s even more true of batteries sitting in a battery swap facility.

Tesla Motors autopilot (photo:Tesla)

An interesting article in last week’s Wall Street Journal spawned a series of unfortunate headlines (in a variety of publications) suggesting that Tesla had somehow “solved” the “problem” of “liability” by requiring that human drivers manually instruct the company’s autopilot to complete otherwise-automated lane changes.

(I have not asked Tesla what specifically it plans for its autopilot or what technical and legal analyses underlie its design decisions. The initial report may not and should not be the full story.)

For many reasons, these are silly headlines.

First, “liability” is a word that nonlawyers often use in clumsy reference to either (a) the existence of vague legal or policy questions related to the increasing automation of driving or (b) the specific but unhelpful question of “who is liable when an automated vehicle crashes.”

If this term is actually meant to refer, however imperfectly, to “driver compliance with the rules of the road,” then it is not clear to what problem or for what reason this lane-change requirement is a “solution.” For a foundational analysis of legality, including my recommendation that both developers and regulators carefully examine applicable vehicle codes, please see Automated Vehicles Are Probably Legal in the United States.

This term probably refers instead to some concept of fault following a crash. As a forthcoming paper explains, liability (or the broader notion of “responsibility”) is multifaceted:

Moreover, although liability is frequently posited as an either/or proposition (“either the manufacturer or the driver”), it is rarely binary. In terms of just civil (that is, noncriminal) liability, in a single incident a vehicle owner could be liable for failing to properly maintain her vehicle, a driver could be liable for improperly using the vehicle’s automation features, a manufacturer could be liable for failing to adequately warn the user, a sensor supplier could be liable for poorly manufacturing a spotty laser scanner, and a mapping data provider could be liable for providing incorrect roadway data. Or not.

The relative liability of various actors will depend on the particular facts of the particular incident. Although automation will certainly impact civil liability in both theory and practice, merely asking “who is liable in tomorrow’s automated crashes” in the abstract is like asking “who is liable in today’s conventional crashes.” The answer is: “It depends.”

Second, the only way to truly “solve” civil liability is to prevent the underlying injuries from occurring. If automated features actually improve traffic safety, they will help in this regard. As a technical matter, however, it is doubtful that requiring drivers merely to initiate lane changes will ensure that they are engaged enough in driving that they are able to quickly respond to the variety of bizarre situations that routinely occur on our roads. This is one of the difficult human-factors issues present in what I call the “mushy middle” of automation, principally SAE levels 2 and 3.

These levels of automation help ground discussions about the respective roles of the human driver and the automated driving system.

At SAE level 2, the human driver is still expected, as a technical matter, to monitor the driving environment and to immediately respond as needed. If this is Tesla’s expectation as well, then the lane-change requirement alone is manifestly inadequate: Many drivers will almost certainly consciously or unconsciously disengage from actively driving for long stretches without lane changes.

At SAE level 3, the human driver is no longer expected, as a technical matter, to monitor the driving environment but is expected to resume actively driving within an appropriate time after the automated system requests that she do so. This level raises difficult questions about ensuring and managing this human-machine transition. But it is technically daunting in another way as well: The vehicle must be able to drive itself for the several seconds it may take the human driver to effectively reengage. If this is ultimately Tesla’s claim, then it represents a significant technological breakthrough—and one reasonably subject to requests for substantiation.

Third, these two levels of automation raise specific liability issues that are not necessarily clarified by a lane-change requirement.

At SAE level 2, driver distraction is entirely foreseeable, and in some states Tesla might be held civilly liable for facilitating this distraction, for inadequately warning against it, or for inadequately designing against it. Victims of a crash involving a distracted Tesla driver would surely point to more robust (though hardly infallible) ways of monitoring driver attention—like requiring occasional contact with the steering wheel or monitoring eye movements—as proof that arguably safer designs are readily available.

At SAE level 3, Tesla would be asserting that its autopilot system could handle the demands of driving—from the routine to the rare—without a human driver monitoring the roadway or intervening immediately when needed. If this is the case, then Tesla may nonetheless face significant liability, particularly in the event of crashes that could have been prevented either by human drivers or by better automated systems.

Fourth, this potential liability does not preclude the development or deployment of these technologies—for reasons that I discuss here and here and here.

In summary, merely requiring drivers to initiate lane changes raises both technical and legal questions.

However, the idea that there can be technological fixes to legal quandaries is a powerful one that is explored in Part III of Proximity-Driven Liability.

A related example illustrates this idea. In the United States, automakers are generally required to design their vehicles to be reasonably safe for unbelted as well as belted occupants. (In contrast, European regulators assume that anyone who cares about their safety will wear their seatbelt.) In addition, some US states restrict whether or for what purpose a defendant automaker can introduce evidence that an injured plaintiff was not wearing her seatbelt.

An understandable legal response would be to update these laws for the 21st Century. Regardless, some developers of automated systems seem poised to adopt a technological response: Their automated systems will simply refuse to operate unless all of the vehicle’s occupants are wearing their seatbelts. Given that seatbelt use can cut the chance of severe injury or death by about half, this is a promising approach that must not be allowed to suffer the same fate as the interlocks of 40 years ago.

More broadly, any automaker must consider the complex interactions among design, law, use, and safety. As I argue in a forthcoming paper by the same name, Lawyers and Engineers Should Speak the Same Robot Language.

Delphi has made waves by undertaking a road trip from San Francisco to New York in their test car, which is equipped with an impressive array of sensors. The trip is now underway, and on their page you can see lots of videos of the vehicle along the trek.

The Delphi vehicle is one of the most sensor-laden vehicles out there, and that’s good. In spite of all those who make the rather odd claim that they want to build robocars with fewer sensors, Moore’s Law and other principles teach us that the right procedure is to throw everything you can at the problem today, because those sensors will be cheap when it comes time to actually ship — especially for those who say they won’t ship for a decade.

At the same time, according to Kristen Kinley at Delphi, their test involves mostly highway driving, with very minimal urban street driving. They are attempting off-map driving, which is possible on highways due to their much simpler environment. Like all testing projects these days, there are safety drivers in the cars ready to intervene at the first sign of a problem.

Delphi is doing a small amount of dedicated short-range communications (DSRC) vehicle-to-infrastructure testing as well, though this is only done in Mountain View where they used some specially installed roadside radio infrastructure equipment.

Delphi is doing the right thing here — getting lots of miles and different roads under their belt. This is Google’s giant advantage today. Based on Google’s announcements, they have more than a million miles of testing in the can, and that makes a big difference.

Hype and reality of Tesla’s autopilot announcement

Telsa has announced they will do an over-the-air upgrade of car software in a few months to add autopilot functionality to existing models that have sufficient sensors. This autopilot is the “supervised” class of self driving that I warned may end up being seen as boring. The press have treated this as something immense, but as far as I can tell, this is similar to products built by Mercedes, BMW, Audi and several other companies, and even sold in the market (at least for traffic jams) for a couple of years now.

The other products have shied away from doing full highway speed in commercial products, though rumours exist of it being available in commercial cars in Europe. What is special about Tesla’s offering is that it will be the first car sold in the US to do this at highway speed, and they may offer supervised lane change as well. It’s also interesting that, since they have been planning this for a while, it will come as a software upgrade to people who bought their technology package earlier.

UK project budget rises to £100 million

What started with a £10 million pound prize in the UK has grown become over 100m in grants in the latest UK budget. While government research labs will not provide us with the final solutions, this money will probably create some very useful tools and results for the private players to exploit.

MobilEye releases their EyeQ4 chip

MobilEye from Jerusalem is probably the leader in automotive machine vision, and their new generation chip has been launched, but won’t show up in cars for a few years. It’s an ASIC packed with hardware and processor cores aimed at doing easy machine vision. My personal judgement is that this is not sufficient for robocar driving, but MobilEye wants to prove me wrong. (The EQ4 chip does have software to do sensor fusion with LIDAR and Radar, so they don’t want to prove me entirely wrong.) Even if they are not good enough on their own, ME chips offer a good alternate path for redundancy.

Chris Urmson gives a TED talk about the Google Car

Talks by Google’s team are rare — the project is unusual in trying to play down its publicity. I was not at TED, but reports from there suggest Chris did not reveal a great deal new, other than repeating his goal of having the cars be in practical service before his son turns 16. Of course, humans will be driving for a long time after robocars start becoming common on the roads, but it is true that we will eventually see teens who would have gotten a licence never get around to getting one. (Teens are already waiting longer to get their licences, so this is not a hard prediction.)

The war between DSRC and more wifi is heating up.

Two years ago, the FCC warned that since auto makers had not figured out how to benefit from the DSRC spectrum at 5.9ghz, it was time to repurpose it for unlicensed use, like more WiFi. There is now a bill to force this proposition.

I think this would be the best thing ever for V2V, and I outlined 2 years ago how doing V2V in phones instead of cars is far more likely to get success. Phones are replaced every two years with a whole new generation, and smartphones are already in 75% of cars in the USA. V2V, even under a mandate, won’t get close to that until after 2030, and by that time it will be 15-year-old technology, long obsolete.

Google Airbags on the outside

A Google patent for placing airbags of a sort on the outside of vehicles to protect others has gotten a lot of press. While I was working at Google, I started the push for this (though the design in the patent is not mine.) I’m glad to see more investigation of it, though. The reality, as everybody knows, is that perfection is not possible, so planning for the hopefully small number of inevitable accidents and mitigating them is the right strategy.

Nvidia

For the past year, Nvidia has been pitching their latest Tegra GPU in the mobile space. They argue that a general purpose chip with very high power, thanks to their giant research budget, will produce better results than a dedicated machine vision chip from a smaller company like MobilEye that takes several years between generations. As a result, they put a big focus on automotive and self-driving at their recent GPU conference. I was in Europe and unable to attend.

More news from China

Reports suggest that Chinese internet companies Baidu and Tencent (WeChat) are starting robocar projects. (Previously Baidu provided some maps for BMW’s projects.) I’ve spoken at a couple of events for Baidu and encouraged this direction. Details are scarce though, and there is some speculation this is more in the “smart” car direction (similar to Tesla) than a full robocar. We should note that the legal situation in China is a lot different, and these large companies might be able to easily get more advanced products to be declared legal well before those in other countries can.

BMW crash

A BMW driver testing night vision collision avoidance systems crashed into a police van in Stuttgart. This was not a robocar, and of course it is amusing that if you have a crash, you might as well do it into a speeding police van! It does raise issues about risks of testing things with safety drivers — in this case, perhaps the test driver was distracted by his testing and didn’t pay enough attention to the environment. Most of the supervised class vehicles being developed today require the driver to keep paying attention, something that Google has decided not to pursue.

This article first appeared on robocars.com.

Tesla is certainly an important company to watch. As the first successful start-up car company in the USA, they are showing they know how to do things differently, taking advantage of the fact that they don’t have a baked in knowledge of “how a car company works” the way other companies do.

Tesla’s announcements of plans for more self-driving features are important. Unfortunately, the announcements around the new dual-motor Model S involve offerings quite similar to what can be found already in cars from Mercedes, Audi and a few others. Namely advanced ADAS and the combination of lane-keeping and adaptive cruise control to provide a hands-off cruise control where you must keep your eyes on the road.

One notable feature demonstrated by Tesla is automatic lane change, which you trigger by hitting a turn signal. That’s a good interface, but it must be made clear to people that they still have the duty to check that it’s safe to change lanes. It’s not that easy for a robocar’s sensors, especially the limited sensor package in the Telsa, to see a car coming up fast behind you in the next lane. On some highways relative speeds can get pretty high. You’re not likely to be hit by such cars, but in some cases that’s because they will probably brake for you, not because you did a fully safe lane change.

Much more interesting is Elon Musk’s prediction of a real self-driving car in 5 to 6 years. He means one where you can read a book, or even, as he suggests, go to sleep. Going to sleep is one of the greatest challenges, almost as hard as operating unmanned or carrying a drunk or disabled person. You won’t likely do that just with cameras — but 5 to 6 years is a good amount of time for a company like Tesla.

Another unusual thing about Tesla is that while they are talking about robocars a lot, they have also built one of the finest driver’s cars ever made. The Model S is great fun to drive, and has what I call a “telepathic” interface sometimes — the motors have so much torque that you can almost think about where you want to go and the vehicle makes it happen. (Other examples of telepathic interfaces include touch-typing and a stickshift.) In some ways it is the last car that people might want to automate. But it’s also a luxury vehicle, and that makes self-driving desirable too.

Audi Racing

Another recent announcement creating buzz is Audi’s self-driving race car on a test track in Germany. Audi has done racing demos several times now. They are both important but also unimportant. It definitely makes sense to study how to control a car in extreme, high performance situations. To understand the physics of the tires so fully that you can compete in racing will teach lessons of use in danger situations (like accidents) or certain types of bad weather.

http://youtu.be/eOYsI1cqUrw

At the same time, real-world driving is not like racing, and nobody is going to be doing race-like driving on ordinary streets in their robocar. 99.9999% of driving consists of “staying in your lane” and some other basic maneuvers and so racing is fun and sexy but not actually very high on the priority list. (Not that teams don’t deserve to spend some of their time on a bit of fun and glory.) The real work of building robocars involves putting them through all the real-world road situations you can put them through, both real, and in some cases, simulated on a track or in a computer.

Google first showed its system to many people by having it race figure-8s on the roof parking lot at the TeD conference. The car followed a course through a group of cones at pretty decent speeds and wowed the crowd with the tight turns. What most of the crowd didn’t know was that the cones were only there for show, largely. The car was guiding itself from its map of all the other physical things in the parking lot — line markers, pavement defects and more. The car is able to localize itself fine from those things. The cones just showed the public that it really was following the planned course. At the same time, making a car do that is something that was accomplished decades ago, and is used routinely to run “dummy cars” on car company test tracks.

A real demo turns out to be very boring, because that’s how being driven should be.

This doesn’t mean we won’t see more impressive demos soon. Many people have shown off automatic braking. Eventually we will see demos of how vehicles respond in danger situations — accidents, pedestrians crossing into the road and the like. A tiny part of driving but naturally one we care about. And we will want them to understand the physics of what the tires and vehicle are capable of so that they perform well, but not so they can find the most efficient driving line on the track.

There was some debate about having a new self-driving car contest like the DARPA grand challenges, and a popular idea was man vs. machine, including racing. That would have been exciting. We asked ourselves whether a robot might have an advantage because it would have no fear of dying. (It might have some “fear” of smashing its owners very expensive car.) Turns out this happens on the racetrack fairly often with new drivers who try to get an edge by driving like they have no fear, that they will win all games of chicken. When this happens, the other drivers get together to teach that new driver a lesson. A lesson about cooperating and reciprocation in passing and drafting. So the robots would need to be programmed with that as well, or their owners would find a lot of expensive crashes and few victories.

This article originally appeared on robocars.com.

[AK] With investments in two accelerators now (Bolt in US and Navigator Campus in Russia), as in well as a range of robotics companies, Grishin Robotics has a bird’s eye view of robot startups. What do you think about the recent high profile announcements about robotics investments … how will this impact the start-up scene? 

[VK] It’s definitely great news. Amazon, for example, announced in a very significant way their interest in robotics a few years ago when they bought Kiva for $775 million. So this isn’t the first time that big corporations not traditionally involved in robotics are starting to pay attention to the field. It really  begun at least in 2012, when Grishin started to invest into robotics startups as well.

However, this time it’s even more about perception. These high profile media announcements are getting picked up by the general public, investors, and entrepreneurs – they attract attention to the industry. No doubts, it is important from an entrepreneurial perspective – there should be more success stories for them, more ‘benchmarks’.

These days many talented mechanical and electrical engineering students and professionals (particularly in the Bay Area) are attracted by the large salaries being offered by big internet companies like Facebook and therefore decide to switch into software and postpone their startup, hardware and robotics dreams indefinitely.

But when they hear these big announcements about new products or M&As, they realize the market is actually happening and they begin to believe that they can start their own company … they don’t have to go work for a big corporation. They can continue to work in a profession that they like and have expertise in. So from that perspective, these announcements are very important.

From the investors’ perspective, it’s largely the same. They need inspiration, they need proof, they need exits, and they need public companies. Of course there are already public companies like iRobot , but still we don’t hear about robotics acquisitions or IPOs every day – not yet. The announcements of the Amazon drones and the Google acquisitions of Nest and other companies really attracted the attention of the venture and finance communities and reminded them that something big is happening. It forced them to pay attention … it forced them to think that maybe another one or two years is too long to wait to make their bet on this space. And that’s really great news.

Google have invested in quite an assortment of robot startups, particularly if you include their investments and resources in AI and NLP. What do you think their plans are?

Interesting question to speculate on. Google itself said that their primary interest at this point lies in the area of manufacturing. But the acquisition of Nest is definitely a consumer play. Frankly, at this point, I think, it may be not so much about any coherent behind-the-scenes grand strategy as much as that they are starting to feel like something big is going to happen to follow the revolutions that came with the personal computer, the internet and the smart phone. Many people are seeing signs that the next big revolution may be robotics.

For Google, Amazon etc. this may be more about announcing some kind of interest in this space and to jump into the game as early as possible. I think there’s a lot of R&D going on internally to figure out where and how exactly to go and approach it.

So while it’s definitely exciting times and more and more startups are launching with interesting ideas, there are still some unknown variables. It’s important for big corporations like Google to figure out the unknowns in this multi-billion dollar market as early as possible, and to do this before the competition does.

Do you see any down side to large companies getting involved in robotics?

Despite all these announcements the market is still at an early stage of development, so the really fierce competition is ahead. It’s not that the market is divided between big companies, after all these news – far from that. Just as a comparison … it’s difficult to launch another smart phone in the market today because the market is fairly mature and is divided between the big companies. But in the robotics market, and in the consumer robotics market specifically, it’s the opposite.

The barrier of entry for robotics startups has been getting lower and lower the last few years. And all these announcements lead to increased investor interest, which means more money for robotics and hardware startups … more investors ready to consider these kinds of companies in addition to the traditional Internet and mobile companies. I just can’t think of any significant downside – at least not at this point and not for the next two or three years.

Possible problems could include losing top talent, or losing access to components or research, or support for community and open source projects, which all have impact on research and startups …

All good points and let’s consider them one by one.

Regarding top talent, all these companies that were bought are fairly small. I deeply respect all these people and I know many of them, but I hope these acquisitions will inspire more robot students or engineers to go out there and start their robotics companies, or it will inspire younger students to select robotics when they pick their majors at universities. I think that the cumulative positive impact will be so much greater than the loss of these people to big corporations.

I also agree that it’s startups that are primarily innovating in new markets rather than the big companies … but it’s not like these people won’t continue to do research. And Google has a lot of money and resources and these people will continue their work with other talents they meet there. It’s very possible that these talented people will be able to accomplish more given the vast resources they’ll have access to inside these big companies. So I urge you consider not just the negative side of this, but the  positive sides as well – word ‘loss’ has too much of a negative connotation, in my opinion.

The second concern you mention is lack of access to components …

Yes, to components, ROS, research findings…

Not necessarily. When big company is buying component technology, of course, it often ceases to be open-source. Discussion of open-source ideals are wholly different topic, but my point is that it is not necessarily means decreased access – may be quite the opposite. Often there is an interest of a big company to make certain technology a dominant platform, therefore, great investments are made to attract more developers to it, improve it, explain benefits etc. And if it will turn out not to be the best platform – competitors or same open-source community will be quick to offer alternatives, especially, again, at this stage of market development, when barriers to entry are quite low.

If you could ask Andy Rubin any question what would it be?

One of the main challenges for consumer robotics market – whether we are talking about connected devices and the Internet of Things or more traditional consumer robotics like Roomba – it pretty much comes down to applications and business models. We can imagine some of the interesting applications or verticals where robotics could make a significant impact, but we can’t always imagine all the possible  areas and ways where robotics can be applied to make our lives better – there are just so many of them, and many are far from being obvious. So I would be interested to know how he thinks robotics can make our lives better in a significant fashion, beyond the usual suspects we already know about, like agriculture, drones etc.

And the same question about business models can be asked from a different perspective: at the moment hardware companies are selling hardware, but it’s getting commoditized … and you have to think about software and data and the cloud connectivity of the hardware components you are making. So, how can business models that are familiar to Google (like advertising, subscriptions, internet) be applied to robotics and consumer hardware companies? And of course, what kinds of new business models will emerge as a result?

We’ve talked about Google and Amazon. Who are the other big new players in the robotics space that are known for things other than robotics but are starting to have an impact in the field …

There may be many more than we are aware of. Like Tesla – of course it’s an automobile company but more and more its top management is starting to say that they are essentially an IT company, considering how connected the car is and how advanced the driving functions and so on. And the same goes with the other automobile companies that are working on self-driving cars.

And the problems they are solving, whether they are BMW or someone else, are very related to the problems the robotics industry in general is facing right now. The advancement of computer vision is basically what the whole robotics industry is reliant on in terms of future growth and potential. Also autonomous capabilities … We’re not talking about the holy grail of full autonomy any time in the foreseeable future, but the algorithms and the decision-making processes needed for these new (semi) autonomous cars may spill over into the startup world tomorrow. Eventually it will impact the whole market in a very significant and positive way.

Many of the big corporations today may essentially be involved in robotics already. For example Foxconn and other manufacturing companies are starting to use robots. And more and more companies in agriculture and mining are beginning to use unmanned vehicles. They are starting to understand that adopting these technologies earlier than their competitors is a necessity if they are going to survive. Robotic capabilities are becoming the basis of their long-term competitive advantage.

To make a comparison, almost everyone involved in physical commerce today is finding that their internet capabilities are getting closer and closer to the heart of their business. I think we will find the same thing happening with robotics. It’s not so much about the specific companies’ names we can come up with today … because in the future it will be almost everybody, just like how almost everybody these days is involved in the internet or is using software or computers.

What do you think are the most interesting recent robotics acquisitions?

It depends on how you define robotics. Is Nest as a robotics company? I for one define it as a consumer robotics company but I can imagine that many people will not agree with me. I believe everybody will eventually agree with this definition but not maybe today. I think the Deep Mind announcement is also very interesting. AI and related software is a very important enablers of robotics.

What do you think about Dyson’s announcement of the $8million (5 million pound) UK research facility specifically into home robots?

It’s an awesome announcement. Dyson himself is an iconic figure, as well as his company. They have been making products that we are used to seeing both in the home and the office, and they’ve been able to make them differently, both in terms of design and in terms of capability. And it’s great when such a company pays significant attention to robotics.

The thing is – it’s not enough to make a good-looking coffee maker or washing machine anymore because people are getting more and more used to smart devices in their homes and cars. Increasingly they are expecting smart capabilities for the same amount of money. And it is possible now to deliver a much smarter device – with Internet connectivity – for almost the same price as a so-called ‘dumb device’.

All this attention is just another confirmation that very soon the fact that you are making smart devices with robotic capabilities will stop to be the competitive advantage and will start becoming the threshold capability. I think Dyson is foreseeing this (probably somewhat ahead of their competitors) – but  eventually everyone will follow.

Grishin Robotics is investing in consumer robotics… Do the recent announcements change any of your plans? 

No. There are a lot of very interesting application areas and verticals in the robotics market, whether it’s health care, agriculture, or manufacturing, but Grishin focuses on consumer robotics simply because we see huge potential, both from the market perspective and from the perspective of having a positive impact on people’s lives. This potential is much greater when we’re talking about robotic products that the average person can buy for $100-$200, and when there are so many prospective customers out there. It’s really all about impact. Big corporations and governments have been benefiting from robotics for so long, but average people have yet to benefit. We think it’s time to change that.

You could say, then, that the recent announcements have validated Grishin Robotics’ interests …

Exactly. In June 2012, when Dmitry Grishin launched the firm, a fair number of people told us that we were absolutely crazy. When so many smart people are telling you that you’re crazy, you have to think that you may not necessarily be right, but at least you’re probably doing something interesting. But when in the end it turns out that you are even may be right … it does feel good.

 [Edited by Hallie Siegel]

If you liked this post, you may also be interested in:

• Robot Launch 2014: The first global startup competition for robotics is here!
• Doggone it, there are no exciting consumer robots
• Dmitry Grishin on evaluating robotic investment opportunities
• Accelerators and opportunities for robotics startups
• Robots Podcast: Grishin Robotics

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.