Source: Intel

Intel is establishing an autonomous driving division; hacker George Hotz is open-sourcing his self-driving software in a bid to become a network company; LiDAR and distancing devices are changing. What does it all mean?

Intel

Semiconductor giant Intel, in a strategic move to become THE supplier of chips to the fast-emerging self-driving industry, is re-organizing and creating a new division. Fortune magazine said:

“The decision is a bet not just on a future of connected devices; it’s also a wager that vehicles will be the centerpiece or at least a core component in the world of IoT. Intel CEO Brian Krzanich has predicted that by 2020, the Internet of Things will include 50 billion devices and each user of those gadgets will generate 1.5 gigabytes of data every day. But, he has said, the average autonomous car will create about 40 gigabytes of data each minute. That’s a lot of data to process and who better to handle the load? Intel.”

Intel in recent months has made key acquisitions and strategic partnerships in pursuit of their autonomous driving division’s plans:

• A partnership with German automaker BMW and Israeli Mobileye to produce self-driving cars for city streets. The companies plan to develop the technology as an open platform that can be used by other automakers or ride-sharing companies.
• Delphi and Mobileye announced they will use Intel’s “system on chip” for autonomous vehicle systems.
• Intel acquired three companies with related products and plans for self-driving cars:
  1. Yogitech is a semiconductor design firm that specializes in adding safety functions to chips used in self-driving cars and other autonomous devices.
  2. Arynga, which was acquired by Intel’s Wind River Unit, develops software that enables vehicles to received over-the-air updates—a capability used frequently by automaker Tesla to enhance performance and fix security bugs in its cars.
  3. Itseez, a company that specializes in machine vision technology that lets computers see and understand their surroundings – a necessity for autonomous vehicles which need to be able to perceive the world around it.

Comma.ai and George Hotz

In a move toward becoming a network provider, and in response to a recent NHTSA letter which caused Hotz to cancel his self-driving aftermarket kit, Hotz announced his plans to open his software code and hardware designs to the world. On November 30th all of the code and plans were posted on Github.

Hotz is encouraging people to build their own self-driving kits. His software enables adaptive cruise control and lane-keeping assist for certain Honda and Acura models.

Fortune magazine quoted Hotz as saying:

“The future is not hardware. If you look at a lot of companies’ business models now it has a lot more to do with ownership of networks than hardware sales. We want to make this as easy as possible for as many people as possible,” Hotz told Fortune, noting that the company even redesigned parts of the Comma Neo to make it easier for hobbyists to build.

nuTonomy

NuTonomy, an MIT spin-off and autonomous vehicle systems startup that’s been testing its technology in Singapore, announced it will begin testing its self-driving Renault Zoe electric vehicles in an industrial park in South Boston in the next few weeks. Although the area is a public area, NuTonomy will, because of the nature of the industrial park, still be somewhat removed from normal vehicle and pedestrian traffic.

Credit: nutonomy

Oryx Vision

Oryx Vision, an Israeli startup which recently got $17 million in a Series A funding round, shows off its new coherent optical radar which uses a laser to illuminate the road ahead, but radar to treat reflected signals as a wave rather than a particle. In an article in IEEE Spectrum, it was revealed that:

“The laser in question is a long-wave infrared laser, also called a terahertz laser because of the frequency at which it operates. Because human eyes cannot focus light at this frequency, Oryx can use higher power levels than can lidar designers. Long-wave infrared light is also poorly absorbed by water and represents only a tiny fraction of the solar radiation that reaches Earth. This means the system should not be blinded by fog or direct sunlight, as lidar systems and cameras can be.”

One of the potential cost savings of Oryx’s technology comes from the fact that its laser does not need to be steered with mechanical mirrors or a phased array in order to capture a scene. Simple optics spread the laser beam to illuminate a wide swathe in front of the vehicle. (Oryx would not say what the system’s field of view will be, but if it is not 360 degrees like rooftop lidars, a car will need multiple Oryx units facing in different directions.)

The clever bit—and what has prevented anyone from building a terahertz lidar before now—is what happens when the light bounces back to the sensor. A second set of optics direct the incoming light onto a large number of microscopic rectifying nanoantennas. These are what Oryx’s co-founder, David Ben-Bassat, has spent the past six years developing.

“Today, radars can see to 150- or 200 meters, but they don’t have enough resolution. Lidar provides great resolution but is limited in range to about 60 meters, and to as little as 30 meters in direct sunlight,” says Wellingstein. He expects Oryx’s coherent optical radar to accurately locate debris in the road at 60 meters, pedestrians at 100 meters, and motorcycles at 150 meters—significantly better than the performance of today’s sensor systems.

Regarding other industry improvements and inventions, the IEEE Spectrum article also said:

“But don’t write off lidar just yet. Following a $150 million investment by Ford and Chinese search giant Baidu this summer, Velodyne expects an “exponential increase in lidar sensor deployments.” The cost of the technology is still dropping, with Velodyne estimating that one of its newer small-form lidars will cost just $500 when made at scale [at present, the price on the website is $7,999 for the new PUCK shown on the right].

Solid-state lidars are also on the horizon, with manufacturers including Quanergy, Innoluce, and another Israeli start-up, Innoviz, hinting at sub-$100 devices. Osram recently said that it could have a solid-state automotive lidar on the market as soon as 2018, with an eventual target price of under $50. Its credit card–size lidar may be able to detect pedestrians at distances up to 70 meters.

That’s not quite as good as Oryx promises but is probably fine for all but the very fastest highway driving. The window for disrupting lidar’s grip on autonomous vehicles is closing fast.”

What’s it all mean:

Fast-emerging may be a catch-all phrase but it seems appropriate to the various parts of the self-driving vehicle marketplace. There are multiple stakeholders about to be disrupted, as I wrote about in April:

“The stakeholders are numerous and the effects can be devastating. Think secretaries, airline reservationists, and all the others disrupted during the emergence of the digital era. The transition to self-driving vehicles and car-sharing systems is likely to cause similar worldwide disruptions.

Consider the insurance industry. If accident rates go down by 90%, as many are predicting, premiums will need to go down too because the reserves for payouts, which are built into the rates, will go down as well. Also, there will likely be fewer insured drivers and car owners thereby lowering the pool of insured people.

Consider hotels and motels and their real estate values and employees – particularly those along the major highways. Why stop at a motel when you can snooze comfortably in a long-haul high-speed driverless car? This will likely affect short-haul airlines and airline employees as well.

Think about professional drivers – truckers, taxies, limos, buses, shuttles. It is expected that many will be replaced by on-demand point-to-point self-driving devices.”

The list of stakeholders also goes on in the auto manufacturing industry and all the Tier 1, 2, 3 and 4 component providers and wannabees. As self-driving vehicles show their needs – for example, the need for low-cost distancing vision systems – new products will be developed and new vendors could replace old ones, hence the posturing by Velodyne, LeddarTech, Quanergy, Bosch, Valeo and all the other LiDAR providers.

As the technologies emerge, it becomes clear that vehicles will be the core component in a larger connected world, streaming massive amounts of data to be analyzed in ways we haven’t even fathomed yet, hence Intel’s new division and the new chips from Nvidia, Qualcomm and others. It’s a fast-emerging marketplace on multiple fronts.

Times and our definition of cars are changing day by day – and fast emerging – a true moving target!

George “Geohot” Hotz Presents the Comma One at Disrupt SF. Source: TechCrunch/YouTube

Silicon Valley and other technology centers have their share of brilliant minds. Some of them have similarly outstanding egos. A few of those have very short fuses. Such is the story of George Hotz.

Source: YouTube

Hotz is certainly brilliant. He was the first person to unlock the iPhone and hack the PlayStation3. Recently his 2,000-line deep learning code, plus car hacks to capture the car’s digital systems, and his driving model, has learned how and written code to drive a car. He demonstrated this to a Bloomberg reporter earlier this year after which he got some funding, hired a team of engineers and turned that software into an inexpensive product – the green thing in his hand – which he promoted at TechCrunch Disrupt SF in September. The green device, the Comma One, is a $999 aftermarket add-on kit with a $24 monthly subscription fee for select Honda and Acura cars. The device replaces the rear-view mirror and captures the car’s radar, steering, braking system and gas pedal. It also adds a camera.

Hotz said that there were 750 of these devices at work in Beta test vehicles around the area, state and country.

The TechCrunch video is in two parts: Hotz’s 7-minute product presentation and an in-depth Q&A with Darrell Etherington which is also informative and worth watching.

Part of Hotz’s talk was a review and critique of other companies making self-driving kits and self-driving cars. “Shippability” was his criteria and by that token, only Tesla has shipped and delivered a self-driving car. He was particularly hostile to the Cruise Automation people saying that they had sold out their original intention to build and sell an aftermarket kit. [For another spin on Cruise, see below.] The most politically correct thing I can say about that portion of his presentation was that his remarks were youthful.

Enter the NHTSA (National Highway Transportation Safety Agency) with a strongly worded query and special order about the safety of Comma One.

“Recently, NHTSA has become aware that your company intends to sell a product, the “comma one,” that you claim allows certain Honda vehicles (and potentially certain Acura vehicles) to operate in a semi-autonomous mode. In a blog post dated October 20, 2016, you described your product as an “advanced driver assist system,” or “in traditional auto manufacturer terms: as “lane keeping assist” and “adaptive cruise control.” You assert that the product “should be legal.”

“This “aftermarket upgrade,” as you describe it, is replacement motor vehicle equipment and your company is a manufacturer of motor vehicle equipment subject to the requirements of the National Traffic and Motor Vehicle Safety Act (Safety Act), 49 U.S.C. Chapter 301, under NHTSA’s oversight. It is essential that you are aware of and in full compliance with your legal responsibilities to ensure vehicle safety before introducing this product into commerce.”

The NHTSA Special Order (SO) said that Comma.ai must respond under oath and accompanied by an affidavit, to 15 “requests,” or be fined up to $21,000 per day and/or become the subject of a civil action to compel responses. The SO asked Hotz to describe, in detail, how the Comma One is installed, what features it offers, how the driver can use the product, what conditions constitute safe operations of the device, what types of operator controls, roads, areas, speeds and environmental conditions such as weather or time of day may a car equipped with the Comma One operate safely and what testing procedures have been implemented to attest to those claims. Each of the questions requires a detailed description of the basis to determine safe operation conditions.  Finally, by replacing the rearview mirror, how will the Comma One block or obstruct the drivers’ ability to have sufficient rear view.

[I don’t know if the NHTSA’s letter and SO are particularly harsh but, certainly, their concern for safety is evident as it should be; and as it should be for Hotz and the Comma.ai team. Governments and their regulations are an integral part of the process of bringing a new product like the Comma One to market. Certainly, it was in their domain to request testing and safety methods from Comma. But Comma’s response was definitely not predictable.]

A day later Hotz Tweeted the bottom three messages you see (read from bottom to top). Thus Hotz cancelled Comma’s first official product even though an admirer says it was the best autonomous driving experience (top Tweet).

Hotz said that Comma would turn its attention to “other products and markets.” It’s unclear what those new products will be. But it is clear that this is a major change in direction at Comma, perhaps even the beginning of its demise.

There’s a saying that at some point every startup needs to bring in an adult to handle the difficult, distasteful, and often diplomatic tasks that happen in business and with strong-willed inventors.

Perhaps now would be a good time for Comma.ai to hire that adult (although it might be a little late).

Cruise Automation

For those of you who thought the billion dollar acquisition of Cruise was a fluke, read on. One billion dollars! That’s what GM paid earlier this year. $300 million in cash; $300 million in GM stock and the remainder in incentives to retain key employees.

Up until news about the acquisition, Cruise was known as the developer of an aftermarket kit to turn select cars into self-driving cars. Actually Cruise had abandoned the kits almost a year earlier to build software instead. That’s what attracted GM.

A new article in Fortune by Erin Griffith sheds some insight and details into Cruise, its two founders, Kyle Vogt and Daniel Kan, and its importance to GM.

“Since the deal closed, Cruise has more than doubled its staff. The company’s office is filling up quickly. Office space in San Francisco is already scarce and pricey, and Cruise has a unique challenge: It needs to drive cars in and out of the office. Beneath Cruise’s barnlike warehouse is a fleet of Chevy Bolts, each outfitted with LiDAR laser sensors. There’s only one ramp in and out of the basement. The cars are starting to feel crowded too.

Cruise could solve its problem by moving its operations to the suburban sprawl of Silicon Valley proper. Like all the automakers, GM has an innovation outpost in Palo Alto. But no one at Cruise has even visited it. Staying independent is too important.”

Read the complete Fortune article here.

If you liked this article, you may also want to read these articles about autonomous cars:

• Comma.ai cancels comma-one add-on box after threats from NHTSA
• What to expect from autonomous cars
• How much can customers test robocars?
• The inevitability of the global march toward self-driving vehicles
• Two New Year’s resolutions for developers of automated vehicles
• comma.ai’s neural network car and new technology in robocars

See all the latest robotics news on Robohub, or sign up for our weekly newsletter

Source: commaai

Comma.ai, the brash startup attempting to make a self-driving system entirely from a neural network has announced it will cancel the “comma one” add-on box it has planned to sell to owners of certain Honda vehicles. The box stuck on the rear-view mirror and used the car’s own bus commands to provide an autopilot similar to those offered by car makers, with lane-keeping and adaptive cruise control.

Of particular importance is the letter from NHTSA to comma.ai which I suggest you read. This letter creates several big issues:

1. There are many elements of this letter that would also apply to Tesla and other automakers which have built supervised autopilot functions.
2. Of particular interest is the paragraph that says: “it is insufficient to assert, as you do, that the product does not remove any of the driver’s responsibilities” and “there is a high likelihood that some drivers will use your product in a manner that exceeds its intended purpose.” That must be very scary for Tesla.
3. I noted before that the new NHTSA regulations appear to forbid the use of “black box” neural network approaches to the car’s path planning and decision making. I wondered if this made illegal the approach being done by Comma, NVIDIA and many other labs and players. This may suggest that.
4. We now have a taste of the new regulatory regime and had it existed before systems like Tesla’s Autopilot, Mercedes Traffic Jam Assist, and Cruise’s original aftermarket autopilot would never have been able to get off the ground.
5. George Hotz of comma declares, “Would much rather spend my life building amazing tech than dealing with regulators and lawyers. It isn’t worth it. The comma one is cancelled. comma.ai will be exploring other products and markets. Hello from Shenzhen, China.”

To be clear, comma is a tiny company taking a radical approach, so it is not a given that what NHTSA has applied to them would have been or will be unanswerable by the big guys. Because Tesla’s Autopilot is not a pure machine learning system, they can answer many of the questions in the NHTSA letter that comma can’t. They can do much more extensive testing that a tiny startup can’t. But even so, a letter like this sends a huge chill through the industry.

It should also be noted that in Comma’s photos the box replaced the rear-view mirror, and NHTSA had reason to ask about that.

Source: commaai

George’s declaration that he’s in Shenzen gives us the first sign of the new regulatory regime pushing innovation away from the United States and California. I will presume the regulators will say, “We only want to scare away dangerous innovation” but the hard truth is that is a very difficult thing to judge. All innovation in this space is going to be a bit dangerous. It’s all there trying to take the car — the 2nd most dangerous legal consumer product — and make it safer, but it starts from a place of danger. We are not going to get to safety without taking risks along the way.

I sometimes ask, “Why do we let 16-year-olds drive?” They are clearly a major danger to themselves and others. Driver testing is grossly inadequate. They are not adults so they don’t have the legal rights of adults. We let them drive because they are going to start out dangerous and then get better. It is the only practical way for them to get better, and we all went through it. Today’s early companies are teenagers. They are going to take risks. But this is the fastest and only practical way to let them get better and save millions.

“…some drivers will use your product in a manner that exceeds its intended purpose”

This sentence, though in the cover letter and not the actual legal demand, looks at the question asked so much after the Tesla fatal crash. The question which caused Consumer Reports to ask Tesla to turn off the feature. The question which caused MobilEye, they say, to sever their relationship with Tesla.

The paradox of the Autopilot is this: The better it gets, the more likely it is to make drivers over-depend on it. The more likely they will get complacent and look away from the road. And thus, the more likely you will see a horrible crash like the Tesla fatality. How do you deal with a system which adds more danger the better you make it? Customers don’t want annoying countermeasures. This may be another reason that “Level 2,” as I wrote yesterday is not really a meaningful thing.

NHTSA has put a line in the sand. It is no longer going to be enough to say that drivers are told to still pay attention.

Black box

Comma is not the only company trying to build a system with pure neural networks doing the actual steering decisions (known as “path planning”.) NVIDIA’s teams have been actively working on this, as have several others. They plan to make commentary to NHTSA about these elements of the regulations, which should not be forbidding this approach until we know it to be dangerous.

It is challenging. It’s hard to do QA on neural networks. You can examine any single state of them, but not really understand them. You can fix the errors they make, but not know how you fixed it or whether your fix is going to work in other cases. On their own that sounds too scary, but the problem is they are outperforming other algorithms at many of the problems they are being applied to.

If you have two systems:

1. A conventionally coded system which you fully understand, that has one safety incident per 100,000 miles,
2. A black box machine learning system that has one safety incident per 150,000 miles, but you have minimal understanding of.

Which is the one that is better to put on the road? It’s not a no-brainer.

Scaring people away

George Hotz is about as brash as they come, so some might discount his bluster about moving his projects to China. But this is a real thing. When regulations come, they tend to favour the big players. The big players have large QA teams and compliance departments. They can afford them, and they move conservatively because of them. Big players help write the regulations to favour their way of doing things — sometimes selfishly but sometimes just because it is the only thing they know.

The USA and California led the way in robocars in part because it was unregulated. In the USA, everything is permitted unless it was explicitly forbidden and nobody thought to write “no robots” in the laws. Progress in other countries where everything is forbidden unless it is permitted was much slower. The USA is moving in the wrong direction.

Countries like China are another animal altogether. There, everything is forbidden unless a smart autocrat allows it but once the smart (or stupid) ruling official makes it happen, the other red tape can vanish. It is not the best system overall, but it can win at things they put their minds to.

Tesla

Has Tesla received such a letter? What would they have done if they received it in their early days? While Tesla and Elon Musk love the Autopilot project, the hard truth is Tesla’s biggest problem is going to be making enough Model 3s to meet demand and doing it profitably. Taking out Autopilot isn’t going to cut their sales at all — they have too many sales and not enough cars. Only their forward vision and guts will keep them going if NHTSA does the same thing to them. I do believe Tesla can answer questions like those in the demand letter more easily than comma.ai. But not trivially, and they almost surely will need to bump the countermeasures.

Most car companies are remaking themselves into tech startups as they move toward offering fully autonomous self-driving vehicles. Yet thousands of tractors already use self-driving kits as do hundreds of tugs, lifts and trucks. Self-driving aftermarket kits are beginning to be offered in multiple markets. What's going on?

Cruise Automation just got acquired by GM for $1 billion. Up until now, Cruise was known as the developer of a $16,000 aftermarket kit to turn select cars into self-driving cars. Does GM want to supplement their parts business with self-driving kits? Or did they spend all that money to acquire talent?

Cruise Automation isn't the only kit developer: Comma.ai promises a $1,000 kit by the end of this year and just got a $3.1 million seed round of funding; Perrone Robotics is already selling a high-end kit for R&D and testing with funding from DARPA and the IIHS. Other companies are listed below. What's going on? Are aftermarket kits really that much in demand? 

Media today seems most focused on suggesting that the industry is quickly moving toward preventing crashes from ever occurring instead of just protecting occupants from crashes even though there are many other reasons for self-driving cars (eg, fuel efficiency, lowering carbon emissions, enabling safe transportion for the elderly, and improving traffic congestion).

Andrew Ng's timetable for self-driving cars

From an interview in Wired of Andrew Ng, Chief Scientist of Baidu; and Chairman and Co-Founder of Coursera.

I think self-driving cars are a little further out than most people think. There’s a debate about which one of two universes we’re in. In the first universe it’s an incremental path to self-driving cars, meaning you have cruise control, adaptive cruise control, then self-driving cars only on the highways, and you keep adding stuff until 20 years from now you have a self-driving car. In universe two you have one organization, maybe Carnegie Mellon or Google, that invents a self-driving car and bam! You have self-driving cars. It wasn’t available Tuesday but it’s on sale on Wednesday.

I’m in universe one. I think there’s a lot of confusion about how easy it is to do self-driving cars. There’s a big difference between being able to drive a thousand miles, versus being able to drive anywhere. And it turns out that machine-learning technology is good at pushing performance from 90 to 99 percent accuracy. But it’s challenging to get to four nines (99.99 percent).

Scramble for talent

As car companies move in the direction of providing self-driving cars, most have proprietary roadmaps and big needs for software and other engineering talent. They have all set up research facilities in Silicon Valley and throughout academia. The Wall Street Journal mapped car company locations in Silicon Valley (shown below) which attests to their proliferation. Not included on this map are the new tech startups profiled in this article.

Self-driving kits and embedded software for cars, buses and trucks

Because the auto industry is so much larger and visible to the consumer public, the remainder of this article will focus on self-driving kits and other software and AI offered to the auto, transportation and trucking industries, as well as to the consumer aftermarket.

Self-driving kits

• Cruise Automation just got acquired by General Motors for what Forbes reported as “north of a billion dollars.” GM has stated their goal of creating the industry’s biggest driverless-vehicle fleet which is why GM paid such a steep purchase price for Cruise, invested $500 million into Lyft, and another $25 million to acquire the employees and technology left over from Sidecar, a ride-sharing service that shut down last December. Certainly these acquisitions are both a talent and a technology grab to help them facilitate their fleet-vehicle goal.
• Comma.ai, a San Francisco startup headed by George Hotz, the hacker that, at age 17, was the first person to unlock an iPhone, and at 21, cracked the encryption schema on the PlayStation3, got $3 million in funding from Andressen Horowitz and others for his plans to produce a self-learning, self-driving kit for under $1,000 and deliver that kit by the end of 2016. Nvidia donated $30K of their new GPUs, which are tailor-made to run deep learning algorithms, to Comma. Nvidia GPUs already run the infotainment systems in Tesla cars.
• Perrone Robotics, a Charlottesville, VA funded by DARPA and the Insurance Institute for Highway Safety (IIHS), has created and patented a general purpose autonomous vehicle operating system to assist researchers and auto engineers in developing and testing their vehicle applications including testing their security to protect against hacking.
• Hi-Tech Robotic Systemz, a Gurgaon, India-based robotics integrator, has branched out into driver assistance systems in the form of aftermarket solutions primarily built onto the embedded ADAS (Advanced Driver Assistance Systems) provided by the car manufacturers.
• ASI (Autonomous Solutions), a Utah-based provider of autonomous driving kits for the mining and farming industries, and testing kits for the automotive and R&D communities, has a whole product line of hardware components that allow users to control vehicles both manually and robotically as well as their Mobius command and control AI and software system. ASI, like Andrew Ng, believes that the charge to fully autonomous self-driving vehicles will likely be a 20-year trickle-down process.
• RobotCar, a project with the UK Department of Transportation and researchers at the University of Oxford, is testing what could be a $150 kit that learns as it goes: “As you drive your RobotCar to work or to the store, the car's systems will be keeping a close eye on the route and, eventually, will know enough to take over for you.”

Embedded software and specialty processors

• Elektrobit is a Finnish supplier of embedded car software for Mercedes, Ford, VW, BMW and Audi, and sold its business to Continental AG last year for $680 million. Elektrobit provides connected car, sensor and autonomous driving systems as well as car company factory automation systems so that the car companies can focus on engineering and building cars.
• Nauto, which just got $12M from Andy Rubin's new incubater Playground, is building automotive networking and safety features for self-driving cars. One particularly valuable technology will be similar to airline black boxes: stored sensor and visual data to help fleet managers and insurance companies detect and understand the cause of accidents and reduce liability claims.
• nuTonomy is developing perception, mapping, localization, and decision-making software for autonomous cars. They are developing algorithms to navigate complex environments, handle parallel, head-in and back-in parking and software that interprets the rules of the road to safely navigate complex itersections.
• Drive.ai, a Silicon Valley AI startup founded by former graduate students out of Stanford University’s Artificial Intelligence Lab, is using deep learning methods to provide solutions for autonomous vehicles from perception to motion planning to controls. Drive.ai just got $12 million in equity funding from undisclosed sources. (Drive.ai is not to be confused with DriveAI, a NJ non-profit working on similar activities and funded by an insurance company.)
• NVIDIA, a long-time provider of graphic processing units (GPUs) to gamers, data centers and the auto market particularly for their infotainment systems, at their recent GPU Tech conference, showed their deep learning self-driving car which boasted 3,000 miles of learning. Nvidia also introduced their new super-computer in a box specially designed to run deep learning programs, the DGX-1. Nvidia donated $30K of these new GPUs, which are tailor-made to run deep learning algorithms, to George Hotz's Comma startup.

Public transport, AGVs and governmental services

• RoboSoft, a French provider of mobility systems to government defense departments and municipal transportation systems. RoboSoft has worked with public transport groups on shuttles, buses and off-public-road shuttles and car fleets and also with warehouse and factory people on AGVs. 
• Zoox is building an autonomous robot taxi fleet and a comprehensive set of systems to support them. “Zoox is developing fully autonomous vehicles and the supporting ecosystem required to bring the technology to market at scale. Zoox aims to provide the next generation of mobility-as-a-service in urban environments.” Zoox joins 11 other companies given permits by California's DMV to test drive autonomous vehicles in California.
• Robot Taxi, a joint venture of Dena and ZMP, is also building an autonomous robot taxi to operate, at least on a test scale, by 2020 in time for the Japanese-hosted Olympics. ZMP's deep learning technology and multi-neural networks will learn about the objects that need to be recognized by their appearance. This technology will then be capable of detecting and recognizing pedestrians, vehicles and other obstacles in real time. Combined with multiple laser sensors the aim is to develop a robust and reliable environment recognition technology that is aware of the vehicle’s situation in ever-changing, real world scenarios.

Key players working on self-driving vehicles

• Google Driverless Car Project, US
• GM-Carnegie Mellong Autonomous Driving Collaborative Research Lab, US
• Uber Advanced Technologies Center, US
• University of Berlin, DE
• Technical University of Braunschweig, DE
• Karlsruhe Institute of Technology, DE
• Bundeswehr University Munich, DE
• VisLab, University of Parma, IT
• Oxford Mobile Robotics Group, UK
• INRIA IMARA, FR
• Navya Technology, FR
• Griffith University, Intelligent Control Systems Lab, AU
• Singapore-MIT-Alliance for Research & Technology, SG
• ETRI, KR

Bottom line

Perhaps the Deere tractor example applies to cars. Deere has had self-driving, autonomous, cab-less concept tractors such as the one shown on the right available for discussion and possible sale but thus far they haven't replaced the self-driving kits which they have been selling for 15 years. Farmers want the air-conditioned and cushioned cabin on their tractors to house all the monitors and other computer and automation technologies for the implements the tractors are towing. They want to remain in charge and the cab has become the hub of computerized farm automation. They say they are waiting for the right time – when fully autonomous tractor systems provide all the services they want and need. And that time is far off in the future.

Maybe car drivers want to remain in charge but are willing to have fun experimenting with low-cost partial self-driving kits or incremental add-on features on new cars until the time – far off in the future – when fully trustworthy autonomous self-driving vehicles take over the transportation marketplace. Maybe the public isn't ready yet to fully trust cars, trucks and buses without drivers. Perhaps self-driving kits are an interim step along the way. 

]]> https://robohub.org/comma-ais-neural-network-car-and-new-technology-in-robocars/ Fri, 15 Apr 2016 15:25:04 +0000 http://robohub.org/comma-ais-neural-network-car-and-new-technology-in-robocars/ Perhaps the world’s most exciting new technology today are deep neural networks, in particular the convolutional neural networks such as “Deep Learning.” These networks are conquering some of the most well known problems in artificial intelligence and pattern matching, and since their development just a few years ago, milestones in AI have been falling as computer systems that match or surpass human capability have been demonstrated.

This is particularly true in image recognition. Over the past several years, neural network systems have gotten better than humans at problems like recognizing street signs in camera images and even beating radiologists at identifying cancers in medical scans.

These networks are having their effect on robocar development. They are allowing significant progress in the use of vision systems for robotics and driving, making those progress much faster than expected. 2 years ago, I declared that the time when vision systems would be good enough to build a safe robocar without lidar was still fairly far away. That day has not yet arrived, but it is definitely closer, and it’s much harder to say it won’t be soon. At the same time, LIDAR and other sensors are improving and drooping in price. Quanergy (to whom I am an advisor) plans to ship $250 8-line LIDARS this year, and $100 high resolution LIDARS in the next couple of years.

The deep neural networks are a primary tool of MobilEye, the Jerusalem company which makes camera systems and machine-vision ASICs for the ADAS (Advanced Driver Assistance Systems) market. This is the chip used in Tesla’s autopilot, and Tesla claims it has done a great deal of its own custom development, while MobilEye claims the important magic sauce is still mostly them. NVIDIA has made a big push into the robocar market by promoting their high end GPUs as the supercomputing tool cars will need to run these networks well. The two companies disagree, of course, on whether GPUs or ASCICs are the best tool for this — more on that later.

In comes comma.ai

In February, I rode in an experimental car that took this idea to the extreme. The small startup comma.ai, lead by iPhone hacker George Hotz, got some press by building an autopilot similar in capability to many others from car companies in a short amount of time. In January, I wrote an introduction to their approach including how they used quick hacking of the car’s network bus to simplify having the computer control the car. They did it with CNNs, and almost entirely with CNNs. Their car feeds the images from a camera into the network, and out from the network come commands to adjust the steering and speed to keep a car in its lane. As such, there is very little traditional code in the system, just the neural network and a bit of control logic.

Here’s a video of the car taking us for a drive:

The network is built instead by training it. They drive the car around, and the car learns from the humans driving it what to do when it sees things in the field of view. To help in this training, they also give the car a LIDAR which provides an accurate 3D scan of the environment to more absolutely detect the presence of cars and other users of the road. By letting the network know during training that “there is really something there at these coordinates,” the network can learn how to tell the same thing from just the camera images. When it is time to drive, the network does not get the LIDAR data, however it does produce outputs of where it thinks the other cars are, allowing developers to test how well it is seeing things.

This approach is both interesting and frightening. This allows the development of a credible autopilot, but at the same time, the developers have minimal information about how it works, and never can truly understand why it is making the decisions it does. If it makes an error, they will generally not know why it made the error, though they can give it more training data until it no longer makes the error. (They can also replay all other scenarios for which they have recorded data to make sure no new errors are made with the new training data.)  read more »

]]>

Source: commaai

Perhaps the world’s most exciting new technology today are deep neural networks, in particular convolutional neural networks, such as “Deep Learning.” These networks are conquering some of the most well known problems in artificial intelligence and pattern matching. Since their development, milestones in AI have been achieved, with computer systems that match or surpass human capability. Playing Go is the more recent example.

This is particularly true in image recognition. Over the past several years, neural network systems have gotten better than humans at problems like recognizing street signs in camera images, even beating radiologists at identifying cancers in medical scans.

These networks are having an effect on robocar development. They are allowing significant progress in the use of vision systems for robotics and driving, allowing progress to happen much faster than expected. Two years ago, I declared that the time when vision systems would be good enough to build a safe robocar without lidar was still far away. That day has not yet arrived, but it is definitely closer and it’s much harder to say it won’t be soon. At the same time, LIDAR and other sensors are improving and dropping in price. Quanergy (to whom I am an advisor) plans to ship $250 8-line LIDARS this year and $100 high resolution LIDARS in the next couple of years.

The deep neural networks are a primary tool of MobilEye, the Jerusalem company which makes camera systems and machine-vision ASICs for the ADAS (Advanced Driver Assistance Systems) market. This is the chip used in Tesla’s autopilot, and Tesla claims it has done a great deal of its own custom development, while MobilEye claims the important magic sauce is still mostly them. NVIDIA has made a big push into the robocar market by promoting their high end GPUs as the supercomputing tool cars will need to run these networks well. The two companies disagree, of course, on whether GPUs or ASCICs are the best tool for this — more on that later.

In comes comma.ai

In February, I rode in an experimental car that took this idea to the extreme. The small startup comma.ai, lead by iPhone hacker George Hotz, got some press by building an autopilot similar in capability to many others from car companies in a short amount of time. In January, I wrote an introduction to their approach including how they used quick hacking of the car’s network bus to simplify having the computer control the car.

They did it with CNNs, and almost entirely with CNNs. Their car feeds the images from a camera into the network, and out from the network come commands to adjust the steering and speed to keep a car in its lane. As such, there is very little traditional code in the system, just the neural network and a bit of control logic.

Here’s a video of the car taking us for a drive:

The network is built instead by training it. They drive the car and the car learns from the humans driving it what to do when it sees things in the field of view. To help in this training, they also give the car a LIDAR which provides an accurate 3D scan of the environment to more absolutely detect the presence of cars and other users of the road. By letting the network know during training that “there is really something there at these coordinates,” the network can learn how to tell the same thing from just the camera images. When it is time to drive the network does not get the LIDAR data however it does produce outputs of where it thinks the other cars are, allowing developers to test how well it is seeing things.

This approach is both interesting and frightening. This allows the development of a credible autopilot, but at the same time, the developers have minimal information about how it works, and never can truly understand why it is making the decisions it does. If it makes an error, they will generally not know why it made the error, though they can give it more training data until it no longer makes the error. (They can also replay all other scenarios for which they have recorded data to make sure no new errors are made with the new training data.)

You need a lot of training data. To drive using vision, you have to segment the 2D images that cameras provide into independent objects to create a 3D map of what you see. You can use parallax, relative motion and two cameras to do that, as humans do, but you also need to see patterns in the shapes to classify them when you see them from every distance and angle, and in every lighting condition. That’s been one of the big challenges for vision — you must understand things in sunlight, in diffuse light, in night-time (illuminated by headlights or other lights), when the sun is pointed directly into your camera, and when complex objects like trees and casting moving shadows on the desired objects. You must tell pavement from lane markers from shoulders from debris from other road users in all those situations.

And you must do it so well that you only make a dangerous mistake perhaps once every million miles. A million miles is perhaps 2.7 billion frames of video. Fortunately, you don’t have to be that good. It’s OK to not perceive an object every so often. It’s OK to miss it in one frame if you get it the next. Indeed, you as a human are probably looking at multiple frames at once to track motion, and so you must just make sure you don’t miss something for more like a couple of seconds. Human brains, good as they are, can’t figure out everything in an arbitrary still photograph, but they do a good job at spotting everything in a second or two of video.

For now, most people are not willing to have the neural network be the entire system as comma.ai wishes. This approach is probably capable of producing an autopilot that requires human supervision, it is less clear if this is a good path to a vehicle capable of unmanned or unsupervised operation. If you do have a camera, however, the neural networks are going to do a lot to help you understand what the camera sees.

More LIDAR vs. vision

It may surprise some people to know that Google’s early generation cars barely made use of their cameras. The LIDAR is so useful that mostly the camera was there to see things like the difference between red and green lights (LIDAR doesn’t see lights and colour.) At the same time since then, Google has established itself as one of the world leaders on neural network technology, so it’s a likely supposition that Google has made strong internal efforts to do “sensor fusion” of the LIDAR, cameras and other sensors, and is probably very good at using neural networks to assist their vehicle. Other companies, such as Zoox and Daimler have shown good skill at fusing camera and LIDAR information.

In 2013, I published an article on the contrast between using LIDAR and cameras. In the article, I pointed out that LIDARS work today and are assured to get cheaper, while vision does not, and needs a breakthrough to get the job done. While we have not yet crossed that threshold new neural network technology holds the promise of being the technology to make the leap.

One of LIDAR’s flaws is that it generally is low resolution. As such, while it is very unlikely to not sense an obstacle in front of the car, it might have trouble figuring out just what the obstacle is. Fusion of LIDAR and camera with CNNs will make those systems much better at this, and knowing what things are means making better predictions about where they will go in the future.

Cameras might also help with LIDAR’s other limitation — the approximate 100m range of near infrared LIDAR for dark objects (such as black cars.) At highway speed, you want more than 100m of range. Vision doesn’t have a range limit (except at night) though the further things are from you, the more resolution you need in your image, at least in the areas you are paying attention to you (mostly the road far ahead.) The more resolution, the more CPU it takes to run the vision processing. That’s why MobilEye’s newer units feature 3 fields of view. One is a “telephoto” view for seeing things further away directly in front and there are two wider views to see things closer and more to the sides. This is a good strategy for making use of vision. It’s even better if, knowing the curvature of the road ahead, you can focus your attention only only the road, and not waste pixels or processing on the things to the side of the road.

Some people hope vision systems will get good enough to make a car that, like a human, can drive any road without a map. Mostly this technique is applied to very simple roads, like highways, which are all similar and easy to understand. However, this is an example of the wrong type of thinking about AI. Just as airplanes do not fly by flapping their wings like birds, robotic systems should not necessarily be aimed and doing what humans do the way that we do it. A vision system that can classify everything in its view sufficiently well to drive without a map is also a system that’s very useful in building a map with less or no human assistance. The map-making system can do its work with the benefit of a car driving over the territory more than once, and with the benefit of as much cloud supercomputer time as is necessary to do the best job. It would be foolish to throw away the great things a map can give you for the false goal of driving unknown roads with no data. Of course, cars need some ability to drive when the real world has changed from their map, but that’s a fairly rare event (which only happens to the very first car to encounter the change) and so it is not necessary that the vehicle be quite as capable in those situations — or if it has a human driver on board, it need not be capable at all.

Source: ideas.4brad.com

Where this approach (probably) falls down.

I wrote in January about how testing is the real blocking problem in robocars — that while there are many challenges in getting robocars working, one of the biggest is proving (to yourself and others) that you have really done it.

Neural networks face a problem here because it’s harder to know that they’re working. You don’t know why they are working, you can only measure their performance. You can re-test your network on all your old sensor data but you have a hard time being sure that the latest training you have given it won’t create a problem that wasn’t there before in some new situation.

On the other hand, traditional systems are so complex that it is difficult to judge their performance to. If the test is, “Drive one million km with less than 2 incidents, including every complex situation imagined in the simulator or recorded in the real world” then it could be the regimen is the same no matter how the system makes its decisions.

Strange legal benefit

Neural networks may also face a bizarre — I might even say perverse — advantage in the legal system. When there is an accident there will be a scramble to figure out why. In particular the plaintiff’s lawyers will be keen to show some negligence on the part of the developers.

With traditional code, you mind discover the cause was a classic old-style bug, like the famous off-by-one error or any other such problem. You will see the cause of the bug (and fix it) but you might now be able to claim that programmer, or the QA process, were negligent in some way.

With a neural network, the bug is not traditional code. If the network makes an error, we won’t know a lot about why, and so there is less likely to be a particular negligent human or negligent act, unless the court decides the whole idea of using the neural network is negligent.

That’s a more complex question, but generally if a team is following established good practices, it’s harder to find negligence. Negligence isn’t any sort of mistake, it’s a mistake that good and diligent people should have avoided, but they didn’t because they got careless.

The perverse factor here is that knowing less about how your system works makes it less likely somebody can claim you were careless.

]]>