I was recently asked about the differences between RADAR and LIDAR. I gave the generic answer about LIDAR having higher resolution and accuracy than RADAR. And RADAR having a longer range and performing better in dust and smokey conditions. When prompted for why RADAR is less accurate and lower resolution, I sort of mumbled through a response about the wavelength. However, I did not have a good response, so this post will be my better response.

LIDAR

LIDAR—which is short for Light Detection and Ranging—uses a laser that is emitted and then received back in the sensor. In most of the LIDAR sensors used for mapping (and self driving vehicles) the time between the emission and reception is computed to determine the time of flight (ToF). Knowing the speed of light and (1/2 of the) time for the wave to return (since the signal traveled out and back) we can compute how far away the object was that caused the light to bounce back. That value is the range information that is reported by the sensor. LIDAR’s generally use light in the near-infrared, visible (but not really visible), and UV spectrum’s.

There are some sensors that use triangulation to compute the position (instead of ToF). These are usually high accuracy, high resolution sensors. These sensors are great for verifying components on an assembly lines or inspecting thermal tile damage on the space shuttle. However that is not the focus of this post.

LIDAR data. The top shows the reflectivity data. The bottom shows the range data with brighter points being farther away.

The laser beam can also be focused to have a small spot size that does not expand much. This small spot size can help give a high resolution. If you have a spinning mirror (which is often the case) then you can shoot the laser every degree or so (based on the accuracy of the pointing mechanism) for improved resolution. It is not uncommon to find LIDAR’s operating at 0.25 degrees of angular resolution.

RADAR

RADAR—which is short for Radio Detection and Ranging—uses radio waves to compute velocity, and/or range to an object. Radio waves have less absorption (so less attenuation) than the light waves when contacting objects, so they can work over a longer distance. As you can see in the image below the RF waves have a larger wavelength than the LIDAR waves. The down side is that if an object is much smaller than the RF wave being used, the object might not reflect back enough energy to be detected. For that reason many RADAR’s in use for obstacle detection will be “high frequency” so that the wavelength is shorter (hence why we often use mm-wave in robotics) and can detect smaller objects. However, since LIDAR’s have the significantly smaller wavelength, they will still usually have a finer resolution.

Electromagnetic spectrum showing radio waves all the way on the left for RADAR and near-infrared/visible/ultra-violet waves towards the right for LIDAR usage.

Most of the RADAR’s that I have seen will have a narrow field of view (10’s of degrees) and then just return a single value (or they might have up to a few dozen channels, see multi-mode RADAR below) for the range of the detected object. There are tricks that some systems can do using multiple channels to also get the angle for the range measurement. The angle will not be as high resolution as most LIDAR’s. There are also some RADAR’s on the market that scan to get multiple measurements. The 2 approaches are a spinning antenna (such as what you see at airports or on ships) or electronically “spinning” which is a device using multiple internal antennas with no moving parts. More advanced (newer) RADAR’s can do things like track multiple objects. In many cases they will not actually return the points that outline the objects (like a LIDAR), but will return a range, bearing, and velocity to an estimated centroid of the detected item. If multiple objects are near each other, the sensor might confuse them as being one large object and return one centroid range [Here is the source and a good reference to read].

Using the Doppler frequency shift, the velocity of an object can also be easily determined with relatively little computational effort. If the RADAR sensor and the detected object are both moving than you get a relative velocity between the two objects.

With RADAR there are commonly two operating modes:

1. Time of Flight – This operates similar to the LIDAR sensor above, however it uses radio wave pulses for the Time of Flight calculations. Since the sensors are pulsed, it knows when the pulse was sent. So, computing the range can be easier than with the continuous wave sensors (described below). The resolution of the sensor can be adjusted by changing the pulse width and the length of time you listen for a response (a ping back). These sensors often have fixed antennas leading to a small operating field of views (compared to LIDAR).

There are some systems that will combine multiple ToF radio waves into one package with different pulse widths. These will allow for various ranges to be detected with higher accuracy. These are sometimes called multi-mode RADAR

2. Continuous Wave – This approach frequency modulates (FMCW) a wave and then compares the frequency of the reflected signal to the transmitted signal in order to determine a frequency shift. That frequency shift can be used to determine the range to the object that reflected it. The larger the shift the farther the object is from the sensor (within some bounds). Computing the frequency shift and the corresponding range is computationally easier than ToF, plus the electronics to do so are easier and cheaper. This makes continuous frequency modulated systems popular. Also, since separate transmit and receive antennas are often used, this approach can continuously be transmitting and receiving at the same time unlike the pulsed ToF approach, which needs to transmit then wait for a response. This feature and the simple electronics can make FMCW RADAR very fast at detecting objects.

There is another version of the Continuous Wave RADAR where the wave is not modulated. These systems are cheap and good for quickly detecting speed using the Doppler effect, however they can not determine range. They are often used by police to detect vehicle speed where the range is not important.

SONAR

Unrelated, I should note that SONAR or Sound Navigation and Ranging, can work in both of the modes as RADAR. The wavelength used is even larger than RADAR. It is farther to the left in the spectrum image shown before, and is off the chart.

I should point out that there are cool imaging SONAR sensors. The general idea is that you can make the sensing wave vertical so that the horizontal resolution is very fine (<1 degree) and the vertical resolution is larger (10+ degrees). You can then put many of these beams near each other in a sensor package. There are similar packages that do this with small wave length RADAR.

Cost

LIDAR Sensors tend to cost more than RADAR sensors for several reasons:

1. LIDAR using ToF needs high speed electronics that cost more.
2. LIDAR sensors needs CCD receivers, optics, motors, and lasers to generate and receive the waves used. RADAR only needs some stationary antennas.
3. LIDAR sensors have spinning parts for scanning. That requires motors and encoders. RADAR only needs some stationary antennas. (I know this is sort of similar to the line above).

Computationally Speaking

RADAR sensors tend to generate a lot less data as they just return a single point, or a few dozen points. When the sensor is multi-channel it is often just returning a range/velocity to a few centroid(ish) objects. The LIDAR sensors are sending lots of data about each individual laser point of range data. It is then up to the user to make that data useful. With the RADAR you have a simple number, but that number might not be the best. With LIDAR it depends on the roboticist to generate algorithms to detect the various objects and discern what is being viewed by the sensor.

If your goal is to detect a car in front of you (or driving towards you) and get its velocity, the RADAR can be great. If you are trying to determine the precise location of an item, generate a surface map, or find a small fence-post, a LIDAR might do better. Just remember that if you are in dust or rain the LIDAR might return a cloud of points right near the sensor (as it reads all of those particles/drops); while the RADAR might do much better.

I recently found a SICK LMS291 in the trash (dumpster diving time!). It is a bit of an older model—manufactured in August 2000—however the fundamentals should be similiar to the current generation of sensors. If you look at robots from that era you will often see them covered in this type of LIDAR sensor.

Here is a series of posts about how LIDAR works and evaluating multiple sensors.

If you have always wanted to know how a SICK LIDAR worked inside, now is your chance. If you did not always want to know, this is still your chance to see. Keep reading for some cool pictures.

The sensor feels solid and feels like a brick. I could probably throw it off a cliff and it will still work, not that you should try…

SICK LMS 291. You can see the front black area where the laser shoots out from. The 3 indicator lights at the top front, and the cutout in the back for the connectors.

SICK LMS 291 power and serial communications ports. There is a large sealed connector that plugs into each slot.

Front Plastic Shield

Mirror and encoder wheel. The laser shoots down from above the spinning mirror. The green post on the left reads the position from the encoder wheel shown on the bottom. The encoder wheel is the metal wheel with the slots cut into it. The 6 black tubes on the bottom I think are IR LEDs that are used for detecting dust or anything else occluding the front black visor.

Front Metal Case removed. Walls are about 3/10″ or 7.6mm thick

With the front removed here are the internals of the sensor. On the bottom you can see the metal ring which is a motor, above it is the spinning mirror encoder. The top section has the control boards and the laser hidden behind them.

Bottom view of the open sensor. The motor for spinning the mirror and encoder wheel is clearly visible at the bottom. The black rectangles on the top board are for detecting if the bottom LEDs are visible for fault detection.

These are the wings that go below and above the spinning mirror assembly. On the left side you can see the LEDs and the board on the right has the black square receivers.

The encoder is directly mounted to the mirror for increased accuracy of the laser position. It is very solidly built, and you can see that a designer was having fun with cutting the SICK logo into the metal disk. I was surprised when I saw this since I expected a higher resolution encoder wheel with more slots. But I guess the larger thicker slots can be more reliable. Towards the top of the wheel it looks like one slot is missing, I wonder if it is some sort of index.

Laser and receiver black can assembly visible after removing the front PCB. You can see the 2 gold coax connectors probably for some high frequency signals from the receiver. On the top board in the front you can see the 3 indicator lights that are visible on the exterior of the enclosure. On the left of the front board you can see the board interconnect connector that connects to the board on the right side of this image. When the boards are screwed down that connector is not coming loose.

Front board that I think converts the analog RF receiver signal to digital for the processor board. The board on top has the reciever, a coax cable goes from there to the gold can. Followed by going to this board. I am really curious what is under the gold can.

Look at the washers used above the screw holes. Excellent attention to detail to avoid screws coming loose from vibration. Not shown but there are many really nice touches, such as captive standoffs. I was surprised that I did not see any loctite.

Close up of the front board. Perhaps some RF signal conditioning for the receiver of the laser signal.

The main processor board that was connected to the front board shown before from the left side board connector. You can see the Xilinx FPGA that is running the sensor.

Bodge wire on the rear of the main processing board. I am surprised to see this. However it is really well done. Look at the epoxy blob holding the wire down.

Metal gold can to shield some components (or so I thought) from EM interference. Desiccant packet on the left is to control any moisture that might get into the sensor.

Inside of the shielding can. Is that for creating a time delay or some phase shift in the RF signal coming from the receiver?

Receiver of the incoming laser signal on the top board center. The black can has some mirrors and the laser.

Laser receiver module closeup. What is all the metal heatsinking for?

The motor and mirror removed. You can sort of look up into the laser and reciever assembly. The flat metal part between the ring and the laser assembly is the back of the power supply module.

Looking up from the mirror into the laser assembly. Most of the center that you see is a mirror. The slot in the plastic has its own mirror that points the laser from the gold block down to the spinning mirror. The returned signal goes on the sides of that slot to the receiver above.

Laser that is connected to the slot in the black mirror can.

Connected to gold pins above to (control) the laser. What is under that shield??

Under the shield. The DM74ALS seems to be a quad 2 input nand gate.

The power supply for the sensor that was located behind the spinning mirror module.

All of the components in the LIDAR sensor. The PCB that is still in the rear metal enclosure is the board that connects to the power and serial connectors that we saw on the external of the sensor before.

I hope you enjoyed seeing the internals of this sensor. It is interesting to see how they made it and the spinning mirror that we often hear about. Leave comments below to help identify parts or other questions you may have.

Luminar, a bay area startup, has revealed details on their new LIDAR. Unlike all other commercial offerings, this is a LIDAR using 1.5 micron infrared light. They hope to sell it for $1,000.

Source: Luminar

1.5 micron LIDAR has some very special benefits. The lens of your eye does not focus medium depth infrared light like this. Ordinary light, including the 0.9 micron infrared light of the lasers in most commercial LIDARS is focused to a point by the lens. That limits the amount of power you can put in the laser beam, because you must not create any risk to people’s eyes.

Because of this, you can put a lot more power into the 1.5 micron laser beam. That, in turn, means you can see further, and collect more points. You can easily get out to 250 meters, while regular lidars are limited to about 100m and are petering out there.

What doesn’t everybody use 1.5 micron? The problem is silicon sensors don’t react to this type of light. Silicon is the basis of all mass market electronics. To detect 1.5 micron light, you need different materials, which are not themselves that hard to find, but they are not available cheap and off the shelf. So far, this makes units like this harder to build and more expensive.

If Luminar can do this, it will be valuable.

Why do you need to see 250m? Well, you don’t for city driving, though it’s nice. For highway driving, you can get by with 100m as well, and you use radar to help you perceive, at very low resolution, what’s going on beyond that. Still, there are things that radar can’t tell you. Rare things, but still important. So you need a sensor that sees further to spot things like stalled cars under bridges. Radar sees those, but can’t tell them from the bridge.

To this point, Google has been the only company to say they have a long range LIDAR, but it has not been for sale. And as we all know, there is a famous lawsuit underway accusing Uber/Otto of copying Google’s LIDAR designs.

The Luminar point clouds are impressive. This will be a company to watch. (In the interests of disclosure, I am an advisor to Quanergy, another LIDAR startup.)

Article first appeared on robocars.com

First ride: Encountering a school bus on real city streets in Austin, Texas. Credit: Waymo/Google

Recently we’ve seen a series of startups arise hoping to make robocars with just computer vision, along with radar. That includes recently unstealthed AutoX, the off-again, on again efforts of comma.ai and at the non-startup end, the dedication of Tesla to not use LIDAR because it wants to sell cars today before LIDARs can be bought at automotive quantities and prices.

Their optimism is based on the huge progress being made in the use of machine learning, most notably convolutional neural networks, at solving the problems of computer vision. Milestones are dropping quickly in AI and particularly pattern matching and computer vision. (The CNNs can also be applied to radar and LIDAR data.)

There are reasons pushing some teams this way. First of all, the big boys, including Google, already have made tons of progress with LIDAR. There right niche for a startup can be the place that the big boys are ignoring. It might not work, but if it does, the payoff is huge. I fully understand the VCs investing in companies of this sort, that’s how VCs work. There is also the cost, and for Tesla and some others, the non-availability of LIDAR. The highest capability LIDARs today come from Velodyne, but they are expensive and in short supply — they can’t make them to keep up with the demand just from research teams!

Note, for more detailed analysis on this, read my article on cameras vs. lasers.

For the three key technologies, these trends seem assured:

1. LIDAR will improve price/performance, eventually costing just hundreds of dollars for high-resolution units, and less for low-res units.
2. Computer vision will improve until it reaches the needed levels of reliability, and the high-end processors for it will drop in cost and electrical power requirements.
3. Radar will drop in cost to tens of dollars and software to analyse radar returns will improve.

In addition, there are some more speculative technologies whose trends are harder to predict, such as long-range LWIR LIDAR, new types of radar, and even a claimed lidar alternative that treats the photons like radio waves.

These trends are very likely. As a result, the likely winner continues to be a combination of all these technologies, and the question becomes which combination.

LIDAR’s problem is that it’s low resolution, medium in range and expensive today. Computer Vision (CV)’s problem is that it’s insufficiently reliable, depends on external lighting and needs expensive computers today. Radar’s problem is super low resolution.

Option one — high-end LIDAR with computer vision assist

High end LIDARs, like the 32 and 64 laser units favoured by the vast majority of teams, are extremely reliable at detecting potential obstacles on the road. They never fail (within their range) to differentiate something on the road from the background. But they often can’t tell you just what it is, especially at a distance. It won’t know a car from a pickup truck or 2 pedestrians from 3. It won’t read facial expressions or body language. It can read signs but only when they are close. It can’t see colours, such as traffic signals.

The fusion of the depth map of LIDAR with the scene understanding of neural net based vision systems is powerful. The LIDAR can pull the pedestrian image away from the background, and then make it much easier for the computer vision to reliably figure out what it is. The CV is not 100% reliable, but it doesn’t have to be. Instead, it can ideally just improve the result. LIDAR alone is good enough if you take the very simple approach of “If there’s something in the way, don’t hit it.” But that’s a pretty primitive result that may brake too much for things you should not brake for.

Consider a bird on the road or a blowing trash bag. It’s a lot harder for the LIDAR system to reliably identify those things. On the other hand, the visions systems will do a very good job at recognizing the birds. A vision system that makes errors 1 time every 10,000 is not adequate for driving. That’s too high an error rate as you encounter thousands of obstacles every hour. But missing 1 bird out of 10,000 means that you brake unnecessarily for a bird perhaps once every year or two, which is quite acceptable.

Option two — lower end LIDAR with more dependence on vision

Low end lidars, with just 4 or so scanning planes, cost a lot less. Today’s LIDAR designs basically need to have an independent laser, lens and sensor for each plane, and so the more planes, the more cost. But that’s not enough to identify a lot of objects, and will be pretty deficient on things low to the ground or high up, or very small objects.

The interesting question is, can the flaws of current computer vision systems be made up for by a lower-end, lower cost LIDAR. Those flaws, of course, include not always discerning things in their field. They also include needing illumination at night. This is a particular issue when you want a 360-degree view — one can project headlights forward and see as far as they see, but you can’t project headlights backwards or to the side without distracting drivers.

It’s possible one could use infrared headlights in the other directions (or forward for that matter.) After all, the LIDAR sends out infrared laser beams. There are eye safety limits (your iris does not contract and you don’t blink to IR light) but the heat output is also not very high.

Once again, the low end lidar will eliminate most of the highly feared false negatives (when the sensor doesn’t see something that’s there) but may generate additional false positives (ghosts that make the vehicle brake for nothing.) False negatives are almost entirely unacceptable. False positives can be tolerated but if there are too many, the system does not satisfy the customer.

This option is cheaper but still demands computer vision even better than we have today. But not much better, which makes it interesting.

Other options

Tesla has said they are researching what they can do with radar to supplement cameras. Radar is good for obstacles in front of you, especially moving ones. Better radar is coming that does better with stationary objects and pulls out more resolution. Advanced tricks (including with neural networks) can look at radar signals over time to identify things like walking pedestrians.

Radar sees cars very well (especially licence plates) but is not great on pedestrians. On the other hand, for close objects like pedestrians, stereo vision can help the computer vision systems a lot. You mostly need long range for higher speeds, such as the highways, where vehicles are your only concern.

Who wins?

Cost will eventually be a driver of robocar choices, but not today. Today, safety is the only driver. Get it safe, before your competitors do, at almost any cost. Later make it cheap. That’s why most teams have chosen the use of higher end LIDAR and are supplementing in with vision.

There is an easy mistake to make, though, and sometimes the press and perhaps some teams are making it. It’s “easy” on the grand scale to make a car that can do basic driving and have a nice demo. You can do it with just LIDAR or just vision. The hard part is the last 1%, which takes 99% of the time, if not more. Google had a car drive 1,000 miles of different roads and 100,000 total roads in the first 2 years of their project back in 2010, and even in 2017 with by far the largest and most skilled team, they do not feel their car is ready. It gets easier every day, as tech advances, to get the demo working, but that should not be mistaken for the real success that is required.

This post originally appeared on robocars.com.

Source: Intel

Intel announced plans to acquire Israel-based Mobileye, a developer of vision technology used in autonomous driving applications, for $15.3 billion. Mobileye share prices jumped from $47 to $61 (the tender offering price is $63.54) on the news, a 30% premium. The purchase marks the largest acquisition of an Israeli hi-tech company ever.

Source: Frost & Sullivan;VDS Automotive SYS Konferenz 2014/

This transaction jumpstarts Intel’s efforts to enter the emerging autonomous driving marketplace, an arena much different than Intel’s present business model. The process to design and bring a chip to market involves multiple levels of safety checks and approvals as well as incorporation into car company design plans – a process that often takes 4 to 5 years – which is why it makes sense to acquire a company already versed in those activities. As can be seen in the Frost & Sullivan chart on the right, we are presently producing cars with Level 2 and Level 3 automated systems. Intel wants to be a strategic partner going forward to fully automated and driverless Level 4 and Level 5 cars.

Mobileye is a pioneer in the development of vision systems for on-board Driving Assistance Systems; providing data for decision making applications such as Mobileye’s Adaptive Cruise Control, Lane Departure Warning, Forward Collision Warning, Headway Monitoring, High Beam Assist and more. Mobileye technology is already included in BMW 5-Series, 6-Series, 7-Series, Volvo S80, XC70 and V70 models, and Buick Lucerne, Cadillac DTS and STS.

Last year, Intel reorganized and created a new Autonomous Driving Division which included strategic partnerships with, and investments in, Delphi, Mobileye and a bunch of smaller companies involved in the chipmaking and sensor process. Thus, with this acquisition, Intel gains the ability to offer automakers a larger package of all of the components they will need as vehicles become autonomous and perhaps gaining, as well, on their competitors in the field: NXP Semiconductors, Freescale Semiconductor, Cypress Semiconductor, and STMicroelectronics, the company that makes Mobileye’s chips.

Mobileye’s newest chip, the EyeQ4, designed for computer vision processing in ADAS applications, is a low-power supercomputer on a chip. The design features are described in this article by Imagination Technology.

Bottom line:

“They’re paying a huge premium in order to catch up, to get into the front of the line, rather than attempt to build from scratch,” said Mike Ramsey, an analyst with technology researcher Gartner in a BloombergTechnology article.

Professor Daniela Rus stands with autonomous SMART vehicles on the MIT campus. Photo: SMART

By: Catherine Marguerite | Singapore-MIT Alliance for Research and Technology

Daniela Rus loves Singapore. As the MIT professor sits down in her Frank Gehry-designed office in Cambridge, Massachusetts, to talk about her research conducted in Singapore, her face starts to relax in a big smile.

Her story with Singapore started in the summer of 2010, when she made her first visit to one of the most futuristic and forward-looking cities in the world. “It was love at first sight,” says the Andrew (1956) and Erna Viterbi Professor of Electrical Engineering and Computer Science and the director of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). That summer, she came to Singapore to join the Singapore-MIT Alliance for Research and Technology (SMART) as the first principal investigator in residence for the Future of Urban Mobility Research Program.

“In 2010, nobody was talking about autonomous driving. We were pioneers in developing and deploying the first mobility on demand for people with self-driving golf buggies,” says Rus. “And look where we stand today! Every single car maker is investing millions of dollars to advance autonomous driving. Singapore did not hesitate to provide us, at an early stage, with all the financial, logistical, and transportation resources to facilitate our work.”

Since her first visit, Rus has returned each year to follow up on the research, and has been involved in leading revolutionary projects for the future of urban mobility. “Our team worked tremendously hard on self-driving technologies, and we are now presenting a wide range of different devices that allow autonomous and secure mobility,” she says. “Our objective today is to make taking a driverless car for a spin as easy as programming a smartphone. A simple interaction between the human and machine will provide a transportation butler.”

The first mobility devices her team worked on were self-driving golf buggies. Two years ago, these buggies advanced to a point where the group decided to open them to the public in a trial that lasted one week at the Chinese Gardens, an idea facilitated by Singapore’s Land and Transportation Agency (LTA). Over the course of a week, more than 500 people booked rides from the comfort of their homes and came to the Chinese Gardens at the designated time and spot to experience mobility-on-demand with robots.

The test was conducted around winding paths trafficked by pedestrians, bicyclists, and the occasional monitor lizard. The experiments also tested an online booking system that enabled visitors to schedule pickups and drop-offs around the garden, automatically routing and redeploying the vehicles to accommodate all the requests. The public’s response was joyful and positive, and this brought the team renewed enthusiasm to take the technology to the next level.

Since the Chinese Gardens public trial, the autonomous car group has introduced a few other self-driving vehicles: a self-driving city car, and two personal mobility robots, a self-driving scooter and a self-driving wheelchair. Each of these vehicles was created in three phases: In the first phase, the vehicle was converted to drive-by-wire control, which allows a computer to control acceleration, braking, and steering of the car. In the second phase, the vehicle drives on each of the pathways in its operation environment and makes a map using features detected by the sensors. In the third phase, the vehicle uses the map to compute a path from the customer’s pick-up point to the customer’s drop-off point and proceeds to drive along the path, localizing continuously and avoiding any other cars, people, and unexpected obstacles. The devices also used traffic data from LTA to model traffic patterns and to study the benefits of ride-sharing systems.

Last April, the team conducted a new test with the public at MIT. This time, they deployed a self-driving scooter that allowed users to use the same autonomy system indoors as well as outdoors. The trial included autonomous rides in MIT’s Infinite Corridor. A significant challenge in this type of space is localization, or accurately knowing the location of the robot in a long and plain corridor that does not have many distinctive features. The system proved to work very well in this type of environment, and the trial completed the demonstration of a comprehensive uniform autonomous mobility system.

“One can easily see the usefulness of such a system between self-driving city cars, golf buggies, and scooters,” Rus says. “A mobility-impaired user could, for example, use a self-driving scooter to get down the hall and through the lobby of an apartment building, take a self-driving golf buggy across the building’s parking lot, and pick up an autonomous car on the public roads to go to a similarly equipped amusement park or shopping centre.”

Daniela Rus, a Class of 2002 MacArthur Fellow and member of the USA National Academy of Engineering, knows that each successful step into urban mobility will bring a positive contribution of artificial intelligence to the public. According to the World Health Organization, 3,400 people die each day in the world from traffic-related accidents.

“It is a new space race,” she says, convinced that autonomy is part of the solution to safe transportation. Daniela Rus will continue visiting her beloved Singapore, where she particularly enjoys the food, the beautiful flowers, the kindness of its people, and the smartness of its youth. “Singapore is definitely a model in many fields,” she concludes.

Source: Ford Velodyne LiDAR/YouTube

In anticipation of the need for LiDAR devices in cars with assisted steering and other self-driving technologies, both Velodyne and Quanergy received funding. Quanergy raised $90 million and Velodyne got $150 million.

Source: Frost & Sullivan;VDS Automotive SYS Konferenz 2014/

As can be seen in the chart provided by Frost & Sullivan, LiDARs will play a big role in all phases of the emergence of self-driving vehicles. Hence the immediacy of these two investments in LiDAR technology.

Velodyne LiDAR, a Morgan Hill, Calif.-based provider of technology that allows self-driving cars see and avoid what’s around them, has raised $150 million in equity funding from Ford Motor Co. and Baidu. Velodyne expects an exponential increase in LiDAR sensor deployments in autonomous vehicles and ADAS applications over the next several years, driving high revenue growth. The funds will be used to help Velodyne lower the cost of their LiDAR to between $300 and $500, the price point at which mass adoption becomes practical.

“LiDAR continues to prove itself as the critical sensor for safe autonomous vehicle operation,” said David Hall, founder and CEO, Velodyne LiDAR. “This investment will accelerate the cost reduction and scaling of Velodyne’s industry-leading LiDAR sensors, making them widely accessible and enabling mass deployment of fully autonomous vehicles. We are determined to help improve the goal of safety for automotive vehicles as soon as possible, as well as empower the efficiency autonomous systems offer.”

“We want the cost to be low enough to be used for all cars. We envision a safer world for the millions of automotive drivers across the globe,” said Marta Hall, Velodyne President of Business Development.

Quanergy Systems Inc., a Sunnyvale, Calif.-based provider of solid state LiDAR sensors and smart sensing solutions, has raised $90 million. Backers include Sensata Technologies, Delphi Automotive, Samsung Ventures, Motus Ventures and GP Capital. Quanergy’s solid state LiDAR is included in the frontal piece of the Mercedes test car shown on the right. The solid-state nature of the LiDAR is a significant cost savings and the expected trade price (once full production begins) will be less than $250.

LiDARs aren’t the only components getting funded

In a NY Times story, Delphi and Mobileye said that they had formed a partnership to provide a core self-driving platform to automakers by the end of 2019.

The Mobileye-Delphi collaboration “will accelerate the time to market” and enable carmakers to produce vehicles capable of driving themselves “without the need for huge capital investments,” said Amnon Shashua, chairman and chief technology officer of Mobileye.

Delphi, the former parts division of General Motors, and Mobileye, an Israeli company known for producing the sensors and other technology that help cars recognize obstacles in the road, are both major component providers to the auto industry.

UPDATE: Only 7 days left to back Sweep! They are now almost 80% funded.

Sweep is a small, lightweight and low-cost laser scanner. It spins a laser range finder to form a 360 degree view of the world, up to 10 times per second. What makes Sweep different is who it’s designed for. Until now, sensors like Sweep were too expensive to be used in cost sensitive products, relegating them to research, industrial, or military applications. With Sweep, we aim to bring the awesome accomplishments of the last two decades of LiDAR research out of the lab and into consumer products!

Sweep is the culmination of years of development work, and like so many inventions, it came as a bi-product of wanting to make something else. We started off developing outdoor consumer robots for the home. We wanted to make a smart product that could navigate through typical yard environments, avoiding obstacles and keeping track of its charging station location. We didn’t like the hassle of other solutions, like buried guide wires, and knew a better way would be to have the robot form a map of the environment. We quickly realized to do any kind of mapping we would need a scanning LiDAR (or something similar) that works outdoors. It was then we discovered that there were no affordable LiDAR sensors on the market that met the requirements.

So, we set out to make one ourselves!

We designed Sweep to be very small and light! It fits right in the palm of your hand and weighs a tad less than most smart phones today. Sweep is also really quiet, something you may have never thought about until you’ve used one that makes annoying sounds as it spins. You won’t be hearing this scanner.

After more than a dozen iterations, we arrived at our current production ready design. We’ve discovered that the key innovation with Sweep will be its ability to perform much of the required data filtering internally. Most LiDAR sensors output a tremendous amount of data, which is great for applications where power and cost are not an issue, but cumbersome for consumer applications. For example, aerial vehicles may only want to know where the closest obstacles are. Sweep is equipped with several sensing modes designed to make interfacing with lower power controllers possible.

Testing with beta units has been successful!

For more than a month we’ve had beta units in the hands of testers. We’ve received great feedback on how people want to use Sweep.

Companies like Dispatch Robotics want a sensor to help guide their delivery robot through city streets. Others want to use it to navigate through grocery isles and office spaces. They are all excited about how Sweep can cut down the parts cost of their products by thousands of dollars.

 At Dispatch, we are very excited by Sweep. LiDAR is a very powerful sensing modality and low-cost LiDAR will have significant impact as we develop and grow our autonomous vehicle fleet.”

Uriah Baalke, Dispatch Robotics

The Sweep is a great fit for high school robotics. It is easy to setup and use, robust, and is at an incredible price point. Students can quickly explore mapping and navigation problems with little prior experience. Best yet, the Sweep is virtually student-proof – it can survive a high school lab environment.”

Charles Williams Mentor, Cupertino High School Robotics

How does Sweep compare to other sensors used for obstacle avoidance?

Traditionally low cost robotic vehicles have used sonar, structured light cameras, or stereo vision to sense their environment, but they all have a limited range. When following user controls or GPS directions they may only have milliseconds to react when obstacles come into their field of view. In contrast, Sweep has a range of 40 meters, giving much more time to react to approaching obstacles. It also works in all lighting conditions, making it more reliable in the real world.

We’re targeting the initial release of Sweep toward early adopters and developers looking to try out the technology. As we receive more insight from our customers about their specific applications, we will refine future versions of Sweep to meet the demand. Likely areas of improvement include additional internal data filtering modes, more compact packaging, faster update rates, longer ranging capability and more environmental hardening.

The ultimate goal of Sweep is to empower new and existing applications with a smart and affordable scanning LiDAR solution!

Contact

More information can be found at their website: www.scanse.io

Thier kickstarter can be found here: https://www.kickstarter.com/projects/scanse/sweep-scanning-lidar

You can contact them directly at: Team@scanse.io

Google’s prototype is what is known as a “Neighbourhood Electric Vehicle” or NEV. There are special NEV rules in place that make such vehicles much less subject to the complex web of regulations required for a general purpose vehicle. They need to be electric, must not travel on roads with a speed limit over 35mph and they must themselves not be capable of going more than 25mph. The Google car was doing 24mph when the officer asked the safety driver to pull over, so there was nothing to ticket. Of course, that does not mean an officer can’t get confused and need an explanation of the law — even they don’t know all of them.

The NEV regulations are great for testing, though there is indeed an issue around how the earliest robocars will probably want to go a little slow, because safety really is the top priority on all teams I know. As such, they may go as slowly as the law allows, and they may indeed annoy other drivers when doing that. This should be a temporary phase but could create problems while cars learn to go faster. I have suggested in the past that cars wanting to go slow might actually notice anybody coming up behind them and pull off the road, pausing briefly in driveways or other open spots, so that the drivers coming up behind never have to even brake. A well behaved unmanned vehicle might go slowly, but it doesn’t necessarily present a burden to hurried humans.

Ford may also avoid standby supervision

Recent reports suggest that Ford, like Google, may have concluded that there is not an evolutionary path from ADAS to full self driving — in particular, the so-called “level 3” (which I call standby supervision) where a human driver can be called on with about 10 seconds notice (but not less) to resolve live driving problems or to take the wheel when the car enters a zone it can’t drive. This transition may just be too dangerous, according to Google and others.

Cheaper LIDAR etc.

Noted without much comment, Quanergy (on whose advisory board I sit) has announced progress on the plans for an inexpensive solid state LIDAR, and plans to ship the first on schedule, in 2016. This sub-$1000 LIDAR keeps us on a path to even cheaper LIDAR, which should eliminate all the people who keep saying they want to build robocars without LIDAR (I am looking at you, Elon Musk). Nobody will make their first full robocar less safe just to save a few hundred dollars.

Also related to Starship (another company I advise) is the arrival of not one but two somewhat similar startups to build small delivery robots. “Dispatch Network” involves US roboticists who participated in a Chinese-based hardware accelerator and have a basic prototype, larger than the Starship robot. “Sidewalk” — a Lithuanian company, also has a prototype model and a deal with DHL to do research together with them on last mile robots.

More to come.

This post originally appeared on robocars.com.

image: Steve Jurvetson/flickr

Security researcher Jonathan Petit claims to be able to hack the multi-thousand-dollar laser ranging – LIDAR – systems that self-driving cars rely on. How does he do it and is it as dangerous as it sounds?

Much has been made of Petit’s recent disclosure of an attack on some of the LIDAR systems used in robocars. I saw his presentation on this in July, but he asked me for confidentiality until they released their paper in October. However, since he has decided to disclose it, there’s been a lot of press, with truth and misconceptions.

There are many security aspects to robocars. By far the greatest concern would be compromise of the control computers by malicious software, and great efforts will be taken to prevent that. Many of those efforts will involve having the cars not talk to any untrusted sources of code or data that might be malicious. The car’s sensors, however, must take in information from outside the vehicle, so they are another source of concern.

There are ways to compromise many of the sensors on a robocar. GPS can be easily spoofed and there are tools out there to do that (fortunately real robocars will only use GPS as one source to gauge their location). Radar is also very easy to compromise — far easier than LIDAR, agrees Petit — but his team’s goal was to see if LIDAR is vulnerable.

The attack was a real one, but not a particularly frightening one, in spite of the press it received. This type of attack may cause a well designed vehicle to believe there are “ghost” objects that don’t actually exist, so that it might brake for something that’s not there or even swerve around it. It might also overwhelm the sensor, so that it feels the sensor has failed and thus the car would go into failure mode, stopping or pulling off the road. This is not a good thing, of course, and it has some safety consequences, but it’s also fairly unlikely to happen. Essentially, there are far easier ways to do these things that don’t involve LIDAR, so it’s not too likely anybody would want to mount such an attack.

Indeed, to carry out this type of attack you need to be physically present either in front of the car (or perhaps to the side of the road) and you need a solid object that’s already in front of the car, such as the back of a truck that it’s following. This is a higher bar than with an attack that might be done remotely (such as a computer intrusion) or via a radio signal (such as with hypothetical vehicle-to-vehicle radio, should cars decide to use that tech).

Here’s how it works: LIDAR sends out a very short pulse of laser light and then waits for it to reflect back. The pulse is a small dot and the reflection is seen through a lens aimed tightly at the place it was sent to. The time it takes for the light to come back tells you how far away the target is and the brightness tells you how reflective it is, like a black-and-white photo.

To fool LIDAR, you must send another pulse that comes from, or appears to come from, the target spot, and it has to come in at just the right time, before the real pulse from what’s really in front of the LIDAR does.

The attack requires knowing the characteristics of the target LIDAR very well. You must know exactly when it is going to send its pulses before it does so, and thus precisely (to the nanosecond) when a return reflection will arrive from a hypothetical object in front of it. Many LIDARs are quite predictable. They scan a scene with a rotating drum and you can see the pulses coming out and know when they will be sent.

In the simplest version of the attack, the LIDAR is scanning something like a wall in front of it. There aren’t such walls on a highway, but there are things like signs, bridges and the backs of trucks and some cars.

The attack laser sends a pulse of light at the wall or other object, but does it so that it will hit the wall in the right place, earlier than the real pulse from the target LIDAR. This pulse will then bounce back, go in the lens of the LIDAR and make it appear that there is something closer than the wall.

The attack pulse does not have to be bright, but it needs to be similar to the pulse from the LIDAR so that the reflection looks the same. The attack pulse must also be at the very same wavelength and launched at just the right time, to the nanosecond.

If you send out lots of pulses without good timing, you won’t create a fake object, but you can make noise. That noise would blind the LIDAR about the wall in front of it, but it would be very obviously noise. Petit and crew did tests of this type of noise attack, and they were a success.

The fancier type of attack knows the timing of the target LIDAR perfectly and paints a careful series of pulses so it looks like a complex object is present, closer than the wall. Petit’s team were able to do this on a small scale.

This kind of attack can only make the ghost object appear in front of another object, like another vehicle or perhaps a road sign or bridge. It could also be reflected off the road itself. The ghost object, being closer than the road in question, would appear to be higher than the road surface.

There are LIDAR designs which favour the strongest or the last pulse, or even report multiple ones. The latter would probably be able to detect a spoof.

Petit also described an attack from in front of the target LIDAR, such as in a vehicle further along the road. Such an attack, if I recall correctly, was not actually produced but is possible in theory. In this attack, you would shine a laser directly at the LIDAR. This is a much brighter pulse and, in theory, it might shine on the LIDAR’s lens from an angle, but be bright enough to generate reflections inside it that would be mistaken for the otherwise much dimmer return pulse (similar to the bright spots known as “lens flare” in a typical camera shooting close to the sun.) In this case, you could create spots where there is nothing behind the ghost object, though it is unknown just how far an angle from the attack laser (which will appear as a real object) needs to be. The flare pulse would have to be aimed quite precisely to hit the lens, if you want a complex object.

image: Steve Jurvetson/flickr

As noted, all this does is create a ghost object, or noise, that temporarily blinds the LIDAR. The researchers also demonstrated a scarier (but less real) “denial of service” attack against the LIDAR processing software. This software’s job is to take the cloud of 3D points returned by the LIDAR and turn it into a list of objects perceived by it. Cars like Google’s, and most others, don’t use the standard software from the manufacturer.

The software they tested here was limited, and could only track a fixed number of objects – around 64, I believe. If there are 65 objects (which is actually quite a lot) it will miss one. This is a far more serious error, and is called a ‘false negative’ — there might be something right in front of you and your LIDAR would see it but the software would discard it and so it would be invisible to you. Thus, you might hit it, if things go badly.

The attack involves creating lots of ghost objects — so many that they overload the software and go over the limit. The researchers demonstrated this but, again, only on fairly basic software that was not designed to be robust. Even software that can only track a fixed number of objects should detect when it has reached its limit and report that, so the system can mark the result as faulty and take appropriate action.

As noted, most attacks, even with an overwhelming number of ghost objects, would only cause the car to decide to brake for no reason. This is not that hard to do. Humans do this all the time actually, braking by mistake or for lightweight objects like tumbleweeds or animals dashing over the road. You can make a perfectly functioning robocar brake by throwing a ball on the road. You can also blind humans temporarily with a bright flash of light or a cloud of dust or water. This attack is much more high tech and complex but amounts to the same thing.

It is risky to suddenly brake on the road, as the car behind you may be following too closely and hit you. This happens even if you brake for something real. Swerving is a bit more dangerous, but can normally only be done when there is a very high likelihood the path being swerved into is clear. Still, it’s always a good idea to avoid swerving. But again, you would also do this for the low-tech example of a ball thrown onto the road.

It is possible to design a LIDAR to return the distance of the first return it sees (the closest object or the last or perhaps the most solid). Some equipment may report the strongest or multiple returns. The Velodyne used by many teams reports only one return.

If a LIDAR reports the first, you can fake objects in front of the background one. More frighteningly, if it reports the last you can fake an object behind the background and possibly hide it (though that would be quite difficult and require you making your fake object very large). If it reports the strongest, you can put your fake object in either place.

Because this attack is not that dangerous, makers of LIDARs and software may not bother to protect against it. If they do take action, there are a number of potential fixes:

If their LIDAR is not predictable about when it sends pulses, the attack can only create noise, not coherent ghost objects. The noise will be seen as noise, or an attack, and responded to directly. For example, even a few nanoseconds of random variation in when pulses are sent would cause any attempted ghost object to explode into a cloud of noise.
LIDARs can detect if they get two returns from the same pulse. If they see that over a region, it is likely there is an attack going on, depending on the pattern. Two returns can happen naturally, but this pattern should be quite distinctive.

LIDARs that scan rather than spin have been designed. They could scan in an unpredictable way, again reducing any ghost objects to noise.

In theory, a LIDAR could send out pulses with encoded data and expect to see the same encoded data return. This requires fancier electronics, but would have many side benefits — a LIDAR would never see interference from other LIDARs, for instance, and might even be able to pull out information away from the background illumination (i.e. the sun) with a better signal to noise ratio, meaning more range. However, total power output in the pulses remains fixed so this may not be doable.

Of course, software should be robust against attacks and detect their patterns, should they occur. That’s a relatively cheap and easy thing to do, as it’s just software.

This topic brings up another common question about LIDAR: whether they might interfere with one another when every car has one. The answer is that they can interfere, but only minimally. A LIDAR sends out a pulse of light and waits for about a microsecond to see the bounce back (a microsecond means the target is 150 meters away). In order to interfere, another LIDAR (or attack laser) has to shine on the tight spot being looked at with the LIDAR’s return lens during the exact same microsecond. Because any given LIDAR might be sending out a million pulses every second, this will happen, but rarely and mainly in isolated spots. As such, it’s not that hard to tune it out as noise. To our eyes, the world would get painted with lots of laser light on a street full of LIDARs. But, to the LIDARs, which are looking for a small spot to be illuminated for less than a nanosecond during a specific microsecond, the interference is much smaller.

Radar is a different story. Radar today is very low resolution. It’s quite possible for somebody else’s radar beam to bounce off your very wide radar target at a similar time. But auto radars are not like the old “pinging” radars used for aviation; they send out a constantly changing frequency of radiation and look at that of the return to figure out how long the signal took to come back. They must also allow for how much that frequency changed due to the Doppler effect. This gives them some advantage, as two radars using this technique should differ in their patterns over time, but it’s a bigger problem: attack against radar is much easier because you don’t need to be nearly so accurate and you can often predict a pattern. Although radars that randomize their pattern could be robust against interference and attack.

This post originally appeared on Robocars.com.

By Ilia Baranov

In this new Robots 101 series, we will be taking a look at how robots work, what makes designing them challenging, and how engineers at Clearpath Robotics tackle these problems. To successfully operate in the real world, robots need to be able to see obstacles around them, and locate themselves. Humans do this mostly through sight, whereas robots can use any number of sensors. Today we will be looking at lasers, and how they contribute to robotic systems.

Overview

When you think of robots and lasers, the first image that comes to mind might come from science fiction; robots using laser weapons. However, almost all robots today use lasers for remote sensing. This means that the robot is able to tell, from a distance, some characteristics of an object. This can include size, reflective, color, etc. When the laser is used to measure distance in an arc around the robot, it is called LIDAR. LIDAR is a portmanteau of Light and Radar, essentially think of a sweeping radar beam shown in the films, using light.

Function and Concepts

By Mike1024, via Wikimedia Commons

All LIDAR units operate using this basic set of steps:

1. Laser light is emitted from the unit (usually infrared)
2. Laser light hits an object and is scattered
3. Some of the light makes it back to the emitter
4. The emitter measures the distance (more on how later)
5. Emitter turns, and process begins again

A great visual deception of that process to the right.

Types of LIDAR sensing

How exactly the laser sensor measures the distance to an object depends on how accurate the data needs to be. Three different methods commonly found on LIDAR sensors are:

Time of flight
The laser is emitted, and then received. The time difference is measured, and distance is simply (speed of light) x (time). This approach is very accurate, but also expensive due to the extremely high precision clocks needed on the sensor. Thus, it is usually only used on larger systems, and at longer ranges. Rare to use on robots

Phase difference
In this method, the emitted laser beam is modulated to a specific frequency. By comparing the phase shift of the returned signal on a few different frequencies, the distance is calculated. This is the most common way laser measurement is done. However, it tends to have limited range, and is less accurate than time of flight. Almost all of our robots use this.

Angle of incidence
The cheapest and easiest way to do laser range-finding is by angle of incidence. Essentially, by knowing what angle the reflected laser light hits the sensor, we can estimate the distance. However, this method tends to be of low quality. The Neato line of robotic vacuum cleaners use this technology.

Another thing to note is that so far, we have only discussed 2D LIDAR sensors. This means they are able to see a planar slice of the world around them, but not above or below. This has limitations, as the robot is then unable to see obstacles lower or higher than that 2d plane. To fix this issue, either multiple laser sensors can be used, or the laser sensor can be rotated or “nodded” up and down to take multiple scans. The first solution is what the Velodyne line of 3D LIDAR sensors employ, while the second solution tends to be a hack done by roboticists. The issue with nodding a LIDAR unit is a drastic reduction of refresh rate, from tens of Hz, down to 1 Hz or less.

Manufacturers and Uses

Some of the manufacturers we tend to use are SICK, Hokuyo, and Velodyne.
This is by no means an exhaustive list, just the ones we use most often.

SICK
Most of our LIDARs are made by SICK. Good combo of price, size, rugged build, and software support. Based in Germany

Hokuyo
Generally considered cheaper than SICK, in larger quantities. Some issues with fragility, and software support is not great. Based in Japan

Velodyne
Most expensive, used only when robot MUST see everything in the environment. Based in US.

Once the data is collected, it can be used for a variety of reasons. For robots, these tend to be:

• Mapping
  • Use the laser data to find out dimensions and locations of obstacles and rooms. This allows the robot to find its position (localisation) and also report dimensions of objects. The title image of this article shows the Clearpath Robotics Jackal mapping a room.
• Survey
  • Similar to mapping, however this tends to be outside. The robot collects long range data on geological formations, lakes, etc. This data can then be used to create accurate maps, or plan out mining operations.
• Obstacle Avoidance
  • Once an area is mapped, the robot can navigate autonomously around it. However, obstacles that were not mapped (for example, squishy, movable humans) need to be avoided.
• Safety Sensors
  • In cases where the robot is very fast or heavy, the sensors can be configured to automatically cut out motor power if the robot gets too close to people. Usually, this is a completely hardware-based feature.

Selection Criteria

A lot of criteria is used to select the best sensor for a given application. The saying goes that an engineer is someone who can do for $1 what any fool can do for $2. Selecting the right sensor for the job not only reduces cost, but also ensures that the robot has the most flexible and useful data collection system.

• Range
  • How far can laser sensor see? This impacts how fast a robot is able to move, and how well it is able to plan.
• Light Sensitivity
  • Can the laser work properly outdoors in full sunlight? Can it work with other lasers shining at it?
• Angular Resolution
  • What is the resolution of the sensor? More angular resolution means more chances of seeing small objects.
• Field of view
  • What is the field of view? A greater field of view provides more data.
• Refresh rate
  • How long does it take the sensor to return to the same point? The faster the sensor is, the faster the robot can safely move.
• Accuracy
  • How much noise is in the readings? How much does it change due to different materials?
• Size
  • Physical dimensions, but also mounting hardware, connectors, etc
• Cost
  • What fits the budget?
• Communication
  • USB? Ethernet? Proprietary communication?
• Power
  • What voltage and current is needed to make it work?
• Mechanical (strength, IP rating)
  • Where is the sensor going to work? Outdoors? In a machine shop?
• Safety
  • E-stops
  • regulation requirements
  • software vs. hardware safety

For example, here is a collected spec sheet of the SICK TIM551.

• ROS 101: Intro to the Robot Operating System
• ROS 101: A practical example
• Arduino for Makers #1: Setting up a development station
• Arduino for Makers #2: Deconstructing Arduino programs
• Robots Podcast: Soft Robotics Toolkit, with Donal Holland

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Many hope to be able to solve robotics (and thus car) problems with just cameras. While LIDAR is going to become cheap, it is not yet, and cameras are much cheaper. I outline many of the trade-offs between the systems in my article on cameras vs lasers. Everybody hopes for a research breakthrough or computer vision breakthrough to make vision systems reliable enough for safe operation.

The Michigan lab’s approach is a special machine vision one. They map the road in advance in 3D and visible light by using a mapping car equipped with lots of expensive LIDAR and other sensors. They build a 3D representation of the road similar to what you need for a video game engine, and from that, with the use of GPUs, they can indeed create a 2D image of what a camera should see from any given point.

The car goes out into the world and its actual camera delivers a 2D frame of what it sees. Their system then compares that with generated 2D images of what the camera should see until it finds the closest match. Effectively, it’s like you looking out a window and then going into a video game and wandering around looking for a place that looks like what you see out that window, and then you know where the window is.

Of course it is not “wandering,” and they develop efficient search algorithms to quickly find the location that looks most like the real world image. We’ve all seen video games images, and know they only approximate the real world, so nothing will be an exact match, but if the system is good enough, there will be a “most similar” match that also corresponds with what other sensors, like your GPS and your odometer/dead reckoning system, tell you about where you probably are.

Localization with cameras has been done before, and this is a new approach taking advantage of new generations of GPUs, so it’s interesting. The big challenge is simulating the lighting, because the real world is full of different lighting, high dynamic range, and shadows. The human system has no problem understanding a stripe on the road as it moves through the shadow of a tree, but computer systems have a pretty tough time with that. Sun shadows can be mapped well with GPUs, but shadows from things like the moving limbs of trees are not possible to simulate, as are the shadows of other vehicles and road users. At night, light and shadows come from car headlights and urban lights. The team is optimistic about how well they will handle these problems.

The much larger challenge is object perception. Once you have a simulation of what the camera should see, you can notice when there are things present that are not in the prediction — like another car or pedestrian, or a new road sign. (Right now their system mostly is looking at the ground.) Once you identify the new region, you can attempt to classify it using computer vision techniques, and also by watching it move against the expected background.

This is where it gets challenging, because the bar is very high. To be used for driving it must effectively always work. Even if you miss 1 pedestrian in a million you have a real problem because there are billions of pedestrians encountered by a billion drivers every day. This is why people love LIDAR — if something (other than a mirror or sheet of glass) sufficiently large is sufficiently close you, you’re going to get laser returns from it, and not from what’s behind it. It has the reliability number that is needed.
The challenge of vision systems is to meet that reliability goal.

This work is interesting because it does a lot without relying on AI “computer vision” techniques. It is not trying to look at a picture and recognize a person. Humans are able to look at 2D pictures with bizarre lighting and still tell you not just what the things in the picture are, but often how far away they are and what they are doing. While we can be fooled in a 2D image, once you have a moving dynamic world, humans are, generally reliable enough at spotting other things on the road. (Though of course, with 1.2 million dead each year, and probably 50 million or more accidents, the majority because somebody was “not looking,” we are far from perfect.)

Some day, computer vision will be as good at recognizing and understanding the world as people are — and in fact surpass us. There are fields (like identifying traffic signs from photos) where they already surpass us. For those not willing to wait until that day, new techniques in perception that don’t require full object understanding are always interesting.

I should also point out that while lowering cost is of course a worthwhile goal, it is a false goal at this time. Today, maximal safety is the overriding goal, and as such, nobody will actually release a vehicle to consumers without LIDAR just to save the estimated 2017 cost of LIDAR, which will be sub-$500. Only later, when cameras get so good they completely replace LIDAR safety capabilities for less money would people release such a system to save cost. On the other hand, improving cameras to be used together with LIDAR is a real goal; superior safety, not lower cost.

A version of this article originally appeared on robocars.com. Want to learn more about the the University of Michigan research cited in this article? Check out the research paper by Ryan Wolcott and Ryan Eustice (Best Paper Award at IROS 2014!), as well as Wolcott’s IROS 2014 Webcam research pitch: 

Of course, I had an extra day to spend at Stanford University after the conference and couldn’t miss out on the event.

The fun really began when I got there. I was greeted by a gigantic inflatable Keepon, followed by booth after booth of robots. Among them were Puzzlebox, a robot controlled using EEG, PR2 from Willow Garage, and a self-driving car demonstrating LIDAR technology from Velodyne. With a lot of help from Dr. Peter Asaro, an expert in roboethics and professor at The New School, and my labmate Mr. Ergun Calisgan from the CARIS lab (University of British Columbia) I captured some of the highlights from Robot Block Party on video.

Puzzlebox:

Puzzlebox Orbit is a helicopter controlled using EEG signals. It was programmed to launch the helicopter whenever the user’s attention level goes above a threshold. As you’ll see in the video, my attention level proved to be quite intense for the Puzzlebox Orbit whenever I listen to someone speak.

Willow Garage:

Willow Garage‘s PR2 is one of the more sophisticated robots in the world today. In this video, Kaijen Hsiao talks about the robot and Willow Garage’s teach by demonstration studies.

Velodyne:

LIDAR is a remote sensing technology that is key to autonomous driving. I had the opportunity to speak with Velodyne‘s president Marta Hall, who discusses Velodyne’s technology and self-driving cars.