Waymo One launches

[CarlK]

You're missing the forest for the trees. If lidar can help (which you admit in your statement), then why not have it? More information is always better. So what if cameras are better than humans? If you can make an even better car with lidar, then why not build one? A bicycle can run faster than my legs, but that didn't stop people from building cars that run faster than bikes.

Yes, cost is an issue, but that's a business problem, not a technical one. (And LIDAR costs are bound to come down.)

By the way, "NN trained from that will not be as smart as that trained by camera only" sentence does not make sense.

OK to give you an example here. Have you ever seen how a blind person gets around? You think you could do the same once you lose your eyesight without having to go through a lot of training? LIDAR is so expensive, and so obtrusive that people would want to have on their cars, there is little chance that it will be put on cars we buy in the foreseeable future if ever. Can you tell me when you think compact and affordable LIDAR will be available? Relying on something no one is sure when or if ever you will get it is a lazy and not a very sound strategy.

 

[ricebucket]

To give you an example. Have you seen how a blind person gets around? You think you can do the same once you lose your eyesight without having to go through a lot of training? LIDAR is so expensive, and obtrusive that people would want to have on their cars, there is little chance that it will be put into cars we buy in the foreseeable future if ever. Can you tell me when you think compact and affordable LIDAR will be available? Relying on something no one is sure when or if ever you will get it is a lazy and not a very sound strategy.

The blind person example has no relevance here. It's not a matter of a person suddenly losing or gaining sight. It's simply two different people with completely different starting and ending points. The ending points are different, because no blind person can out navigate a person with sight -- more info is always better.

Your cost question is again why I said you're missing the forest. In my bike/car example, does it matter that a car is not as cheap as a bike? It matters in a direct sense, if you could only buy one. But that's not how the world works, and cars have plenty of sales. I can see realistic business models where expensive LIDAR based autonomous cars would have a market (e.g. taxis). In addition, although I don't know the trajectory of LIDAR prices over time, I think we can all agree that prices will drop once tech improves and volume increases (by many orders of magnitude). LIDAR right now is a niche product with low production numbers.

 

[acoste]

Elon says using LIDAR is like on crutches, I say a camera based solution is going to be the cheap man's autopilot due to its limitations even when it's ready.

Cameras lack the resolution of the human eye in dark (become noisy) and also weak on the dynamic range. It can be blinded. Plus there is no redundancy, no backup system. Stereo camera system at least has one more choice and possibly the distance information. But it needs too much computing power as of now.

So why not use a system that is even better than human? Adding lidar is one option, but there are developments on microbolometer as well.

And certain businesses are already happy to pay for the shorter range lidars.

 

[electronblue]

I think when Karpathy joined, they restarted the NN side of thing. The training data was likely still usable in some form.
My guess: the NN exists as one object. However, the internals of the NN (lane detection, sign detection, vehicle gap adjusting) could still be tested (and thus trained) individually. What the suspected SW 2.0 approach removes is having separate groups working on separate aspects with separate NNs and then trying to merge them at the end.

General can also mean that they are using some more typical, research paper supported, methods, but they do everything over the complete data set. Whereas some people may start at a neighborhood, then region, then city, then country, then another country, then worldwide with changes each time, Tesla could be going for all or nothing.

Analogy: training a robot to walk, progressive: flat, high mu ground, bumpy ground, hill, ruts, mountains, ice. General: all topology cases at once.

Similar to the Lidar debate: Lidar gets to a working system sooner, but Tesla is going for pure vision, which Lidar system will need also for completeness. So, in terms of apparent progress, Lidar system start out ahead, but then run into many of the same problems.

End results: the Tesla approach may look poor until it works, then it looks good (perhaps great).

Just my thoughts.

Thank you. A fair theory.

 

[CarlK]

The blind person example has no relevance here. It's not a matter of a person suddenly losing or gaining sight. It's simply two different people with completely different starting and ending points. The ending points are different, because no blind person can out navigate a person with sight -- more info is always better.

Your cost question is again why I said you're missing the forest. In my bike/car example, does it matter that a car is not as cheap as a bike? It matters in a direct sense, if you could only buy one. But that's not how the world works, and cars have plenty of sales. I can see realistic business models where expensive LIDAR based autonomous cars would have a market (e.g. taxis). In addition, although I don't know the trajectory of LIDAR prices over time, I think we can all agree that prices will drop once tech improves and volume increases (by many orders of magnitude). LIDAR right now is a niche product with low production numbers.

If both person can not be relying on their eyes to get around then the blind person with better trained brain will do much better. The underlaying question is still when and if LIDAR will be practical enough for the purpose. No one would bet the bottom dollar to give you an answer. It likely will be a long time if ever. Unless if you have a definitive path to get there it's just a lazy solution, or actually non-solution, for that.

LIDAR of course can be used in autonomous taxi or ride hailing application which likely is Waymo's goal. Not a problem with that but it's just not the same as cars with general autonomous capability everyone can buy and take it anywhere he wants. Even when the day of cheap and compact LIDAR comes, which is probably between a very long time and never, companies like Waymo will still need to retrain their NN with a lot of cars on the road like Tesla does in order to make that to work. Starting with camera and put it in every car from the beginning is just a brilliant strategy. It's not just a dumb luck, assuming camera eventually will work which there is no reason to believe it won't, but another case of first principle engineering at work.

 

[heltok]

I think the single model karpathy has been talking about is similar to the one waymo showed. Train a feature layer that is shared between all tasks based on all sensor input data and then many sublayers for various tasks.

 

[electronblue]

Not problem with that but it's just not the same as cars with general autonomous capability everyone can buy and take it anywhere he wants.

Depends totally on how Lidar is used though. Mapping the world in Lidar is one thing but Lidar can also simply be the ultimate crash avoidance sensor. Latter is much cheaper, much smaller and still immensely useful for redundancy compared to the thing on a Waymo’s roof. Lidar is very good at avoiding false negatives.

For example — to simplify a little — the Level 3 Audi uses Lidar in that manner. EyeQ3 watches the road while radars watch other cars. Lidar makes sure there is nothing else they can hit, nothing fooling the vision and nothing outside the capabilities of the vision... and just a second opinion for the vision. An active light, Lidar also works in the dark.

 

[mongo]

Plus there is no redundancy, no backup system.

Tesla has three forward looking cameras.

You're missing the forest for the trees. If lidar can help (which you admit in your
statement), then why not have it? More information is always better.

For any numer of sensors (n), an additional sensor will provide more data.

If the rate of failure if a single sensor is x, the rate of failure of n of them is x^n.
If you have two data sources that disagree, who wins? If that sensors wins, why do you have the other one?
Never Take Two Chronometers to Sea « ipSpace.net blog

 

[acoste]

Tesla has three forward looking cameras.

They are all for different distances, 60m, 150m, 250m and in the same spot. If the 250m camera fails, the system has less margin. If a bird or a rock hits that area, all cameras are gone.

Redundancy is more valuable if it is based on different technologies.

 

[acoste]

If the rate of failure if a single sensor is x, the rate of failure of n of them is x^n.
If you have two data sources that disagree, who wins? If that sensors wins, why do you have the other one?

The failing sensor will most likely report an error.

If they don't report any errors (this is the situation with Tesla's radar and camera), the more threatening signal has more weight (phantom braking).

 

[strangecosmos]

If you’re designing an affordable, mass produced self-driving car for 2030, you should probably include long-range, high-resolution lidar.

If you’re designing an affordable, mass produced self-driving car for 2020, you probably can’t include long-range, high-resolution lidar — it’s too expensive, and possibly too unreliable as well. (You can probably throw in some crummy, short-range, low-resolution lidars... but why?)

Well, why do self-driving cars need to be affordable? Why can’t we just have $300,000 robotaxis? We can, but we can’t use $300,000 robotaxis to collect 10 billion miles of driving data (at least not without spending tens of billions of dollars). If you need 10 billion miles of driving data to train neural networks to master the perception tasks and action tasks require for driving, then the fastest way to do that is deploy an affordable, mass market car now, without lidar.

 

[mongo]

They are all for different distances, 60m, 150m, 250m and in the same spot. If the 250m camera fails, the system has less margin. If a bird or a rock hits that area, all cameras are gone.

Redundancy is more valuable if it is based on different technologies.

My point: In the event of a failure of a sensor, are the remaining sensors sufficient to produce a failsafe condition? Can the car pull off the road if it loses a front camera? Not saying it should continue driving as if nothing is wrong.

The failing sensor will most likely report an error.

If they don't report any errors (this is the situation with Tesla's radar and camera), the more threatening signal has more weight (phantom braking).

Then your system braking false positove rate is the union of the false positive rates of the individual sensors. With heterogeneous sensor suites, that means the worst attributes of each type. Does that improve things?

 

[acoste]

Then your system braking false positove rate is the union of the false positive rates of the individual sensors. With heterogeneous sensor suites, that means the worst attributes of each type. Does that improve things?

Yes, it improves safety. And this is also the answer why we don't have Level 3 systems on the highways yet.
The sensors that are prone to report false objects or can't report objects have to be improved. Until then: used only in the case of a complete failure of the other. Radar is on its way to get better resolution.


Forgot to add, of course in case of 3 different type of sensors, 1 can be ignored. For level 2 system, one sensor is the driver.

 

[strangecosmos]

Interesting that, in a dev version of the software, Tesla is using one or more neural networks for sensor fusion. From Amir Efrati at The Information:

“Before artificial intelligence can take over the car, Tesla must take baby steps. That means letting neural networks take over writing parts of Autopilot software that were previously written by humans, the beginning of what Autopilot vision group chief Andrej Karpathy calls “Software 2.0.” In a development version of Autopilot that isn’t available to customers, the system is relying on such networks to figure out which readings from different sensors on the car are “in agreement,” which is part of a process known as as sensor fusion, said a person familiar with the project. (Some rival autonomous vehicle developers use varations of this method.) Already, the networks appear to perform better than the hand-coded software that told the system how to find those areas of agreement, this person said.”
​

I think sensor fusion essentially attempts to make a virtual mega-sensor that is “aware” (so to speak) of the strengths and weaknesses of each sensor modality.

If you’re walking in a dark, snowy forest at night, and you hear a crunch (a footfall on the snow?), you are likely to pay attention to that signal even if you can’t see anything yet. You know your ears are better at detecting things behind you or behind a tree. Whether a sensory modality is evincing strength or weakness at a particularly moment is situation-dependent; it depends on the information landscape at the time. In another context, you would trust your eyes to read the lyrics of a song better than your ears can hear them.

A neural network might be able to learn this kind of situation dependency. In a way that would be difficult to explicitly hand-code.

 

[acoste]

Interesting that, in a dev version of the software, Tesla is using one or more neural networks for sensor fusion. ...

One of such methods I heard is the Kalman filter. Kalman filter - Wikipedia

The way I understand it: if there is a noise from a sensor that is implausible data based on the previous couple of seconds, then it can be ignored. Or maybe it has a specific storage of every type of events, for example "false positive radar event", collects all the occurrences and averages them. 99 times out of 100 it was a false positive. That one last time it is a real object. Would the filter ignore it based on the statistics? How could one teach it that one of the detections was a real object if they didn't hit it? If camera detected it then, it's all good. But if it didn't, the NN learned something, it shouldn't.

 

[electronblue]

Some points on sensor redundancy:

1. Tesla has sensor redundancy only towards a very narrow cone on the front where three cameras overlap together with a single radar. Ultrasonics may provide a rather fallible level of redundancy in some specific speed scenarios but in some cases even they are alone in the vision blind spots like immediately around front corners.

2. Most others where car responsible driving is the target are aiming at triple redundancy at speed: vision, radar and Lidar in every direction or scenario where the car is automatically driving towards (ie 360 degrees for urban). There may also be the ultrasonics as the fourth level for low speeds. This does not mean sensor fusion can’t still be a problem but it does mean it is not merely a question of one sensor disagreeing with the other — you get two out of three situations for example.

In the case of Tesla’s suite the lack of redundancy does not manifest itself merely in the lack of Lidar but also in lack of all around radar for example.

The oft-discussed question of unprotected left turns with merely vision means that basically a Tesla has to rely on B pillar cameras alone to make the call towards left and right. Even the addition of corner radars would increase the confidence in saying no fast cars are approaching a lot. Instead Tesla’s sole radar covering is not even heated so Elon talks of ignoring it in the future — and it does not help towards the sides anyway.

 

[heltok]

One of such methods I heard is the Kalman filter. Kalman filter - Wikipedia

The way I understand it: if there is a noise from a sensor that is implausible data based on the previous couple of seconds, then it can be ignored. Or maybe it has a specific storage of every type of events, for example "false positive radar event", collects all the occurrences and averages them. 99 times out of 100 it was a false positive. That one last time it is a real object. Would the filter ignore it based on the statistics? How could one teach it that one of the detections was a real object if they didn't hit it?

There are many types of filters. Kalman filters are baysian and generally work best with ~gaussian measurements. There are other filter types such as particle filters that can have less gaussian distributions, but you pay in compute. For a radar you can just add 20% probability of miss and outliers will just be considered 20% likely. Often all hypotheses will be ~equally unlikely for misses so they don’t change the estimate that much.

Neural networks learn these by just being provided enough data.

If the rate of failure if a single sensor is x, the rate of failure of n of them is x^n.

1-(1-x)^n

 

[heltok]

They are all for different distances, 60m, 150m, 250m and in the same spot. If the 250m camera fails, the system has less margin. If a bird or a rock hits that area, all cameras are gone.

Redundancy is more valuable if it is based on different technologies.

For ASIL D you need redundancy, but you don’t need to be able to drive with the redudant system, you just need to be able to stop. Which usually means that you need to be able to slow down to a speed that doesn’t kill the driver(70km/h) before any accident, stay in lane(position +-10cm) and move to the side of the road. If they go blind with front camera they might be able to position themselves good enough while stopping using only Radar, Sonar and side cameras. This is preferably tested exhaustively in simulation.

 

[electronblue]

@J1mbo Out of curiosity which part of my post #116 do you disagree with? I think it is pretty much just stating facts. If you think I missed some fact let me know.

 

[J1mbo]

Elon says using LIDAR is like on crutches, I say a camera based solution is going to be the cheap man's autopilot due to its limitations even when it's ready.

Cameras lack the resolution of the human eye in dark (become noisy) and also weak on the dynamic range. It can be blinded. Plus there is no redundancy, no backup system. Stereo camera system at least has one more choice and possibly the distance information. But it needs too much computing power as of now.

Tesla AP cameras already have a higher dynamic range than the human eye - check out verygreen's example of a red stop light right in front of the sun. Also, comparing "resolution" of a cam vs human eye is fundamentally meaningless. Not only are they very different approaches to solving a problem, but the NN (or brain) receiving the input doesn't need to use every "pixel" anyway.

Also, I think you confuse lidar as being somehow equivilent to vision. It's not. A lidar unit only detects it's own signal when (if) that signal returns to the lidar unit. It can't use ambient light, and many things can distort the return signal (heat haze, rain, heavy fog, exhaust fumes on a cold day, etc.).

Lidar could replace radar, but has the disadvantage that everything in the return signal has to be "understood". Example: with radar, you know immediately with high certainty that a large reflection in front of the car is an obstacle. With lidar, that could be a puddle, or paint on the road, or a tractor wheel ¯\_(ツ)_/¯

Some points on sensor redundancy:

1. Tesla has sensor redundancy only towards a very narrow cone on the front where three cameras overlap together with a single radar. Ultrasonics may provide a rather fallible level of redundancy in some specific speed scenarios but in some cases even they are alone in the vision blind spots like immediately around front corners.

2. Most others where car responsible driving is the target are aiming at triple redundancy at speed: vision, radar and Lidar in every direction or scenario where the car is automatically driving towards (ie 360 degrees for urban). There may also be the ultrasonics as the fourth level for low speeds. This does not mean sensor fusion can’t still be a problem but it does mean it is not merely a question of one sensor disagreeing with the other — you get two out of three situations for example.

I would argue that it is not redundancy unless each sensor is part of a pair, and each unit in the pair has i) two independently cabled dedicated feeds ii) two independantly cabled and dedicated power sources iii) dual independant ECUs to receive and process the four data feeds.

Anyway, you probably already saw this. It's clear that all the cameras all overlap to some extent.