Autonomous Car Progress

[stopcrazypp]

@drtimhill, if point cloud data isn't helpful then why is Tesla working on a NN to generate a point cloud from camera data?

That's not what he's arguing, he arguing to make use of point clouds to reliably attach a velocity and identity to an object (especially stationary ones) as opposed to points you need still neural nets just like you do for camera vision. So that is a problem you have to solve even if you use lidar. It is not like what a lot of people seem to assume that the raw data automatically spits out a unique id for an object for every point (thus basically you can trivally tell which group of points is a single "object" and which is a separate one).

Radar has an easier time with doppler shift, where it can filter out a lot of objects it would not be interested in (basically how ACC has worked for a long time), but for more advanced applications like imaging radar (and also if you want to classify stationary objects too, instead of only moving ones), it'll still need to use some form of machine learning.

 

[Daniel in SD]

That's not what he's arguing, he arguing to make use of point clouds to reliably attach a velocity and identity to an object (especially stationary ones) as opposed to points you need still neural nets just like you do for camera vision. So that is a problem you have to solve even if you use lidar. It is not like what a lot of people seem to assume that the raw data automatically spits out a unique id for an object for every point (thus basically you can trivally tell which group of points is a single "object" and which is a separate one).

Radar has an easier time with doppler shift, where it can filter out a lot of objects it would not be interested in (basically how ACC has worked for a long time), but for more advanced applications like imaging radar (and also if you want to classify stationary objects too, instead of only moving ones), it'll still need to use some form of machine learning.

If point cloud data is helpful then wouldn‘t accurate point cloud data be better?
Are you sure object tracking from a point cloud requires machine learning?

 

[stopcrazypp]

If point cloud data is helpful then wouldn‘t accurate point cloud data be better?

Depends on how well it integrates into your other data. The difference with vision based point cloud data is every point is already directly associated with a point in your image, so you don't have to sync them. TOF cameras do the same thing in sort of a hybrid between cameras and lidar.

Are you sure object tracking from a point cloud requires machine learning?

You can probably build examples that just uses hard coded heuristics, but from what I can tell practically every advanced solution out there uses some sort of machine learning.

The idea however being presented I believe is that the theory is it is impossible to achieve L5 without solving all the vision based problems (car must be smart enough to drive on vision alone and very basic navigation maps just like a human does), and at that point sensors like lidar or radar is unnecessary. If the end goal is geofenced L4 fleets, Waymo and others have already clearly beat Tesla to it (or any of the vision focused companies, there was an article up thread of others that are taking the same approach). If the end goal is L5 that works everywhere in regular cars, then it's still anyone's game.

I should point out that people living in metro areas that Waymo and other fleet based companies like to target will likely have a different perspective than people that live in less populated areas that those companies will take forever to reach (maybe never). For those people, geofenced fleet L4 is practically useless given it makes no business sense, so companies will not want to develop for those areas (it doesn't even make business sense yet in populated areas at the moment).

 

[Dewg]

How does Waymo handle debris in the road? Say a small wooden plank with a nail sticking up?

 

[drtimhill]

@drtimhill, if point cloud data isn't helpful then why is Tesla working on a NN to generate a point cloud from camera data?

I didn't say it wasn't helpful, I said deriving accurate object estimations purely from point clouds is very hard (JPDAF has all sorts issues with merging, and was developed with other applications in mind where merging wasn't considered a problem).

With a camera, you can train an NN to notice that two very closely co-moving cars are still two cars, since the NN has been trained to understand what delineates a car (tail lights, wheels etc). Point clouds contain no such data, so you left with velocity bundles and similar techniques to figure out if you are looking at two side by side cars or a bus .. all very unreliable without vision.

Let me be very clear .. Lidar and Radar clearly do offer additional data that pure cameras do not. However, there are quite a few posts in this thread that more or less say "just add Lidar, that can figure it all out easily", which are just plain wrong. Lidar/Radar are not panaceas, and deliver some additional data I the form of a 3D point map with some approach velocity information. Properly integrated, this can, under some conditions, reinforce the accuracy of the camera views. However, no-one at present knows (not even Tesla), if this reinforcement is necessary, since the accuracy achievable by pure cameras is still in active development (as we see with the FSD beta). Tesla have taken a bet that the necessary accuracy and reliability can be achieved with cameras alone, and that is a big bet. If they succeed, they emerge with a huge advantage over the competition .. if they fail, they look silly and have some major re-work to do.

 

[drtimhill]

I'm still unclear on why Waymo and Cruise "don't count" as examples of the helpfulness of LIDAR. I could see that argument if they only used LIDAR for localization on the HD map but they also use it for objects that are not mapped.

I'm sure lidar is very helpful to Waymo, and in fact I'm sure, right now, they can't do without it. As I noted elsewhere, Waymo started out long enough ago that a pure vision approach was never going to fly as the raw processing power way not then available at a viable price point. The result, I suspect, is that their whole self-driving stack is based on all those different sensors being integrated, and to go pure vision now would be a massive undertaking .. something management would probably never buy into.

The point isnt if Lidar or radar can help .. they can, kind of, if you are careful. The point is, are they necessary. And that is ultimately a question about how good vision only systems can be .. if they are fine, then why bother with lidar etc? Tesla are going down a vision only path because, if they do succeed, they have a significant business advantage over the competition. They have given themselves a hard problem to solve though, and no-one really knows if they can do it or not.

 

[Daniel in SD]

Depends on how well it integrates into your other data. The difference with vision based point cloud data is every point is already directly associated with a point in your image, so you don't have to sync them. TOF cameras do the same thing in sort of a hybrid between cameras and lidar.

This doesn't make sense to me. If you know the positions of your camera and lidar sensors then you know the position of your camera pixels.

I'm sure lidar is very helpful to Waymo, and in fact I'm sure, right now, they can't do without it. As I noted elsewhere, Waymo started out long enough ago that a pure vision approach was never going to fly as the raw processing power way not then available at a viable price point. The result, I suspect, is that their whole self-driving stack is based on all those different sensors being integrated, and to go pure vision now would be a massive undertaking .. something management would probably never buy into.

The point isnt if Lidar or radar can help .. they can, kind of, if you are careful. The point is, are they necessary. And that is ultimately a question about how good vision only systems can be .. if they are fine, then why bother with lidar etc? Tesla are going down a vision only path because, if they do succeed, they have a significant business advantage over the competition. They have given themselves a hard problem to solve though, and no-one really knows if they can do it or not.

It sounds like we are all now agreeing. Though I would argue that adding LIDAR is the easy solution to never hitting any fixed objects. Waymo claims to have gone 6 million miles without hitting a fixed object. And back in the DARPA urban challenge the winning vehicle didn't hit anything either (and that was without HD maps!).

I actually don't think Tesla should add LIDAR unless they're sure they can use it to achieve driverless or L3 operation. I expect more companies to go to camera only systems (Honda and Subaru already do camera only). It will be interesting to see if Tesla adds any sensors to the upcoming robotaxi model.

 

[Daniel in SD]

It will be interesting to see if Tesla adds any sensors to the upcoming robotaxi model.

I suspect this is related to the robotaxi model development. Clearly Tesla engineers also think range finding sensors are helpful.

https://twitter.com/x/status/1489841690601041924

 

[diplomat33]

Paid driverless rides for Cruise are now rolling out in SF:

https://twitter.com/x/status/1539997375661342721

 

[diplomat33]

Pony.ai launches robotaxi service in China:

https://twitter.com/x/status/1540003825653649413

 

[Daniel in SD]

Pony.ai launches robotaxi service in China:

https://twitter.com/x/status/1540003825653649413

I don't think it should be called a robotaxi service if they still have a safety driver.
Though I would be curious what the Uber rating of an FSD Beta "robotaxi service" would be. haha.

 

[diplomat33]

The Verge has some details on the pricing for a Cruise driverless ride:

The cost for riding in one of Cruise’s driverless vehicles will vary depending on the length of the trip and the time of day. According to an example provided by the company, a customer taking a 1.3-mile trip would pay $0.90 per mile and $0.40 per minute, in addition to a $5 base fee and 1.5 percent city tax, for a total of $8.72. (By comparison, an Uber ride for the same trip would cost at least $10.41.)

Cruise is now charging for rides in its driverless vehicles in San Francisco

It’s an important milestone for the GM-backed company.

www.theverge.com

So it looks like Cruise will be cheaper than Uber.

 

[Daniel in SD]

The Verge has some details on the pricing for a Cruise driverless ride:

Cruise is now charging for rides in its driverless vehicles in San Francisco

It’s an important milestone for the GM-backed company.

www.theverge.com

So it looks like Cruise will be cheaper than Uber.

At this point what Cruise charges has nothing to do with profitability, the robotaxi service is R&D. The amount of revenue they'll collect from rides in SF is minuscule relative to their burn rate. I'm looking forward to the Origin and expansion to other cities.

 

[diplomat33]

At this point what Cruise charges has nothing to do with profitability, the robotaxi service is R&D. The amount of revenue they'll collect from rides in SF is minuscule relative to their burn rate. I'm looking forward to the Origin and expansion to other cities.

Yes but I think the pricing rate is still informative. There are many here who have asked what the cost of a robotaxi will be compared to an Uber. Now, we have some idea. I do think the pricing of the Origin rides, especially when it scales, will be different than the pricing of these Bolt rides. And the ability to undercut Uber will be an advantage when Cruise scales.

 

[stopcrazypp]

This doesn't make sense to me. If you know the positions of your camera and lidar sensors then you know the position of your camera pixels.

Not true at all, you still need to map them separately because your sensors are not positioned in the exact same place. The relative points between the two sensors can change depending on the the relative positions of the object with the two sensors, so a simple static mapping does not work. Think of parallax and stereo cameras: the distance between cameras and other sensors typically is much further than even those.

As an example, look at the issues Tesla have had syncing radar and vision data. It can result in something worse than camera alone when it maps the output to the wrong pixel.

That doesn't happen with vision based point clouds or TOF cameras because the image pixel and the point cloud source is literally the same one so there is a direct mapping that does not change with object position.

Another example I can think of is AF calibration for DSLRs, necessary because of the different relative positions of the AF sensor and image sensor. This need was eliminated when mirrorless cameras came out due to AF pixels on the sensor itself.

It sounds like we are all now agreeing. Though I would argue that adding LIDAR is the easy solution to never hitting any fixed objects. Waymo claims to have gone 6 million miles without hitting a fixed object. And back in the DARPA urban challenge the winning vehicle didn't hit anything either (and that was without HD maps!).

I actually don't think Tesla should add LIDAR unless they're sure they can use it to achieve driverless or L3 operation. I expect more companies to go to camera only systems (Honda and Subaru already do camera only). It will be interesting to see if Tesla adds any sensors to the upcoming robotaxi model.

Tesla is hitting curbs Iiterally because the cars can't even see them directly (it's just extrapolating like a human does). It doesn't even require advanced sensors, just something like the 360 parking cameras that have been out for ages to cover the blind areas in the front.

 

[Daniel in SD]

Not true at all, you still need to map them separately because your sensors are not positioned in the exact same place. The relative points between the two sensors can change depending on the the relative positions of the object with the two sensors, so a simple static mapping does not work. Think of parallax and stereo cameras: the distance between cameras and other sensors typically is much further than even those.

As an example, look at the issues Tesla have had syncing radar and vision data. It can result in something worse than camera alone when it maps the output to the wrong pixel.

That doesn't happen with vision based point clouds or TOF cameras because the image pixel and the point cloud source is literally the same one so there is a direct mapping that does not change with object position.

Tesla is hitting curbs Iiterally because the cars can't even see them directly (it's just extrapolating like a human does). It doesn't even require advanced sensors, just something like the 360 parking cameras that have been out for ages.

There must be something I'm missing. This is what I'm thinking. The points of a LIDAR point cloud are exactly known in all three dimensions relative to the sensor. Each pixel is somewhere along a cone the position of which is also known. Where the cone intersects the point cloud is the position of the pixel.
By the time a curb is in the blindspot of the current camera positions it's already too late unless you think it's acceptable to be doing emergency type maneuvers to avoid curbs...

EDIT: Now I see that this mapping only works well at distances that are multiples of the distance between the sensors. Still it doesn't seem that hard to mount them close enough together for it not to be an issue.

 

[stopcrazypp]

There must be something I'm missing. This is what I'm thinking. The points of a LIDAR point cloud are exactly known in all three dimensions relative to the sensor. Each pixel is somewhere along a cone the position of which is also known. Where the cone intersects the point cloud is the position of the pixel.

You are assuming a zero distortion (or near zero distortion) lens, which I doubt any of the cameras being in use for ADAS use.

By the time a curb is in the blindspot of the current camera positions it's already too late unless you think it's acceptable to be doing emergency type maneuvers to avoid curbs...

Most of the FSD curb hitting (including the sole "crash" into a bollard) involved a slow speed hit (a bunch of them happened at stop signs) that a parking camera could have easily prevented.

EDIT: Now I see that this mapping only works well at distances that are multiples of the distance between the sensors. Still it doesn't seem that hard to mount them close enough together for it not to be an issue.

 

[drtimhill]

It sounds like we are all now agreeing. Though I would argue that adding LIDAR is the easy solution to never hitting any fixed objects. Waymo claims to have gone 6 million miles without hitting a fixed object. And back in the DARPA urban challenge the winning vehicle didn't hit anything either (and that was without HD maps!).

I would be interested in your comments contrasting this position to the PB issues, since the two are closely related...

As fro DARPA .. there is always this (to add a little humor to the debate)...

 

[Daniel in SD]

You are assuming a zero distortion (or near zero distortion) lens, which I doubt any of the cameras being in use for ADAS use.

No, I'm assuming a known distortion camera. Outside of the camera light travels in a straight enough line.

Most of the FSD curb hitting (including the sole "crash" into a bollard) involved a slow speed hit (a bunch of them happened at stop signs) that a parking camera could have easily prevented.

The bollard is definitely visible to the center camera when the car hits it! The curb strikes I have seen have all been moderate speed right turns. Though many people claim it would hit them during higher speed left turns but no one seems to want to prove it.

 

[Daniel in SD]

I would be interested in your comments contrasting this position to the PB issues, since the two are closely related...

PB?