Autonomous Car Progress

[Daniel in SD]

Nope, wasn't assuming anything, I was merely pointing out that there was an unvoiced assumption that adding lidar/radar would yield a fast(er) track to some level or other of self-driving.

Waymo has operational robotaxis which I would classify as some level of self-driving. Are you arguing they could have achieved that with the current state of the art in computer vision?

 

[MarkDave]

I'm not understanding the huge optimism surrounding Tesla's vision-heavy approach to solving self driving cars.

In what area is Tesla truly ahead of the curve? Even in their bread and butter: battery tech and EVs, Tesla is maybe a few years ahead of other manufacturers at best. You want an electric truck, you get a Rivian or Ford. You want 300 mile range, you get a Hyundai. You want acceleration, you get a Porsche. You want fast charging, you get a Hyundai. Etc.

I just don't see why it's reasonable to believe that Tesla is somehow ahead or going to be ahead in computer vision, which is well outside Tesla's core competencies.

Even if CV were a Tesla core conptency like batteries, ramge, and actually building an EV, it's not as if Tesla has some sort of insurmountable advantage over other players. There are lots of competent EVs now that aren't from Tesla.

 

[Dewg]

I'm not understanding the huge optimism surrounding Tesla's vision-heavy approach to solving self driving cars.

In what area is Tesla truly ahead of the curve? Even in their bread and butter: battery tech and EVs, Tesla is maybe a few years ahead of other manufacturers at best. You want an electric truck, you get a Rivian or Ford. You want 300 mile range, you get a Hyundai. You want acceleration, you get a Porsche. You want fast charging, you get a Hyundai. Etc.

I just don't see why it's reasonable to believe that Tesla is somehow ahead or going to be ahead in computer vision, which is well outside Tesla's core competencies.

Even if CV were a Tesla core conptency like batteries, ramge, and actually building an EV, it's not as if Tesla has some sort of insurmountable advantage over other players. There are lots of competent EVs now that aren't from Tesla.

Not sure, but their approach is unique in that they are going for AI and neural nets. Even if they end up adding additional sensors or cameras, their system has the potential to outpace others who are relying heavily on mapping data. I think the goal is that a Tesla will be able to drive nearly anywhere with basic map data. I don't know if they'll get there, but that's the goal. You'll know they are getting somewhere when other manufacturers start using a similar approach, even with additional sensors.

 

[MarkDave]

but their approach is unique in that they are going for AI and neural nets.

Definitely not unique. Everyone is using neural nets for a large part of the perception layer. Audi in 2017's A8 L3 driving system (that never got publicly enabled) used convolutional NNs. Mercedes' actual production L3 that you can test in Germany at the moment uses CNNs as well. (Congratulations Mercedes-Benz and Veoneer!)

The future appears to involve more than CNN's but also Transformer NNs.

 

[drtimhill]

Waymo has operational robotaxis which I would classify as some level of self-driving. Are you arguing they could have achieved that with the current state of the art in computer vision?

No, I'm arguing that when people say "Tesla should have used xxx" such as radar/lidar, they have no factual basis for that argument. They MAY turn out to be correct, but since no-one has yet demonstrated if these augmentations really do help, its all speculation at this point. And too many posts in these threads take it as axiomatic that they WILL help .. again without substantiating that.

Waymo in fact are an indication that they DONT solve the issues (yet) .. do you think they would still faff about with HD maps and geofencing if all those sensors had in fact solved general self-driving a la Tesla (L2 but no goefencing etc)?

 

[drtimhill]

It is not an assumption. It has yielded a faster track to some level of self-driving. We have real self-driving (at L4 level) in several areas with the vision+lidar+radar approach. To-date, we have no real self-driving with vision-only. By "real self-driving", I mean driverless or L4/L5.

So Waymo could turn off HD maps and geofencing and everything would work just fine?

 

[Daniel in SD]

No, I'm arguing that when people say "Tesla should have used xxx" such as radar/lidar, they have no factual basis for that argument. They MAY turn out to be correct, but since no-one has yet demonstrated if these augmentations really do help, its all speculation at this point. And too many posts in these threads take it as axiomatic that they WILL help .. again without substantiating that.

Waymo in fact are an indication that they DONT solve the issues (yet) .. do you think they would still faff about with HD maps and geofencing if all those sensors had in fact solved general self-driving a la Tesla (L2 but no goefencing etc)?

How could Lidar possibly solve “the issues”? It’s just a high resolution range finder. That’s the only issue it solves.
I don’t think it’s speculation to say it helps when every driverless vehicle currently uses Lidar.

I guess I agree that if the only goal is L5 no geofence then there is no point in using LiDAR because I think that will require AGI and I expect whatever technology that requires to also achieve human level vision.

 

[diplomat33]

Mobileye just announced "cloud enhanced driver assist". My understanding is that it combines the base driver assist with Mobileye's AV maps. So with "cloud enhanced driver assist", a car with Mobileye's base driver assist system will be able to access the AV maps in the cloud for better capabilities.

The article gives some examples of how it will improve the base driver assist:

Imagine you’re driving along a multi-lane road. Maybe it’s snowing, or raining, or foggy, or dark. Maybe the lane markers have worn away with time and haven’t been repainted in a while. You can’t see them with your own eyes, and the vehicle’s onboard sensors may not be able to, either. But Mobileye’s system knows where they are, and our Cloud-Enhanced Driver-Assist solution helps the vehicle to stay centered in the lane – keeping you on the proverbial straight and narrow.

Or picture arriving at a busy intersection. There’s a whole mess of traffic lights – one or more sets for each lane in each direction of traffic. Cloud-Enhanced Driver-Assist knows exactly which lights are relevant for the lane you’re in, enabling the vehicle to alert you (or even apply the brakes itself) if you’re about to roll through a red light.

That construction zone around the next bend or over the next rise? A toll gate coming up fast on that highway exit ramp? The speed at which traffic actually travels (independent of the posted limit) on a given stretch of road? Or the line drivers typically take around a corner? Cloud-Enhanced Driver-Assist can access all that information – thanks to innovations from our autonomous-vehicle program – to augment the ADAS performance in today’s human-driven vehicles.

Cloud-Enhanced Driver-Assist™ Takes ADAS to the Next Level | Mobileye Blog

Mobileye’s latest solution combines our expertise in advanced driver-assistance systems with the power of our cloud-based, crowdsourced mapping technology.

www.mobileye.com

 

[hgmichna]

When I look at actual accidents, caused by human drivers, the cause is overwhelmingly a driver mistake in good visibility. Even in poor visibility, accidents do not rise sharply. You get a strong increase in the accident rate only when at least two problematic conditions come together, such as poor visibility (rain, snow, etc.) and darkness, i.e. under conditions not so often encountered by the human driver.

Even for an accident under such unfavorable conditions one could argue that the driver has simply not adapted to the conditions. An autonomous car would adapt, I presume, so no more accidents due to poor visibility. The properly designed autonomous car would rather give up and park than cause an accident.

This leaves only the argument that an autonomous car could drive faster with additional sensors. This, I guess, is not a high priority for Tesla right now.

 

[diplomat33]

How could Lidar possibly solve “the issues”? It’s just a high resolution range finder. That’s the only issue it solves.

No. Lidar is more than just a range finder. HD lidar creates a very dense 3D point cloud that is good enough to create a 3D video of the world around you, good enough to drive on. And you can use the 3D point cloud to classify objects, determine speed of objects, detect road edges, etc... You can navigate a route and avoid obstacles on lidar-only. That's one reason why all AVs use lidar. It is a very powerful sensor.

I guess I agree that if the only goal is L5 no geofence then there is no point in using LiDAR because I think that will require AGI and I expect whatever technology that requires to also achieve human level vision.

Lidar would still be very useful for L5 no geofence. Remember that you can use lidar without HD maps. So even if you ditch HD maps because you don't think HD maps are viable for L5, you can still use lidar. Lidar is just a powerful sensor for seeing the world in 3D. AGI is not incompatible with lidar. Lidar helps with perception which in turn will help your prediction/planning make smarter decisions. So you can use lidar to help see the world and use AGI to make smart decisions based on the vision-lidar-radar data.

 

[Daniel in SD]

No. Lidar is more than just a range finder. HD lidar creates a very dense 3D point cloud that is good enough to create a 3D video of the world around you, good enough to drive on. And you can use the 3D point cloud to classify objects, determine speed of objects, detect road edges, etc... You can navigate a route and avoid obstacles on lidar-only. That's one reason why all AVs use lidar. It is a very powerful sensor.

You just described a high resolution range finder.

Lidar would still be very useful for L5 no geofence. Remember that you can use lidar without HD maps. So even if you ditch HD maps because you don't think HD maps are viable for L5, you can still use lidar. Lidar is just a powerful sensor for seeing the world in 3D. AGI is not incompatible with lidar. Lidar helps with perception which in turn will help your prediction/planning make smarter decisions. So you can use lidar to help see the world and use AGI to make smart decisions based on the vision-lidar-radar data.

Yes, I wouldn't mind having a Lidar sensor interfaced to my brain. My prediction is that once the technology exists to achieve "L5 no geofence" on the prediction and planning side the same AI technology will also allow super human perception using only cameras. So, @drtimhill is right that LiDAR is not necessary for "general self-driving." Now I also predict that what he wants won't happen for decades and there are useful, much more limited, driverless vehicles that will be deployed in the meantime. The designers of those vehicles clearly find LIDAR helpful.

 

[Goose66]

I think front 3D radar and two long range side radars in front bumper would be more valuable than lidar. But again, utilizing highly detailed, crowd sourced AV maps -- at least where available -- would have far more benefit than additional sensors, IMO. Doesn't mean you have to limit operation to where the maps were available. Just means it will be smoother and more confident in those areas.

 

[diplomat33]

Zoox completes "critical checkpoint" in their robotaxi testing:

Jesse Levinson, the Zoox founder and CTO, told the audience at Amazon’s re:MARS event here Tuesday night that Zoox “recently completed a critical checkpoint that we haven’t talked about publicly yet.”

Specifically, he said, the Zoox vehicle operated “with no one inside, no chase vehicle, and no emergency stop, all on open, private roads with non-Zoox agents, including pedestrians, cyclists, cars and trucks.”

The vehicle satisfied safety requirements in these tests, which means it “can operate in an unstructured environment at human-plus safety levels,” he said. “And we’re almost ready to do that on public roads.”

https://twitter.com/x/status/1539625355404644352

 

[MarkDave]

Even if they end up adding additional sensors or cameras, their system has the potential to outpace others who are relying heavily on mapping data. I think the goal is that a Tesla will be able to drive nearly anywhere with basic map data.

I think the issue with "basic" map data is that detailed map data is incredibly useful. Think of how many spots where the foliage or overgrown bushes or statues or parked cars block your ability to see enough of the intersection to make an unprotected left turn onto a busy road. Or even a right turn. A true autonomous vehicle can't just drive up to the white stop line and stop on the line and claim to be "safe" because it's following the letter of the law about where to stop.

A powerful autonomous vehicle would be able to "reason" and inch forward to perceive more of the scene to make an informed decision about when to make that unprotected left.

However, it'd be easier with detailed map data. Human drivers are always going to overshoot the white stop line when making that unprotected left. Gather enough data from humans and you'll know exactly just how far past that white stop line you need to be to see the entire traffic scene, without getting hit by oncoming traffic, and make an informed decision about making a left/right turn. This is just one example of how detailed map data can help shoulder the computational burden of making an autonomous vehicle.

Robotics is all about maximizing your givens, as stated by Jim Keller shortly after he left Tesla. A detailed map showing the consensus vantage point at every intersection in a city would be hugely helpful versus forcing the car to figure it out on the fly. And this is exactly the data that Mobileye has been collecting, en masse.

The current Tesla approach to forget everything that happened on yesterday's commute and just figure it out over and over again is just plain inefficient.

 

[diplomat33]

So Waymo could turn off HD maps and geofencing and everything would work just fine?

Yes, but define "just fine". Waymo can drive without HD maps but safety might be worse. HD maps improve safety so Waymo thinks it would be silly to turn them off and lose that safety advantage. Waymo is not interested in FSD with supervision. Waymo is only interested in fully driverless with very high safety. HD maps help make that happen.

 

[Dewg]

I think the issue with "basic" map data is that detailed map data is incredibly useful. Think of how many spots where the foliage or overgrown bushes or statues or parked cars block your ability to see enough of the intersection to make an unprotected left turn onto a busy road. Or even a right turn. A true autonomous vehicle can't just drive up to the white stop line and stop on the line and claim to be "safe" because it's following the letter of the law about where to stop.

A powerful autonomous vehicle would be able to "reason" and inch forward to perceive more of the scene to make an informed decision about when to make that unprotected left.

However, it'd be easier with detailed map data. Human drivers are always going to overshoot the white stop line when making that unprotected left. Gather enough data from humans and you'll know exactly just how far past that white stop line you need to be to see the entire traffic scene, without getting hit by oncoming traffic, and make an informed decision about making a left/right turn. This is just one example of how detailed map data can help shoulder the computational burden of making an autonomous vehicle.

Robotics is all about maximizing your givens, as stated by Jim Keller shortly after he left Tesla. A detailed map showing the consensus vantage point at every intersection in a city would be hugely helpful versus forcing the car to figure it out on the fly. And this is exactly the data that Mobileye has been collecting, en masse.

The current Tesla approach to forget everything that happened on yesterday's commute and just figure it out over and over again is just plain inefficient.

With storage being as cheap as it is, it's possible the systems will learn and record their information for future drives. I agree it's better to learn a route that's been driven before, just as humans start to learn routes as they drive. However, I can plop you down in a different place that you've never been to, and you can drive in that place pretty well with just your two eyes and brain. Hopefully that's the place FSD will get to at some point. Then it doesn't really matter to the CPU whether it's been somewhere before, it's all the same to the system.

 

[diplomat33]

Informative marketing video that details Pony.AI FSD for consumer cars.

The video is sped up but shows and entire trip on streets and highways with zero intervention. It can even handle toll booths with no human intervention. The lane change logic is also quite good. It knows the stay in the left lane when the cars in the right lane are too slow. So unlike NoA, it does not lane change unnecessarily back and forth.

It uses the Nvidia Orin chip and surround cameras, surround radar and only 1 forward lidar.

 

[MarkDave]

No, I'm arguing that when people say "Tesla should have used xxx" such as radar/lidar, they have no factual basis for that argument. They MAY turn out to be correct, but since no-one has yet demonstrated if these augmentations really do help, its all speculation at this point. And too many posts in these threads take it as axiomatic that they WILL help .. again without substantiating that.

Waymo in fact are an indication that they DONT solve the issues (yet) .. do you think they would still faff about with HD maps and geofencing if all those sensors had in fact solved general self-driving a la Tesla (L2 but no goefencing etc)?

In my view, nearly every FSD beta release notes document highlights how Tesla is throwing an inordinate amount of man-hours and salaries at the problem of getting around the lack of information that an active perception system would grant.

Lidar and Radar interrogate the environment unlike passive sensors, such as the cameras.

Lidar and especially radar can return near-instantaneous positions, velocities, and accelerations of objects at a reliability far greater than any current computer vision techniques.

Release notes examples:

Improved creeping for visibility using more accurate lane geometry and higher resolution occlusion detection.

This isn't exactly releated to this post but this goes to highlight a point I made in my post above about the usefulness of HD maps. Knowing where 100,000 human drivers stopped relative to a white stop line costs a lot less, computationally, than having the self-driving software have to figure out how far to creep forward at every stop line.

Improved angular velocity and lane-centric velocity for non-VRU objects by upgrading it into network predicted tasks.

Ah ha, there's one. Tesla is forced to estimate velocities because Tesla doesn't want to rely on active sensors such as lidar or radar, both of which would return the true velocities, rather than an estimation.

Increased reliance on network-predicted acceleration for all moving objects, previously only longitudinally relevant objects.

More estimations rather than precise and exact measurements enabled by active sensors (such as radar or lidar).

Now, one might ask--does it matter? In my estimation, it matters a lot. Neural networks can only ever hope to approximate reality. The Universal Approximation Theorem, in sum:

Universal approximation theorems imply that neural networks can represent a wide variety of interesting functions when given appropriate weights. On the other hand, they typically do not provide a construction for the weights, but merely state that such a construction is possible.

Active sensors measure reality rather than merely "represent" reality. Heavy reliance on NN's to estimate things that other teams measure just leads to an unfavorable stackup of probabilities.

Let's say the Tesla NN for estimating lead car velocity is correct 99% of the time. The Tesla NN for estimating lead car distance is correct 99% of the time. Naively assuming independence: the probability of them being jointly correct would be 0.99*0.99=0.9801 or jointly correct 98% of the time. A team using radar would get the lead car velocity and lead car distance jointly correct basically 100% of the time.

Does this difference, however slight, matter? I'd say it does, given just how many millions of miles humans collectively drive on a daily basis. The millions, if not billions, of lead cars that human drivers interact with, on a daily basis.

Also, there's a lot more to driving than simply knowing lead car velocity and the distance to the lead car. Every single one of these things that Tesla has to estimate rather than measure just makes the stackup of probabilities even more unfavorable. As the release notes show, it's not just linear velocities that Tesla has to estimate. It also has to estimate angular velocities, and Tesla is still working on improving angular velocity estimation. Meanwhile, the other teams with radar and lidar consider that issue solved; that's basically a trivial task.

Another term comes to mind when contemplating Tesla's approximation-heavy approach to estimating the physics of other road users: error propagation. There's inevitably going to be some error between reality and what Tesla's NNs output. The hope is that this error is vanishingly small, or at least small enough that it doesn't result in a crash. The problem is that the motion planner has to deal with bad information when this error is large. Garbage in, garbage out. The hope is that the motion planner can be fed data that's as accurate as possible. The shortcut to feeding the motion planner accurate physics of other road users is to take direct measurements. The Tesla approach is to refine its estimators (neural networks) of road user physics. If Tesla's estimators are far off, then even the most brilliantly coded Tesla motion planner is reduced to a case of garbage in and garbage out.

We can finally ground our discussion back into reality by considering the state of Tesla's auto high beam functionality. From what I understand, it's hit-and-miss, to say the least. Auto high beams simply need to a) identify nearby vehicles forward of the vehicle and b) estimate the distances of nearby vehicles. The problem is that Tesla's 3 forward-facing cameras and Tesla's Computer Vision prowess apparently aren't enough for reliable AHB functionality. Which leads me to doubt Tesla's CV prowess in other areas. If Tesla's ability to identify forward vehicles and Tesla's estimators of forward vehicle distance can't reliably function to enable reliable AHB, then that leads to a lot of other questions ...

 

[drtimhill]

Release notes examples:


This isn't exactly releated to this post but this goes to highlight a point I made in my post above about the usefulness of HD maps. Knowing where 100,000 human drivers stopped relative to a white stop line costs a lot less, computationally, than having the self-driving software have to figure out how far to creep forward at every stop line.



Ah ha, there's one. Tesla is forced to estimate velocities because Tesla doesn't want to rely on active sensors such as lidar or radar, both of which would return the true velocities, rather than an estimation.


More estimations rather than precise and exact measurements enabled by active sensors (such as radar or lidar).

But this is my point ... everyone is treating lidar and radar as "magical solutions". Can't get accurate motion? Add lidar/radar! Can't estimate xxx? Add lidar/radar!! Tooth decay? Try Cream Toothpaste now with Lidar protection!

Are you so sure that lidar/radar could indeed handle those issues mentioned in the release notes? Based on what?

Active systems have some advantages (better 3D perception and invariance of external illumination for example), but they are NOT a magical cure-all. First of all, the data coming from such systems needs the same kind of AI as cameras if its to be useful. Second, as active systems, they are subject to cross-interference (and this is already becoming a problem, though my apologies for not having the relevant papers to quote at hand). Finally, the resolution is way below that of cameras, unless you go for a high-end exotic system that can cost more than the car itself. Meaning that even when lidar/radar says "ohh big and shiny" its the cameras and the NN that have to figure out if "big and shiny" is an obstacle or not.

And Lidar/radar cannot return "true" velocities .. they can only directly return a component of the velocity based on effects such as doppler shift .. they are more or less useless for lateral velocity, which requires .. that's right .. an NN just like the cameras!

 

[diplomat33]

And Lidar/radar cannot return "true" velocities .. they can only directly return a component of the velocity based on effects such as doppler shift .. they are more or less useless for lateral velocity, which requires .. that's right .. an NN just like the cameras!

For regular radar yes but I don't think this is true for lidar. Lidar creates a 3D point cloud. You can identify the group of points that match an object and measure the velocity of that group of points to get the true velocity of the object. HD radar can also create a high res 3D point cloud. So you could probably do the same with HD radar. And yes, it uses NN. But everything uses NN.