Autonomous Car Progress

[powertoold]

He probably thought it was a solved problem because the technique was available (computer vision), and it was a matter of maturing the software and labeling workflow. As of now, he's still right, it's a "solved problem," judging from the advances in computer vision and green's recent tweet about the depth perception outputs (they're very good IMO).

Elon's job isn't to see things in monthly time-frames. He's good at seeing things far into the future and judging whether it's viable at all. So far, his long term track record is amazing. Vision-only fsd seems more and more viable every day. Many of us didn't think so when he was first saying it's possible or "solved."

 

[Soda Popinski]

when not explicitly stated it could be assumed I am running current prod versions

Isn't there also an additional distance estimation based on visual segmentation? I wonder how well that does past 62 meters.

 

[mikes_fsd]

Anyway, we have another cool new twitter thread from @verygreen which shows the output from the depth NN, which seems pretty impressive for passive vision only:

I wanted to add one note to this:
Even though green is stating this initial 160x120 resolution as a negative (eg 1/8th the native camera resolution) this is WAY better than the resolution on the radar.

Radar can do 40 points per frame at 10 frames per second.
Radar: 40 points at 10fps
vs
Vision: 19200 points at 36fps.

See both screenshots below:

https://twitter.com/x/status/1412795353011736576

About distance, this particular example is from the fisheye feed, but it's a NN, can be applied to ALL camera feeds (I would argue for FSD it needs to be running on all the feeds).
About fidelity, again, this is not a hardware problem but a NN training problem, if they can get this accuracy at this stage, this should see improvement going forward.

Speed cannot be gleaned from a single frame in Tesla Vision approach, but since everything is done in video (i.e. with time) you can get speed from that (indeed they already do with radar-less 3/Y's in production).

 

[dgatwood]

And that optimism is why we have Tesla and SpaceX today. But it's also why he's pursuing some ridiculous ideas like Hyperloop and Mars colonization. He thinks hard problems are easy so he throws himself into them and expects them to be solved soon. Without that optimism, we would not be driving electric cars today.

 

[daniel]

I would posit that no one take cold fusion seriously because after decades of false promises, everyone stopped paying attention to the constant "cold fusion in 10 years" predictions.

However, there are still a few people that take cold fusion seriously, including a small company some of us may be familiar with:

"... cold fusion research has been funded by private and small governmental scientific investment funds in the United States, Italy, Japan, and India. For example, it was reported in Nature, in May, 2019, that Google had spent approximately $10 million on cold fusion research."

Conclusion: 10 years away! Joking, actual conclusion was:

"A group of scientists at well-known research labs (e.g, MIT, Lawrence Berkeley National Lab, and others) worked for several years to establish experimental protocols and measurement techniques in an effort to re-evaluate cold fusion to a high standard of scientific rigor. Their reported conclusion: no cold fusion."

When Pons and Fleischmann (sp?) first announced cold fusion I was skeptical, as were many others. When their results were thoroughly debunked, I dismissed it as a crank idea. I thought pretty much everybody did also. Ten million dollars is chump change for Google. It's not even a serious investment.

Controlled hot fusion is the thing that's always been "just a few years away" and has never come to anything in spite of the massive sums invested. (Uncontrolled fusion --bombs-- they've had down pat for decades.)

I think that's a very naive view of Musk. I view his approach as:
(1) Identifying a difficult engineering problem with a large reward
(2) Sell stories of hope to investors and customers
(3) Invest in engineering to try to solve the problem
(4) If problem not solved, go to (2)
(5) Profit

And the result is electric cars are now mainstream, a private company is sending astronauts and supplies to the ISS at a fraction of the cost NASA was spending, and a few other ideas have not panned out.

Investing carries risks. A wise investor weighs those risks. Selling FSD before it exists is, in my opinion, dishonest. I think Elon really screwed up on that one. But as for "selling hope to investors," you're a fool if you invest in a project without learning about it from all angles and deciding if the risk is worth it to you.

 

[verygreen]

Isn't there also an additional distance estimation based on visual segmentation? I wonder how well that does past 62 meters.

there are other NNs that do distance estimation and even speed and acceleration. Those work beyond 62 meters

 

[verygreen]

Even though green is stating this initial 160x120 resolution as a negative (eg 1/8th the native camera resolution) this is WAY better than the resolution on the radar.

Radar can do 40 points per frame at 10 frames per second.
Radar: 40 points at 10fps
vs
Vision: 19200 points at 36fps.

well, that was mostly a comparison to lidar, anyway.

Do note you cannot directly compare radar to this point NN here because:
1. Radar gives you speed for every point too
2. Radar can give you 40 points of those 19200 (actually likely even more) it's just some tiny ECU selects which points it thinks you are more interested in without you having a direct input. (and it tends to select something moving vs something that does not)

 

[mikes_fsd]

1. Radar gives you speed for every point too

Covered/pointed to that at the end of my post.

2. Radar can give you 40 points of those 19200 (actually likely even more) it's just some tiny ECU selects which points it thinks you are more interested in without you having a direct input. (and it tends to select something moving vs something that does not)

No matter how you slice it, in this example, the vision approach gives you more granular data points and more consistent data than the legacy radar stack (at least about depth/distance).
No down selection, you always get the 19200 points for every frame.

 

[verygreen]

No matter how you slice it, in this example, the vision approach gives you more granular data points and more consistent data than the legacy radar stack (at least about depth/distance).
No down selection, you always get the 19200 points for every frame.

yes, when it works, anyway (I am sure you have heard about low visibility scenarios)

 

[mikes_fsd]

yes, when it works, anyway (I am sure you have heard about low visibility scenarios)

Like fog, rain, snow - where lidar performs so much better?!

Cool, come back with, "but radar"....

well, that was mostly a comparison to lidar, anyway.

Anyway!

 

[Daniel in SD]

Hopefully they’ll use this depth estimation for autopark soon. It will be interesting to see how reliable it is. Didn’t they demo this at autonomy day?

 

[Bladerskb]

I wanted to add one note to this:
Even though green is stating this initial 160x120 resolution as a negative (eg 1/8th the native camera resolution) this is WAY better than the resolution on the radar.

Radar can do 40 points per frame at 10 frames per second.
Radar: 40 points at 10fps
vs
Vision: 19200 points at 36fps.

See both screenshots below:
View attachment 682315
View attachment 682316

https://twitter.com/x/status/1412795353011736576

About distance, this particular example is from the fisheye feed, but it's a NN, can be applied to ALL camera feeds (I would argue for FSD it needs to be running on all the feeds).
About fidelity, again, this is not a hardware problem but a NN training problem, if they can get this accuracy at this stage, this should see improvement going forward.

Speed cannot be gleaned from a single frame in Tesla Vision approach, but since everything is done in video (i.e. with time) you can get speed from that (indeed they already do with radar-less 3/Y's in production).

State of the Art digital software-defined 4D imaging radars used by Mobileye and Waymo has over 500k PPS.

Tesla’s 2011 & 2014 2D Radar is simply worthless in comparison.

@verygreen

 

[verygreen]

Like fog, rain, snow - where lidar performs so much better?!

reportedly modern lidar works in those conditions now. But I have no direct experience with lidar myself so who knows.

But lidar works in the complete darkness and cameras don't (I know you'll bring headlights, but the car does not have 360 degree coverage of headlights and on lowbeams the range is sort of short anyway). This is what sets aside active sensors from passive. Active sensors just work no matter what's outside and passive need some favorable outside conditions (e.g. a light source in case of the camera)

 

[dgatwood]

reportedly modern lidar works in those conditions now. But I have no direct experience with lidar myself so who knows.

But lidar works in the complete darkness and cameras don't (I know you'll bring headlights, but the car does not have 360 degree coverage of headlights and on lowbeams the range is sort of short anyway). This is what sets aside active sensors from passive. Active sensors just work no matter what's outside and passive need some favorable outside conditions (e.g. a light source in case of the camera)

On the flip side, you mostly need to see in the direction you're going. If another vehicle is coming at you from any other direction, it should have headlights.

 

[verygreen]

On the flip side, you mostly need to see in the direction you're going. If another vehicle is coming at you from any other direction, it should have headlights.

may be it's an animal

Also like I said, on lowbeams even the headlights have very limited range.

 

[dgatwood]

may be it's an animal

Also like I said, on lowbeams even the headlights have very limited range.

On the other hand, if it is behind you, you probably won't hit it at all (unless you're backing slowly, in which case that's why you have backup lights), and if it is beside you, either it hits you or it doesn't. Chances are, you won't be able to evade a side impact even if you know it's about to happen. So why worry about it?

Low beams should light up the road for about 200 feet. That's about as long as it takes a human to stop a car at 55 MPH, including reaction time. A self-driving car should do even better. I wouldn't be surprised if Tesla stopped including high beams as soon as they think that self-driving is good enough.

 

[uujjj2]

Can you define what you mean by “unequivocally on top?”

Based on Elon’s comments, it does not seem likely at all that the V9 software release will significantly alter market positioning.

I mean, he only just realized in retrospect that this was difficult (something nearly everyone else knew in advance!!!). Not a great sign! (Probably already posted here but I haven’t been reading all the posts.)

View attachment 682071

Mr. Uujjj personally knows several of the past or former leaders of Tesla's autopilot program, who are stellar engineers. People inside Tesla have been trying to tell Elon for years now that his timeline for FSD is impossible. Elon wouldn't listen. If you told him, you could get fired on the spot. Like Chris Lattner, who Mr. Uujjj knew from a compiler class back at U of I. Internally, the engineering team has always been executing to a much longer-term plan for self-driving than Elon was promoting.

 

[powertoold]

People inside Tesla have been trying to tell Elon for years now that his timeline for FSD is impossible.

Well, that’s for sure, Elon was expecting robotaxis years ago. At least Tesla has “stellar” engineers working on it. There’s a good chance they’ll achieve human level+ safety by the end of this year (judging by the pace of progress).

Chris Lattner is a big shot name in tech, but he was definitely an odd pick for AP / AI. It’s quite refreshing that he left Tesla quickly.

 

[Soda Popinski]

reportedly modern lidar works in those conditions now. But I have no direct experience with lidar myself so who knows.

But lidar works in the complete darkness and cameras don't (I know you'll bring headlights, but the car does not have 360 degree coverage of headlights and on lowbeams the range is sort of short anyway). This is what sets aside active sensors from passive. Active sensors just work no matter what's outside and passive need some favorable outside conditions (e.g. a light source in case of the camera)

I suppose the question I have is, will the car be able to get to better than average human safety with the current sensor suite? Clearly, we're not getting Lidar in our Teslas.

 

[daktari]

@verygreen If I understand you correctly, they need to "fuse" output from this special NN with output from the others anyway. The output seem kind of low res.
-But how long should it take them to get to a higher (more precise) resolution?
-if they do data fusion on outputs anyway, doesn't that mean Karapthy's claim that fusion was "barking at the wrong tree" a bit strange?
I know they have a "no sensor change in any Tesla since late 2016" paradigm but I get a feeling this is a risk for the sw progress in 2023, going down the wrong path, or will sw progress be reusable when they end to change sensor suite?