FSD v12.x (end to end AI)

[Mardak]

There is no way to demonstrate ANY increase in use of driver assist from these data, even though FSD does seem to be better than AP on the freeway (in my experience)

The number of vehicles with FSD Beta was relatively steady earlier this year, so the main factors affecting the cumulative miles driven by FSD Beta each month is mostly attributable to changes in usage of the driver assist.

Very roughly estimating the FSD Beta population over time, the original 100 Safety Score group of ~15k used FSD Beta 10.2 on average nearly 3 miles/day/vehicle, and as this early access group expanded to 97+ scores with ~60k on 10.10, the average nearly dropped to 2 miles/day/vehicle probably not because of less capability of the software but because people selectively limit their use especially for those who weren't specially driving to get admitted sooner.

This average basically halved to 1 mile/day/vehicle after Tesla pushed FSD Beta 10.69.x to ~363k vehicles with FSD Capability including those who didn't opt-in to Safety Score for early access, and this included a lot of people who weren't interested in testing out FSD Beta such as my wife who saw the popup suggestion to try FSD Beta but didn't bother actually using it on city streets. However with 11.x, she does actually use FSD Beta now primarily on highways, and that's reflected in the data with the average jumping up to ~5.6 miles/day/vehicle -- 75% of an order of magnitude increase.

I think what you're getting at is FSD Beta replacing highway basic Autopilot miles doesn't result in significantly more miles driven with Autopilot in general, but it's still a significant step in FSD Beta overall capabilities and usage especially with 11.x development delayed so many times with many 10.x versions needed to test architectural changes needed for safe and reliable single stack at highway speeds.

Hopefully we see this FSD Beta cumulative miles usage rate significantly increase with 12.x end-to-end presumably from increased capabilities and comfort on city streets while maintaining or improving safety.

 

[GWord]

And eyes are on double gimbals (eyeball and neck), substantially higher resolution in the fovea than the cameras, and stereoscopic. And the real problem is now in planning more than perception.

The Sony IMX490 image sensor used in HW4 has approximately 110000 pixels per sq*mm. Not that dissimilar to an average foveal cone cell density? But regardless, I am in strong agreement with you that the big hurdle to FSD usefulness in current builds seems to be around planning.

 

[spacecoin]

Right now, e2e is reliable enough for demos but not reliable enough to be driverless everywhere. ML and computing are making big improvements every year. So it is possible that in the future, maybe 10 or 20 years from now, e2e will be L5.

I think you mean anywhere.

 

[spacecoin]

The Sony IMX490 image sensor used in HW4 has approximately 110000 pixels per sq*mm. Not that dissimilar to an average foveal cone cell density? But regardless, I am in strong agreement with you that the big hurdle to FSD usefulness in current builds seems to be around planning.

It's not really a camera problem. The main problem is to reliably interpret (classify and figure out intent) the images/video at sub optimal conditions like rain at night or fog/smoke - and perform planning based on that - at high enough reliability to remove the human from the loop.

We're probably 10-20 years away from general autonomy even with more and better sensors even in optimal conditions.

 

[GWord]

It's not really a camera problem. The main problem is to reliably interpret (classify and figure out intent) the images/video at sub optimal conditions like rain at night or smoke - and perform planning based on that - at high enough reliability to remove the human from the loop.

Where probably 10-20 years away from general autonomy even with more and better sensors even in optimal conditions.

I think that we may be saying the same thing. Camera sensors are good enough at this point. Perception and planning is the next frontier.

 

[AlanSubie4Life]

I think what you're getting at is FSD Beta replacing highway basic Autopilot miles doesn't result in significantly more miles driven with Autopilot in general,

Yes, that is correct. There's just not a way to determine whether anyone is using Autopilot any more than they used to. Shows as a big step up in the plot, but it's not indicative of a step change in performance of that magnitude (though I do think FSD on the highway is better than the older AP).

Hopefully we see this FSD Beta cumulative miles usage rate significantly increase with 12.x end-to-end

I am definitely not betting on that. Hopefully there will be incremental improvement. There has to be incremental improvement of course, otherwise it will never be released.

We could be looking at a 1-year wait. At least. Just not particularly optimistic about anything other than incremental improvements.

Certainly our cars will not be driving us around in a year. It'll be basically the same as what we have now.

 

[spacecoin]

I think that we may be saying the same thing. Camera sensors are good enough at this point. Perception and planning is the next frontier.

Yeah, mostly agree. HW4 cameras aren'y likely safe enough (to remove the driver) at highways speeds though. You need to see 250-300 meters out with decent resolution.

Other sensor modalities complement cameras in many conditions though and aren't as reliant on ML to safely detect an object in the planned path.

No amount of ML will fix sensor weaknesses.

Like here (this is not on AP, but to prove my point):

https://twitter.com/x/status/1584726822830252032

 

[mongo]

Yeah, mostly agree. Other sensor modalities complement cameras in many conditions though and aren't as reliant on ML to safely detect an object in the planned path.

No amount of ML will fix sensor weaknesses.

Like here (this is not on AP, but to prove my point):

https://twitter.com/x/status/1584726822830252032

Crank up the brightness/ contrast on the image and the car becomes more visible (not so much in Twitter potato version though).
Radar mostly ignores objects that are motionless relative to the scene.

 

[spacecoin]

Crank up the brightness/ contrast on the image and the car becomes more visible (not so much in Twitter potato version though).
Radar mostly ignores objects that are motionless relative to the scene.

Passive (light) sensing only go as far, regardless of digital enhancement, in the same way as a FLIR wouldn't see a block of concrete. Try upping the brightness/contrast in dense fog

And I wouldn't make general statements about a technology, like radar. Perhaps your statement is true for an implementation using a $300 cruise control radar like the Conti one Tesla was using. Most high end radars would easily handle the scenario above.

This scene could be pitch dark. Multiple sensing modalities to complement vision will always be safer. It's very hard to argue otherwise. the only reason NOT to add more and better sensing is cost.

 

[DrChaos]

As one who has a surface knowledge of e2e as the “new” FSD approach, I don’t understand how it can work when our rules of the road differ significantly among the 50 states and even within each state. For example here in the Phoenix area, there are no car pool lane entry/exit limitations like California and some other states. So will my car know it should copy the behavior of good Phoenix drivers rather than good Los Angeles drivers when interacting with car pool lanes?

There obviously are many other examples. I suppose it would help if rules of the road were set nationally instead of at the state and even local level, but that won’t be happening in my lifetime.

That's the difference between a cool demo (acquire lots of data automatically and train) and a product.

All those events would have to be annotated manually as to what is happening, and also semi-automatically as to the location (state or region perhaps with similar rules) and encoded in the training dataset, and then the ML system trained to use those as input labels so it could do certain behaviors differentially depending on region, and then humans have to develop evaluation procedures to make sure the performance is well balanced across various regions (and not good only in California which dominates the dataset numerically, which would happen naturally). Then synthetic labels & data duplication, like show a positive example for California, but then change the annotation to Arizona with a negative (disapprove) weight or vice versa so it can pick up on the direct contrast and make the network use that state-id label to distinguish policy as opposed to other irrelevant inputs which ML tends to do statistically.

It's tons of work and you see why Waymo rolls things out geographically.

I bet Tesla is going to give up on L4 in practice (though not in PR) because it's not yet scalable or profitable. Selling L2 at L4 prices is good business.

 

[DrChaos]

Passive (light) sensing only go as far, regardless of digital enhancement, in the same way as a FLIR wouldn't see a block of concrete. Try upping the brightness/contrast in dense fog

And I wouldn't make general statements about a technology, like radar. Perhaps your statement is true for an implementation using a $300 cruise control radar like the Conti one Tesla was using. Most high end radars would easily handle the scenario above.
View attachment 983888

This scene could be pitch dark. Multiple sensing modalities to complement vision will always be safer. It's very hard to argue otherwise. the only reason NOT to add more and better sensing is cost.

I personally think high-res radar is the best additional sensor technology: eventually cheaper than lidar as it's all silicon, higher power efficiency, no mechanical/optical pieces which are fragile (AESA scanning like on military), more resistant to poor weather than lidar or vision, and it can get physical relative velocity with high precision instantly with doppler effect and not need multiple frames and estimation. The resolution will continue to increase.

 

[DrChaos]

The Sony IMX490 image sensor used in HW4 has approximately 110000 pixels per sq*mm. Not that dissimilar to an average foveal cone cell density? But regardless, I am in strong agreement with you that the big hurdle to FSD usefulness in current builds seems to be around planning.

The HW4 camera is starting to get good enough, but it's not clear if there is enough processing power to use it with current hardware. And most people still have HW3.

This is not an original thought of mine but one I read: Waymo using lidar since 2007 (they are out of the DARPA challenge winners) for perception meant that they could concentrate on refining their planning for many years before high-ML based perception was feasible. Tesla didn't have any of that and they built up ML-based perception, which is decent given hardware limitations, but is far behind on planning.

Much of the other planning difficulty is likely due to mapping problems, and Tesla doesn't want to pay for, or acquire the accurate, validated maps that are needed. The L4 companies test many routes in certain areas and no doubt additionally annotate or correct whatever maps they are using when their cars interpret them wrong or are misled by them. That takes expense and labor.

Even around my home the Tesla planning map gives a substantially wrong and circuitous route (that Apple & Google don't) and that's just basic nav, not the fine lane-by-lane semantics needed.

 

[spacecoin]

I personally think high-res radar is the best additional sensor technology: eventually cheaper than lidar as it's all silicon, higher power efficiency, no mechanical/optical pieces which are fragile (AESA scanning like on military), more resistant to poor weather than lidar or vision, and it can get physical relative velocity with high precision instantly with doppler effect and not need multiple frames and estimation. The resolution will continue to increase.

There is so much innovation in sensing. We have solid state lidars, there is talk about hybrid lidar/camera sensors and so on. mmW radar in nice right now, but still not something for Tesla, based on Elons comments on the last earnings call. He called their radar "an experiment in the S and X". It's not really an imaging radar either.

At the end of the day, it seems Elon will remove anything that is needed for autonomy and safety if it saves money and keep saying (incorrectly) that humans drive with eyes only to justify it.

 

[DrChaos]

At the end of the day, it seems Elon will remove anything that is needed for autonomy and safety if it saves money and keep saying (incorrectly) that humans drive with eyes only to justify it.

I've said it here before: anytime Tesla does something weird and uncomfortable and annoying, it's Always About The Money.

I really wish now JB Straubel would take over as CEO

 

[jgt1942]

Slapping the word "beta" on there has no actual legal relevance in a liability suit.

All the "you remain responsible at all times" disclaimers however DO- which would go away if they offered an L3 or higher system regardless of still calling it beta or not.

Thanks for the info!

 

[mongo]

Multiple sensing modalities to complement vision will always be safer.

Only if you accept all positives both false or true from all sensors.
Otherwise, you are choosing to ignore one sensor's data over another's which makes one of them extraneous and puts you back into the original situation.

 

[JB47394]

Like here (this is not on AP, but to prove my point):

I'm failing to understand your point because I don't know what your target product is. I'm assuming that Tesla wants a product that is capable of operating in the most common driving environments. As Mongo pointed out, even that wrecked vehicle at night could be perceived with that video stream. I've seen other videos of better-than-human perception in fog. How far does Tesla need to go with perception in order to have a viable L2 product? Or any autonomy level beyond that? It's obvious that other modalities will do better, but how much is needed for each?

 

[AlanSubie4Life]

I've seen other videos of better-than-human perception in fog.

My only experience with fog was in Yosemite this June. In the daytime, with high levels of attention, attempting to use FSD just resulted in a lot of panic crashing of FSD, red wheel of death, and unable to re-enable for a time (as I recall; don't remember the details on re-enabling). It was a dense fog, but drivable at reasonably low speeds; could see people's tail lights maybe 50-100 feet ahead (though it varied). My perception was definitely better than FSD's. It was not viable L2 for those conditions.

Not as dense as the next night though, lol. Didn't use FSD then (it would have just immediately given the red wheel of death I suspect)! Had a very hard time at 2-3mph finding the turnoff to Yosemite West. Did it mostly by feel.

I'd ignore any posted videos and base your assessment on personal experiences in terrifying fog conditions.

 

[spacecoin]

I'm failing to understand your point because I don't know what your target product is. I'm assuming that Tesla wants a product that is capable of operating in the most common driving environments. As Mongo pointed out, even that wrecked vehicle at night could be perceived with that video stream. I've seen other videos of better-than-human perception in fog. How far does Tesla need to go with perception in order to have a viable L2 product? Or any autonomy level beyond that? It's obvious that other modalities will do better, but how much is needed for each?

L2 is not autonomy. It's driver assistance - as in "watch the road and monitor it at all times or it might kill you".

My target product is L3 at highway speeds with a minimum 10s handover procedure.

 

[spacecoin]

Only if you accept all positives both false or true from all sensors.
Otherwise, you are choosing to ignore one sensor's data over another's which makes one of them extraneous and puts you back into the original situation.

This is Tesla misinformation. You train the NN:s on all inputs from all the sensors. Sensor fusion is a solved problem, but it requires good input. The £300 Conti cruise control radar isn't. Shocker.