FSD v12.x (end to end AI)

[spacecoin]

I'm not using your sentence out of context at all. The entire premise of your post was that Tesla is a long way away from higher levels of autonomy because they have not yet developed any of the requisite features.

I'm telling you that you're verifiably wrong, because one of the key features you point to is in development. And that's just what we have some insight into through internal sources.

I am not going to have a meta discussion. Let’s assume that you are right and that I am wrong about FSD:s ability to detect issues.

I have only two questions for you:

1) Why does it fail without warning if it has the capabIlity you claim it has. It should be able to give 5-10 seconds advance warning then.

2) In your opinion, what’s needed to get an L2 to L3?

 

[willow_hiller]

1) Why does it fail without warning is it has the capabIlity you claim it has. It should be able to give 10 seconds advance warning then.

2) In your opinion, what’s needed to get an L2 to L3?

1) It's a L2 system. If it gave advanced warning, that would give a false-sense of security that it's safe to not pay attention without the warning.

2) It's not a matter opinion. You already listed above the things needed to get to L3. I'm just letting you know that you're wrong when you say FSD Beta has had no development toward them.

 

[spacecoin]

1) It's a L2 system. If it gave advanced warning, that would give a false-sense of security that it's safe to not pay attention without the warning.

2) It's not a matter opinion. You already listed above the things needed to get to L3. I'm just letting you know that you're wrong when you say FSD Beta has had no development toward them.

Great to hear that you acknowledge the rest of what I wrote except the sentence that you singled out. We have different opinions about that then.

 

[willow_hiller]

Great to hear that you acknowledge the rest of what I wrote except the sentence that you singled out. We have different opinions about that then.

I accept your premise (the SAE definitions of autonomy), but reject your conclusions about Tesla's progress.

 

[spacecoin]

I accept your premise (the SAE definitions of autonomy), but reject your conclusions about Tesla's progress.

So let’s discuss the specifics then, shall we? What was it that you disagreed with specifically, excluding that wording about advance knowledge of failure that you apparently have running in shadow mode because it’s L2 for safety reasons (sic!).

 

[willow_hiller]

So let’s discuss the specifics then, shall we? What was it that you disagreed with specifically, excluding that wording about advance knowledge of failure that you apparently have running in shadow mode because it’s L2 for safety reasons (sic!).

I generally don't like those massive posts where someone quotes each individual sentence to argue them. It's better just to pick the central thesis and refute that. But if you really want me to:

There is nothing so far indicating that v12 will be vastly better than v11.

Even just based on the one livestream of V12 that we've seen, it drives smoother and more human-like than V11. That alone is evidence of improvement. And theoretically we know that it has the potential to be much better, based on the higher frame rates and potential for holistically interpreting a driving scene.

The cameras are still susceptible to rain, dirt, sun et.c. There is no cleaning.

Although the firmware can detect when performance is degraded by rain/dirt/sun, the threshold for disengaging based on these factors is very high. Weather needs to be to the point where it's difficult for me to see before FSD Beta determines it cannot drive and throws up the "Take over immediately" message. The three forward facing cameras are capable of cleaning themselves, including the wide-angle lens. If all of the other cameras were simultaneously blinded, it could clean the forward-facing cameras in order to find a safe location to achieve a minimal risk condition.

To start with FSD currently has no clue on when it is going to fail.

Already addressed this.

There are no code for ODD-detection and boundaries afaik

FSD Beta briefly had a geofence, where it was disallowed in a certain part of downtown Toronto that had a streetcar. So that code does exist: Tesla might remove downtown Toronto geofence with FSD Beta 10.69.3 [Update]

If they got the reliability up by 1000x tomorrow in some limited ODD, it would probably still take 12-18 month just to design the new UX (ODD detection, handover procedure etc) and adapt it to current (non-US) regulation like R157 detection of emergency vehicles, and then validation...

Tesla redesigns their main UI with a high frequency, compared to any other automaker. I doubt it would take nearly that long for them to extend the development they already have and put together a workable UX.

 

[spacecoin]

I generally don't like those massive posts where someone quotes each individual sentence to argue them. It's better just to pick the central thesis and refute that. But if you really want me to:


Even just based on the one livestream of V12 that we've seen, it drives smoother and more human-like than V11. That alone is evidence of improvement. And theoretically we know that it has the potential to be much better, based on the higher frame rates and potential for holistically interpreting a driving scene.


Although the firmware can detect when performance is degraded by rain/dirt/sun, the threshold for disengaging based on these factors is very high. Weather needs to be to the point where it's difficult for me to see before FSD Beta determines it cannot drive and throws up the "Take over immediately" message. The three forward facing cameras are capable of cleaning themselves, including the wide-angle lens. If all of the other cameras were simultaneously blinded, it could clean the forward-facing cameras in order to find a safe location to achieve a minimal risk condition.


Already addressed this.


FSD Beta briefly had a geofence, where it was disallowed in a certain part of downtown Toronto that had a streetcar. So that code does exist: Tesla might remove downtown Toronto geofence with FSD Beta 10.69.3 [Update]



Tesla redesigns their main UI with a high frequency, compared to any other automaker. I doubt it would take nearly that long for them to extend the development they already have and put together a workable UX.

Thanks. Appreciate it. So your guess for L3 from Tesla is what? 2024? 2025? ODD speed and road types?

 

[Mardak]

Look at actual performance. There is nothing so far indicating that v12 will be vastly better than v11.

We only have the livestream demo for actual performance of in-development v12, but even then it already showed off control behavior quite different from 11.x. If you can believe Tesla saying they're replacing 99% of control code with neural networks, it would seem to be a rewrite much more significant than other architectural changes we've seen so far.

The decision to basically abandon in-vehicle behavior that has been deployed and safety evaluated over many years and miles presumably was based on actual evidence of what v12 could already do and its potential. Similarly, presumably Tesla is well aware of all the problems of 11.x (based on usage in US/Canada as well as internal testing worldwide) and maybe even early actual performance was vastly better.

Of course we won't actually know until Tesla releases the next version for people to determine performance, and even then it won't be clear for a while if its limits will require a v13. I suppose there's still a possibility of Tesla just hyping things even if they somehow know it won't work, and Tesla engineers are good at faking excitement.

 

[spacecoin]

We only have the livestream demo for actual performance of in-development v12, but even then it already showed off control behavior quite different from 11.x. If you can believe Tesla saying they're replacing 99% of control code with neural networks, it would seem to be a rewrite much more significant than other architectural changes we've seen so far.

The decision to basically abandon in-vehicle behavior that has been deployed and safety evaluated over many years and miles presumably was based on actual evidence of what v12 could already do and its potential. Similarly, presumably Tesla is well aware of all the problems of 11.x (based on usage in US/Canada as well as internal testing worldwide) and maybe even early actual performance was vastly better.

Of course we won't actually know until Tesla releases the next version for people to determine performance, and even then it won't be clear for a while if its limits will require a v13. I suppose there's still a possibility of Tesla just hyping things even if they somehow know it won't work, and Tesla engineers are good at faking excitement.

Good post. At this point I stopped listening to anything Tesla says about FSD goals and performance. I hope v12 gets to a point that is significantly better than previous versions.

I doubt it will ever be autonomous in a meaningful ODD though (on existing cars). And if they knock it out of the park with v12 and surprise me, I’m thinking L3 is earliest at 2026 based on what I wrote before.

 

[willow_hiller]

Thanks. Appreciate it. So your guess for L3 from Tesla is what? 2024? 2025? ODD speed and road types?

It's less a matter of what Tesla is capable on what timescale, and more a matter of what they want to release, and whether it would benefit their customers.

If Tesla wanted to release an L3 suite similar to what Mercedes has, I don't think that would take them more than a couple of months to work on the UX. They can already detect lead cars (highlighted on the current UI), follow lead car speeds, and detect emergency vehicles (flashing lights trigger UI warning message and slow down). All they would need to do is combine those existing elements into a handoff. Given FSD Beta's performance on the highway, I'd venture that it would be safer on average than Mercedes L3.

But I don't think they want to do that. Contrary to popular belief, I think Tesla avoids releasing features just because they can, and only releases features when they are verifiably safe and genuinely useful for the end users.

 

[S100D]

It's important to differentiate between current software / hardware limitations and potential. Similar to how your 2016 couldn't get Autosteer on city streets until many years later, there's potential for end-to-end to better handle raining situations without the "degraded" message or disengaging. The initial v12 release probably will still have those messages, but hopefully it won't take years to improve the software for the deployed hardware.

It'll be interesting how regulators and the public assess automated vehicle safety in different scenarios such as if it'll be acceptable or even desirable for say maximum speed to be reduced even more than 10 mph when raining. Would getting to the destination safely but potentially much later still be better than not completing a trip in the rain?

A lot of this will depend on how end-to-end actually behaves in the rain where maybe it already is good enough or can do fine at lower speeds or could need more significant changes to move from driver assist.

I am ok with FSD decreasing the speed by 10 mph when it is raining. I am not ok with FSD cancelling because the cameras have too much rain on them to drive safely. Remember, Elon said that my 2016 Model with FSD Radar, Ultrasonic Sensors, and cameras could achieve Level 5 autonomy. My 2023 Model S LR with Tesla Vision thinks the weather is poor in cloudy/sunny days, and can't see well enough to allow FSD to keep working when it is raining.

The bottom line is that whatever hardware is going to be installed in Tesla's RoboTaxi needs to be installed in all new Teslas being built now. A driver-less RoboTaxi cannot stop driving when it is raining. This may require Lidar, Radar, Ultrasonic Sensors, and Cameras. If that hardware was installed now, Tesla would have millons of FSD miles and years of experience with that new hardware.

Tesla will never achieve Level 5 autonomy with the current hardware. Tesla is guilty of false advertising when is said that my 2016 Model S had the hardware required to achieve Level 5 autonomy. Level 5 autonomy should be able to drive in rain without cancelling.

 

[spacecoin]

If Tesla wanted to release an L3 suite similar to what Mercedes has, I don't think that would take them more than a couple of months to work on the UX. They can already detect lead cars (highlighted on the current UI), follow lead car speeds, and detect emergency vehicles (flashing lights trigger UI warning message and slow down). All they would need to do is combine those existing elements into a handoff. Given FSD Beta's performance on the highway, I'd venture that it would be safer on average than Mercedes L3.

Given that it took them 4+ years to get to acceptable auto high beam (not blinding anymore, but still turns on three secs too late and feel dangerous) and I'm still counting on the autowipers, I think you severely underestimate the complexity of this problem (not including liability and verification).

Contrary to popular belief, I think Tesla avoids releasing features just because they can, and only releases features when they are verifiably safe and genuinely useful for the end users.

Would you say any version of FSDb is "verifiably safe and genuinely useful for the end users"? I would tend to disagree with both those statements.

 

[MP3Mike]

My 2023 Model S LR with Tesla Vision thinks the weather is poor in cloudy/sunny days, and can't see well enough to allow FSD to keep working when it is raining.

Have you ever had it stop working because of rain? (Not just the "may" be degraded warning.)

I haven't ever had that happen, even when the rain was so bad that I could barely see. It stayed engaged and drove perfectly fine.

 

[Supcom]

I am ok with FSD decreasing the speed by 10 mph when it is raining. I am not ok with FSD cancelling because the cameras have too much rain on them to drive safely. Remember, Elon said that my 2016 Model with FSD Radar, Ultrasonic Sensors, and cameras could achieve Level 5 autonomy. My 2023 Model S LR with Tesla Vision thinks the weather is poor in cloudy/sunny days, and can't see well enough to allow FSD to keep working when it is raining.

The bottom line is that whatever hardware is going to be installed in Tesla's RoboTaxi needs to be installed in all new Teslas being built now. A driver-less RoboTaxi cannot stop driving when it is raining. This may require Lidar, Radar, Ultrasonic Sensors, and Cameras. If that hardware was installed now, Tesla would have millons of FSD miles and years of experience with that new hardware.

Tesla will never achieve Level 5 autonomy with the current hardware. Tesla is guilty of false advertising when is said that my 2016 Model S had the hardware required to achieve Level 5 autonomy. Level 5 autonomy should be able to drive in rain without cancelling.

I've driven V11 in some very heavy thunderstorms without it disengaging. It does put up the degraded warning pretty quickly and will repeatedly do so, with the warning tone each, and every time, which is annoying. In heavy rain, it slows down, sometime more than I would, but it keeps driving. Rather impressive actually.

I haven't has 11.4.8.1 in the rain, but the previous version, with the dodgey wipers would give the red hands of death in even light rain if I didn't manually adjust the wiper speed. But, that's a wiper issue, not FSDb.

 

[willow_hiller]

Would you say any version of FSDb is "verifiably safe and genuinely useful for the end users"? I would tend to disagree with both those statements.

Hard for anyone outside of Tesla to actually verify the safety, but given my experience of actually driving with it every day: yes. It's much safer than the average driver, and much less exhausting for me to just monitor my surroundings than to actually drive most days.

 

[S100D]

In the beginning. Then sleeping drivers died. So, Tesla added the steering wheel nag, which avoided replacing existing steering wheels. Finally, they implemented Elon Mode so that a privileged few can continue driving hands free.

There was also some idiot that got into the back seat creating a video while FSD was engaged and uploaded the video on YouTube. Tesla then put the nag on the steering wheel and checks to see if you are in the sitting in the drivers seat.

 

[S100D]

I've driven V11 in some very heavy thunderstorms without it disengaging. It does put up the degraded warning pretty quickly and will repeatedly do so, with the warning tone each, and every time, which is annoying. In heavy rain, it slows down, sometime more than I would, but it keeps driving. Rather impressive actually.

I haven't has 11.4.8.1 in the rain, but the previous version, with the dodgey wipers would give the red hands of death in even light rain if I didn't manually adjust the wiper speed. But, that's a wiper issue, not FSDb.

I am on 11.4.7.3 and during heavy rain on an interstate highway, FSD did slow down 10 mph due to poor weather detected, then disengaged to Enhanced Auto Pilot, Enhanced Auto Pilot then disengaged to Auto Pilot due to poor weather detected. Auto Pilot would follow the lane and maintain speed but it would not take exits, etc. which is normal.

I watch Teslascope and so far I have not seen any firmware updates to 11.4.9 on any 2023 Model S LR with HW 4.0. I do not believe that any future firmware updates is going to resolve the rain issues with Tesla Vision and FSD. As I have said before, all Teslas need the hardware that is going to be installed on RoboTaxi or we will never see level 2 autonomy, especially when it is raining.

 

[uscbucsfan]

I am on 11.4.7.3 and during heavy rain on an interstate highway, FSD did slow down 10 mph due to poor weather detected, then disengaged to Enhanced Auto Pilot, Enhanced Auto Pilot then disengaged to Auto Pilot due to poor weather detected. Auto Pilot would follow the lane and maintain speed but it would not take exits, etc. which is normal.

I watch Teslascope and so far I have not seen any firmware updates to 11.4.9 on any 2023 Model S LR with HW 4.0. I do not believe that any future firmware updates is going to resolve the rain issues with Tesla Vision and FSD. As I have said before, all Teslas need the hardware that is going to be installed on RoboTaxi or we will never see level 2 autonomy, especially when it is raining.

The rain algorithm was pretty recently broken eith V11. It worked fine prior, so they can definitely fix it again.

Also, I've seen a few 23 MS with 11.4.9

 

[S100D]

I should note, you can achieve L4 and not handle rainy weather, it is not a required prerequisite. The order page previously suggested some features that can be interpreted as L4, but never mentioned specific levels (much less a specific definition like SAE's) or that it would handle all weather.

I agree with you, it might even be possible to achieve level 4 in perfect sunny conditions. I have seen the poor weather detected/FSD maybe degraded messages when it is not raining on cloudy and sunny days.

If you go back to Elon/Tesla's statemennt that the 2017 hardware would achieve level 5 autonomy. Below is a picture that shows that level 5 autonomy can handle all driving tasks under all conditions with zero human attention. This means it can drive in rain with zero disengagements.

I go back to Elon's statement in 2017 that all Tesla's with FSD could join the RoboTaxi fleet and use the car to make money. Can you imagine a driver less Tesla on a RoboTaxi call and it starts raining. The Tesla then slows down 10 mph, due to it starting to rain harder, the Tesla then disengages FSD and runs into other cars or obstacles. If Tesla added Radar, Ultrasonic Sensors, and Lidar to their current FSD hardware, I believe that it could drive in rain without all of the current weather issues.

Elon/Tesla has made so many promises regarding FSD that never came true. One promise that was broken, I received an e-mail from Tesla saying that since I was an early FSD adopter because I purchased FSD in December 2016, that I would be one of the first to receive FSD. Even after spending the money to upgrade from MCU2 to MCU3 on my 2016 Model S, I did not get FSD installed untl November 2022, six years after purchasing FSD.

 

[Mardak]

At this point I stopped listening to anything Tesla says about FSD goals and performance

Do you believe Tesla is actually trying to reach robotaxi capabilities? Outside of Autopilot, Tesla has probably already spent a significant amount in vehicle design and engineering for a potential robotaxi future, so it would seem reasonable to expect Tesla focusing efforts on what they believe will get to robotaxi sooner.

Even if you disagree with their timelines, understanding their goals helps provide context of why Tesla may not waste resources with potential distractions such as conditional driving automation. Training end-to-end to specially handle behaviors not needed for robotaxi could make things worse for the long-term goal, but then again Tesla can change directions when they need to such as this v12 rewrite.