FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Genie]

Tesla gets a lot of data from cars on FSDb without your needing to manually report. The car knows whenever you disengage or intervene with the accelerator and likely self-reports those incidents with no action required by the driver.

Is the self-reporting confirmed or is that speculation? I have intervened so many times the reporting chip must get hot

 

[JulienW]

Is the self-reporting confirmed or is that speculation? ....

Kinda both, since none of us can know for sure what data Tesla acquires or discards and even what Tesla does or doesn't do with any data it gets. So other than just forum speculation there is really no need to concern about it since there is nothing we can know or do to influence it.

 

[powertoold]

11.4 doing some great maneuvering around a parked bus and van, although I'm not sure if it's a law in San Jose to yield to a bus signaling left:

 

[powertoold]

This looks to be a new caption for 11.4:

 

[LowlyOilBurner]

11.4 is gonna be hotter than a microwave oven.

 

[Ramphex]

What was the car doing or trying to do when the red hands came up? It just seems so unusual that you got red hands this quickly? I go months between red hands and have never had a strike since using FSD in Oct 2021.

What car? I feel like the experiences are vastly different by models, even thought they shouldn’t be.

 

[LowlyOilBurner]

What car? I feel like the experiences are vastly different by models, even thought they shouldn’t be.

Referring to camera placement across the 4 models?

 

[Ramphex]

Referring to camera placement across the 4 models?

No idea what causes it. Camera placement is a possible factor for sure.

 

[jebinc]

11.4 is gonna be hotter than a microwave oven.

A “PhD” of FSD(j), so to speak…. “Piled high & Dry” on the junk heap!

 

[powertoold]

Another new caption in 11.4? It seems like Tesla is eventually going to tell you everything the car is thinking. This is actually a very interesting situation (see the video). It seems as though fsd saw the "double parked" car far ahead and decided to move over way in advance:

 

[Kablooie]

Is the self-reporting confirmed or is that speculation? I have intervened so many times the reporting chip must get hot

I heard that some disengagements are automatically sent to Tesla but not all of them. It has to fit defined requirements. The details of the requirements are unknown

 

[drtimhill]

I hope 11.4.x comes soon. I've had a lot of blatantly unsafe situations with 11.3.6.

The worst and most consistent one is attempting to turn in front of oncoming 100 km/h traffic where it most definitely does not have the room. Turning right at this corner (2 way stop sign), it does not yield to traffic coming from the north, it tries to turn right in front of very fast oncoming traffic.

There is a very small (50m maybe) turn/acceleration lane, but even goosing it in insane mode isn't enough acceleration to merge with full speed oncoming traffic, so one realistically has to wait until the other car is roughly abeam before turning, accelerating and merging in behind.

FSD just tries to turn right in front. I'm not sure if it's trying to use that acceleration lane or not, as I disengage before I let it try. To let it go would cause an accident, 100%.

View attachment 932699

I hope you reported this to Tesla...

 

[jj3884]

Sigh two strikes on this road trip I-15 through Vegas. Hopefully a new release and reset soon as I have 1500+ miles to go

I swear they want me swerve into another lane to prove I have my hands on the wheel, and the inconsistency on how long it will warn is annoying (like ok, not responding to wheel turn, use the volume wheel... Nope not quick enough this round). I also don't know if it's because it's dark outside, but if I have my head tilted to the side it also thinks I'm not paying attention. Like f, my eyes are on the road.

 

[TK211X]

Sigh two strikes on this road trip I-15 through Vegas. Hopefully a new release and reset soon as I have 1500+ miles to go

I swear they want me swerve into another lane to prove I have my hands on the wheel, and the inconsistency on how long it will warn is annoying (like ok, not responding to wheel turn, use the volume wheel... Nope not quick enough this round). I also don't know if it's because it's dark outside, but if I have my head tilted to the side it also thinks I'm not paying attention. Like f, my eyes are on the road.

This has pissed me off as well. It’s broken. When I’m looking at the road and jerking the wheel when it’s spazzing out I just disable and send a voice report saying DM is broken then re-enable.

 

[nvx1977]

Tesla gets a lot of data from cars on FSDb without your needing to manually report. The car knows whenever you disengage or intervene with the accelerator and likely self-reports those incidents with no action required by the driver.

Based on my router stats, my model 3 has only been uploading data to tesla if I submit a voice feedback. The chunk of data is substantial (over 1gb, though it varies a lot, maybe video quality, time of day, compression, etc). But I haven't seen any spikes that aren't accounted for (an intervention for which I didn't leave voice feedback).

There's a chance that intervention triggers send much smaller data back, but seems unlikely.

 

[jj3884]

This has pissed me off as well. It’s broken. When I’m looking at the road and jerking the wheel when it’s spazzing out I just disable and send a voice report saying DM is broken then re-enable.

Yeah I realizing that's what I'm going to have to do too and just training the muscle memory now disabled report back cuss a little bit

 

[Mardak]

I find it so odd that FSD and feature releases are separated. Is it a big thing, where it's easier to hunt down bugs if the two are separated?

There's different types of bugs that can be introduced by regular production feature releases and FSD Beta releases, and there's additional types of bugs when merging the two together. For example, 2023.12.1.1 is currently in a large push (~40%) to replace 2023.12.1, which went to about 20% then stalled, maybe because there was a new bug introduced in the latter. Similarly, FSD Beta 11.3(.0) probably introduced new changes since 11.2.x that gets evaluated with internal employee testing before going wider with later 11.3.x. And for merging, 10.69.25 / 2022.44.25.5 later updated to 10.69.25.1 / 2022.44.30.5 and slowly rolled out to make sure there were no new regressions from moving 10.69 to a newer production release.

Overall, the rollout process tries to reduce exposure of risk from new code behaviors.

 

[Mardak]

Based on my router stats, my model 3 has only been uploading data to tesla if I submit a voice feedback

Is your vehicle consistently uploading data? In the past as well as with 11.3.x, I've needed to turn on camp mode (not sentry) to keep the upload going probably because I leave the car parked outside unplugged. It seems to more aggressively go into a deeper sleep mode that avoids network traffic.

 

[Mardak]

Maintaining these separate branches (and separate AP stacks) must be a burden for Tesla. I'm surprised they haven't done it already but I'd be shocked if they don't merge FSDb into production within the next few months.

Yeah, it seems like Tesla has been wanting to merge FSD Beta into the main production branch for a while now, but practically keeping things separate can actually speed up development and targeted deployments. For example, S/X with HW4 are now getting 2023.12.1.x which is also going to many HW3 vehicles and seems like so far only access to regular Autopilot but not FSD Beta, so if FSD Beta was already merged, it might have delayed releasing that software to the HW3 vehicles to ensure functionality was working on all hardware configurations. Basically, merging sooner could have forced HW4 + FSD Beta to be ready sooner at the expense of delaying updates to the majority of the fleet on HW3, and this could even delay non-FSD-Beta related changes like security or NHTSA recall issues.

I do agree that it should merge soon as there are also tradeoffs of maintaining separate branches.

 

[Mardak]

Is the self-reporting confirmed or is that speculation?

At minimum, Tesla reports trip logs to calculate aggregate Autopilot statistics for their safety reports, and these include how Autopilot was disengaged. These might have been updated for Safety Score type data that is able to show where/when Autopilot was active when viewing in the app, and presumably the data sharing agreement in turning on FSD Beta allows for this to be sent even when the app doesn't show Safety Scores.

Pretty sure my wife driving the other vehicle hasn't used the voice drive-notes recording feature but now actually is using FSD Beta at least on highways. That vehicle has uploaded almost 100GB in the last 30 days according to Google Wi-Fi / Home. Although unclear if that upload was from disengagements or general shadow mode data collection.