FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[tpatana]

Is it wrong to hope that they have option to use old highway stack even after single stack has been implemented?

I love how well it currently works on highways, and while FSD on streets is ok, I'm afraid the highway experience might go down. So having option to drive with the current stack would be nice. Some days want to just relax and not worry, other days test the new stack.

 

[Dewg]

Is it wrong to hope that they have option to use old highway stack even after single stack has been implemented?

I love how well it currently works on highways, and while FSD on streets is ok, I'm afraid the highway experience might go down. So having option to drive with the current stack would be nice. Some days want to just relax and not worry, other days test the new stack.

My guess is that the combined stack will be released to FSD Beta users first for testing. Once testing for highway/freeway is completely, they'll release to the general users.

 

[tpatana]

My guess is that the combined stack will be released to FSD Beta users first for testing. Once testing for highway/freeway is completely, they'll release to the general users.

Well I'm FSD beta user, and I'd hope to be able to select between the old stack and the new combined one.

 

[JulienW]

Well I'm FSD beta user, and I'd hope to be able to select between the old stack and the new combined one.

I give the odds of that between 0 and 0.01.

 

[Dewg]

Well I'm FSD beta user, and I'd hope to be able to select between the old stack and the new combined one.

You can disable FSD Beta, which should return you to the older code.

 

[tpatana]

I give the odds of that between 0 and 0.01.

 

[sleepydoc]

You can disable FSD Beta, which should return you to the older code.

That assumes the older code is retained with the new software. It may not be.

I sincerely doubt they would let people choose. That adds another layer of complexity to the software and makes debugging that much more difficult.

 

[Mardak]

That assumes the older code is retained with the new software. It may not be.

I would think the toggle for FSD Beta will still be available, e.g., different driver profiles keep the old behavior.

Technically, the current behavior seems to always run the old stack and sometimes the current beta stack at the same time, and it's been speculated that the new single stack might require all compute not allowing immediate fallback to the old stack. Also supporting the old stack with the toggle could require being in Park to allow time for neural networks to get loaded from disk and start up. I.e., new single stack must be safe/reliable enough without the immediate fallback.

 

[sleepydoc]

I would think the toggle for FSD Beta will still be available, e.g., different driver profiles keep the old behavior.

Technically, the current behavior seems to always run the old stack and sometimes the current beta stack at the same time, and it's been speculated that the new single stack might require all compute not allowing immediate fallback to the old stack. Also supporting the old stack with the toggle could require being in Park to allow time for neural networks to get loaded from disk and start up. I.e., new single stack must be safe/reliable enough without the immediate fallback.

No one knows for sure how they have it configured. Most people speculate as you said that they have both running simultaneously to allow seamless switching. I really don’t know enough about the software architecture to say whether it would be possible to have the old and new together or not. The most logical way to switch would be what you suggested, by turning off FSD - you can turn that off on the fly, just not on. The startup time wouldn’t really matter, either - you can always drive the car manually until it’s ready.

AP on freeways is really quite good. I rarely have to intervene and it's generally for things that are more personal preference, not necessarily mistakes by the algorithms. I would expect/hope it to be pretty good with the integration as well. My main concern would be lane selection - that is it's weakest point, just as it has been with the FSD stack. The advantage now with the AP stack is you have to approve a lane change first so it can't go ahead with a stupid choice like it does on smaller roads. That's my big concern with combining the two.

 

[Mardak]

My main concern would be lane selection - that is it's weakest point, just as it has been with the FSD stack.

I feel like this FSD Beta behavior is actually partially inherited from NoA stack where mapped highway lanes data is much more accurate and complete. At highway speeds, Autopilot needs do lane selection from miles away to have enough time to make multiple lane changes that won't be predictable by the vision neural networks (until it can understand the green guidance signs that humans use). Whereas city streets have a lot more incorrect or missing mapped lanes data where FSD Beta gets confused by overreliance on mapped data, e.g., thinking it needs to get out of a mapped right-turn-only lane not realizing an auxiliary/temporary right-turn pocket lane appears just before the intersection.

I wonder how much special casing will need to happen for "single stack" and if they end up being temporary crutches to get FSD Beta 11 released sooner.

 

[sleepydoc]

I feel like this FSD Beta behavior is actually partially inherited from NoA stack where mapped highway lanes data is much more accurate and complete. At highway speeds, Autopilot needs do lane selection from miles away to have enough time to make multiple lane changes that won't be predictable by the vision neural networks (until it can understand the green guidance signs that humans use). Whereas city streets have a lot more incorrect or missing mapped lanes data where FSD Beta gets confused by overreliance on mapped data, e.g., thinking it needs to get out of a mapped right-turn-only lane not realizing an auxiliary/temporary right-turn pocket lane appears just before the intersection.

I wonder how much special casing will need to happen for "single stack" and if they end up being temporary crutches to get FSD Beta 11 released sooner.

Some of it may be inherited, but a lot/most of it seems to be more unique to FSD, like all the times it will randomly put the blinker on for one or two flashes, then turn it off. I've also had numerous occasions where it will merge out of the lane it needs to be in less than a half mile before the turn, the opposite of what you describe above, needing a longer lead time. FSD will also randomly switch lanes for no clear reason - it doesn't need to turn for > 10 miles and there's no traffic in either lane yet it decides it doesn't like the lane it's in.

The biggest issue with lane selection is FSD's seeming difficulty in determining what's a turn land and what's a through lane. My single biggest source of disengagements comes from FSD veering into a turn lane then apparently thinking it's obligated to make the turn even though it needs to go straight. I don't know if this is related to maps, the neural net, local processing or some combination.

 

[Supcom]

Some of it may be inherited, but a lot/most of it seems to be more unique to FSD, like all the times it will randomly put the blinker on for one or two flashes, then turn it off. I've also had numerous occasions where it will merge out of the lane it needs to be in less than a half mile before the turn, the opposite of what you describe above, needing a longer lead time. FSD will also randomly switch lanes for no clear reason - it doesn't need to turn for > 10 miles and there's no traffic in either lane yet it decides it doesn't like the lane it's in.

The biggest issue with lane selection is FSD's seeming difficulty in determining what's a turn land and what's a through lane. My single biggest source of disengagements comes from FSD veering into a turn lane then apparently thinking it's obligated to make the turn even though it needs to go straight. I don't know if this is related to maps, the neural net, local processing or some combination.

Both NOA and FSD beta have lane selection issues. With NOA, the issues are not as critical because if NOA moves into a lane that is ending, there is always a gradual merge to allow it to get back into a through lane. With beta, moving into an turnout lane is a big deal because the lane ends abruptly and the car may not have room to merge back in at speed.

NOA also has a propensity to switch back and forth between lanes for no apparent reason even when there is no other traffic anywhere nearby. Given that NOA planning is supposedly old procedural code instead of neural nets, that behavior is especially curious to me.

Hopefully, single stack will sort out some of this.

 

[Mardak]

it seems to be more unique to FSD, like all the times it will randomly put the blinker on for one or two flashes, then turn it off

Totally agree FSD Beta introduced more types of unwanted lane changes. Just making sure, the message on screen says it's switching lanes to follow route (and not something like switching away for cones that were false positive prediction)? If so, I have noticed plenty of those in the past especially in places with center turn lanes or special/exclusive lanes where it seems like map data is incorrect/incomplete so FSD Beta falls back to assuming there's 1 lane ahead and tries to get out of what it assumes is a right lane that ends or merges, but vision is able to see the 2 lanes continuing straight and decides to cancel the blinker.

That vision ability to override map data seems to have gotten more weighting in recent versions probably due to the vector space neural network architecture introduced in 10.11 and improved in 10.12, but it also has regressed some intersections that you might be encountering too. One particular intersection here curves left with a right-only lane that appears during the curve, and 10.12 gets confused in multiple ways, e.g., switching left incorrectly thinking it's in the right-only lane or switching into the right-only when it should stay to continue through.

This type of failure might not need architectural changes and "just" needs additional training data. I.e., the neural network has the capability to learn but hasn't yet. These might not be explicitly called out in 10.x release notes as additional training on a large variety of autolabeled mispredictions generally will reduce errors without needing someone to explicitly understand each failure.

Unclear if solving these issues are on the critical path for having FSD Beta 11 ready, e.g., would FSD Beta 10.12 on interstates misread painted arrows and switch lanes unnecessarily due to lack of training? Similarly with 10.69's focus on unprotected left turns, maybe those architectural improvements tie into single stack needing to accurately predict vehicles from further away? Or maybe it's more of a practical engineering of tweaking heuristics or even turning off certain city-streets-only behaviors for interstates to release as FSD Beta 11.

 

[tpatana]

Also I've noticed interesting bug with the FSD stack interfering current freeway-EAP-stack.

Normally I have lane changes disabled on freeways, so it can only do them if really needed for navigation, or if I use the signal stalk to indicate lane change.

But, if I enabled FSD on normal roads (which naturally includes lane changes if needed) and then I go to freeway (even if I disabled FSD before entering freeway), it will do lane changes on freeway based on speed difference between lanes. If I don't use FSD before freeway, it will not do lane changes. So it somehow remembers "hey I have permission to do lane changes" if FSD was on during the drive.

 

[Mardak]

With 10.69 notes, there's an entry that might be critical for single stack highway driving:

• Upgraded to a new two-stage architecture to produce object kinematics (e.g. velocity, acceleration, yaw rate) where network compute is allocated O(objects) instead of O(space). This improved velocity estimates for far away crossing vehicles by 20%, while using one tenth of the compute.

At highway speeds, there's a lot more space covered and visible in a given amount of time, so the existing network architecture from 10.x so far might have been resulting in compute bottlenecks leading to poor predictions on interstates. Overall, if we assume single stack hasn't happened yet due to compute bottlenecks and Autopilot team is actively optimizing and trying out more compute-efficient approaches, there should be hints like these in 10.x release notes related to better predictions from far away necessary for FSD Beta 11.

 

[Mardak]

Hmmm………

Tesla transitioning its first radar vehicles to 'Tesla Vision'

Tesla appears to be transitioning its first radar-equipped vehicles to Tesla Vision

www.notateslaapp.com

Did Tesla remove radar stack with 2022.20.9 to free up space to be able to run single stack FSD Beta 11 in shadow mode on non-FSD-beta vehicles?

 

[GlockWorkOrnge]

Well I'm FSD beta user, and I'd hope to be able to select between the old stack and the new combined one.

Do we get such granular options with the betas? I purchased a used Y that had FSD and I've been on the beta for like a week, so I am super ignorant--be nice to me!

 

[Dewg]

Do we get such granular options with the betas? I purchased a used Y that had FSD and I've been on the beta for like a week, so I am super ignorant--be nice to me!

No. Tesla's goal is to combine the older AP stack with the new FSD Beta stack, so that freeway/highways use the newer neural nets. It should improve performance as the car is not switching between both stacks, freeing up resources.

 

[Mardak]

Was this a hint that the next FSD Beta (presumably 10.69.3 for soon(?) after AI Day [at end of this month]) will have partial single stack with the combination of city streets and parking lot stacks?

https://twitter.com/x/status/1572090491050799107

It would make some sense that "actually smart" summon depended on the introduction of the upgraded video occupancy network in 10.69 to drive around shopping carts, parking sign poles, etc. in a parking lot.

And perhaps getting a bit off topic, I wonder if some of the occupancy network initiative came from Optimus needing to interact with many more types of objects such as walking around a table. Both of these uses (parking lots and Optimus) match up with "control for arbitrary low-speed moving volumes" that are probably not the limiting factor for single stack highway driving major release.

So I wonder if FSD Beta 11 will be delayed to next year to prioritize wide release of FSD targeting this year?

https://twitter.com/x/status/1561607860101996544

 

[impastu]

Any opinions based on past Tesla FSD beta behavior whether candidates who enter the beta qualification criteria (score >= 80; autopilot minimum miles within last 30 days, etc) this week will be added to the beta in the coming days? Asking for a friend...