FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[MP3Mike]

Anyone know what Cz means?

I assume a shortcut for 'cause/because.

 

[Gbills]

Anyone know what Cz means?

It looks like an attempt to shorten cameraz

 

[willow_hiller]

https://twitter.com/x/status/1614376296267632642

 

[powertoold]

So basically two weeks

 

[willow_hiller]

So basically two weeks

One (which would be on his last week's two week schedule) to two.

Also some new details about the new neural nets:

https://twitter.com/x/status/1614377287557984256

 

[willow_hiller]

FWIW. Sounds like Tesla now considers FSD Beta to be out of "Early Access." So new releases need to be very stable before they go public:

https://twitter.com/x/status/1614405339411288065

 

[kabin]

Hopefully it doesn't add phantom steering to the mix.

 

[swedge]

FWIW. Sounds like Tesla now considers FSD Beta to be out of "Early Access." So new releases need to be very stable before they go public:

https://twitter.com/x/status/1614405339411288065

Good point! Not only stable, but also very much like the long standing standard version, what we call fix it but don't change it. No wonder they are being cautious!

Just an idea on this. I expect the roll out will be incremental, but there may be an additional option.

Do we know if V11 will apply the "single stack" to cars without FSD? Or might it go out to FSD users first for validation and fine tuning?

I ask because all owners are already well trained on how standard Autopilot operates. Replacing that with FSD-derived Autopilot will change AP behavior some - hopefully for the better, but bugs will be bugs, and the highway edge cases are different from those on which FSD on city streets has been trained. Breaking a safe and well established product is a huge risk.

But If swapping out the highway AP with new code is initially applied only in FSD cars, it would be a release only to folks who have signed signed (and who comply with) the agreement to always, always be vigilant. It would also give these users the option to revert to the old highway AP version if needed by simply turning off FSD beta.

Limiting V11 to FSD owners might be a reasonable half step in the roll out.

SW

 

[EVNow]

We’re starting to make use of neural nets for vehicle navigation & control, not just vision.

I wonder what that means. Are they trying "end-to-end" ? My guess is some convoluted control/navigation tasks are now handed over to NN since the code has gotten too much like spaghetti and can no longer be maintained ...

 

[powertoold]

I wonder what that means. Are they trying "end-to-end" ? My guess is some convoluted control/navigation tasks are now handed over to NN since the code has gotten too much like spaghetti and can no longer be maintained ...

NNs for navigation / planning make sense, but NNs for control? That's confusing. Isn't control just the amplitude of power or braking / steering sent to the motors by the planner? Why would that need to be a NN as well?

 

[Mardak]

NNs for control?

There's probably some hardcoded control behaviors that were added to be safe/conservative such as slowing down to 0 when one of multiple overlapping roads has an intersection:

I've encountered similar behavior on a bridge. It's slowing down to confirm if there's traffic control probably because it's confused by map data that indicates there's an intersection at that same position. The intersection is actually for the road that runs directly underneath the bridge at that location: Way: 25355805 | OpenStreetMap

On my regular commute, FSD Beta has consistently wanted to slow down unless I continue to press the accelerator until passing the intersection below the bridge, so it'll be pretty obvious if FSD Beta 11.x changes behavior. A neural network would be able to visually see that this particular control doesn't make sense in this case to avoid unnecessary slowing/stopping.

 

[powertoold]

There's probably some hardcoded control behaviors that were added to be safe/conservative such as slowing down to 0 when one of multiple overlapping roads has an intersection:

On my regular commute, FSD Beta has consistently wanted to slow down unless I continue to press the accelerator until passing the intersection below the bridge, so it'll be pretty obvious if FSD Beta 11.x changes behavior. A neural network would be able to visually see that this particular control doesn't make sense in this case to avoid unnecessary slowing/stopping.

But in this case, isn't the coarse map data and/or perceptual prediction inaccurate, and this inaccuracy is causing the planner to slow the car down on the bridge? What do the actual control algorithms have to do with it?

Maybe my understanding of "control" is incorrect. Perception -> Planner -> Control

Unless Elon is referring to control as the planner?

 

[EVNow]

But in this case, isn't the coarse map data and/or perceptual prediction inaccurate, and this inaccuracy is causing the planner to slow the car down on the bridge? What do the actual control algorithms have to do with it?

Maybe my understanding of "control" is incorrect. Perception -> Planner -> Control

Unless Elon is referring to control as the planner?

I think Elon is referring to control/Nav/Planner as a single unit/group.

 

[powertoold]

I think Elon is referring to control/Nav/Planner as a single unit/group.

Interesting, ya, maybe he means using NNs for navigation through to control.

So using NNs throughout the whole stack:

Perception -> Navigation -> Planning -> Control

 

[Mardak]

But in this case, isn't the coarse map data and/or perceptual prediction inaccurate, and this inaccuracy is causing the planner to slow the car down on the bridge? What do the actual control algorithms have to do with it?

In this case of slowing down on the bridge for the intersection below it, it feels very much like some explicitly programmed control because visualization shows just lanes continuing straight and navigation knows I'm on the bridge with no strangeness of rerouting, so something else is overriding the controls to be cautious every single time.

For example, if Tesla had some hardcoded control logic to always stop if it detected a parked school bus to override the usual "go around" behavior, incorporating a neural network for that decision could be preferable to explicitly programming a way to define which situations are good or not to pass.

 

[Mardak]

Do we know if V11 will apply the "single stack" to cars without FSD? Or might it go out to FSD users first for validation and fine tuning?

It seems a bit risky to make FSD Beta 11 driving behavior available to everyone without a need to toggle FSD Beta as part of its initial release, but that could be part of the overall release plan. Similar to how even most of us in the "Safety Score group" could be split into say 10% rollout then 20% rollout then 40%, etc. for a given FSD Beta version, Tesla can evaluate whether the software is ready to expand from those who purchased FSD Capability to a much wider audience with basic Autopilot.

Given that Tesla is preparing a 10.69.25.2 for rollout to those with FSD Beta, it might suggest 11's "Should start rolling out later this week" will be quite gradual as opposed to incorporating whatever (time-critical?) changes from 10.69.25.1 -> 10.69.25.2 into 11's release.

Presumably, part of what has taken single stack on highway so long is matching the safety / comfort / quality of the legacy stack in most situations, so continuing that slow rollout from internal alphas to strict NDA employees to wider might maintain that pace to make sure there were no new unknown unknowns found.

I originally figured the strict NDA was to maintain the holiday update type of surprise, but with the original deadline missed, I wonder what Tesla is still wanting to keep secret?

 

[EVNow]

Do we know if V11 will apply the "single stack" to cars without FSD? Or might it go out to FSD users first for validation and fine tuning?

Eventually yes ... but I think it will rollout to FSD Beta users first. After a few months, may be they replace AP in the main branch with V11 derived one.

BTW, cars with < HW3 will continue to run old AP.

 

[jeremymc7]

Is there any indication that people on HW3 will free bump to HW4 in hen we move to 11?

New cameras seem to be coming at the least and they would almost seem necessary at some point.

 

[Supcom]

It seems a bit risky to make FSD Beta 11 driving behavior available to everyone without a need to toggle FSD Beta as part of its initial release, but that could be part of the overall release plan. Similar to how even most of us in the "Safety Score group" could be split into say 10% rollout then 20% rollout then 40%, etc. for a given FSD Beta version, Tesla can evaluate whether the software is ready to expand from those who purchased FSD Capability to a much wider audience with basic Autopilot.

It's going to be interesting to see how the rollout is executed. This will be the first significant update to FSDb since it went to all FSD cars.

 

[Supcom]

Is there any indication that people on HW3 will free bump to HW4 in hen we move to 11?

New cameras seem to be coming at the least and they would almost seem necessary at some point.

There are no indications one way or the other regarding whether HW3 cars will be upgraded to HW4 at all. Given that V11 wide release could begin at any time, and there has been no announcement as to what HW4 consists, there is no possibility that HW4 will be coordinated with the V11 release.

Any upgrade from HW3 to HW4, if it offered, will likely take many months to implement. I'm still waiting for a CCS1 charge module retrofit almost a year and a half since I took delivery. That's for one little circuit card.