FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[EVNow]

So what’s the difference going to be between beta 10.9 and beta 11?

See the first post of this thread

 

[black knight 3]

My guess is they are trying to delicately balance the number of users to prevent the possibility of a major accident and pubic relations disaster. However, they also need more data collection to improve FSD so catch-22 there. The software is 100% beta as expected but tesla is absolutely on point when they say that it might do the wrong thing at the worst time.

 

[sleepydoc]

See the first post of this thread

So that’s what this thread’s about!

Based on that, it sounds like it probably will be a significant rewrite. I wonder how buggy 11.0 will be

 

[EVNow]

Based on that, it sounds like it probably will be a significant rewrite. I wonder how buggy 11.0 will be

It "is" a significant rewrite - and apparently mostly done. They are working to fix regressions / bugs. Some of the features are also being released in 10.x.

My hope is it is on par with 10.x when released - ofcourse, we should expect some regressions and somethings that are better. But by the end of this year, I'd be disappointed if we don't see significant improvements - but unlike Musk I don't expect human level driving which needs about 1,000x improvement in disengagement rates.

 

[sleepydoc]

My hope is it is on par with 10.x when released - ofcourse, we should expect some regressions and somethings that are better. But by the end of this year, I'd be disappointed if we don't see significant improvements -

yes - if they’re doing that much of a rewrite I’d definitely expect bugs and regression. Nothing against Tesla, That’s just part of releasing new code. Hopefully they get those quashed in short order and we can start seeing the real benefits.

 

[S4WRXTTCS]

SAE refers to a skilled human driver. That's the criteria to judge Level 5.

That's really the only hope L5 has.

Every year that goes by the number of skilled human drivers dwindles.

Eventually autonomous cars will get to L5, but not because they met or exceed skilled human drivers of Germans in 2000. But, Americans in 2025.

 

[drtimhill]

So that’s what this thread’s about!

Based on that, it sounds like it probably will be a significant rewrite. I wonder how buggy 11.0 will be

Initially it is bound to be a regression .. with the commensurate rants here about how outraged people are and how Tesla will never deliver FSD without LIDAR, RADAR, SONAR, Mac n Cheese, new cameras, new drivers and new underwear protection for the testers.

 

[Raurele]

S

Good choice. No point doing the safety score nonsense while the FSD rollouts are halted. As an fyi, At any point in the future, you can opt back in, and maintain a 98 or 99 for 100 miles, and you’ll get in.

no one knows for sure if its 100 miles, or 7 days, or 100 miles AND 7 days... I've read of people getting in after 50 miles the day after... Best to not set people up with expectations at this point.

 

[EVNow]

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Bladerskb]

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

I will give the guy credit for the effort. But the article focused mainly on what Tesla is doing. The actual “comparison” are mostly footnotes.

 

[EVNow]

I will give the guy credit for the effort. But the article focused mainly on what Tesla is doing. The actual “comparison” are mostly footnotes.

Agreed. It’s more for the links and notes than any real analysis.

BTW, whatever happened to Uber … don’t hear much about them after than accident.

 

[FloridaJohn]

Agreed. It’s more for the links and notes than any real analysis.

BTW, whatever happened to Uber … don’t hear much about them after than accident.

Uber sold off their self driving division in 2020.

Uber Sells Its Autonomous Vehicle Research Division

Uber has sold its autonomous vehicle research division to a startup named Aurora. It's a major shift for a company that once saw self-driving technology as a core investment for its future.

www.npr.org

 

[Mardak]

https://twitter.com/x/status/1496068893835681792

The forum's emoji support indicates Collision :boom: for , but independent of whether the emoji is a good or bad, he seems to be using present tense "is" maybe because he's been using FSD Beta 11 builds? Previously in tweets or interviews, he referred to it as "hoping" or "will," so maybe it's well past those aspirational stages?

Maybe he's choosing to talk about it as it's getting close to FSD Beta release? Probably not surprising if it's closer to "beta" quality now that it's been hinted at for over 7 months now, but maybe there were too many regressions initially to be usable even as "alpha."

 

[EVNow]

The forum's emoji support indicates Collision :boom: for , but independent of whether the emoji is a good or bad, he seems to be using present tense "is" maybe because he's been using FSD Beta 11 builds? Previously in tweets or interviews, he referred to it as "hoping" or "will," so maybe it's well past those aspirational stages?

Maybe he's choosing to talk about it as it's getting close to FSD Beta release? Probably not surprising if it's closer to "beta" quality now that it's been hinted at for over 7 months now, but maybe there were too many regressions initially to be usable even as "alpha."

I think he gets information of new releases builds from three sources
- actual driving
- project reviews / presentations
- Weekly project status reports

The team would be showing the % improvement of various metrics to Elon in presentations / status reports.

So, not sure what his source of "is" is, this time.

But I do feel they are closer to releasing 11. At this point I won't be surprised to see 11 released within the next month. It won't be very surprising to see it slip into next quarter too.

 

[Mardak]

They may need single stack if technical improvements to improve upon the human accident rate are dependent on the single stack architecture

Basic Autopilot is primarily used on highways, and single stack from FSD Beta replacing the current highway driving behavior probably indicates Tesla believes it will be even better, e.g., detecting partially-in-lane vehicles, reacting to unusual objects, handling "NoA limited" scenarios.

https://twitter.com/x/status/1496416738304565252

Unclear how soon that will happen after FSD Beta 11 as regressions are likely with major changes, but at least there's basically confirmation that "everyone" (probably requiring FSD computer hardware but not software) will get on the unified platform for standard Autopilot capabilities including standard active safety.

 

[EVNow]

Basic Autopilot is primarily used on highways, and single stack from FSD Beta replacing the current highway driving behavior probably indicates Tesla believes it will be even better, e.g., detecting partially-in-lane vehicles, reacting to unusual objects, handling "NoA limited" scenarios.

Unclear how soon that will happen after FSD Beta 11 as regressions are likely with major changes, but at least there's basically confirmation that "everyone" (probably requiring FSD computer hardware but not software) will get on the unified platform for standard Autopilot capabilities including standard active safety.

It is not clear to me what this "single stack" actually means. Does it mean
- Single perception Neural Network or
- Single planning unit
or both ?

If we go back to all the upgrades intended for 11, there are a lot of perception related items and one major planning related item. Given the simplicity of planner for Highways I think they will keep highway & city planners separate. They may implement MCTS for highway planner, but I doubt it is really needed. But merging the NN might be beneficial to highways if it really increases accuracy of perception.

Perception :
- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video

Planner :
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)

 

[momo3605]

But merging the NN might be beneficial to highways if it really increases accuracy of perception.

I think 100% the FSD Beta code has a much higher accuracy of perception. If you think about it, autopilot has very basic perception. It's really only watching lines, other cars (and now emergency lights as well), and speed limit signs. But other than that, it's not looking for obstacles, or signs or anything.

Question is, is it even beneficial to increase perception? I suppose it could be for potholes, barriers...or maybe the extremely unlikely event of a person that could potentially stand in the middle of a freeway. As an L2 system, i don't think it's necessary, but for L3+, these edge cases do need to be solved.

 

[jabloomf1230]

It is not clear to me what this "single stack" actually means. Does it mean
- Single perception Neural Network or
- Single planning unit
or both ?

I think you're correct about none of us really knows what Tesla means by single stack. I've always looked at the car's code in the abstract. In my simple view it is represented by the Autopilot

diagram that shows each component in separate boxes and assigned a color as to how the car was using the code. I always assumed that single stack meant that where there were two different pieces of code that were responsible, for example changing lanes, that some attempt would be made to consolidate the two pieces into one. That's a totally simplistic view of things, but my thought is that Tesla has been doing this consolidation all along and testing it internally.

 

[drtimhill]

It is not clear to me what this "single stack" actually means. Does it mean
- Single perception Neural Network or
- Single planning unit
or both ?

I think the safe bet is that it means the NN/vision system is unified, so that the inputs to the planners etc are coming from one sensor stack that builds the cars world view of roads, cars, pedestrians etc. The planner side is more fuzzy, and over time the car may choose different planners based on environment, or simply use this information as input into a single planner. However, while the planner is important, to my mind the world view is the critical component here, and from the pure effort involved in training etc a unified NN stack makes sense, especially given that they are running close to the limit of the HW3 capacity, and I'm sure unifying the stack will help rationalize the NN load somewhat.

 

[Mardak]

The planner side is more fuzzy, and over time the car may choose different planners based on environment, or simply use this information as input into a single planner

What do you think the current FSD Beta 10.x is lacking to replace standard Autopilot (in particular driving on highways)? I've used FSD Beta driving 65mph+ on highways, so both the FSD Beta perception and planning are in use already. But indeed I don't think I've had a situation where it needed to go around a pedestrian at those speeds to see how it behaves.

Tesla is probably well aware of phantom braking issues, and potentially false positive predictions on the production Tesla Vision stack are significantly reduced with FSD Beta 11 stack incorporating multiple camera videos and memory types. That could also be a reason to work on single stack and get it ready to deploy to the whole fleet to solve numerous issues with standard Autopilot.