FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[DarkForest]

Was this a hint that the next FSD Beta (presumably 10.69.3 for soon(?) after AI Day [at end of this month]) will have partial single stack with the combination of city streets and parking lot stacks?

Good guess! I’m optimistic about getting single stack this year since Elon said at AI Day that he’s already running the Alpha with single stack on his car. I think they also mentioned the parking lot stack coming in a month or two?

 

[AlanSubie4Life]

Good guess! I’m optimistic about getting single stack this year since Elon said at AI Day that he’s already running the Alpha with single stack on his car. I think they also mentioned the parking lot stack coming in a month or two?

Yep, and it wasn’t even Elon who said it!

Smart Parking and (presumably) Smart Summon will be by the end of the year. You heard it at AI Day first!

So great. Too bad FSD is currently unusable by anyone who remotely cares about driving skill.

Seems like single stack won’t be too long since they are currently driving with it. Have to say Elon does not seem to be a very good Alpha tester though.

 

[Mardak]

Elon does not seem to be a very good Alpha tester though

To be fair, there's only so much alpha testers can cover as there's so much "real world" corner cases that take time to validate with the fleet. If the production Autopilot stack won't be available as a fallback for FSD stack when deployed, Tesla needs to make sure FSD Beta isn't just better in some or even most cases but also needs to basically be as good as the production version in basically all cases.

Here's a transcript of the answer about single stack during Q&A:

John (presented Lanes & Objects [Vision]): There used to be a lot of differences between production Autopilot and the Full Self-Driving Beta, but those differences have been getting smaller and smaller over time. I think just a few months ago, we now use the same Vision-only object detection stack in both FSD and in the production Autopilot on all vehicles. There’s still a few differences — the primary one being the way that we predict Lanes right now, so we upgraded the modeling of Lanes so that it could handle these more complex geometries like I mentioned in the talk. In production Autopilot, we still use a simpler lane model, but we’re extending our current FSD Beta models to work in all sorts of highway scenarios as well.​

​

Elon: The version of FSD Beta that I drive actually does have the integrated stack so it uses the FSD stack both in city streets and highway. It works quite well for me, but we need to validate it in all kinds of weather like heavy rain, snow, dust; and just make sure it’s working better than the production stack across a wide range of environments. We’re pretty close to that. I think it’s definitely before the end of the year and maybe November.​

​

Paril (presented Planning): Yeah in our personal drives, the FSD stack on highway drives already way better than the production stack we have. We do expect to also include the parking lot stack as part of the FSD stack before the end of this year. So that will basically bring us to: you sit in the car in the parking lot and drive until the end of the parking lot at a parking spot before the end of this year.​

​

Elon: In terms of the fundamental metric to optimize against is how many miles between a necessary intervention. So just massively improving how many miles the car can drive on full autonomy before an intervention is required that is safety critical. That’s the fundamental metric we’re measuring every week and we’re making radical improvements on that.​

So from John, it sounds like there's some actual new functionality that was added in 10.69 with the Lanes network ("deep lane guidance") primarily focused on city streets but needs some work / training on highway data, and I would guess that's quite the departure from the old stack that relied very heavily on map data for correct lane selection potentially for interchanges miles away with very different kinds of visual hints at that distance.

Similarly from Paril, sounds like there's special Planning related updates to get parking lot behaviors working well such as selecting a parking spot in a parking lot. But indeed, good to hear from not just Elon Musk about their planned timing of release of FSD Beta 11 single stack before the end of this year.

I'm guessing the safety-critical interventions metric is what they're monitoring to decide when it's ready for wide release without Safety Score. Although not very clear what they consider safety critical vs convenience/polish.

 

[Zane Liu]

I’m quite excited about all this and hope to be able to receive the Beta in time for these features. Hoping another round of beta users will be getting access soon.

 

[AlanSubie4Life]

To be fair, there's only so much alpha testers can cover as there's so much "real world" corner cases that take time to validate with the fleet

Yeah I was just referencing the basic driving behavior, which is clearly garbage. It’s completely and totally unusable and unpleasant, just a pile of turds. Amazing what it can do though! I would never have let that fly but Elon has other ideas! He has all the money and I don’t though, so whatever works for him.

Certainly many different conditions have to be accounted for as they said. Thanks for the long transcript - informative for those who have not listened and captures the only worthwhile information from AI Day.

I'm guessing the safety-critical interventions metric is what they're monitoring to decide when it's ready for wide release without Safety Score

I make safety-critical interventions (obviously that does not mean avoiding an accident - I just mean safety-critical interventions to avoid any other traffic having to make adjustments) all the time whenever I turn on FSD Beta, so not sure what they mean.

 

[sleepydoc]

Yep, and it wasn’t even Elon who said it!

Smart Parking and (presumably) Smart Summon will be by the end of the year. You heard it at AI Day first!

So great. Too bad FSD is currently unusable by anyone who remotely cares about driving skill.

Seems like single stack won’t be too long since they are currently driving with it. Have to say Elon does not seem to be a very good Alpha tester though.

you don't seem to have a firm grasp of what an alpha version vs a beta version is, nor the difference between personal preference and actual unsafe driving.

Beta is by definition incomplete, both in feature set and with regards to software bugs. Alpha is essentially the first version they test on roads and is expected to be very buggy, crash and have numerous issues. The fact that the beta versions we get are not perfect is fully expected and says nothing about the quality of alpha testing. Aside from that I would be floored if Elon was the only alpha tester.

In terms of FSD being unusable, we've discussed that before. It's quite usable in many circumstances. The fact that it doesn't have the exact acceleration profile you desire or that it may use the brakes more than you like simply means you have different preferences.

I posted a few days ago how FSD took me from work to a store, then another store, then home, a total of 30-40 minutes and 18 miles including residential streets, feeder roads, highways and 3 traffic circles with zero interventions. Was it perfect? No - it hesitated some at the traffic circles and at some turns but it was never unsafe.

 

[AlanSubie4Life]

I posted a few days ago how FSD took me from work to a store, then another store, then home, a total of 30-40 minutes and 18 miles including residential streets, feeder roads, highways and 3 traffic circles with zero interventions. Was it perfect? No - it hesitated some at the traffic circles and at some turns but it was never unsafe.

Post the video.

 

[Dewg]

Post the video.

Why? You'll just pick it to death about how it didn't accelerate smoothly and quickly enough, decelerated too late and used brakes, and had a pulsing regen line, left too much of a gap between it and the car ahead when leaving an intersection, accelerated too much on a turn, and was too timid around pedestrians, bicyclists, etc. Can't handle "basic driving behavior, which is clearly garbage. It’s completely and totally unusable and unpleasant, just a pile of turds."

 

[AlanSubie4Life]

Why? You'll just pick it to death about how it didn't accelerate smoothly and quickly enough, decelerated too late and used brakes, and had a pulsing regen line, left too much of a gap between it and the car ahead when leaving an intersection, accelerated too much on a turn, and was too timid around pedestrians, bicyclists, etc. Can't handle "basic driving behavior, which is clearly garbage. It’s completely and totally unusable and unpleasant, just a pile of turds."

Yes. That is what I would do. I mean I think it is more than reasonable to ask that it be able drive well. And reject it if it cannot. We absolutely cannot be compromising on this. How else can we get it to be good ASAP? Not by accepting a steaming pile, that’s for sure. I turned it on today and disengaged on the first stop as it surged stupidly towards a light that was clearly changing to red.

As I said, it is still amazing what it can do. Wonder if the promise will ever be realized though?

 

[Dewg]

Yes. That is what I would do. I mean I think it is more than reasonable to ask that it be able drive well. And reject it if it cannot. We absolutely cannot be compromising on this. How else can we get it to be good ASAP? Not by accepting a steaming pile, that’s for sure.

As I said, it is still amazing what it can do. Wonder if the promise will ever be realized though?

Never for you.

 

[AlanSubie4Life]

Never for you.

At least I can say I am a lot more optimistic than you are! And believe me, that is saying something.

 

[sleepydoc]

At least I can say I am a lot more optimistic than you are! And believe me, that is saying something.

@Dewg is optimistic enough, he's just not optimistic that you'll ever be satisfied with the performance of FSD. Having read your posts I can't disagree with him. The general tone of your posts has been overwhelmingly negative, continually criticizing every detail of how FSD drives. I'm no fanboy and FSD still makes plenty of boneheaded mistakes for me but I've also seen plenty of progress over the past year and find more and more situations where it used to fail but now succeeds.

 

[AlanSubie4Life]

@Dewg is optimistic enough, he's just not optimistic that you'll ever be satisfied with the performance of FSD. Having read your posts I can't disagree with him. The general tone of your posts has been overwhelmingly negative, continually criticizing every detail of how FSD drives. I'm no fanboy and FSD still makes plenty of boneheaded mistakes for me but I've also seen plenty of progress over the past year and find more and more situations where it used to fail but now succeeds.

I keep on saying how much it has improved though. Honestly a bit confused about the backlash.

It’s possible for it to be dramatically better and amazing and still nowhere near good enough.

I’ll continue to provide my analysis which I think has been appreciated, and definitely will comment when some of these things improve assuming I am still active on the forum (which I may not be).

I certainly will not bash something which is good.

Every single thing I gripe about is also echoed by tons of other people; there is nothing unique about the underlying complaints.

In fact, many of these specific issues have been called out in Tesla’s release notes!

 

[impastu]

Any speculation on when the 2022-20 s/w branch will converge with 2022-28? I understand that FSD beta is only running on 2022-20 branch (and I unfortunately now qualify but am on 28 branch).

 

[sleepydoc]

Any speculation on when the 2022-20 s/w branch will converge with 2022-28? I understand that FSD beta is only running on 2022-20 branch (and I unfortunately now qualify but am on 28 branch).

No one know for sure. FSD has always lagged behind. I’ve seen many speculate that 69.3 may be on a later version but no real data on it.

 

[jabloomf1230]

There are leaked release notes for 10.69.2.3 which is on 2022.20.18:

FSD Beta 10.69.2.3 (2022.20.18) Official Tesla Release Notes - Software Updates

Tesla software update 2022.20.18 (FSD Beta 10.69.2.3) includes Full Self-Driving (Beta) Suspension, FSD Beta v10.69.2.3 Release Notes, Full Self-Driving (Beta), Cabin Camera, Cabin Camera, Seat Belt System Enhancement, Tire Configuration, Tesla Adaptive Suspension, Car Colorizer, Improvements to...

www.notateslaapp.com

I'm not sure if anyone outside of Tesla this minor release.

 

[Mardak]

the difference between personal preference and actual unsafe driving

I wonder what Tesla's threshold is for single stack highway driving? Current production Navigate on Autopilot is quite polished except for the cases it doesn't understand, e.g., emergency vehicle partially blocking the lane, so FSD Beta stack is probably safer in handling these odd scenarios. Similarly, FSD Beta might fix some "phantom braking," e.g., smarter FSD Beta behavior for a merging vehicle including switching away instead of matching speed; but it could introduce other types of unwanted braking that people have noticed on city streets.

I would guess FSD Beta 11 highway driving will get released without the full polish of NoA to the Beta fleet, and then it'll be some time for polish to deploy that to production for the rest of the Tesla fleet. I suppose Tesla could even monitor usage statistics to see if FSD Beta usage changes if people toggle it on/off for the new single stack behaviors.

 

[sleepydoc]

I wonder what Tesla's threshold is for single stack highway driving? Current production Navigate on Autopilot is quite polished except for the cases it doesn't understand, e.g., emergency vehicle partially blocking the lane, so FSD Beta stack is probably safer in handling these odd scenarios. Similarly, FSD Beta might fix some "phantom braking," e.g., smarter FSD Beta behavior for a merging vehicle including switching away instead of matching speed; but it could introduce other types of unwanted braking that people have noticed on city streets.

I would guess FSD Beta 11 highway driving will get released without the full polish of NoA to the Beta fleet, and then it'll be some time for polish to deploy that to production for the rest of the Tesla fleet. I suppose Tesla could even monitor usage statistics to see if FSD Beta usage changes if people toggle it on/off for the new single stack behaviors.

In general I'd say the AP stack is better than FSD but it also has far less to do. NoA's big fault is it is very slow to change lanes, often causing it to miss exits where a rapid lane change is required.

FSD doesn't handle emergency vehicles well so I wouldn't be too confident that that would improve on the highway simply by combining them.

 

[Zane Liu]

In general I'd say the AP stack is better than FSD but it also has far less to do. NoA's big fault is it is very slow to change lanes, often causing it to miss exits where a rapid lane change is required.

FSD doesn't handle emergency vehicles well so I wouldn't be too confident that that would improve on the highway simply by combining them.

I think the biggest challenge I’ve dealt with on NoA is how it struggles with lane lines on occasion. Certain highway splits will cause the car to bounce or suddenly swerve a bit which can be jarring for cars behind you. Other thing I noticed was when a car merged in front of me the Tesla could no longer see the lane dash lines and swerved a bit as well. Just some things which I wonder will improve with FSD

 

[Mardak]

FSD doesn't handle emergency vehicles well so I wouldn't be too confident that that would improve on the highway simply by combining them

Yeah, FSD Beta doesn't really react to emergency lights. I was more thinking of emergency vehicles parked partially into the lane to act as a physical barrier for people standing behind where there's not enough shoulder. FSD Beta 10.69 shows a message for switching lanes away from path blockage, and even if there's not a lane to switch to, it knows to bias away instead of just stopping.

I would hope FSD Beta 11 single stack would have noticed at least some parts of these crashed vehicles via the occupancy network if failing to identify them as vehicles: