FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[sleepydoc]

so what’s the difference between v11 and v10.69.42.2011.99? The big thing everyone was talking about was merging Autopilot and FSD. Merging the code will take a fair amount of resources and likely lead to sow regression. Since they’re obviously still working on things like lane planning, lane selection, turns, etc it makes just as much sense to delay the merger.

Beyond that, I really don’t care what version name they give it as long as they keep improving it.

 

[JulienW]

so what’s the difference between v11 and v10.69.42.2011.99? The big thing everyone was talking about was merging Autopilot and FSD. Merging the code will take a fair amount of resources and likely lead to sow regression. Since they’re obviously still working on things like lane planning, lane selection, turns, etc it makes just as much sense to delay the merger....

But Tesla HAS been working on merging the Stacks for WAY over a year now and still no V11 in sight. Not sure how many more months, quarters or even years they can delay it if we are ever going to get to at least L3. I'm not unhappy with the progress Autosteer on City Streets (FSD Beta) is making but we NEED for AP to be ported over so we can make some meaningful progress towards REAL FSD (L3).

Just musing that V11 is looooog in the tooth.

 

[Mardak]

I really don’t care what version name they give it as long as they keep improving it

Right, there has been plenty of fundamental improvements in the 10.x series of FSD Beta. If Elon Musk hadn't announced single stack as version 11 over a year ago, we probably would be on maybe FSD Beta 15 now, but as you point out, the name doesn't really matter.

Clearly he thought it would have been relatively simple to "just" turn on city streets behavior for highway driving, and while some like me probably would have even enjoyed the quirks of both the higher highs and lower lows, most people probably want and expect the high safety of the existing stack.

And that's probably the practical reality that FSD Beta single stack hasn't matched the existing safety in all highway situations, and it isn't the top priority as safety can be improved for unprotected left turns, high velocity cross traffic and other non-highway situations. But fortunately that also means we have experienced those improvements sooner that Autopilot team focusing solely on replacing interstate driving.

 

[Mardak]

Planner optimization using NN

Looks like this is now in 10.69.3 release notes:

• Reduced runtime of candidate trajectory generation by approximately 80% and improved smoothness by distilling an expensive trajectory optimization procedure into a lightweight planner neural network.

This seems to indicate it's able to run 5x trajectory plans in the same time. Potentially this was critical for smoothly handling high-speed single-stack situations like interstates? There were also a number of items for latency that could help for FSD Beta 11:

• Upgraded the Object Detection network to photon count video streams and retrained all parameters with the latest autolabeled datasets (with a special emphasis on low visibility scenarios). Improved the architecture for better accuracy and latency…
• Converted the VRU Velocity network to a two-stage network, which reduced latency…
• Converted the NonVRU Attributes network to a two-stage network, which reduced latency…
• Reduced best case object photon-to-control system latency by 26% through adaptive planner scheduling, restructuring of trajectory selection, and parallelizing perception compute. This allows us to make quicker decisions and improves reaction time.

 

[misdemeanor]

V11 rollout to internal staff yesterday at 11:11PM PST
https://www.teslaoracle.com/2022/11/12/tesla-starts-rollout-of-fsd-beta-v11-2022-40-5-confirms-elon-musk-and-it-most-probably-is-single-stack/

 

[Mardak]

Is the FSD Beta 11 actually ready or last night 11/11 11:11 was too good to pass up?

https://twitter.com/x/status/1591327799469801475

Single stack has been in development for months -- at least the highway stack, and parking lot stack had a target deadline of AI day, so quite likely the software is actually pretty close. Hopefully not too many dot releases of 11 before it's ready for the rest of us.

Also nice that everyone including latest firmware 2022.40.x should be compatible.

 

[AlanSubie4Life]

Is the FSD Beta 11 actually ready or last night 11/11 11:11 was too good to pass up?

Last night it claimed 1 vehicle was downloading. How they generate their metrics I have no idea.

 

[Mardak]

Last night it claimed 1 vehicle was downloading

Oh I'm quite confident Teslascope actually detected an employee downloading that version and installed. My comment was more of how far the capabilities of this software are for wider release to Safety Score testers. The previous 10.69 major update had to go through several weeks of ironing out bugs updates before getting to us.

 

[sleepydoc]

Oh I'm quite confident Teslascope actually detected an employee downloading that version and installed. My comment was more of how far the capabilities of this software are for wider release to Safety Score testers. The previous 10.69 had to go through several weeks of updates before getting to us.

It probably is an employee rollout - that’s how every other release has happened. Since V11 is a major rewrite I fully expect there to be significant issues with it so I wouldn’t be surprised if there‘s a more protracted delay between the employee rollout and the influencer rollout. My only question is whether Twitter will still be around and functional by that time. How is Omar going to gush about how totally smooth V11 is if he can’t pay his $8?

 

[EVNow]

Partial release notes :

2022.40.5 Official Tesla Release Notes - Software Updates

Tesla software update 2022.40.5 (FSD Beta 11) includes FSD Beta v11 Release Notes, Full Self-Driving (Beta).

www.notateslaapp.com

- Enabled FSD Beta on highway. This unifies the vision and planning stack on and off-highway and replaces the legacy highway stack, which is over four years old. The legacy highway stack still relies on several single-camera and single-frame networks, and was setup to handle simple lane-specific maneuvers. FSD Beta's multi-camera video networks and next-gen planner, that allows for more complex agent interactions with less reliance on lanes, make way for adding more intelligent behaviors, smoother control and better decision making.

Improved Occupancy Network's recall for close by obstacles and precision in severe weather conditions with 4x increase in transformer spatial resolution, 20% increase in image featurizer capacity, improved side camera calibration, and 260k more video training clips (real-world and simulation).

- Reduced the predicted velocity error of very close-by motorcycles, scooters, wheelchairs, and pedestrians by 63.6%. To do this, we introduced a new dataset of simulated adversarial high speed VRU interactions. This update improves autopilot control around fast-moving and cutting-in VRUs.

- Improved creeping profile with higher jerk when creeping starts and ends.

- Improved control for nearby obstacles by predicting continuous distance to static geometry with the general static obstacle network.

- Reduced vehicle "parked" attribute error rate by 17%, achieved by increasing the dataset size by 14%. Also improved brake light accuracy.

- Improved clear-to-go scenario velocity error by 5% and highway scenario velocity error by 10%, achieved by tuning loss function targeted at improving performance in difficult scenarios.

- Improved detection and control for open car doors.

- Improved smoothness through turns by using an optimization-based approach to decide which road lines are irrelevant for c[unreadable] given lateral and longitudinal acceleration and jerk limits as we[unreadable] vehicle kinematics.

- Improved stability of the FSD UI visualizations by optimizing ethernet data transfer pipeline by 15%.

- Improved recall for vehicles directly behind ego, and improv precision for vehicle detection network.

 

[Mardak]

Partial release notes

Only the first two are actually from FSD Beta 11. The other thread pointed out the rest are copied from 10.11. Maybe the threads could be cleaned up somehow to avoid confusion.

 

[jabloomf1230]

How is Omar going to gush about how totally smooth V11 is if he can’t pay his $8?

Omar has his white on blue checkmark as do I and lot of other people.

 

[sleepydoc]

Omar has his white on blue checkmark as do I and lot of other people.

Serious question - what exactly does the blue checkmark mean now?

 

[Mardak]

It probably is an employee rollout - that’s how every other release has happened

Yup, seems like this initial FSD Beta 11 release is even smaller than the usual employee rollouts in the past. So that does seem like 11/11's single stack release was more symbolic than actually ready:

https://twitter.com/x/status/1591490459066564610

Hopefully it does result in it getting ready sooner for the rest of us than if it wasn't released. Although that might also mean no additional 10.69 related releases either for us in the meantime as getting FSD Beta 11 to wide release is probably the top priority and not so much keeping people on old software happy.

 

[jabloomf1230]

Serious question - what exactly does the blue checkmark mean now?

It means:

1. You had a Twitter account either on or prior to 9 November 2022.
2. You have an iPhone, backed with an Apple account and a valid credit card.
3. You subscribed to Twitter Blue via the App Store for $8/month + tax.

 

[sleepydoc]

Since V11 is a major rewrite I fully expect there to be significant issues with it so I wouldn’t be surprised if there‘s a more protracted delay between the employee rollout and the influencer rollout.

Yup, seems like this initial FSD Beta 11 release is even smaller than the usual employee rollouts in the past. So that does seem like 11/11's single stack release was more symbolic than actually ready:

https://twitter.com/x/status/1591490459066564610

Hopefully it does result in it getting ready sooner for the rest of us than if it wasn't released. Although that might also mean no additional 10.69 related releases either for us in the meantime as getting FSD Beta 11 to wide release is probably the top priority and not so much keeping people on old software happy.

Yup - a few Elon weeks means we'll be lucky to see it before spring.

 

[Mardak]

• Enabled FSD Beta on highway. This unifies the vision and planning stack on and off-highway and replaces the legacy highway stack, which is over four years old. The legacy highway stack still relies on several single-camera and single-frame networks, and was setup to handle simple lane-specific maneuvers. FSD Beta's multi-camera video networks and next-gen planner, that allows for more complex agent interactions with less reliance on lanes, make way for adding more intelligent behaviors, smoother control and better decision making.

I wonder why the special emphasis on lanes? FSD Beta indeed ignores lanes to go around parked vehicles, and that would be quite the change for someone only familiar with legacy highway behavior. Maybe also hinting at a change to the existing lane-centering behavior for when a lane widens for an onramp?

What other situations on highways might be improved without a strict following of lane lines?

 

[EVNow]

What other situations on highways might be improved without a strict following of lane lines?

Lane splits.

Anyway, it will indeed be "interesting" to see how FSDb performs - fortunately I won't be the one testing

Even now I prefer AP to NOA.

 

[Knightshade]

I wonder why the special emphasis on lanes? FSD Beta indeed ignores lanes to go around parked vehicles, and that would be quite the change for someone only familiar with legacy highway behavior. Maybe also hinting at a change to the existing lane-centering behavior for when a lane widens for an onramp?

What other situations on highways might be improved without a strict following of lane lines?

Hitting emergency vehicles not entirely pulled onto the shoulder.

Which is a thing highway code has had problems with for years- honestly that, and object/debris detection, were the only 2 things keeping it from at least L3 if not 4, strictly on freeways IMHO.

 

[Mardak]

At least for "make way for adding more intelligent behaviors," I was reminded last night when driving on an interstate with 8 lanes in each direction that legacy highway Autopilot basically only looks at the current and immediately adjacent lanes. Hopefully the intelligent behaviors means it can see if upcoming traffic is moving faster two lanes away to switch.

The unifying of driving stacks might result in some repriortization of features as highway driving counts for a lot of daily miles, and some of these features should also improve city streets driving too.