FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Mardak]

FSD Beta 11 full release notes:

• Enabled FSD Beta on highway. This unifies the vision and planning stack on and off-highway and replaces the legacy highway stack, which is over four years old. The legacy highway stack still relies on several single-camera and single-frame networks, and was setup to handle simple lane-specific maneuvers. FSD Beta's multi-camera video networks and next-gen planner, that allows for more complex agent interactions with less reliance on lanes, make way for adding more intelligent behaviors, smoother control and better decision making.
• Improved Occupancy Network's recall for close by obstacles and precision in severe weather conditions with 4x increase in transformer spatial resolution, 20% increase in image featurizer capacity, improved side camera calibration, and 260k more video training clips (real-world and simulation).
• Improved merging behavior by leveraging lane geometry and lane bounds, association with coarse map information and better gap selection algorithms, allowing for smoother and safer experience.
• Added highway behavior to offset away from blocked lanes and generic obstacles like road debris while also adding a smooth hand-off between in-lane offsetting and lane changing.
• Improved speed-based lane change decisions to better avoid slowing down traffic in fast lanes, and interfere less with navigation.
• Reduced sensitivity for speed-based lane changes in CHILL mode.
• Improved lane changes to allow higher jerk maneuvers if required to stay on-route or to move away from lane blockages.
• Improved smoothness at highway lane splits by being less strict about centering between lane lines and allowing lower jerk maneuvers, where safe to do so.
• Reduced latency of trajectory optimization by 20% on average, without sacrificing behavior, by leveraging numerical tricks for more efficient computations.

 

[Mardak]

It seems like all of the FSD Beta 11 improvements were prioritized for highway driving, but quite a few of them will also benefit regular city street driving too such as higher jerk to quickly switch out of a lane blockage (instead of slowing down to a stop, which is less appropriate on a highway).

Also interesting is the item about lower jerk maneuvers, which I believe means FSD Beta will prefer a gradual smooth shifting over to a newly created exit lane. Although curious why it needs a special distinction for highway splits instead of the existing 10.10's "Smoother fork maneuvers and turn-lane selection using high fidelity trajectory primitives."

 

[Mardak]

With the note about sensitivity of lane changes in CHILL mode, it seems to suggest Navigate on Autopilot settings (Enable At Start Of Every Trip, Speed Based Lane Changes including Mad Max, Exit Passing Lane, Require Lane Change Confirmation) are still ignored. That's how it has been with city streets, so not that surprising FSD Beta only follows Full Self-Driving (Beta) Profile settings (CHILL, AVERAGE, ASSERTIVE).

As with any major change, there's plenty of decisions on what existing functionality should remain vs replaced vs removed. It makes sense to delete things that were added because of system capabilities/deficiencies of the time. Like for cars, manufacturers needed to decide whether to keep a cassette player, and people even had special adapters to play arbitrary things to their cars and "depended" on that interface and were more impacted by the removal.

Of course, we have plenty of ways to play music in the car now that most people will find better than using cassettes, so similarly, will FSD Beta on highways be better for most people to not want the controls from Navigate on Autopilot? I wouldn't be surprised if Navigate on Autopilot gets removed with single stack while also reintroducing some settings/controls eventually.

 

[T2v]

Just got V11 2022.36.20 on my new MYP after around 2500+ miles of auto pilot and 99 score . Will update after testing

 

[Mardak]

Just got V11 2022.36.20

It's a bit confusing, but Tesla basically has 3 main FSD Beta versions happening at the same time for now:

• 2022.20.19 = FSD Beta 10.69.2.4 where most existing testers are waiting
• 2022.36.20 = FSD Beta 10.69.3.1 adding new testers as of last night
• 2022.40.5 = FSD Beta 11 only to a small number of employees for early testing of single stack / highway

 

[boonedocks]

Just got V11 2022.36.20 on my new MYP after around 2500+ miles of auto pilot and 99 score . Will update after testing

Not V11. That’s 10.69.3.1

 

[swedge]

As with any major change, there's plenty of decisions on what existing functionality should remain vs replaced vs removed.

My hunch is that the feature set of FSD is not what we are seeing in beta. Instead, they are stressing the hard parts by not enabling options which would make it easier. For example, vision is harder without mapping, but necessary for unmapped areas. So if they want to improve vision quickly, they would disable map inputs and so use even well mapped areas in testing and training the vision capability. I expect that chill and optional lane change confirmation and other adjustment will be added to city FSD before it becomes a standard, wide release product. We'll see...

 

[DarkForest]

Since the follow distance set by the steering wheel controls doesn’t apply to FSD, I wonder if that’s just going away in v11. That might explain why they never bothered giving us back the 1 setting.

If it does go away, I really hope they let us change the driving mode with the steering wheel controls between Chill, Average, and Assertive.

 

[swedge]

Since the follow distance set by the steering wheel controls doesn’t apply to FSD, I wonder if that’s just going away in v11. That might explain why they never bothered giving us back the 1 setting.

If it does go away, I really hope they let us change the driving mode with the steering wheel controls between Chill, Average, and Assertive.

I expect they will eventually include these user comfort adjustments. But for now, perhaps they are focused on getting it to respond safely with a stable set of those parameters. We are testing and supplying training input on the stuff they don't have completely right yet. I think those adjustments will be pretty easy for them to include later, and it won't degrade the response to back off or slow down a bit.

For example, in some older versions, FSDβ would accelerate and turn pretty aggressively. It is good that it knows how to do that if it ever needs to, but I find it more comfortable now that it has mellowed out a bit.

It used to take some turns faster than it could handle, and would cross well into opposing lanes, oops. After reporting those a few time, I just dialed back the speed manually for a more relaxing ride. The newer version I have now can stay in it's lane OK at speed, and also slows down in advance of some tighter turns so that it can handle those too. So now that it can deal with these twisties, chilling down should not cause problems. And If a driver does dial up the speed, it now knows how to slow down for the tight turns. I hope. Hmmm... Maybe I should test that...

We are in a development program, not a product evaluation. They need us near the edges to help them teach the car how handle those edges safely. I expect the "public" versions will include more passenger comfort concessions and options, but the car will know how to drive hard when the need arrises.

That is my guess anyway.

 

[wknickless]

For example, vision is harder without mapping, but necessary for unmapped areas. So if they want to improve vision quickly, they would disable map inputs and so use even well mapped areas in testing and training the vision capability.

My experience tends to disconfirm that hypothesis.

On my way to/from work there’s an intersection with clearly marked straight/left and right turn lanes:

I have yet to see FSD Beta get into the straight/left turn lane when going left to right; it always wants to be in the right lane even though that’s a right turn lane. Going the other way (right to left) it routinely tries to jump into the right turn lane if there’s a bit of a delay on the left straight-through lane.

And yes, the “mind of car” visualization shows the left/straight and right turn road markings. So it’s not like the vision system can’t pick up on them.

 

[Mardak]

the “mind of car” visualization shows the left/straight and right turn road markings. So it’s not like the vision system can’t pick up on them.

I suspect these road markings visualizations were part of the separate old stack when they were introduced in 2019(!) as part of Full Self Driving Visualization Preview. The way FSD Beta predicts intersections has most likely gone through multiple rewrites over the past few years to focus on the raw visual data to make sure they have the right fundamental architecture and intentionally minimized additional dependencies like map data and road markings.

Only recently with 10.69, did they reintroduce coarse map data:

• Added a new "deep lane guidance" module to the Vector Lanes neural network which fuses features extracted from the video streams with coarse map data, i.e. lane counts and lane connectivites. This architecture achieves a 44% lower error rate on lane topology…

And 10.69.3, which most of us haven't gotten yet reintroduces road markings:

• Added a new "road markings" module to the Vector Lanes neural network which improves lane topology error at intersections by 38.9%.

Hopefully this fixes issues like predicting left turn lanes tend to be left-only but in this case it's actually a left-straight lane.

So similarly, it's interesting that FSD Beta 11 reintroduces coarse map data for better highway behavior:

• Improved merging behavior by leveraging lane geometry and lane bounds, association with coarse map information and better gap selection algorithms, allowing for smoother and safer experience.

Navigate on Autopilot has always been heavily dependent on map data as at highway speeds, the visual system often cannot see far enough by itself for smooth experiences. And this "reintroducing" process will probably happen for some of the user facing controls too.

 

[Mardak]

they are stressing the hard parts by not enabling options which would make it easier

Makes sense, and I believe this follows directly from Elon Musk's "the best part is no part, the best process is no process" where Autopilot team tries really hard to make sure the fundamentals are good as often times it results in not needing other parts that then need to be optimized and maintained. Like you suggest, this has made the vision system more unified and capable overall, and if things turn out to still need an additional part, then it can be added back.

 

[Mardak]

Since the follow distance set by the steering wheel controls doesn’t apply to FSD, I wonder if that’s just going away in v11

Ah right, that's a good point of something else currently ignored by FSD Beta. We have it set to the max 7, and on highways it's approximately 2 seconds behind the lead vehicle. FSD Beta on CHILL seems to follow on city streets at roughly 1 second. I suppose another possible behavior is Tesla decides follow setting is important enough for highways and finally decides to prioritize integrating it to FSD Beta which then could potentially mean it gets used on city streets too.

 

[Mardak]

They need us near the edges to help them teach the car how handle those edges safely

I agree that Tesla wants to discover the difficult problems, and something like a long follow distance makes things "easier" with more time to react as well as better visibility, e.g., too close behind a tall vehicle sometimes requires looking for other visual clues of what's ahead.

Although curiously that FSD driving profiles exists with CHILL and ASSERTIVE. Then again, it seemed like this was introduced to allow users to turn off California rolling stops, and with the NHTSA recall removing that feature, maybe there isn't as much need to have this setting especially as a development program newly introducing highway driving and wanting to uncover the "difficult" unknown unknowns.

 

[swedge]

Just got V11 2022.36.20 on my new MYP after around 2500+ miles of auto pilot and 99 score . Will update after testing

Looking at TeslaFi, most of the downloads so far of that version are to folks new to Beta like you. I think prior beta releases have gone to cars already in the program first.

I look forward to reading your first impressions, those of a new beta recruit.

SW

 

[willow_hiller]

Wide release still on track for December:

https://twitter.com/x/status/1594458275013181443

 

[Hayseed_MS]

If I have FSD purchased with the car but am not in the beta program, will I get it with this wide release? I am currently on 2022.40.4 1

 

[Supcom]

If I have FSD purchased with the car but am not in the beta program, will I get it with this wide release? I am currently on 2022.40.4 1

Tesla has a goal to release city streets navigation to all FSD licensees by the end of the year. If they achieve that, then you will receive it without being in the beta program.

It is, of course, very possible that they will miss this goal, but I believe that they will include city streets as part of the holiday release.

 

[JulienW]

......It is, of course, very possible that they will miss this goal......

WHAT, name just once Tesla has missed a goal.

 

[EVNow]

but I believe that they will include city streets as part of the holiday release.

I definitely think that is the plan.

The intriguing thing is .... is that holiday release V11 ?

I'd think V11 would be too "raw" to be released widely for everyone next month (given it is not even available to all employees now, let alone youtubers or beta testers).

So, it is possible the "general" release will include some release of FSD Beta - but current testers will continue to get newer versions of FSDb including V11, which will go to "general" release later.