FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[drtimhill]

Removing legacy networks means they can't be used as an immediate fallback, e.g., currently FSD Beta can revert to basic Autopilot in poor weather and when switching to highway driving, so Tesla must believe/know the new networks are at least as good as what's replaced. This might have also resulted in the extended development time for 10.12 as removing safety-nets probably has a higher threshold.

I'm not sure you can assume "legacy" means they are removing networks currently or previously used by non-FSD (e.g. AP), since that would potentially impact the AP/NoA functionality. These are probably networks that were used by earlier iterations of FSD but are now redundant as their functions have been subsumed by newer sections of the NN.

 

[Mardak]

I'm not sure you can assume "legacy" means they are removing networks currently or previously used by non-FSD (e.g. AP), since that would potentially impact the AP/NoA functionality

Yeah, definitely would be interesting to see which networks were actually removed or potentially just turned off instead of always running.

For example, maybe "FSD Visualization Preview" networks (for traffic lights, road markings, trash cans, etc.) don't need to be active when FSD Beta is active as FSD Beta networks probably have a better understanding anyway, but so far it hadn't been a priority to update the visualizations to use FSD Beta network outputs as the "Preview" one was good enough.

But indeed, these could also be removing early FSD Beta networks that were never used for basic Autopilot and are no longer relevant.

 

[verygreen]

@verygreen, do you have insights into which networks are (already?) removed? This could give a sense of where Tesla feels FSD Beta is already more capable as well as when 10.12 and future versions get released

I don't think I've seen any removed (when comparing to similarly aged non-fsdbeta build)

 

[DrChaos]

I'm not sure you can assume "legacy" means they are removing networks currently or previously used by non-FSD (e.g. AP), since that would potentially impact the AP/NoA functionality. These are probably networks that were used by earlier iterations of FSD but are now redundant as their functions have been subsumed by newer sections of the NN.

Agree: the only ones that really matter would be the nets which are in the real-time execution loop. So they'd all be in FSDb.

If AP vs FSD are different products and codebases, as still seems likely, then the code and nets get swapped out from storage into the computer and turned on and that's an infrequent event vs all the time.

There were probably some layers in existing functionality, or nets producing outputs which turned out to be not needed in a retrain.

 

[MP3Mike]

If AP vs FSD are different products and codebases, as still seems likely, then the code and nets get swapped out from storage into the computer and turned on and that's an infrequent event vs all the time.

No, it switches between the two instantly as you get on/exit a freeway. So unless things have changed they both run all the time.

 

[boonedocks]

No, it switches between the two instantly as you get on/exit a freeway. So unless things have changed they both run all the time.

You can also see the difference of how just TACC looks / works vs FSD Beta on side streets / city streets. It still looks to use the old code as well.

 

[drtimhill]

If AP vs FSD are different products and codebases, as still seems likely, then the code and nets get swapped out from storage into the computer and turned on and that's an infrequent event vs all the time.

Sadly not .. both stacks run at the same time (and must), which is one of the reasons why eventually FSD will take over the AP/NoA tasks also, so Tesla can discard the older and less capable AP stack.

 

[DrChaos]

Sadly not .. both stacks run at the same time (and must), which is one of the reasons why eventually FSD will take over the AP/NoA tasks also, so Tesla can discard the older and less capable AP stack.

Why do the stacks need to run at the same time? Do you think all the nets for conventional AP are processing every image simultaneously with FSD? That would be a major waste of computing & power consumption, and it would hurt FSDb safety by lowering frame rate.

I'm imagining that a swap could take place within half a second or so and wouldn't be very noticable.

 

[MP3Mike]

Why do the stacks need to run at the same time? Do you think all the nets for conventional AP are processing every image simultaneously with FSD? That would be a major waste of computing & power consumption, and it would hurt FSDb safety by lowering frame rate.

I'm imagining that a swap could take place within half a second or so and wouldn't be very noticable.

Because it has to, and does, instantly switch between code bases as you enter/exit a freeway. Having no processing/driving control for even half a second would be totally unacceptable.

 

[Knightshade]

Because it has to, and does, instantly switch between code bases as you enter/exit a freeway. Having no processing/driving control for even half a second would be totally unacceptable.

FWIW my observation is that it switches late during the on-ramp process (sometimes not until the end of the onramp)

So there's be plenty of time to spin up the traditional code before the handoff--- not like it doesn't know it's on an on-ramp.

 

[JulienW]

FWIW my observation is that it switches late during the on-ramp process (sometimes not until the end of the onramp)

So there's be plenty of time to spin up the traditional code before the handoff--- not like it doesn't know it's on an on-ramp.

Watch Frenchie's video as he crosses an Interstate the Stack switches to AP and back to Beta instantly with no time to calculate a handoff, since it is an incorrect switch. Time code 3:55

 

[Knightshade]

That does not appear to be a switch to legacy code, he remarks the display was already being slow/laggy even before that, looks like some bit of code had to restart as there's jumpy/incomplete visualization for a few moments there- and it's still showing things like the traffic poles in the visualization even with the red lines not showing.

 

[Mardak]

I don't think I've seen any removed (when comparing to similarly aged non-fsdbeta build)

Thanks. Just to be clear, "removed" also includes any conditional if/then dynamic loading of various networks (as opposed to deleting the weights from storage/firmware)?

Overall, it seems like the compute budget needs to simultaneously handle basic Autopilot, related existing safety features, pre-FSD-Beta features like green light chime, FSD Beta and any other "shadow mode" networks to power triggers and data collection to improve FSD Beta 11. It would be natural to try to free up compute by conditionally loading networks while maintaining safety.

In some sense, the existing FSD Beta population actively using it has less compute for "shadow mode" as theoretically it could double the compute requirements to run a similarly FSD-beta-sized set of experimental networks just for data collection?

 

[sleepydoc]

Thanks. Just to be clear, "removed" also includes any conditional if/then dynamic loading of various networks (as opposed to deleting the weights from storage/firmware)?

Overall, it seems like the compute budget needs to simultaneously handle basic Autopilot, related existing safety features, pre-FSD-Beta features like green light chime, FSD Beta and any other "shadow mode" networks to power triggers and data collection to improve FSD Beta 11. It would be natural to try to free up compute by conditionally loading networks while maintaining safety.

In some sense, the existing FSD Beta population actively using it has less compute for "shadow mode" as theoretically it could double the compute requirements to run a similarly FSD-beta-sized set of experimental networks just for data collection?

I saw a post in another thread referencing a tweet saying V11 was being 'deployed in shadow mode.' This would make a lot of sense, especially for a big rewrite, but if it's running on the cars running the current FSD beta software it would also essentially double the processing overhead. If they added that to people just using AP they could get data without having to worry about maxing out the computer.

 

[ammulder]

I saw a post in another thread referencing a tweet saying V11 was being 'deployed in shadow mode.' This would make a lot of sense, especially for a big rewrite, but if it's running on the cars running the current FSD beta software it would also essentially double the processing overhead. If they added that to people just using AP they could get data without having to worry about maxing out the computer.

The computer does have two chips... could be running the main stack on one and the shadow stack on the other, unless/until one of the chips fails.

 

[drtimhill]

Why do the stacks need to run at the same time? Do you think all the nets for conventional AP are processing every image simultaneously with FSD? That would be a major waste of computing & power consumption, and it would hurt FSDb safety by lowering frame rate.

I'm imagining that a swap could take place within half a second or so and wouldn't be very noticable.

Because FSD hands over to AP (and vice versa) at arbitrary times (not just when "expected" as for example at an off-ramp). You can see that if you watch FSD "sulk" and decide it cannot drive in some conditions, the car display reverts to the non-FSD stack display. Both stacks are switching control back and forth in real time.

And the hand-over time has to be fast, much faster than the time needed to switch stacks and give the new stack time to warm uo its algorithms/state. Half a second is WAY too slow for handover while the car is moving at 60mph. Or would you be happy taking your hands off the wheel and closing your eyes for ½ second when taking a freeway off-ramp?

 

[verygreen]

Thanks. Just to be clear, "removed" also includes any conditional if/then dynamic loading of various networks (as opposed to deleting the weights from storage/firmware)?

Overall, it seems like the compute budget needs to simultaneously handle basic Autopilot, related existing safety features, pre-FSD-Beta features like green light chime, FSD Beta and any other "shadow mode" networks to power triggers and data collection to improve FSD Beta 11. It would be natural to try to free up compute by conditionally loading networks while maintaining safety.

In some sense, the existing FSD Beta population actively using it has less compute for "shadow mode" as theoretically it could double the compute requirements to run a similarly FSD-beta-sized set of experimental networks just for data collection?

well, there are several "schedules" on what networks to execute depending on operational mode, like summon, highway, city streets (this is the fsdbeta mode) and so on.

the city strreets runs the most of stuff including the fsdbeta bits. There are networks common to all those schedules, but some others are exclusive to certain modes. Like lane lines runs for highway and citystreets, but the experimental depth pointcloud only runs on citystreets.
Since there's no full replacement for anything the compute does not really double when additional NNs are introduced and certain modes (e.g. highway) have more spare cycles, and this is where the "signle stack" is being tried in shadow mode for the obvious reasons.

 

[Knightshade]

The computer does have two chips... could be running the main stack on one and the shadow stack on the other, unless/until one of the chips fails.

Nope.

Just the regular stack ran out of enough compute on a single node like 2 years ago now and has been borrowing significant amounts of compute from node B since then.

This is why folks expect it'll require (at least) HW4) for L3 or greater since the current hardware is insufficient to run "everything" in a single node and thus offer redundant failover.

 

[Mardak]

this is where the "single stack" is being tried in shadow mode for the obvious reasons

Oh of course, that's clever. FSD Beta might need to fall back to basic Autopilot at any moment, but FSD Beta doesn't need to be able to activate at any time, so indeed "highway" mode has spare compute to run experimental networks like those for FSD Beta 11. Thanks for clarifying.

 

[DrChaos]

Nope.

Just the regular stack ran out of enough compute on a single node like 2 years ago now and has been borrowing significant amounts of compute from node B since then.

This is why folks expect it'll require (at least) HW4) for L3 or greater since the current hardware is insufficient to run "everything" in a single node and thus offer redundant failover.

So, do you think that after the 'merge' and legacy Autopilot is completely off, there will be more compute available to improve the frame rates for FSDb? What level? Do you have any specific technical knowledge here?

That might improve performance or safety all on its own.

For better autonomous performance I would want: stereoscopic 4k to 8k cameras---much more data all on its own. Current main front camera is monocular and 1280x960 fixed orientation---humans have better vision than this for sure.

Plus multiple 77 GHz imaging radar sets. Agree on no lidar, but new radar is excellent. Elon was making up bullshit excuses about inability to do sensor fusion between vision and radar. You can train machine learning models for it, and use better sensors.