FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[EVNow]

I dont even think that .. it's pretty clear that V11 has always been about single stack (though now we get confused because the UI is at V11, and we have V10.11, which some people think IS "v11" .. visions of Spinal Tap).

See the first post. Based on what Elon said. Single stack is the least important of all those changes.

 

[drtimhill]

See the first post. Based on what Elon said. Single stack is the least important of all those changes.

Hmm well now .. good thing Elon is never wrong or changes his mind

 

[EVNow]

Hmm well now .. good thing Elon is never wrong or changes his mind

We have discussed a lot about this in this and other threads ... single stack isn't really going to be a game changer. All other updates are.

FSD needs all those changes to move beyond 1 disengagement every 10 miles it seems to be stuck in for a few releases now.

 

[Mardak]

Based on what Elon said. Single stack is the least important of all those changes.

Where did he say it's the least important? I believe you've ranked it as the least important as it's probably the least interesting on the surface as it doesn't directly improve city street disengagements.

But as we've seen with the continuous technical improvements to 10.x such as vector predictions needed to handle all highway driving, the high level product requirement of FSD Beta 11 being capable/safe enough to replace legacy Navigate on Autopilot stack has resulted in innovations that also improve city street behaviors.

Additionally existing Autopilot on highways is already quite capable, so the amount of potential improvement seems relatively small especially when city street disengagements are relatively frequent. But another way to look at single stack is that FSD Beta is currently implicitly geofenced to exclude NoA-capable highways, which I very roughly estimate to be 2000sqmi from ~50,000 miles of interstates, and that area is over 100x larger than the current explicit geofence for Toronto downtown. Getting rid of all geofence restrictions is probably an important milestone.

 

[drtimhill]

We have discussed a lot about this in this and other threads ... single stack isn't really going to be a game changer. All other updates are.

FSD needs all those changes to move beyond 1 disengagement every 10 miles it seems to be stuck in for a few releases now.

Perhaps .. however, you should remember that there is a dramatic difference between the complexity of city street driving and the far more restricted freeway driving. Single stack wont be a game changer for city streets, but it could well be for NoA/AP/TACC.

 

[Mardak]

there is a dramatic difference between the complexity of city street driving and the far more restricted freeway driving

Tesla hasn't shown any indications of wanting to release a system where the driver doesn't need to pay attention sometimes as a step before Full Self-Driving, but if single stack handles highway significantly better than current NoA, it seems like Tesla could introduce some conditional driving automation features for certain conditions/highways. FSD Beta was designed for the complexities of city streets, so there is potential it could do quite well on highways.

 

[EVNow]

Perhaps .. however, you should remember that there is a dramatic difference between the complexity of city street driving and the far more restricted freeway driving. Single stack wont be a game changer for city streets, but it could well be for NoA/AP/TACC.

Could be - but more worried about FSD Beta - since that is the one which needs dramatic improvement. BTW, James thinks the only noticeable difference would be city-freeway handoff. Also, depends on exactly what is meant by "one stack" - is it just NN that is combined ? Also planner ?

 

[drtimhill]

Could be - but more worried about FSD Beta - since that is the one which needs dramatic improvement. BTW, James thinks the only noticeable difference would be city-freeway handoff. Also, depends on exactly what is meant by "one stack" - is it just NN that is combined ? Also planner ?

Indeed .. I think from Teslas standpoint the priority is the NN .. at present they are running both NN stacks, and really want to remove that old AP/NoA NN stack to free up resources to expand FSD capability. That, in turn, gives the freeway stack the benefit of the far richer FSD stack. That will probably require rework on the old NoA planner. I'm not clear about the tradeoffs between one or many planners, since the rules are very different. For example, the FSD stack, if in doubt, will bring the car to a halt .. hardly safe to do on a freeway.

 

[sleepydoc]

BTW, James thinks the only noticeable difference would be city-freeway handoff.

IME, Autopilot/NOA on highways is very good and only rarely makes mistakes so it would be hard to have any dramatic improvements at this point.

 

[Knightshade]

IME, Autopilot/NOA on highways is very good and only rarely makes mistakes so it would be hard to have any dramatic improvements at this point.

I mean, not hitting stationary objects partly on the shoulder would be a dramatic improvement.

Not that it happens often but getting a complete OEDR to handle that kinda stuff is the main thing preventing L4 highway driving right now.

 

[EVNow]

I mean, not hitting stationary objects partly on the shoulder would be a dramatic improvement.

Not that it happens often but getting a complete OEDR to handle that kinda stuff is the main thing preventing L4 highway driving right now.

Building Voxels and using NN, instead of bag of points should help there. But even now FSD is fairly good at estimating where a stationary object is on the side of the road.

ps : I think internally Tesla probably has a good idea of where NOA would benefit from single stack. My basic point is - single stack won't help FSD Beta and it needs a lot more help than NOA.

 

[tmoz]

I mean, not hitting stationary objects partly on the shoulder would be a dramatic improvement.

Not that it happens often but getting a complete OEDR to handle that kinda stuff is the main thing preventing L4 highway driving right now.

Or handling a recently blown tire in the middle of the highway lane, or some other rare but not unheard of abnormal situation.

 

[EVNow]

Not that it happens often but getting a complete OEDR to handle that kinda stuff is the main thing preventing L4 highway driving right now.

Or handling a recently blown tire in the middle of the highway lane, or some other rare but not unheard of abnormal situation.

Tesla will not take liability for freeway driving until
- They are atleast 10x better than humans. Humans on highway are pretty good and the error rate is probably 1 in 100,000 miles or more. That means NOA has to be at 1 error is 1 Million miles. Given that it is probably 1 disengagement every hundred miles or so, now - they have a looong way to go.
- Taking on such liability should make financial sense. i.e. they should make more money selling NOA than they are likely to lose because of very high liability awards they are likely to be handed down in courts. Several million for every accident.

 

[drtimhill]

Building Voxels and using NN, instead of bag of points should help there. But even now FSD is fairly good at estimating where a stationary object is on the side of the road.

ps : I think internally Tesla probably has a good idea of where NOA would benefit from single stack. My basic point is - single stack won't help FSD Beta and it needs a lot more help than NOA.

As I noted earlier .. the benefit is freeing up NN resources used by the overlapping NoA stack to dedicate to FSD .. which are badly needed apparently.

 

[EVNow]

As I noted earlier .. the benefit is freeing up NN resources used by the overlapping NoA stack to dedicate to FSD .. which are badly needed apparently.

But is FSD stack and NOA stack running at the same time ? I don't think so .... so, there are no runtime resources saved by single stack.

 

[MP3Mike]

But is FSD stack and NOA stack running at the same time ? I don't think so .... so, there are no runtime resources saved by single stack.

Don't they have to run at the same time? You can't shut down one stack and load up another stack to do the hand-off in an instant. And you can't have it just drive blind for a few seconds while that happens.

 

[EVNow]

Don't they have to run at the same time? You can't shut down one stack and load up another stack to do the hand-off in an instant. And you can't have it just drive blind for a few seconds while that happens.

I don’t think so. I don’t think it takes that long to switch.

 

[drtimhill]

But is FSD stack and NOA stack running at the same time ? I don't think so .... so, there are no runtime resources saved by single stack.

Yes they are .. the car has to switch between them rapidly when moving from city to freeway driving .. there is no way they can switch one out, switch another one in and initialize it fast enough to seamlessly switch without losing control of the car. Remember that NNs aren’t code running in RAM that can just sit by idling when not needed.

 

[EVNow]

Yes they are .. the car has to switch between them rapidly when moving from city to freeway driving .. there is no way they can switch one out, switch another one in and initialize it fast enough to seamlessly switch without losing control of the car. Remember that NNs aren’t code running in RAM that can just sit by idling when not needed.

I’m actually not sure either way.

Long time back - I proposed exactly what you are saying and replies basically said they don’t run at the same time and they switch.

 

[drtimhill]

I’m actually not sure either way.

Long time back - I proposed exactly what you are saying and replies basically said they don’t run at the same time and they switch.

Of course, none of us outside of Tesla actually know. However, I dont see any way they could switch the NNs in real-time. The hand-over would be a nightmare and there would always be some period of time when the car was essentially running blind. At a minimum, the incoming stack needs to handle several frame cycles to build up a world view and compute velocity vectors. And that's after the old stack has been evicted and the new one loaded .. and initialized.

Of course, this is different for the more traditional C/C++ code, but there are still startup time issues even then. Code takes time to "warm up", doing setup, loading options/settings, cache warming, connecting to I/O channels. Doing this "live" while the car is movingly at 60mph would be suicide.