FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[EVNow]

Of course, none of us outside of Tesla actually know. However, I dont see any way they could switch the NNs in real-time. The hand-over would be a nightmare and there would always be some period of time when the car was essentially running blind. At a minimum, the incoming stack needs to handle several frame cycles to build up a world view and compute velocity vectors. And that's after the old stack has been evicted and the new one loaded .. and initialized.

Of course, this is different for the more traditional C/C++ code, but there are still startup time issues even then. Code takes time to "warm up", doing setup, loading options/settings, cache warming, connecting to I/O channels. Doing this "live" while the car is movingly at 60mph would be suicide.

We should ask James or @verygreen … they may know.

 

[verygreen]

We should ask James or @verygreen … they may know.

they run multiple NNs in parallel, often with duplicate/overlapping functionality, so there's a seamless transition to highway mode from citystreets mode and back for example.
In addition a few releases back they started to ship two sets of NNs in the firmware that could be roughly named "prod" and "experimental", though initially they were identical (inputs/outputs are always identical for obvious reasons too). They should be able to switch between the two sets pretty easily based on triggers or some other signals from the mothership.

There's no way to easily put in new NNs OTA outside of firmware updates

 

[EVNow]

However, I dont see any way they could switch the NNs in real-time.

Of course, none of us outside of Tesla actually know. However, I dont see any way they could switch the NNs in real-time. The hand-over would be a nightmare and there would always be some period of time when the car was essentially running blind. At a minimum, the incoming stack needs to handle several frame cycles to build up a world view and compute velocity vectors. And that's after the old stack has been evicted and the new one loaded .. and initialized.

Of course, this is different for the more traditional C/C++ code, but there are still startup time issues even then. Code takes time to "warm up", doing setup, loading options/settings, cache warming, connecting to I/O channels. Doing this "live" while the car is movingly at 60mph would be suicide.

So - they will benefit from having a single stack.

But if they have not only have highway/city stacks running but also a parallel stack - that is not much of a constraint as such ? Though, would be obviously better in the long run.

 

[sleepydoc]

In addition a few releases back they started to ship two sets of NNs in the firmware that could be roughly named "prod" and "experimental", though initially they were identical (inputs/outputs are always identical for obvious reasons too). They should be able to switch between the two sets pretty easily based on triggers or some other signals from the mothership.

Interesting. This may explain why I’ve seen some behavior changes despite being on the same firmware.

 

[verygreen]

Interesting. This may explain why I’ve seen some behavior changes despite being on the same firmware.

there are multiple factors, maps influence things. Additionally there's some local memory of things the car have seen before, so e.g. if you see some strange behavior at deadend and try to repeapt it and car behaves much better - this is often why.

 

[momo3605]

there are multiple factors, maps influence things. Additionally there's some local memory of things the car have seen before, so e.g. if you see some strange behavior at deadend and try to repeapt it and car behaves much better - this is often why.

Source? I have not noticed this. Where the car fails, It fails every time. I don’t believe there is any memory

 

[sleepydoc]

Source? I have not noticed this. Where the car fails, It fails every time. I don’t believe there is any memory

Yeah, I haven't seen any evidence of this. People talk about the neural net and learning but the car/FSB pretty reliably makes the same mistakes over and over again. The one thing that seems to matter is if there's another car in front of you since it will frequently preferentially follow it.

 

[drtimhill]

Source? I have not noticed this. Where the car fails, It fails every time. I don’t believe there is any memory

There have been several YouTube videos that have caught this .. one specific one in SF had a tester driving around the block a few times with the car have trouble getting the lanes right (there was road construction). On the 3rd and subsequent times it corrected itself before reaching the blockage .. leading to speculation that the car did track some small memory of local map corrections/anomolies.

 

[GlmnAlyAirCar]

There have been several YouTube videos that have caught this .. one specific one in SF had a tester driving around the block a few times with the car have trouble getting the lanes right (there was road construction). On the 3rd and subsequent times it corrected itself before reaching the blockage .. leading to speculation that the car did track some small memory of local map corrections/anomolies.

Right but it’s just speculation. My (probably equally invalid) speculation is that these changes in behavior are due to map corrections rather than changes to FSD code itself. If mapped data disagrees with what the car perceives, this difference is sent to mothership where a map patch is created and distributed. This could explain why regular full map updates are so infrequent, as these would likely invalidate any of the patches. It would also explain why, at least for me, things were so much worse after receiving the latest map update, but have slowly gotten better over the last few months.

But, again, this is just speculation.

 

[jabloomf1230]

My speculation is different. The difference between identical runs is due to the error term in the NN model. Lighting, level of traffic and other factors contribute to the variability

 

[Knightshade]

My speculation is different. The difference between identical runs is due to the error term in the NN model. Lighting, level of traffic and other factors contribute to the variability

Yup... I remember much earlier in the beta someone had a video where they were SURE their car had "learned" how to handle a u-turn, and when you actually looked close enough at the video you realized the initial failed runs were like 50% higher speed than all the later successful ones.


"This try was exactly the same!"

naah...humans are terrible at calibration and measuring.

 

[Supcom]

If the cars learned from their mistakes, Chuck Cook's car would make perfect left turns by now.

 

[drtimhill]

Right but it’s just speculation. My (probably equally invalid) speculation is that these changes in behavior are due to map corrections rather than changes to FSD code itself. If mapped data disagrees with what the car perceives, this difference is sent to mothership where a map patch is created and distributed. This could explain why regular full map updates are so infrequent, as these would likely invalidate any of the patches. It would also explain why, at least for me, things were so much worse after receiving the latest map update, but have slowly gotten better over the last few months.

But, again, this is just speculation.

Agreed, but @verygreen who seems to have some actual knowledge of the stack, mentioned that the car can do this. I've no way to confirm this of course, and the example I cited may have been other factors (e.g. getting a better GPS signal on the later attempts).

However, you should note that these earlier/later attempts were all on the same test drive, where the driver was doing a loop around several blocks and re-trying the same path of road again and again. So code/map patches from the mothership are highly unlikely (and, since it was a diversion around roadworks, would be stale the next day anyway).

 

[boonedocks]

If the car had any "memory" of a location or failure it would be widely discussed. It does not. I have submitted the "video" clip of two locations near my house where FSD Beta will fail at every time. It can't remember them within 10 minutes much less over time.

 

[GlmnAlyAirCar]

Agreed, but @verygreen who seems to have some actual knowledge of the stack, mentioned that the car can do this. I've no way to confirm this of course, and the example I cited may have been other factors (e.g. getting a better GPS signal on the later attempts).

However, you should note that these earlier/later attempts were all on the same test drive, where the driver was doing a loop around several blocks and re-trying the same path of road again and again. So code/map patches from the mothership are highly unlikely (and, since it was a diversion around roadworks, would be stale the next day anyway).

And how well did it perform the next day? Or the next week? Or month? Humans are very good at finding apparent patterns in random data. There is no reason to believe, based on the statistically insignificant sample set, that the changes in behavior are from “learning.” Rather, they are just variations based on slightly differing initial conditions.

 

[boonedocks]

And how well did it perform the next day? Or the next week? Or month? Humans are very good at finding apparent patterns in random data. There is no reason to believe, based on the statistically insignificant sample set, that the changes in behavior are from “learning.” Rather, they are just variations based on slightly differing initial conditions.

Indeed we humans are great at seeing things in random data. Perception to some is a reality they can not escape from.

Pareidolia

 

[EVNow]

Yeah, I haven't seen any evidence of this. People talk about the neural net and learning but the car/FSB pretty reliably makes the same mistakes over and over again. The one thing that seems to matter is if there's another car in front of you since it will frequently preferentially follow it.

As they say - absence of proof is not proof of absence

 

[sleepydoc]

As they say - absence of proof is not proof of absence

True, but if it does learn it doesn’t learn well!

 

[boonedocks]

True, but if it does learn it doesn’t learn well!

It doesn't learn. But that was funny

 

[EVNow]

True, but if it does learn it doesn’t learn well!

Moving goalposts !