FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[RowdyMY]

So, did 11.4.7 die as well? Zero on Teslafi.

Just showed up today. 33 still pending after 90 installs but so far too early to call over way or the other. Bigger roll out than 11.4.6.

 

[Tronguy]

All right, you guys.

I just finished watching a 45-minute live stream of Musk driving around with WholeMarsCatalog in a V12.x-equipped Tesla. This popped up on the Investors's Forum, here's the link to one of the posts.

The whole thing was streamed with what looks like WholeMarsCatalog's cell phone, so it's a little shaky. In the 45 minute drive, there was one intervention.

Now comes the crazy stuff. In no particular order:

• Musk said, repetitively, that they didn't tell the NNET what things were. That stop sign? Nope. The traffic circle? Nope. Other cars? Nope. How to drive smoothly? Nope. Tesla just showed it videos of how good drivers drive and told it to do that.
• Musk stated that there's 300k+ lines of C++ code in V11.x that do tell the car what to do in traffic circles, when cyclists come by, how to handle intersections, how to handle left and right turns, and all that jazz. That code is gone.
• The Hardware load on HW3.0 that he was driving allowed for 50 frames per second of driving NN work. They had to limit it to 36 frames per second because that's the limit of how fast the cameras can run. He stated that HW3 is more than capable of Doing The Job. And, in fact, wasn't working as hard as it does in V11.x.
• His words: It's NNET, all the time.

Guys... From time to time I've made mumbly noises around the FSD-b threads that there's been gradual improvement, but thought that, with software, breakthroughs can happen that take the idea that each version is 1.2X better than the previous (leading to infinite numbers of 1.2X better before we get to 9 9's better than humans) and toss those ideas out the window. If what I just saw tonight is real, and I have no reason to believe that it's not, at least one major breakthrough has happened.

I have no idea if this is leading to robotaxis. But it sure looks like a step in that direction.

Couple of minor points.

Musk says that 12.x isn't ready yet, it's still making mistakes from time to time. OK, I get it: It's a massive re-write, so what else is new.

Musk discussed that infamous, "Must come to a complete stop at a stop sign" fight with the Regulators. He said that real, live data, from real, live drivers showed that humans only came to a complete stop 0.2% of the time. Even when a human thought that they had stopped.. they hadn't. They pointed this out to the regulators: They demanded the complete stop. So, now you know who to blame. (Some of us had thought that around here, but here it is from somebody who was actually there.)

I'm in shock. That was one heck of a drive.

 

[Buckminster]

All right, you guys.

I just finished watching a 45-minute live stream of Musk driving around with WholeMarsCatalog in a V12.x-equipped Tesla. This popped up on the Investors's Forum, here's the link to one of the posts.

The whole thing was streamed with what looks like WholeMarsCatalog's cell phone, so it's a little shaky. In the 45 minute drive, there was one intervention.

Now comes the crazy stuff. In no particular order:

• Musk said, repetitively, that they didn't tell the NNET what things were. That stop sign? Nope. The traffic circle? Nope. Other cars? Nope. How to drive smoothly? Nope. Tesla just showed it videos of how good drivers drive and told it to do that.
• Musk stated that there's 300k+ lines of C++ code in V11.x that do tell the car what to do in traffic circles, when cyclists come by, how to handle intersections, how to handle left and right turns, and all that jazz. That code is gone.
• The Hardware load on HW3.0 that he was driving allowed for 50 frames per second of driving NN work. They had to limit it to 36 frames per second because that's the limit of how fast the cameras can run. He stated that HW3 is more than capable of Doing The Job. And, in fact, wasn't working as hard as it does in V11.x.
• His words: It's NNET, all the time.

Guys... From time to time I've made mumbly noises around the FSD-b threads that there's been gradual improvement, but thought that, with software, breakthroughs can happen that take the idea that each version is 1.2X better than the previous (leading to infinite numbers of 1.2X better before we get to 9 9's better than humans) and toss those ideas out the window. If what I just saw tonight is real, and I have no reason to believe that it's not, at least one major breakthrough has happened.

I have no idea if this is leading to robotaxis. But it sure looks like a step in that direction.

Couple of minor points.

Musk says that 12.x isn't ready yet, it's still making mistakes from time to time. OK, I get it: It's a massive re-write, so what else is new.

Musk discussed that infamous, "Must come to a complete stop at a stop sign" fight with the Regulators. He said that real, live data, from real, live drivers showed that humans only came to a complete stop 0.2% of the time. Even when a human thought that they had stopped.. they hadn't. They pointed this out to the regulators: They demanded the complete stop. So, now you know who to blame. (Some of us had thought that around here, but here it is from somebody who was actually there.)

I'm in shock. That was one heck of a drive.

Copied post here:
FSD V12 (end to end AI)

 

[amadeusneo]

That doesn't help you get it faster.

It actually can. Has worked for me a few times. I’ve almost never had it “pushed” to my car (meaning the app notifies me I have a new software to install).

 

[Buckminster]

https://twitter.com/x/status/1695178555875786811

 

[aronth5]

https://twitter.com/x/status/1695178555875786811

Sun vs shade accounts for a lot of the differences. Not particularly helpful because of that.

 

[JulienW]

Sun vs shade accounts for a lot of the differences. Not particularly helpful because of that.

Plus you could add a million other variables like time of year and time of day. BUT most important this is an image processed and produced for human consumption and doesn't necessarily directly correlate to the computer analyzing pixel information.

 

[Buckminster]

https://twitter.com/x/status/1695412161583915238

 

[fasteddie7]

I posted in another thread, the disengagement from last nights v12 livestream happens to me currently at this intersection, the cross traffic light is recognized as a light for my direction, and when it turns green for them I scare the crap out of them since my car darts before I hit the brake as they start to go.

 

[jebinc]

Huh? It shows 92 installs with another 33 pending.

View attachment 968317

My bad. 2023.7.30 is so old it wasn’t showing on the first page summary of releases.

 

[kpanda17]

It’s a great morning
2023.26.10/11.4.7 is currently loading to my MYP
I’ll do FSD this am
Big jump from 26.1
My M3 just went up to 26.9 last night

Funny they were recently both on 26.1 and then the M3 did a bunch of step upgrades to 26.9
The MYP did one step

 

[JulienW]

It’s a great morning
2023.26.10/11.4.7 is currently loading to my MYP
I’ll do FSD this am
Big jump from 26.1
My M3 just went up to 26.9 last night

Funny they were recently both on 26.1 and then the M3 did a bunch of step upgrades to 26.9
The MYP did one step

Say what?......that makes you unique since 23.26.10 comes with 11.4.4.

 

[kpanda17]

Excuse me......that makes you unique since 23.26.10 comes with 11.4.4.

You are right as when the UG was done saw 11.4.4
But will state:
FSDb is way more confident than 11.3.6
But def not perfect, why we need v12
Wipers are much better, no issue at all
Interesting there is I II III IIII and AUTO now
And auto is very intelligent watching it almost stop with light mist

Love OTA updates

Btw, wearing a hat did not help with the nag
Only rhe glasses do

 

[JulienW]

You are right as when the UG was done saw 11.4.4
But will state:
FSDb is way more confident than 11.3.6
But def not perfect, why we need v12
Wipers are much better, no issue at all
Interesting there is I II III IIII and AUTO now
And auto is very intelligent watching it almost stop with light mist

Love OTA updates

Btw, wearing a hat did not help with the nag
Only rhe glasses do

Is your Y HW4? If so you could be one of the first to get 23.26.10/11.4.4 Beta.

 

[kriskros91]

Is your Y HW4? If so you could be one of the first to get 23.26.10/11.4.4 Beta.

Doubt it. According to teslafi there are no AP4 cars on this version.

 

[Gwgan]

If everyone stops using 11… will they be more likely to deliver 12 sooner

I don’t see any value to testing or improving 11 at this point

 

[archae86]

All right, you guys.

I just finished watching a 45-minute live stream of Musk driving around with WholeMarsCatalog in a V12.x-equipped Tesla. This popped up on the Investors's Forum, here's the link to one of the posts.

...

I'm in shock. That was one heck of a drive.

Comments on that live-stream:

Although early on it appeared that the car briefly was using a lane that was properly a reserved turn lane for the opposite direction, aside from the very bad traffic light misbehavior Elon intervened for the driving looked pretty good.

The verbal comments by Elon and someone seemingly senior in FSD development on training were interesting. There was strong emphasis on the importance of providing GOOD training examples to the training process. Also they said that a way (perhaps THE way) to modify behaviors not desired was to selectively provide training data that would push behavior in the desired direction (for example, actually stopping for stop signs in order to satisfy regulators, even though humans don't).

Regarding sign reading, they appeared to say that just as they never taught the system with C++ code what a bicyclist was, but through training the system learned to behave properly in the presence of a bicycle, they expect quite broadly the system to give proper behavior in the presence of signs.

Now that is fascinating, though I suspect getting anywhere near all the way on the school zone problem will be daunting to solve this way. Imagine having to provide training examples where the drivers are PROPERLY responding to the bewildering variety of school zone situations. Still, I have hope that a considerable majority of real school zone stuff might be well-handled this way, whereas my current release of FSDb pretty much gets all school zones wrong.

Right turn on red varieties, in contrast, I actually think ought to be pretty solvable this way.

Good luck to them. I think the claim this really is different is probably well-founded. I'm less optimistic as to how many 9s they can hope to get this way.

 

[Gwgan]

Pretty sure the guy in the pasenger seat was Ashok

 

[Buckminster]

If everyone stops using 11… will they be more likely to deliver 12 sooner

I don’t see any value to testing or improving 11 at this point

Exactly, I made this point 2 weeks ago. 12 could exceed 11 in one or two weeks. Why bother splitting the team?

 

[JulienW]

If everyone stops using 11… will they be more likely to deliver 12 sooner

I don’t see any value to testing or improving 11 at this point

Elon started promoting V11 a year before it was released. He even says V12 is not ready. So I wouldn't expect it to be before first of next year at best. Still LOTS of V11 to go.