FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[kabin]

I don't think there are any shortage of bizarre v11.3.6 planner examples. Here's one where FSDb actually tried to cut across 5 or 6 lanes with a half dozen cars stopped at a red light. Gotta wonder what data set they used to train for this scenario. Or maybe this is FSDb ad-libbing (i.e. fruits of NN generalization)? If so they need bigger NNs etc.

 

[Alset2]

Sorry if this has been covered but are they releasing to new testers yet or is it still only existing cars?
I have to get new computer this weekend so I will lose FSDb. so I'm hoping the pause has been lifted??

thanks

 

[AlanSubie4Life]

but it takes way too long to go from 1mph to 0mph at stop signs... adds a lot of time. 11.3.4 was already pretty bad

Was already unusable in 11.3.4. Fortunately I have video of my test case so I can compare. I’ll check sometime. Need to wait for similar time of day etc. There is zero traffic. So that variable can be eliminated.

Gotta wonder what data set they used to train for this scenario. Or maybe this is FSDb ad-libbing (i.e. fruits of NN generalization)? If so they need bigger NNs etc.

They aren’t using NNs. Not that it would matter probably. Probably not going to figure out that if needs to check for traffic in the left turn lane, then turn right, before immediately turning back left before sidling up next to traffic waiting in line and cuddle with them until they let you in. Or just wait or whatever (since you have to turn right anyway).

 

[jebinc]

Have yet to drive on Junk 7.0

FSDb; aka, FSDj…

 

[LowlyOilBurner]

11.3.6:

First night drive this evening. Auto high beams are the worst I’ve seen yet on FSDb. Especially on the highway. You’d think the car was doing the Christmas light show going down the road. Constantly switching between hi/lo beam for no apparent reason. The only thing I can think is that it’s picking up on reflections from mile markers/road signs and causing this. It’s awful. 10.69.X and 11.3.2/3 they were actually working pretty good. This latest installment of slop makes it look like a kid is driving, hitting the hi/low stalk for fun.

Maybe I have bugs spattered across the forward looking cameras…I hope.

 

[Ramphex]

11.3.6:

First night drive this evening. Auto high beams are the worst I’ve seen yet on FSDb. Especially on the highway. You’d think the car was doing the Christmas light show going down the road. Constantly switching between hi/lo beam for no apparent reason. The only thing I can think is that it’s picking up on reflections from mile markers/road signs and causing this. It’s awful. 10.69.X and 11.3.2/3 they were actually working pretty good. This latest installment of slop makes it look like a kid is driving, hitting the hi/low stalk for fun.

Maybe I have bugs spattered across the forward looking cameras…I hope.

same issue, luckily Model S lets you turn it off, I don't believe Model Y does. I think Model Y forces auto-highs even on AP.

 

[Ramphex]

Looks like a ton of people moving over to the latest FSDj, wonder if they're going to stop the a/b testing.

 

[jebinc]

11.3.6:

First night drive this evening. Auto high beams are the worst I’ve seen yet on FSDb. Especially on the highway. You’d think the car was doing the Christmas light show going down the road. Constantly switching between hi/lo beam for no apparent reason. The only thing I can think is that it’s picking up on reflections from mile markers/road signs and causing this. It’s awful. 10.69.X and 11.3.2/3 they were actually working pretty good. This latest installment of slop makes it look like a kid is driving, hitting the hi/low stalk for fun.

Maybe I have bugs spattered across the forward looking cameras…I hope.

J

 

[thewishmaster]

Not sure how Tesla sources the velocity data (tire/motor sensors) but would bet the lag has always been there related to normal delays from data communications and filtering noisy data. I would hope the team didn't add a bogus delay giving more processing time when stopped or at intersections.

Yeah there’s definitely something artificial to it. The car slows down, and just when you think it’s about to stop, it takes another 2+ seconds to actually reach 0. I would almost rather prefer that it just braked harder and stopped with more confidence versus a gentle stop that takes several seconds longer than it should.

 

[aronth5]

11.3.6:

First night drive this evening. Auto high beams are the worst I’ve seen yet on FSDb. Especially on the highway. You’d think the car was doing the Christmas light show going down the road. Constantly switching between hi/lo beam for no apparent reason. The only thing I can think is that it’s picking up on reflections from mile markers/road signs and causing this. It’s awful. 10.69.X and 11.3.2/3 they were actually working pretty good. This latest installment of slop makes it look like a kid is driving, hitting the hi/low stalk for fun.

Maybe I have bugs spattered across the forward looking cameras…I hope.

Didn't you know auto high beams are only for dark secondary roads

 

[Scott Chapman]

I have not had that problem so far. My only recommendation would be:

1) Enter the Service Mode and force a reinstall of the existing firmware
2) Recalibrate your cameras (after cleaning them completely)
3) Set back up your settings

Wait, how do you force a reinstall of existing firmware? I went to the service tab in the car, and didn't see anything there...

 

[sandman1330]

I agree with @Julien W. Today I was on a long highway drive and a few times only wanted to change the value of the Minimal Lane Change setting. Every time I did that I had to toggle back to the Average setting since as soon as I clicked on the scroll wheel the setting gets changed. Since you cannot see your setting I've found on more than one occasion I have forgotten which setting I was on so a quick change could be problematic.

At least they fixed the problem I had on 11.3.3 where the scroll wheel would not change the chill/avg/assv setting. I could only do that from the main display. 11.3.6 fixed that for me.

At least you can even do that.

They STILL haven't even enabled the right scroll wheel for the Refresh Model S. The wheel itself controls speed, but the left/right click is non-functional. If we want to change any of those settings (including FSDb profile), we have to go all the way into the menu on the main screen!

 

[MarcG]

Yeah there’s definitely something artificial to it. The car slows down, and just when you think it’s about to stop, it takes another 2+ seconds to actually reach 0. I would almost rather prefer that it just braked harder and stopped with more confidence versus a gentle stop that takes several seconds longer than it should.

Mine actually does that on 11.3.6 and I hate it. It slows down “normally” from city street speeds (25-30mph), until about 10mph, then stops slowing down until just before the stop sign line and slams on the brakes at the last second.

 

[JulienW]

Tried out BMW emulation mode this morning. Worked OK but did make a major mistake and turn on the signal light at the end. As we all know BMW's don't have actual working signal lights.

 

[Jeff N]

Yeah there’s definitely something artificial to it. The car slows down, and just when you think it’s about to stop, it takes another 2+ seconds to actually reach 0. I would almost rather prefer that it just braked harder and stopped with more confidence versus a gentle stop that takes several seconds longer than it should.

And it still isn’t that gentle…

My experience is that FSD comes to a full stop and then often waits one or even two or more seconds before continuing.

Here’s my wish: in order to bring the car to a full stop (at a stop sign etc.), they should handoff to the one-pedal regen system which does a really excellent job of gentle braking. Then, use the final couple of seconds during the one-pedal braking to do whatever observations and calculations it now does when fully stopped. Then, as soon as the one-pedal system has brought the car to a full stop FSD should wait maybe 1/10 of a second and then continue on. This is how humans legally drive whereas FSD gives the appearance of coming to a stop and only then figuring out whether it is safe to go. Hide the latency.

 

[Dewg]

Wait, how do you force a reinstall of existing firmware? I went to the service tab in the car, and didn't see anything there...

 

[kabin]

They aren’t using NNs. Not that it would matter probably. Probably not going to figure out that if needs to check for traffic in the left turn lane, then turn right, before immediately turning back left before sidling up next to traffic waiting in line and cuddle with them until they let you in. Or just wait or whatever (since you have to turn right anyway).

For sure it's an unknown looking in from the outside. The neural planner is suppose to be trained on "human demonstration from the fleet" per 2022 AI Day. I can't say it's working well in 3.6. But you are right, the real time planner is more than neural. Its computationally intensive and very likely a main source for much lag, poor decision making, and bugs.

 

[tivoboy]

Wait, how do you force a reinstall of existing firmware? I went to the service tab in the car, and didn't see anything there...

You can call to Tesla and they can review your logs, etc. and force a re-install remotely. I had it done once before back in Nov 2021 at the beginning of this FSDb journey.

 

[JB47394]

I just did a test for the vanishing FSDb icon to see if I could spot any patterns.

The road I drove was a divided four lane road with a 45 mph speed limit. Grass median. Suburban setting. Multiple traffic lights. Light traffic. Midday with slight overcast, but shadows were visible. Pollen all over the car. I drove the road three times in each direction for an overall total of about 12 miles. The first trip was mixed lanes, the second was left lane only and the third was right lane only. I rarely had a lead car.

For starters, FSDb is never available when passing through a traffic light. It will disable earlier when driving faster through the intersection.

Beyond that, the only correlation that I could pick up on is that roadside clutter is the cause. If the car sees something close along the side of the road that it doesn't classify, then FSDb will not be available. This includes bushes, trees and signs that are not the sort that FSDb would care about. The icon was disabled for the right lane far more often than the left, and that correlates with lots of clutter on the right side of the roadway. One interesting case is that there is a series of no parking signs on the right side along one stretch. The car would consistently (though not perfectly) turn off the icon before reaching each such sign.

Speed limit signs and signs with school traffic lights didn't turn off the icon, and some others such as school crossing signs also didn't.

Guard rails didn't bother it, but a bridge with a three-foot-high concrete retaining wall right on the edge of the roadway did.

I didn't record my car's screen to see later if anything showed up for my claimed "clutter", but I don't recall ever seeing any grey blobs on the display.

This observation is consistent with what I was seeing on that two lane road through the woods where I asserted that the problem seemed to be lane width. It was through a forest, so there were trees very close to the road. The few times that the road widened, the trees and such were either absent or farther away. So the icon could show up.

In the above, "close" means "within about 15-20 feet of the road edge".

 

[thewishmaster]

Mine actually does that on 11.3.6 and I hate it. It slows down “normally” from city street speeds (25-30mph), until about 10mph, then stops slowing down until just before the stop sign line and slams on the brakes at the last second.

Ironically this is how it seemed to drive this morning... lots of last-minute braking