FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[bhakan]

This version became available on the 29th based on TeslaFI information.

If I remember correctly, it was installed to HW4 cars I thought when it started installing on Dec 29th. That's why I was surprised.

 

[kabin]

We talk of self driving cars, these kids show up on lawn tractors with atv tires hammering the trails behind my house. Maybe we have it all wrong?: and what’s worse? I’m complaining I don’t have heat, ac and power windows in my $28k side by side, but, that Polaris Xpedition Northstar is just a click away…I am getting older I guess? Sigh.

Good for them. They look like 1970's Sears catalog yard tractors.

 

[LowlyOilBurner]

Good for them. They look like 1970's Sears catalog yard tractors.

Yep, they are, they put Honda GX 212 motors in them, changed gearing, they moved out pretty well!

 

[TK211X]

Rumor has it V12 will start going out around the same time the new episodes of Bluey hit Disney+

 

[yerEVan]

Rumor has it V12 will start going out around the same time the new episodes of Bluey hit Disney+

Bluey is kinda shaped like a cybertruck so that makes sense.

 

[kabin]

I had another episode of 11.4.9 FSD stopping for no reason. This was on a multi lane 35mph roadway with median separation, clear weather, no traffic, no intersection, and no signage. FSD was displaying nothing on the UI to indicate a stop sign or traffic light - just an end of the blue path where it stopped. I let it play out. After 2 or 3 secs the UI displayed 'creeping for visibility' and then it accelerated back to the speed limit.

 

[yerEVan]

11.4.9 although not amazing, seems to be a solid improvement from prior, recent versions.

 

[Twiglett]

Just rewatched that 12.1 video. Not holding out to much hope of that getting to us for a while.
It's still doing the basic nonsense of moving into the wrong lane before it needs to turn. You can see it showing .1 mile to turn left, so it moves into the right lane.
Or the classic right at the start, sitting there doing nothing with no traffic waiting to turn left, but nothing stopping it with no onscreen messages.
The steering wheel did look much smoother though, much less jerkiness etc.
If you didn't know it was FSD, you'd just think it was a learner driver who was being directed by someone who didn't know where they were going or was drunk.

 

[powertoold]

It's still doing the basic nonsense of moving into the wrong lane before it needs to turn. You can see it showing .1 mile to turn left, so it moves into the right lane.

Nah, not what you think, driver manually changed to the right to test it

12.1 looks very impressive so far, designed to limit liability

There's no real point of picking it apart right now because it hasn't been released to OGs, but based on what we've seen so far, it's working very well on non-rainy days

 

[JulienW]

Been having ALL improved drives since the holiday update but guess it had to end and boy did it. Yesterday went out in the burbs (>40 miles). The drive out was mostly OK and was also getting minimal nags (love this) with one bad move. Oddly Nav was correct and on a HOV divide FSD tried to go to the left when it should have went to the right. It also couldn't figure it out and ping ponged at 75MPH between the 2 and crossing the dividing lines before seeming to choose wrong. Had to snatch it to the right.

Then the pile of crap drive home and it was like I had a drunk 10 year old driving the entire way. For some reason it couldn't change lanes most of the time and would just drive with signal on and not move. In wrong turning lanes, phantom braking out the a$$, random signals and all manor of obtuse driving. It was a terrible experience with sooooooo many disengagements and oddly back to so many nags. It was like I had reverted to 10.2.

The final straw was coming into midtown on I85 south in the HOV lane and it waits until the last mile and HOV "exit" to start getting over. Stupid that it treats starting to merge over the same by waiting to the last mile whether on a rural 2 lane of an urban congested multi lane. While that would have been OK yesterday (holiday) it is not a good option under normal traffic since you only have 1 mile to cross 5 or 6 lanes that may be creeping bumper to bumper. I let it play to see it and when it got to the "exit" dash line it turned on the right signal......but would not change lanes. I let it ride for over ½ a mile (looking like a granny) but it was stuck. So I had to intervene and aggressively move over.

Very discouraging >40 miles.

 

[sleepydoc]

I had another episode of 11.4.9 FSD stopping for no reason. This was on a multi lane 35mph roadway with median separation, clear weather, no traffic, no intersection, and no signage. FSD was displaying nothing on the UI to indicate a stop sign or traffic light - just an end of the blue path where it stopped. I let it play out. After 2 or 3 secs the UI displayed 'creeping for visibility' and then it accelerated back to the speed limit.

Is it repeatable? There’s a spot on a road near me where FSD does the exact same thing. It was recently reconstructed and changed a bit so I don’t know if that’s the issue.

 

[jebinc]

Been having ALL improved drives since the holiday update but guess it had to end and boy did it. Yesterday went out in the burbs (>40 miles). The drive out was mostly OK and was also getting minimal nags (love this) with one bad move. Oddly Nav was correct and on a HOV divide FSD tried to go to the left when it should have went to the right. It also couldn't figure it out and ping ponged at 75MPH between the 2 and crossing the dividing lines before seeming to choose wrong. Had to snatch it to the right.

Then the pile of crap drive home and it was like I had a drunk 10 year old driving the entire way. For some reason it couldn't change lanes most of the time and would just drive with signal on and not move. In wrong turning lanes, phantom braking out the a$$, random signals and all manor of obtuse driving. It was a terrible experience with sooooooo many disengagements and oddly back to so many nags. It was like I had reverted to 10.2.

The final straw was coming into midtown on I85 south in the HOV lane and it waits until the last mile and HOV "exit" to start getting over. Stupid that it treats starting to merge over the same by waiting to the last mile whether on a rural 2 lane of an urban congested multi lane. While that would have been OK yesterday (holiday) it is not a good option under normal traffic since you only have 1 mile to cross 5 or 6 lanes that may be creeping bumper to bumper. I let it play to see it and when it got to the "exit" dash line it turned on the right signal......but would not change lanes. I let it ride for over ½ a mile (looking like a granny) but it was stuck. So I had to intervene and aggressively move over.

Very discouraging >40 miles.

Wow….

 

[kabin]

Is it repeatable? There’s a spot on a road near me where FSD does the exact same thing. It was recently reconstructed and changed a bit so I don’t know if that’s the issue.

Good question. I'll check. It's a stretch of road I don't drive often but there's no recent construction or over/underpasses.

 

[kabin]

Just rewatched that 12.1 video. Not holding out to much hope of that getting to us for a while.
It's still doing the basic nonsense of moving into the wrong lane before it needs to turn. You can see it showing .1 mile to turn left, so it moves into the right lane.
Or the classic right at the start, sitting there doing nothing with no traffic waiting to turn left, but nothing stopping it with no onscreen messages.
The steering wheel did look much smoother though, much less jerkiness etc.
If you didn't know it was FSD, you'd just think it was a learner driver who was being directed by someone who didn't know where they were going or was drunk.

Seems like the planning portion is struggling. It's gotta be a tricky business training each NN section and then knowing where and how to integrate all NNs for end to end training/debug.

And I don't see a reason to keep the 3-5 sec delay/creep at stop signs even with NHTSA requirements. Anything more than one second with no traffic probably indicates the same old systemic issues.

 

[fasteddie7]

No, I usually have a hand or hands "touching" the wheel, just not applying ANY torque unless nagged.

I think I’m going to try taking a cabin recording of me holding the wheel eyes forward and use service mode to place the phone on a suction cup lined up in front of the internal camera and place the clip on 10 second loop and see if it nags for fun.

 

[Mardak]

Had an interesting drive with 11.4.9 where lane lines and road edges showed up fine, but FSD Beta couldn't be activated because it complained about the front camera being blocked. The visualization correctly showed white vs yellow dashed vs double yellow lines just fine but no other vehicles were rendered, so this seems to indicate the various neural networks have different thresholds of what's visible enough.

After arriving, I previewed the camera views from the service menu, and they were actually mostly fine except main camera had a blurry section directly in the middle roughly 15% of the whole view. That was even more surprising when looking from the outside as it would seem to be mostly blocked:

Perhaps the lanes network relies more on the wide camera view and not so much the main camera while vehicle detection wants both (HW4) front cameras to be clear?

 

[WilliamG]

Had an interesting drive with 11.4.9 where lane lines and road edges showed up fine, but FSD Beta couldn't be activated because it complained about the front camera being blocked. The visualization correctly showed white vs yellow dashed vs double yellow lines just fine but no other vehicles were rendered, so this seems to indicate the various neural networks have different thresholds of what's visible enough.

After arriving, I previewed the camera views from the service menu, and they were actually mostly fine except main camera had a blurry section directly in the middle roughly 15% of the whole view. That was even more surprising when looking from the outside as it would seem to be mostly blocked:
View attachment 1005067

Perhaps the lanes network relies more on the wide camera view and not so much the main camera while vehicle detection wants both (HW4) front cameras to be clear?

Tesla messed up with the new defrost lines. There’s not enough coverage, either, when they do work, leaving the center blurry. Highland owners are complaining about it, too.

 

[Porkalletta]

The opposite of FSD.

 

[FSDtester#1]

Is it repeatable? There’s a spot on a road near me where FSD does the exact same thing. It was recently reconstructed and changed a bit so I don’t know if that’s the issue.

I have been having my best drives ever. However out of nowhere, while sitting as front car at a red light, the car just slowly started moving forward. I let it go a couple of feet, since there weren't any cars coming from either direction. My first red light runner in months. Just an FYI to be alert at red lights when you are in pole position. It has never ran this light before.

 

[WilliamG]

I have been having my best drives ever. However out of nowhere, while sitting as front car at a red light, the car just slowly started moving forward. I let it go a couple of feet, since there weren't any cars coming from either direction. My first red light runner in months. Just an FYI to be alert at red lights when you are in pole position. It has never ran this light before.

Definitely had my car run a red light. Even recorded it both times via dashcam. Definitely isn’t the most fun feeling, let me tell you.