FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[old pilot]

Two days in a row now, with the same behavior; four lane road with a center turn lane, the car gets to a point where there is a dedicated right turn lane and puts on the right turn signal (you think its just bluffing because its not part of the route) and then it starts to turn right into the lane, then it realizes the mistake (not part of the route to turn) and it corrects back to the left (using the left turn signal) but instead of stopping in the right most lane it goes to the left most lane but the next turn is less than a mile away and its a right turn. so it puts on the right turn signal to get into the right (correct) lane for the right turn ahead. I make sure there were no other cars close when I allowed this to occur, I just wanted to see what it would do and now I and you know.

 

[uscbucsfan]

I noticed that 11.4.2 has max speed limits at certain weather conditions. Makes sense, I guess, but it was drizzling and it said max speed was 70 in a 70, then it started absolutely pouring and it increased to 75, then it cleared up and it went back to 70 for about 5 min until it went to 80...then it started pouring again and ....85 mph.

I had to use the pedal at 70 because cars were flying by me.

I have noticed that no matter how horrible the conditions, Beta won't disengage.

 

[sleepydoc]

Two days in a row now, with the same behavior; four lane road with a center turn lane, the car gets to a point where there is a dedicated right turn lane and puts on the right turn signal (you think its just bluffing because its not part of the route) and then it starts to turn right into the lane, then it realizes the mistake (not part of the route to turn) and it corrects back to the left (using the left turn signal) but instead of stopping in the right most lane it goes to the left most lane but the next turn is less than a mile away and its a right turn. so it puts on the right turn signal to get into the right (correct) lane for the right turn ahead. I make sure there were no other cars close when I allowed this to occur, I just wanted to see what it would do and now I and you know.

Yup - the annoying lane selection issues continue to persist. They’re slowly improving but it’s hard to understand why it’s such a difficult task to solve.

 

[rlsd]

Dumb question: How do you react to the posts with info, thump up, thump down emoticons? So far I only click the Like button?

 

[Ramphex]

User error - so you wanted to be doing something else at known problem sites. No wonder you have all kinds of issues with FSD.

Dawg, I was wiping my glasses, not grilling a burger, give me a break.

I’m fully capable of supervising the Junk at 20mph while wiping sunglasses. The question is if it’s supposed to be superhuman, why can’t it even perform at normal human level? Even my level? Am I superhuman?!?

 

[EVNow]

Dawg, I was wiping my glasses, not grilling a burger, give me a break.

I’m fully capable of supervising the Junk at 20mph while wiping sunglasses. The question is if it’s supposed to be superhuman, why can’t it even perform at normal human level? Even my level? Am I superhuman?!?

User error was just to confuse you ....

Seriously, when FSD is uncertain it will ask you to nudge the wheel and if you don't quickly throw the red hands. Wondering whether thats what happened.

In general low occlusion objects are difficult for NN and I remember sometime back FSD wouldn't even register things like gates.

 

[Ramphex]

User error was just to confuse you ....

Seriously, when FSD is uncertain it will ask you to nudge the wheel and if you don't quickly throw the red hands. Wondering whether thats what happened.

In general low occlusion objects are difficult for NN and I remember sometime back FSD wouldn't even register things like gates.

There was no warning, but also no strike. Just red take over immediately hands…. and four ways lol. This never happened before 11.4

I’ll try to recreate it in a couple mins when I get back home. The Junk hates me using my phone to record its behavior, let’s see if I get another strike

 

[D Good]

All these posts about 11.4.2 are making me rlly angry. Why is this rollout so slow? It’s never been this slow before.

 

[uscbucsfan]

All these posts about 11.4.2 are making me rlly angry. Why is this rollout so slow? It’s never been this slow before.

You are really angry at a relatively minor update to your car that almost 50% of other users haven't received?

Sounds like you have an easy life. Congrats.

 

[EVNow]

All these posts about 11.4.2 are making me rlly angry. Why is this rollout so slow? It’s never been this slow before.

There have been slow rollouts before. I won't be surprised if they are finding lots of issues being reported & disengagement rate is up or something similar.

Personally not seeing any major improvements over 11.3.x. The turns have got progressively better (and slower) ... really the only improvement worth noting. Definitely seems to be more lane change issues.

 

[uscbucsfan]

There have been slow rollouts before. I won't be surprised if they are finding lots of issues being reported & disengagement rate is up or something similar.

Personally not seeing any major improvements over 11.3.x. The turns have got progressively better (and slower) ... really the only improvement worth noting. Definitely seems to be more lane change issues.

Right. "Basically v12", maybe there's substantial backend improvements, but it's a minor release for testers as you would expect going from 11.3 to 11.4. There are things that are better and things that are worse.

For me taking any left with a car at a stop sign makes the car freak out...it won't go. Really odd behavior, like the car doesn't think it can make it or that the car at the stop sign has the right of way.

 

[Ramphex]

The Junk got me good today. As I was driving slowly through my community, I figured I’ll allow it to drive for a bit while I use a microfiber cloth to clean my sunglasses. So as we pull up to the gate where it “brakes aggressively” and stops like 20 feet before the sensor to trigger the gate to open, it does the usual spill your coffee braking and stops.

Not a problem, I do what I always do and give it a bit of accelerator to move up to the sensor. 11.3 handled that fine, it moved up to the sensor and waited for the gate to open and then proceeded through. NOT TODAY JUNIOR. The Junk threw up red take over hands, turned on hazards, and a message came up saying autosteer is no longer available due to a system error.

I wish the car that was behind me was a Tesla, I would ask for a recording to see how this debacle looked like from the outside view

There was no warning, but also no strike. Just red take over immediately hands…. and four ways lol. This never happened before 11.4

I’ll try to recreate it in a couple mins when I get back home. The Junk hates me using my phone to record its behavior, let’s see if I get another strike

Video Jun 01 2023, 11 50 19.mov

Shared with Dropbox

www.dropbox.com

This red hands behavior did not start until Junk v11.4.x, and I drive through this gate every single day.

 

[Olle]

In the circumstance you describe, I choose to do my nudge with the speed scroll, rather than the accelerator.

That's my typical way as well, but since most drivers speed up to block you from passing, the scroll wheel can be too laggy to speed past, which is why I sometimes nudge the pedal.

I've been perplexed by FSD beta's tendency to park in the left lane in other drivers' blind spots on an otherwise empty highway. Starting to understand what's going on as I read these posts. It's atypical and confusing for humans to be passed at a constant speed. The process can be so slow that the blocking driver on the right has time to forget that you are there once they sped you into their blind spot. Later, once they gradually slowed down to their preferred speed, they see you again, speed up, you disappear and they forget. This may be what perpetuates the situation for miles on end.

Definitely an algorithm that needs to improve.

 

[uscbucsfan]

Video Jun 01 2023, 11 50 19.mov

Shared with Dropbox

www.dropbox.com

This red hands behavior did not start until Junk v11.4.x, and I drive through this gate every single day.

Give your version and warning to the really mad guy.

Sharing is caring.

 

[EVNow]

Video Jun 01 2023, 11 50 19.mov

Shared with Dropbox

www.dropbox.com

This red hands behavior did not start until Junk v11.4.x, and I drive through this gate every single day.

Looks like it saw something and wanted to immediately stop.

ps : Ironically first time I tried to play the video dropbox crashed my browser

 

[Ramphex]

Give your version and warning to the really mad guy.

Sharing is caring.

He’s not mad, he’s angry. Cows get mad, D Good gets angry

 

[uscbucsfan]

Looks like it saw something and wanted to immediately stop.

View attachment 942989

I've notice with a curve and obstacle, FSD thinks for sure it's going to hit it. Sometimes it's for an instant and sometimes it will fully dump.

 

[Ramphex]

Looks like it saw something and wanted to immediately stop.

ps : Ironically first time I tried to play the video dropbox crashed my browser

View attachment 942989

Yea and it used to stop, I would nudge it forward to the sensor with the accelerator and once the gate opened it would continue through. Not anymore.

 

[uscbucsfan]

He’s not mad, he’s angry. Cows get mad, D Good gets angry

Sometimes I like to go to the Investment thread and say that Robotaxi on existing cars is a fairytale, then never go back. I just count the thumbs down and laughing faces I get and come back here to talk about how my car is now scared to make left turns.

Maybe that would make D Good happy.

 

[EVNow]

Yea and it used to stop, I would nudge it forward to the sensor with the accelerator and once the gate opened it would continue through. Not anymore.

But why is it stopping there .... I think it can't figure out the distance to the gate reliably / correctly. If FSD thinks it is about to hit the gate, it will throw red hands. Something has changed in this release that either makes FSD throw red hands sooner or its calculation of distance less reliable.