FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Frunkenstein]

Absolutely hate FSD on the highway mainly around larger trucks. It pulls so hard away in the lane it freaks drivers in the opposite lane out that I’m creeping into them. And going between two trucks, it’s like being in a game of pong. This is on 11.3.6 (2022.45.15).

It almost feels like it does complex areas better than the basics. When I report via the “what happenduhhhhh” prompt, it’s always simple responses like “car was trying to make a left from the U turn lane” or “car was changing lanes away from my exit…at the exit”. I wish my experiences were like some of the “no takeover” videos. Right now it looks like I’m a terrible driver and I hate that feeling.

 

[kabin]

But it isn't just an issue on narrow unmarked roads. Yesterday I was driving in a different area than normal, and things were going fine, then the road I was on, which had very clear lanes lines, narrowed by a foot, or two, and suddenly the car was a scaredy cat, and slammed on the brakes as every car from the opposing direction got near, even though they were clearly fully in their own lane.

So it is interesting how just a foot or two totally changes how FSDb works.

Yep, the facts show it's not an edge case. There are youtube videos demonstrating the issue on normal roadway types some even with double solid yellow lines in the roadway center.

 

[EVNow]

But it isn't just an issue on narrow unmarked roads. Yesterday I was driving in a different area than normal, and things were going fine, then the road I was on, which had very clear lanes lines, narrowed by a foot, or two, and suddenly the car was a scaredy cat, and slammed on the brakes as every car from the opposing direction got near, even though they were clearly fully in their own lane.

So it is interesting how just a foot or two totally changes how FSDb works.

Is this on 11.4.1 ?

 

[EVNow]

Yep, the facts show it's not an edge case. There are youtube videos demonstrating the issue on normal roadway types some even with double solid yellow lines in the roadway center.

The facts are - its an edge case. Otherwise we'd all experience it often. If you have to go to YT to see it ...

ps : What are not an edge cases - school zone, emergency vehicles, school bus ... all of which FSD does not handle yet.

 

[Knightshade]

Absolutely hate FSD on the highway mainly around larger trucks. It pulls so hard away in the lane it freaks drivers in the opposite lane out that I’m creeping into them.

What's funny is, on the old legacy highway code everyone complained about it NOT moving over in the lane when a big truck was next to them.

So now it does, and nobody likes THAT either


BTW interesting thing- Ford recently just updated their blues clues ADAS to ALSO move over in a lane if next to a big truck.

 

[Yelobird]

Absolutely hate FSD on the highway mainly around larger trucks. It pulls so hard away in the lane it freaks drivers in the opposite lane out that I’m creeping into them. And going between two trucks, it’s like being in a game of pong. This is on 11.3.6 (2022.45.15).

It almost feels like it does complex areas better than the basics. When I report via the “what happenduhhhhh” prompt, it’s always simple responses like “car was trying to make a left from the U turn lane” or “car was changing lanes away from my exit…at the exit”. I wish my experiences were like some of the “no takeover” videos. Right now it looks like I’m a terrible driver and I hate that feeling.

As you’re staying in your legal lane why are you so concerned about the feelings of other drivers? If you darted Into their lane sure, but it’s doing what it should as trucks often slide the lane as they have a greater turn circle. It will never be “no take over” if you are constantly concerned about other drivers and how you “assume” they feel. If I did that I would drive every cell phone talker off the road.

 

[willow_hiller]

What's funny is, on the old legacy highway code everyone complained about it NOT moving over in the lane when a big truck was next to them.

So now it does, and nobody likes THAT either


BTW interesting thing- Ford recently just updated their blues clues ADAS to ALSO move over in a lane if next to a big truck.

One of them items in the 11.4.1 release notes addressed making these adjustments with less jerk. That will probably help:

"- Smoothed in-lane offsetting for large vehicles by tuning the amount of lateral jerk allowed for the maneuver."

 

[Yelobird]

BTW interesting thing- Ford recently just updated their blues clues ADAS to ALSO move over in a lane if next to a big truck.

They also increased the annual price to each owner to $800 to do so lol.

 

[MP3Mike]

Is this on 11.4.1 ?

No, 11.3.6

 

[kabin]

Regarding the idea 'V11 is having its ChatGPT moment' ... Maybe so in a very superficial naiive moment as one could almost see it given the automation, commonplace mistakes they make. Otherwise ChatGPT has creativity potential versus FSDj being more mundane, needing to do what is expected, follow roadway laws, drive comfortably, and safe. There's not much comparison here.

ChatGPT appears to have an easier path with an almost immediate utility and endless applications, businesses, governments, minus societal concerns. Whereas FSDj, a more specific application (minus optimus which appears on path for years of hobbled dog and pony show), won't get there until it can meet a primary goal of real world human safety which is likely still years away.

FSDj is having a moment but it ain't a ChatGPT moment.

 

[JulienW]

Regarding the idea 'V11 is having its ChatGPT moment' .....

OT: I wish Siri would have an AI moment.

Hey Siri turn the lights up.
Siri: Playing Lights Up by Harry Styles
Hey Siri turn the lights up.
Siri: Increasing volume
Hey Siri stop playing Lights Up and turn the lights up.
Siri: I found this on the web
Hey Siri you suck a$$.
Siri: I won't respond to that

 

[Dewg]

Absolutely hate FSD on the highway mainly around larger trucks. It pulls so hard away in the lane it freaks drivers in the opposite lane out that I’m creeping into them. And going between two trucks, it’s like being in a game of pong. This is on 11.3.6 (2022.45.15).

It almost feels like it does complex areas better than the basics. When I report via the “what happenduhhhhh” prompt, it’s always simple responses like “car was trying to make a left from the U turn lane” or “car was changing lanes away from my exit…at the exit”. I wish my experiences were like some of the “no takeover” videos. Right now it looks like I’m a terrible driver and I hate that feeling.

It's getting smoothed out in 11.4.

 

[Mardak]

A post on my experience driving 2023.7.5/FSDb 11.4.1: … I drove it in light rain in the dark, on a road where previously the steering was noticeably erratic in that condition. Even though the "FSDb might be degraded" alarm chimed early on and stayed on for the whole drive, the actual steering experience seemed normal. I think it was probably an improvement--though no two rain experiences can ever be quite the same for perfect comparison. This improvement may have come a release or two ago.

Thanks for the 11.4.1 update. Yeah it'll be hard to compare two rainy dark scenarios, but at least on 11.3.6, I've had it suddenly show the large red steering wheel to take over immediately and switching from "degraded" to "unavailable" message. Previously with 10.x, it probably would have fallen back to the legacy Autopilot stack, but maybe the conditions were so bad that legacy stack wouldn't be useful either? Even with FSD Beta 11.x single stack, plain TACC seems to use a different stack when disengaging FSD Beta with steering.

I wonder if the trigger for "unavailable" is more of FSD Beta detecting too much blur from the wipers vs noticing predictions for the road are low confidence vs preventing extremely erratic paths or something else?

 

[EVNow]

No, 11.3.6

So - not a regression in 11.4.x.

In a way it is not surprising that FSD is sensitive to road width. They have to be sure FSD won't brush the coming car or something stationary on the side. I've not seen MibilEye (Zeekr) even try unmarked roads (let alone narrow roads).

 

[sleepydoc]

How is the most basic of driving scenario broken?

One word - beta.

 

[sleepydoc]

The thing we all tend to forget is this is an extraordinarily complex system. When a human learns something in one situation they automatically generalize it to similar situations. Computers don't have 'common sense' in the same way. When they change the algorithm to account for one problem or improve handling characteristics there are unintended consequences elsewhere.

The other trap we all fall in to is expecting each version to be (and believe what Elon says, even when we say we don't.) After a year and a half of FSDb updates I've seen steady, incremental improvements with a few regressions in each version. When I think about driving with FSDb now it is undoubtedly much more capable and much more relaxing to use than it was a year ago, even with all it's flaws and issues.

Yesterday I drove from work in a Minneapolis suburb to our cabin in northern Wisconsin, stopping at a restaurant and a store along the way. I turned on FSDb after turning out of the parking lot and it drove the entire 150 miles with zero disengagements - smaller roads, feeder highways, interstate, state highways, turning off in some small towns and a roundabout that it handled flawlessly (@EVNow) . The only interventions I did were to adjust the speed to my preferred speed and tap the accelerator when it stopped at a right turn yield lane. I'm reasonably confident it would have proceeded on it's own but there were cars directly behind me that I didn't want to annoy.

The one issue I had was it gave me another unprovoked strike without warning right before I arrived at one of my destinations. the car was negotiating a somewhat difficult turn and I was focusing on the road. FSD beeped once, I looked down and saw the screen switch from blue to the red wheel of death. The screen was likely flashing blue and I didn't see it because I was paying attention but there were absolutely no warning beeps. It seems they still haven't fixed that issue.

 

[zazim]

I hope that 11.4.2 will be released to 2023 models. I am sure that Teslafi doesn’t have many 2023 model subscribers, but so far every 11.4.1 install on HW3 is by an existing FSD beta subscriber.

 

[EVNow]

a roundabout that it handled flawlessly (@EVNow) .

I started a Roundabout thread sometime back I've not updated in a while. I'll update with what I find in 11.x. Anyway ...

Roundabouts FSD handles ok
- not-tiny single lane roundabouts that are in the map and are empty
- Sometimes, the above roundabouts that have traffic. But this is a not consistent at all (see Curt's latest video, FSD jumped in front of another car)

Here are all the issues I see / used to see - along with what seems to be currently fixed.

- Doesn't wait for the vehicle coming from the left. <-- This seems to be fixed in 10.69.25.2. <-- Regression in 11.x ?
- Stops for vehicles coming from the right. Those vehicles will wait at the entrance and ego car should proceed without waiting.
- At an empty roundabout, stops and waits for a couple seconds before entering. FSD should slow down (to 10/15 mph) and enter the roundabout without waiting.
- Highly indecisive when navigating a small (or tiny !) roundabout. FSD stops some 50 meters ahead and needs several seconds to figure out how to navigate ... and does so in a very unnatural way i.e. tries to take the "shortest distance" route rather than around the roundabout.
- At multilane entrance of roundabout, creeps too far into the roundabout if there is another vehicle on the left lane blocking the view to the left.

 

[Frunkenstein]

As you’re staying in your legal lane why are you so concerned about the feelings of other drivers? If you darted Into their lane sure, but it’s doing what it should as trucks often slide the lane as they have a greater turn circle. It will never be “no take over” if you are constantly concerned about other drivers and how you “assume” they feel. If I did that I would drive every cell phone talker off the road.

It be one thing if it was gradual. It’s a jerk left or right, and rides the line. And it’s not about feelings. It’s about the short tempers of other drivers thinking my car is about to make a lane change without signal. Road rage is real. LoL

 

[Frunkenstein]

What's funny is, on the old legacy highway code everyone complained about it NOT moving over in the lane when a big truck was next to them.

So now it does, and nobody likes THAT either


BTW interesting thing- Ford recently just updated their blues clues ADAS to ALSO move over in a lane if next to a big truck.

I think it’s a good thing. When not in autopilot/FSD I do the same thing. However, it’s in advance and not riding the opposite lane. Right now it’s too extreme. They can dial it back a bit, especially between trucks! It’s beta. There’s room for improvement.