FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[rlsd]

You are correct, when Tesla changed the software to use cameras only, they degraded FSDs capability when driving in the rain. Prior to Tesla Vision, my Model S with Radar and Ultra Sonic Sensors did not have issues with FSD cancelling because of rain.

Tesla Vision will never reach level 5 because it cannot reliably drive in rain.

I don't have issue with driving in the rain. If the rain is not too heavy FSD gives me the warning "FSD may be degraded". In heavy rain or foggy weather sometimes I see message "FSD is degraded". However it still perform well in the rain. Off course I reduce speed in heavy rain, but still 50 mph or above. I have only vision.

 

[uscbucsfan]

You are correct, when Tesla changed the software to use cameras only, they degraded FSDs capability when driving in the rain. Prior to Tesla Vision, my Model S with Radar and Ultra Sonic Sensors did not have issues with FSD cancelling because of rain.

Tesla Vision will never reach level 5 because it cannot reliably drive in rain.

Wheb my MS used radar FSD would disable in heavy rain. It would revert to AP.

Now with the radar disabled, it never does that. It will annoyingly reduce speed, sometimes when it's not raining hard, but even in the worst rain in Florida, never disables.

It's exponentially better now than it was 2-3 years ago.

 

[aronth5]

First time I've seen FSD properly stop for a school bus. Have others seen this behavior?

https://twitter.com/x/status/1714409718096040401

 

[Supcom]

First time I've seen FSD properly stop for a school bus. Have others seen this behavior?

https://twitter.com/x/status/1714409718096040401

I'm less than impressed. It looked to me like FSDb stopped because the bus is wide and, with the stop signs extended, FSDb did not want to squeeze between it and the center of the roadway.

Had the bus been in the oncoming lane, I would have been impressed. Had the poster stated that the car put up a message stating it was stopping for the school bus, I would be similarly impressed.

 

[Ramphex]

Aaaaaaaand non-FSD Model Y gets another update. I feel like it’s been getting an update every two weeks. Wonder if it’s some important features or bug fixes. It’s okay, we’ll get these updates in 6 months or so.

 

[archae86]

Had an intriguing handling of an odd situation by FSDb running 2023.27.6/FSDb 11.4.7.2 this evening.

Ego pulled up to an intersection on a multilane road in the left-most lane (best choice for what is coming in the next half mile). The light was red. The car ahead was not only stopped but had the driver door open, and the driver was out and doing something at the sidewalk to the left.

Ego turned on the right blinker and changed lanes to the right, while giving me a text announcement something like "changing lanes to avoid obstruction".

The maneuver was not very graceful, steering and acceleration both a little jerky, and the final pose was not very well lined up in the new lane.

But it was good enough, and like the bear that played chess, the stunning thing was that it saw the right thing to do and did it, even if not extremely well.

 

[Bet TSLA]

For the first time since I started using Autopilot in October, 2014, my Tesla made a serious attempt to kill me and my wife. This was on Friday, October 13, 2023 in my December 2017 Model S (HW3) running 2023.7.6 (FSDb 11.4.7.2). My wife was driving on I-80 heading towards Truckee, in the mountains. I was in the front passenger seat.

The vehicle was in the center lane with a semi to the right and a pickup following behind us, but not too close. FSD was on, going 70 on an uphill grade. Suddenly the car slowed a bit. Being used to occasional phantom braking, my wife checked the road ahead, saw nothing of concern, and pushed the go pedal a bit. No response, continued to slow. Now drifting backward compared to the semi on our right. Pushed harder on the go pedal, eventually pushing it all the way down — no response, neither accelerating nor braking, just continuing to slow like the car was in neutral and there was no power going to the wheels.

After a bit the car came to a complete stop in the center lane. Luckily the pickup behind us went around us, and nobody else was very close. My wife put the car in park, then tried the go pedal. Nothing. Oh, wait, got to step on the brake first to get it out of park. Did that and the car then behaved normally.

This is a mystery to us. There’s no way she could have been mistakenly pressing the brake because that would have made the car slow abruptly, but that didn’t happen. And getting no response from pushing the go pedal all the way down is bizarre. The whole thing is bizarre.

So, has anybody here ever experienced anything like this? We haven’t turned on FSD since, despite hundreds of miles of driving. Any explanation other than that it was Friday the 13th?

 

[arnolddeleon]

For the first time since I started using Autopilot in October, 2014, my Tesla made a serious attempt to kill me and my wife. This was on Friday, October 13, 2023 in my December 2017 Model S (HW3) running 2023.7.6 (FSDb 11.4.7.2). My wife was driving on I-80 heading towards Truckee, in the mountains. I was in the front passenger seat.

The vehicle was in the center lane with a semi to the right and a pickup following behind us, but not too close. FSD was on, going 70 on an uphill grade. Suddenly the car slowed a bit. Being used to occasional phantom braking, my wife checked the road ahead, saw nothing of concern, and pushed the go pedal a bit. No response, continued to slow. Now drifting backward compared to the semi on our right. Pushed harder on the go pedal, eventually pushing it all the way down — no response, neither accelerating nor braking, just continuing to slow like the car was in neutral and there was no power going to the wheels.

After a bit the car came to a complete stop in the center lane. Luckily the pickup behind us went around us, and nobody else was very close. My wife put the car in park, then tried the go pedal. Nothing. Oh, wait, got to step on the brake first to get it out of park. Did that and the car then behaved normally.

This is a mystery to us. There’s no way she could have been mistakenly pressing the brake because that would have made the car slow abruptly, but that didn’t happen. And getting no response from pushing the go pedal all the way down is bizarre. The whole thing is bizarre.

So, has anybody here ever experienced anything like this? We haven’t turned on FSD since, despite hundreds of miles of driving. Any explanation other than that it was Friday the 13th?

FSD or some other fault? Did you do a bug report?

It sounds like no power off; just park and then shift to drive. Any chance the car was accidentally put in neutral while driving? I've on a few times gotten the warning about trying to go to neutral when I was trying to cancel FSD with the stalk.

 

[Bet TSLA]

FSD or some other fault? Did you do a bug report?

Some kind of software issue, I imagine. Filed a service request. Will get into that if there's any interest here. We talked to a Tesla service guy about the content of the logs.

It sounds like no power off; just park and then shift to drive. Any chance the car was accidentally put in neutral while driving? I've on a few times gotten the warning about trying to go to neutral when I was trying to cancel FSD with the stalk.

Yeah, there was no power off. I'm sure because the screens happily continued to display. And it's hard to shift into neutral if your hands are simply resting on the wheel and not near the stalk. Nothing was done with the stalk until my wife pressed the button to put it in park after it stopped.

 

[LowlyOilBurner]

First time I've seen FSD properly stop for a school bus. Have others seen this behavior?

https://twitter.com/x/status/1714409718096040401

I’ve had it do this a few times on prior builds than 11.4.7.2. It would be more impressive if it stop for a school bus if you are in the lane approaching the bus (oncoming lane of bus). That would be an improvement.

 

[sleepydoc]

Some kind of software issue, I imagine. Filed a service request. Will get into that if there's any interest here. We talked to a Tesla service guy about the content of the logs.


Yeah, there was no power off. I'm sure because the screens happily continued to display. And it's hard to shift into neutral if your hands are simply resting on the wheel and not near the stalk. Nothing was done with the stalk until my wife pressed the button to put it in park after it stopped.

Let us know. Agree - this doesn’t sound like a FSD or AP issue. Not that it would help much but did you save a dash cam clip?

 

[S100D]

Have you considered the fact that your 2016 Model S didn't know better not to drive that fast? i.e. Tesla did not do a recall on older cars with unsafe software? Just remember you are still responsible on any of the driver assist systems.

I have been driving for 61 years and have never had an accident and I am aware that I am totally responsible when driving my car.

The radar that Tesla removed could see hundreds of feet down the road in rain and snow conditions. Driving the speed limit with radar was much safer than using Tesla Vision. If visibility was low, I always slowed down below the speed limit.

 

[enemji]

v11.4.4. on 2023.26.10

It has gotten alarmingly more confident in the last week. Driving very agressively and changing lanes most of the times while crossing the intersection. Real life rollercoaster.




I was in the rightmost lane and the vehicle ahead of me slowed down to make a right turn into a residential lane. FSDb switched lanes right away to the lane on the left to continue on the path. Generally I have seen it slow down behind the vehicle, and then determine next action. This was confident quick action. Damn!





and the surprise continues...

Today I was driving down a city road 30mph on FSDb and on the right side of the lane there was a bundle of blanket/rug. The car shifted to the left side of the lane, avoided the blanket, and then came back to the center.

 

[KLam]

I have been driving for 61 years and have never had an accident and I am aware that I am totally responsible when driving my car.

The radar that Tesla removed could see hundreds of feet down the road in rain and snow conditions. Driving the speed limit with radar was much safer than using Tesla Vision. If visibility was low, I always slowed down below the speed limit.

All I am trying to say was that how do you know that Tesla engineers know what they are doing with the added radar signals. Radar detection range decreases with rain and snow too. Apparently, Tesla can only detect rain or snow with their cameras. Does radar also indicates there is rain or snow? You may have the false sense that it is saver with the radar if Tesla engineers did not implement the hardware and software correctly and did not warn you of the degraded capabilities of radar with rain and snow.

A simple situation would be while driving in heavy snow. The camera said I cannot see. The radar said there is nothing out there. Should the system just said lets proceed as usual or slow down. Should the system warn the user? What is safer in this situation?

 

[heapmaster]

Allow me to be a bit pedantic… I do it for a living and it is hard to stop…

It’s “brakes” not “breaks”.

lol, you can thank Siri while typing on the mobile, you should see how I text with her stupid auto-correcting BS that changes the words I spell correctly for the sentence. Maybe shes the one teaching how my FSD should drive

 

[rlsd]

lol, you can thank Siri while typing on the mobile, you should see how I text with her stupid auto-correcting BS that changes the words I spell correctly for the sentence. Maybe shes the one teaching how my FSD should drive

"Brakes" is old generation language. "Breaks" is gen Y language.

 

[JHCCAZ]

lol, you can thank Siri while typing on the mobile, you should see how I text with her stupid auto-correcting BS that changes the words I spell correctly for the sentence. Maybe shes the one teaching how my FSD should drive

She's trying, give her a brake...

 

[arnolddeleon]

Some kind of software issue, I imagine. Filed a service request. Will get into that if there's any interest here. We talked to a Tesla service guy about the content of the logs.


Yeah, there was no power off. I'm sure because the screens happily continued to display. And it's hard to shift into neutral if your hands are simply resting on the wheel and not near the stalk. Nothing was done with the stalk until my wife pressed the button to put it in park after it stopped.

I ran some experiments today to try to intentionally put the car in neutral while driving to learn what that would be like. I made sure there were no other cars around me on a quiet road. The short answer is it would be very difficult to miss the car going into neutral. A quick swipe doesn't put the car in neutral.

Apropos to other FSD stuff:

I still wonder if there is some logic somewhere that gets left and right backwards. It's either that or the priority for a "pass slower traffic" is overriding the "we need to stay in the rightmost lane to make the exit". For those in Silicon Valley, you might be familiar with the Northbound 85 to Eastbound 237. It's a tight maneuver and I often find myself "helping" by manually signaling right to get the car to go exit lane sooner. The last two times I've gone through this intersection I've had to disengage to keep the car from moving over to the left lane just as the right exit lane appears.

I've noticed for a while now, but I keep forgetting to mention the phantom signals seem to be slightly worse than release prior to 11.4.7.2. Also, a weird inconsistency, it will signal exits from freeways, but it somehow doesn't like signaling when exiting a limited access road, specifically, Central Expressway (at least for the exit that I take). So sometimes signals when it doesn't need to and doesn't signal when it should.

I need a map update; it keeps attempting rights on red that I thought were fixed in previous releases.

 

[AlekseiNovikov]

I am wondering which version of FSD will come with the new 2023.38.x releases.
Anyone seen anything?

Version 2023.38.1 includes FSD Beta 11.4.4.
My friend received this firmware yesterday.

 

[sleepydoc]

I ran some experiments today to try to intentionally put the car in neutral while driving to learn what that would be like. I made sure there were no other cars around me on a quiet road. The short answer is it would be very difficult to miss the car going into neutral. A quick swipe doesn't put the car in neutral.

Apropos to other FSD stuff:

I still wonder if there is some logic somewhere that gets left and right backwards. It's either that or the priority for a "pass slower traffic" is overriding the "we need to stay in the rightmost lane to make the exit". For those in Silicon Valley, you might be familiar with the Northbound 85 to Eastbound 237. It's a tight maneuver and I often find myself "helping" by manually signaling right to get the car to go exit lane sooner. The last two times I've gone through this intersection I've had to disengage to keep the car from moving over to the left lane just as the right exit lane appears.

I've noticed for a while now, but I keep forgetting to mention the phantom signals seem to be slightly worse than release prior to 11.4.7.2. Also, a weird inconsistency, it will signal exits from freeways, but it somehow doesn't like signaling when exiting a limited access road, specifically, Central Expressway (at least for the exit that I take). So sometimes signals when it doesn't need to and doesn't signal when it should.

I need a map update; it keeps attempting rights on red that I thought were fixed in previous releases.

Agree on the accidental neutral shifting. I've accidentally bumped the stalk and it does not go into neutral simply from a quick bump.

Regarding the left/right issues, I tend to think it's simply poor planning/decision making. Just a few days ago I was driving on the highway in the left lane with the map indicating I needed to take a left exit in 0.8 miles. The car then inexplicably merged into the right lane. There was no one behind me trying to get by (which shouldn't have mattered anyway, that close to the exit,) and no other reason to change lanes. A bit later it realized it needed to get back into the lane and would have missed the exit had I not intervened.