FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[D Good]

Yet you don’t still have it but you’ve already rated it. Impressive.

I do have (an older version of) it. 11.4.2. I’m just analyzing other’s posts that 11.4.4 is behaving poorly as well. That’s the meaning of “Seems like.”

 

[PR0FESSOR]

11.4.4 is performing better for me than 11.4.3 did. I am getting better merges, better stop sign performance on unprotected right and left turns, and smoother performance throughout, especially on narrow two lane high speed country roads during left curves. I noticed a bit of spastic left right left right yoke movements before a left turn from a two lane road at a stop light on to a two lane road where there was a vehicle in the left lane coming in the opposite direction and sitting at the light. I thought for a moment that ego might turn into the vehicle sitting at the light, but it did not. I was a bit apprehensive to say the least, but it stabilized and made the turn fine once it decided it was safe. No other complaints right now, but I have not been in any city driving yet, just rural roads and small rural cities.

Joe

 

[jebinc]

11.4.4 so far for me is worse the. 11.4.3. I was actually pretty pleased with 11.4.3 but on 11.4.4 I’ve noticed it gets really nervous when turning or when someone ahead stops. Almost like someone who wasn’t paying attention realizes a second or two after they should have

I found 11.4.2 to be best, and this 11.4.4 Junk to be the worst in my MS Plaid.

 

[jebinc]

Ok I spoke too soon, 4.4 is worse than 4.2 for me with chill. 4.4 just does so many random things and overreacts more to cross cars and peds. 4.4 is definitely smoother in routine things, but when you have other dynamic objects in the environment, it doesn't perform as well as 4.2 so far.

Progress is not accelerating! (at least not in my area)

Nope. FSDj has hit the glass ceiling of progress - at least for refreshed S/X cars running HW3 with crappy camera resolution.

 

[EVNow]

After a couple of drives on 4.4
- the general driving dynamics seems better. Turns are at the right speed and some were really nicely done.
- haven't had many Lane Network issues. But haven't driven in areas that 4.3 had problems

BTW, on 4.3 I had a rare attempt on a rainy twilight to run on the sidewalk up the curb assuming it to be a lane. Never had this before even on the first FSD version we got.

So, watch out for rain.

 

[top1105]

11.4.4 is terrible for me. It’s the most unsafe version for me so far. At many unprotected left turns, it keeps inching left until it’s in the path of oncoming traffic. Even when there’s a line of cars coming.

During two right turns, it tried to turn into the oncoming traffic.

A few times it tried to drive into a freeway on-ramp barrier.

All at regular locations it never had issues with before. Unpredictable.

 

[MP3Mike]

I will say that 11.4.4 (Average) is the best version so far for me. I just completed ~22 miles of mainly city driving and my insurance app only logged one hard braking event. (I think I caused that event by messing with the set speed as it approached the stop sign. I tried to duplicate it and couldn't get it to do it again.)

Turns/corners were way better as well.

There was one situation where it changed to the right, faster, lane right before we should have been making a left turn, so it missed that turn and had to reroute. But it isn't a safety issue. and it should be an easy fix for them to disable speed difference based lane changes when you are within x distance of a turn,

 

[jebinc]

11.4.4 is terrible for me. It’s the most unsafe version for me so far. At many unprotected left turns, it keeps inching left until it’s in the path of oncoming traffic. Even when there’s a line of cars coming.

During two right turns, it tried to turn into the oncoming traffic.

A few times it tried to drive into a freeway on-ramp barrier.

All at regular locations it never had issues with before. Unpredictable.

Yep. It’s truly amazing how much “review variability” exists with each and every release. Assume the variability is real (for whatever software/hardware reason(s)). At this rate, FSDj will not see L3 anytime soon - and never on existing HW3 vehicles. There goes your rototaxi pot smoking induced dreams (Elon).

 

[WilliamG]

Just tested for the first time 11.4.4 down my side streets that are clearly marked with a yellow line in the middle. Still came slamming to a halt with a car coming the other way in its own lane. Just… wtf.

 

[PR0FESSOR]

Since variation analysis is what I teach (quantitative analysis and regression modeling), I will pipe up here. The variation reported here is a factor of all of the variables associated with both FSD software AND (most importantly), the driving environment. The problem is that there are so many variables that keeping track of them, and reporting them here on the forum when we complete driving reports, that it quickly becomes overwhelming. Some of these variables for example:

Visibility associated with time of day and direction relative to the sun and moon.
Phase of the moon for night driving.
Weather (Rain, clouds, barometric pressure, etc.)
Barometric pressure
Environment (rural, urban)
Actual traffic environment (other vehicles and obstacles)
Direction relative to the sun
Direction relative to the moon during night driving
Road markings or lack thereof
Road obstructions
Road sign obstructions
Wind
Behavioral predictability (both animate and inanimate)
Tire pressures and changes in tire pressures while driving
FSD variability
Map variability
Vehicle cleanliness
Color of the tesla (different colors absorb and reflect heat differently, causing thermal distortion of the air coming off the hood, heat waves if you will)
and
on
and
on

Not only do all of these variables exist, but (in quantitative analysis terms) they interact with each other, and the vehicle. Additionally, even with the simplistic off-the-cuff variables, each of the above variables have near infinite possibilities. For example, let’s pick a variable from the above list that looks simple:

Temperature.

With respect to temperature, we have the variables associated with:

The vehicle interior temperature
The air temperature
The temperature of every single object in the cars field of vision
Temperature deviations (both high and low frequency deviations)
Tire temperatures
Etc.

Each of these sub variables has their own sub variables, for example, with respect to tire temperatures, we have:

Left front tire temperature
Left rear tire temperature
Right rear tire temperature
Right front tire temperature

And each of these tire temperature have sub variables:

Left front tire temperature at time t = 0
Left rear tire temperature at time t = 0
Right rear tire temperature at time t = 0
Right front tire temperature at time t = 0
Left front tire temperature at time t = 1
Left rear tire temperature at time t = 1
Right rear tire temperature at time t = 1
Right front tire temperature at time t = 1
etc.

And every variable, sub variable, sub-sub variable, and sub-sub-sub variable (and so on, you get the idea) interacts with other variables, sub variables, sub-sub variables, sub-sub-sub variables (and so on, you get the idea).

The infinite combination of variables and variable interaction terms should quickly become obvious.

And then, to top everything else off, we have human variability, both with respect to the driver of the Tesla and other drivers and vehicles.

Given the above, not even close to complete list of variables that influence FSD behavior, the variability of driver reports of FSD performance should not be at all surprising.

Joe

Yep. It’s truly amazing how much “review variability” exists with each and every release. Assume the variability is real (for whatever software/hardware reason(s)). At this rate, FSDj will not see L3 anytime soon - and never on existing HW3 vehicles. There goes your rototaxi pot smoking induced dreams (Elon).

 

[JohnB007]

Driven few hundred kms now.

- Definitely stops smoother, and brakes earlier.
- I do believe that the wrong lane selection when an exit lane appears still exist. Had one incident where it was going to choose the right exist lane, but swurve back to go straight. Been voice reporting each incident.
- More sensitive to cars joining the opposite lane for sure. One car was simply turning right on the oncoming lane, but mine reacted to that. Not sure if that happened before.
- I felt the car creeps a bit faster, and takes the opportunity faster if exists. Still need to validate further.

Not sure if it's just me, or it happened recently (or, the changes were so gradual that I'm not noticing), but nags are becoming less frequent for me when I'm paying attention to the road in front. Amount of force needed to register torque is also lighter than before.

I personally prefer to leave one of my hand on the wheel all the time so that they can act as a second set of eyes to keep myself aware of what the car is doing. And it's very easy to keep nags away at the same time without trying.

Average setting is what I've been using without "Minimize Lane Changes". I want to see how good it's lane changes are with this setting. It has been good so far, including moving over to the right lane when a faster car approaches from behind. I only use "Minimize Lane Changes" when the situation is complex around me. I wish if there was a way to check my setting without doing "left" then "right".

Overall, it still remains very usable for me, although I wish they fix that wrong lane selection issue that appeared recently.

 

[LowlyOilBurner]

Just tested for the first time 11.4.4 down my side streets that are clearly marked with a yellow line in the middle. Still came slamming to a halt with a car coming the other way in its own lane. Just… wtf.

Yeah, wasn’t this fix in the release notes too? I’ve never had it happen on marked roads, but narrow unmarked roads it’s been doing this I think forever for me? Really needs to be fixed. It shouldn’t be that hard, once oncoming traffic is detected, move to the far right of your drivable space?

 

[SO16]

I personally prefer to leave one of my hand on the wheel all the time so that they can act as a second set of eyes to keep myself aware of what the car is doing. And it's very easy to keep nags away at the same time without trying.

Yes!!!

 

[jebinc]

Since variation analysis is what I teach (quantitative analysis and regression modeling), I will pipe up here. The variation reported here is a factor of all of the variables associated with both FSD software AND (most importantly), the driving environment. The problem is that there are so many variables that keeping track of them, and reporting them here on the forum when we complete driving reports, that it quickly becomes overwhelming. Some of these variables for example:

Visibility associated with time of day and direction relative to the sun and moon.
Phase of the moon for night driving.
Weather (Rain, clouds, barometric pressure, etc.)
Barometric pressure
Environment (rural, urban)
Actual traffic environment (other vehicles and obstacles)
Direction relative to the sun
Direction relative to the moon during night driving
Road markings or lack thereof
Road obstructions
Road sign obstructions
Wind
Behavioral predictability (both animate and inanimate)
Tire pressures and changes in tire pressures while driving
FSD variability
Map variability
Vehicle cleanliness
Color of the tesla (different colors absorb and reflect heat differently, causing thermal distortion of the air coming off the hood, heat waves if you will)
and
on
and
on

Not only do all of these variables exist, but (in quantitative analysis terms) they interact with each other, and the vehicle. Additionally, even with the simplistic off-the-cuff variables, each of the above variables have near infinite possibilities. For example, let’s pick a variable from the above list that looks simple:

Temperature.

With respect to temperature, we have the variables associated with:

The vehicle interior temperature
The air temperature
The temperature of every single object in the cars field of vision
Temperature deviations (both high and low frequency deviations)
Tire temperatures
Etc.

Each of these sub variables has their own sub variables, for example, with respect to tire temperatures, we have:

Left front tire temperature
Left rear tire temperature
Right rear tire temperature
Right front tire temperature

And each of these tire temperature have sub variables:

Left front tire temperature at time t = 0
Left rear tire temperature at time t = 0
Right rear tire temperature at time t = 0
Right front tire temperature at time t = 0
Left front tire temperature at time t = 1
Left rear tire temperature at time t = 1
Right rear tire temperature at time t = 1
Right front tire temperature at time t = 1
etc.

And every variable, sub variable, sub-sub variable, and sub-sub-sub variable (and so on, you get the idea) interacts with other variables, sub variables, sub-sub variables, sub-sub-sub variables (and so on, you get the idea).

The infinite combination of variables and variable interaction terms should quickly become obvious.

And then, to top everything else off, we have human variability, both with respect to the driver of the Tesla and other drivers and vehicles.

Given the above, not even close to complete list of variables that influence FSD behavior, the variability of driver reports of FSD performance should not be at all surprising.

Joe

Great write up, Joe! As an engineer by education, I appreciate the detail of your post! I still stand by my conclusion (given the plethora of variables and variable interactions); L3 isn’t happening for HW3 cars and may not for the current HW4 cars, either.

 

[g1000x]

11.4.4 has tried to run more stop signs than any other version!

 

[PR0FESSOR]

I agree with your conclusion based on what I see on my HW4 car now, but I would change “isn’t happening” to “isn’t happening anytime soon” for both HW3 and HW4 cars. Most, if not all, of the people on this forum have more knowledge about Tesla’s FSD capabilities than I do… but right now, I do not see how we can get there (comfortable L3) from here (nervous L2) anytime soon.

It would be unfair if I didn’t add that I truly appreciate 11.4.4 on my HW4 MSP. It really does make long drives more comfortable, especially in the rural area I live. If I lived in a metropolitan area, I would not feel the same way. The $15k I plunked down is worth it to me, right now. If FSD continues to get better, great. If it stays exactly the same as it is right now, I am satisfied because it makes my long drives so much easier.

Joe

Great write up, Joe! As an engineer by education, I appreciate the detail of your post! I still stand by my conclusion (given the plethora of variables and variable interactions); L3 isn’t happening for HW3 cars and may not for the current HW4 cars, either.

 

[rlsd]

I tested 11.4.3 early this morning. I drove to work than drove back home.
No regression. Neighborhood turns are accurate, not jerking, but not very smooth yet.

Still late left lane changing to turn into 2 turning lanes to freeway. It crossed the solid white line to get into the rightmost turning lane. It's still better than 11.4.1 at the same spot (late left lane changing and crossing the solid white line to get to the leftmost turning lane). Probably FSD does not anticipate this situation: leftmost lane on the road changes to 2 left turning lanes to freeway. FSD kept staying on the second lane on the road and missed the chance to make left turn.

I repeated this test today with 11.4.4. FSD performed smoothly and passed with colors.
The biggest improvement is FSD keeps the leftmost lane on the road and takes the left most lane on a 2-lane ramp on the left to enter freeway.

 

[X Fan]

2023.7.20

crap update for my ‘18 X. exceedingly numerous nags, didn’t slow down quickly form a 55 to 35 zone and in our community without lines in middle of street vehicle was driving smack in the middle of the road versus imputing lines and staying right.

 

[jebinc]

I agree with your conclusion based on what I see on my HW4 car now, but I would change “isn’t happening” to “isn’t happening anytime soon” for both HW3 and HW4 cars. Most, if not all, of the people on this forum have more knowledge about Tesla’s FSD capabilities than I do… but right now, I do not see how we can get there (comfortable L3) from here (nervous L2) anytime soon.

It would be unfair if I didn’t add that I truly appreciate 11.4.4 on my HW4 MSP. It really does make long drives more comfortable, especially in the rural area I live. If I lived in a metropolitan area, I would not feel the same way. The $15k I plunked down is worth it to me, right now. If FSD continues to get better, great. If it stays exactly the same as it is right now, I am satisfied because it makes my long drives so much easier.

Joe

Fair assessment, Joe. I do think HW3 cars and below are done, and likely won't ever achieve all of the past and public Elon hyped claims used to generate FSD sales. Not sure what DoJo and HW4 v2 may bring, but the current HW4 implementation is only a partial as the full HW4 suite is rumored to include additional cameras and camera placement changes. So, I will revise my forecast to, "L3 likely isn’t happening for HW3 cars and may not for the current HW4 v1 cars..."

Edit: Yes, long drives (highways and sparely populated rural arterials) are relatively pleasant on FSDj; however, the same can be said for most L2 systems offered by other brands for more than a decade... FSD was supposed to be a "mind-blowing" step function disruptor.

 

[kabin]

Since variation analysis is what I teach (quantitative analysis and regression modeling), I will pipe up here. The variation reported here is a factor of all of the variables associated with both FSD software AND (most importantly), the driving environment. The problem is that there are so many variables that keeping track of them, and reporting them here on the forum when we complete driving reports, that it quickly becomes overwhelming. Some of these variables for example:

Visibility associated with time of day and direction relative to the sun and moon.
Phase of the moon for night driving.
Weather (Rain, clouds, barometric pressure, etc.)
Barometric pressure
Environment (rural, urban)
Actual traffic environment (other vehicles and obstacles)
Direction relative to the sun
Direction relative to the moon during night driving
Road markings or lack thereof
Road obstructions
Road sign obstructions
Wind
Behavioral predictability (both animate and inanimate)
Tire pressures and changes in tire pressures while driving
FSD variability
Map variability
Vehicle cleanliness
Color of the tesla (different colors absorb and reflect heat differently, causing thermal distortion of the air coming off the hood, heat waves if you will)
and
on
and
on

Not only do all of these variables exist, but (in quantitative analysis terms) they interact with each other, and the vehicle. Additionally, even with the simplistic off-the-cuff variables, each of the above variables have near infinite possibilities. For example, let’s pick a variable from the above list that looks simple:

Temperature.

With respect to temperature, we have the variables associated with:

The vehicle interior temperature
The air temperature
The temperature of every single object in the cars field of vision
Temperature deviations (both high and low frequency deviations)
Tire temperatures
Etc.

Each of these sub variables has their own sub variables, for example, with respect to tire temperatures, we have:

Left front tire temperature
Left rear tire temperature
Right rear tire temperature
Right front tire temperature

And each of these tire temperature have sub variables:

Left front tire temperature at time t = 0
Left rear tire temperature at time t = 0
Right rear tire temperature at time t = 0
Right front tire temperature at time t = 0
Left front tire temperature at time t = 1
Left rear tire temperature at time t = 1
Right rear tire temperature at time t = 1
Right front tire temperature at time t = 1
etc.

And every variable, sub variable, sub-sub variable, and sub-sub-sub variable (and so on, you get the idea) interacts with other variables, sub variables, sub-sub variables, sub-sub-sub variables (and so on, you get the idea).

The infinite combination of variables and variable interaction terms should quickly become obvious.

And then, to top everything else off, we have human variability, both with respect to the driver of the Tesla and other drivers and vehicles.

Given the above, not even close to complete list of variables that influence FSD behavior, the variability of driver reports of FSD performance should not be at all surprising.

Joe