FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[EVNow]

He wants it to work at higher speeds. Which seems a bit high for the US.

https://twitter.com/x/status/1664365183429386244

Looks like green is a very aggressive driver who drives close to 100 mph and basically tailgates to be "safe". Good luck getting any OEM to support that kind of maniacal driving.

 

[uscbucsfan]

His experience is completely the opposite of mine, it's bizarre:

https://twitter.com/x/status/1664365178899644420

I think FSD Beta follows too closely on the highway.

Many want closer follow distance on the interstate. It's a personal preference. What you feel is too close, many feel is too far. The old 1 follow distance was much better IMO.

 

[MarcG]

Looks like green is a very aggressive driver who drives close to 100 mph and basically tailgates to be "safe". Good luck getting any OEM to support that kind of maniacal driving.

100mph definitely feels excessive to me, but FSD’s following distance at 75-80mph is way too long in my opinion and I keep getting cut-ins in medium/high traffic. To prevent that, I constantly have to press the accelerator, which immediately brings up the “hands on wheel” flashing warning, unless I’m actively torquing the wheel at the time (and of course I’m constantly looking at the road).

 

[dflowerz]

Gave 11.4.2 another chance at the 4 way stop left turn into my housing development. Last time crept forward then started to awkwardly do the turn. This time is crept forward again but did the turn OK. Better, but still not usable when traffic is around because it takes too long to creep forward for no reason. The mapping shows a 4 way stop on screen so the creeping behavior is a mystery. I should try filming it one of these days. The left turn is onto a boulevard with a big entry sign and flowers separating the two lanes on the left side. It must be the sign the is causing the confusion.

 

[MarcG]

All this creeping forward continues to confirm that the combination of the b-pillar camera being too far back and the side repeater not seeing at a wide enough angle is severely hindering FSD’s ability to see oncoming traffic from the sides

 

[Yelobird]

Gave 11.4.2 another chance at the 4 way stop left turn into my housing development. Last time crept forward then started to awkwardly do the turn. This time is crept forward again but did the turn OK. Better, but still not usable when traffic is around because it takes too long to creep forward for no reason. The mapping shows a 4 way stop on screen so the creeping behavior is a mystery. I should try filming it one of these days. The left turn is onto a boulevard with a big entry sign and flowers separating the two lanes on the left side. It must be the sign the is causing the confusion.

If you simply give it a slight “confirmation” accelerator pedal it will go through fine. I find this simple process smooths any known trouble spots without over thinking it every time.

 

[EVNow]

and I keep getting cut-ins in medium/high traffic. To prevent that,

Why do you want to prevent that ? Even when manually driving I follow at a safe distance and maniacal drivers cut in. I just fall back. I have never got late because of a few cut ins and they don't hurt my ego.

Especially when going on interstate for hundreds of miles there's zero reason to aggressively follow.

 

[EVNow]

Many want closer follow distance on the interstate. It's a personal preference. What you feel is too close, many feel is too far. The old 1 follow distance was much better IMO.

My personal preference is to have even more follow distance than max so that car doesn't have to react to every speed change. Infact a good feature would be a follow distance range rather than one number.

 

[uscbucsfan]

My personal preference is to have even more follow distance than max so that car doesn't have to react to every speed change. Infact a good feature would be a follow distance range rather than one number.

I agree. Options are the best, but what we have now is worse than it was before in that regard.

 

[uscbucsfan]

Why do you want to prevent that ? Even when manually driving I follow at a safe distance and maniacal drivers cut in. I just fall back. I have never got late because of a few cut ins and they don't hurt my ego.

Especially when going on interstate for hundreds of miles there's zero reason to aggressively follow.

I travel a lot and very long distances. Many drivers get upset and will pass if there is a large gap like FSD Beta allows and then you will continuously get people passing because you have a gap in front of you...consistently making you fall further behind.

My last rental was a Hyundai and the follow distance was closer than FSD Beta on Aggressive on their '1' setting. Also, the old 1 follow distance was better for me and my preference.

 

[Reddy Kilowatt]

Video Jun 01 2023, 11 50 19.mov

Shared with Dropbox

www.dropbox.com

This red hands behavior did not start until Junk v11.4.x, and I drive through this gate every single day.

I used to have those when I ran through the large X crossover with traffic signals. It would go through the green signal then be uh no what do I do now? then threw the red hands up at me. after a couple cycles and updates, it went away. Almost like it deliberately causes this issue to flag Tesla programmers to fix the issue. In later version the car just ran the red light, then corrected that. In this most recent build, which I posted. The car stopped for the red but told me to hit the gas to continue. After that it was like Tesla wanted me to confirm things were good instead of just going for it. Now it is fine, after three or more Beta updates. This is my understanding of what is going on after drinking the Tesla Kool-Aid. FYI, I also reported this to Beta via the "bug report" and follow up Email.

Now if this is a new issue, ( I haven't read all your posts.) Is this a new vision feature Tesla has activated on NN or whatever there moving towards. (I'm no programmer that is for sure , i just test it as it comes out)

 

[sleepydoc]

100mph definitely feels excessive to me, but FSD’s following distance at 75-80mph is way too long in my opinion and I keep getting cut-ins in medium/high traffic. To prevent that, I constantly have to press the accelerator, which immediately brings up the “hands on wheel” flashing warning, unless I’m actively torquing the wheel at the time (and of course I’m constantly looking at the road).

A bit of a rhetorical question, but partially serious- why is it a problem if someone cuts in? The goal should be to arrive at your destination safely, not necessarily before everyone else. If someone cuts in how much does it affect you? A second at the most?

Back when I took driver’s ed I was taught to leave 3 seconds between me and the next car. It seems like most people want to leave about 500 milliseconds.

Ultimately this comes down to driving styles, philosophy and people being irrationally competitive.

 

[sleepydoc]

All this creeping forward continues to confirm that the combination of the b-pillar camera being too far back and the side repeater not seeing at a wide enough angle is severely hindering FSD’s ability to see oncoming traffic from the sides

Except it does so even when there the view is unobstructed. That indicates it’s more of a software issue than due to the camera location.

 

[sleepydoc]

If you simply give it a slight “confirmation” accelerator pedal it will go through fine. I find this simple process smooths any known trouble spots without over thinking it every time.

While this works it’s really much more of an intervention than people give it credit for. One of the primary functions of FSD is to determine whether it’s safe to proceed. Without that requirement FSD would be done by now. When you tap the accelerator you’ve analyzed the traffic and determined that it’s safe, relieving FSD of that responsibility.

 

[uscbucsfan]

Except it does so even when there the view is unobstructed. That indicates it’s more of a software issue than due to the camera location.

It's likely a combination of both.

I will say 11.4.2 seems to really struggle when situations become overly complex to it. i.e. a car is at a stop sign, pulled a little past the stop sign and you need to turn left. The car just freezes and the steering wheel will start to jerk left and right instead of trying to move forward and assess/complete the left turn.

This occurs pretty often at busy intersections, but I used the previous example of something we find relatively easy, but the car gets confused by the situation.

 

[Yelobird]

While this works it’s really much more of an intervention than people give it credit for. One of the primary functions of FSD is to determine whether it’s safe to proceed. Without that requirement FSD would be done by now. When you tap the accelerator you’ve analyzed the traffic and determined that it’s safe, relieving FSD of that responsibility.

I don’t disagree just noting that sometimes assist works both ways. The system is designed (currently) to do what it is certain is safe. Anything it’s uncertain of it’s looking for an assist hence why your attention is required at this time. If overhead signage and a lane flower garden are an unknown variable I am fine with confirming “looks good to me“ with a light pedal tap. Not an end game solution but better then an I give up on all use cause it can’t figure this out on its own. I would prefer assisting vs the system guessing.

 

[JulienW]

I'm sure the follow distance on Interstate's will be tightened up or at least hope so. Was just on multilane I85 almost to Connector and traffic backed up and was over in 2ed to right lane for the 2ed next exit. Because my car was leaving such a gap I had at one point 3 cars in a row cut in front of me to get over to the I75 north exit. As one cut in Ego fell back and then another and then another. Worried I might be backing up before long . Luckily no one behind me honked.

I'll just say I have my Ego driving.

 

[MarcG]

Why do you want to prevent that ? Even when manually driving I follow at a safe distance and maniacal drivers cut in. I just fall back. I have never got late because of a few cut ins and they don't hurt my ego.

Especially when going on interstate for hundreds of miles there's zero reason to aggressively follow.

A bit of a rhetorical question, but partially serious- why is it a problem if someone cuts in? The goal should be to arrive at your destination safely, not necessarily before everyone else. If someone cuts in how much does it affect you? A second at the most?

Back when I took driver’s ed I was taught to leave 3 seconds between me and the next car. It seems like most people want to leave about 500 milliseconds.

Ultimately this comes down to driving styles, philosophy and people being irrationally competitive.

I’m referring to getting cut in while already in the fast lane, following a car ahead of me, and we’re both trying to pass a slightly slower car in lane #2. Here in the US, people get impatient and pass on the right (I grew up in Europe where this is a no-no, same as hanging out in the leftmost lane while passing no one).

The annoyance is that cars that were also behind me in the fast lane see a big gap in front me while on FSD, thinking I’m not going fast enough, then pass me on the right and cut back in front me.

If I don’t follow closely enough, this keeps happening and I end up never being able to pass the original car I was trying to get by, as FSD keeps letting others cars cut in and drop back further and further.

I understand the argument of “why are you in a high rush anyway”, but this FSD behavior just doesn’t match the environment around me in terms of how other cars drive. I might as well drive below the speed limit and stay on the rightmost lane all the time..

 

[EVNow]

I'm sure the follow distance on Interstate's will be tightened up or at least hope so. Was just on multilane I85 almost to Connector and traffic backed up and was over in 2ed to right lane for the 2ed next exit. Because my car was leaving such a gap I had at one point 3 cars in a row cut in front of me to get over to the I75 north exit. As one cut in Ego fell back and then another and then another. Worried I might be backing up before long . Luckily no one behind me honked.

I'll just say I have my Ego driving.

Thats why the gap needs to be dependent on speed. Yesterday FSD decided to stop atleast a car length behind the stop line at a traffic light - and I had to disengage and move forward since the left turn signal was not getting triggered !

ps : I've to say the follow distance at the minimum currently is something I'd never use. Its just too close and I don't feel safe. Even when I drive in other cities with much more aggressive traffic than Seattle. There used to be a time I used to drive like I was in a video game - not anymore.

 

[EVNow]

I might as well drive below the speed limit and stay on the rightmost lane all the time..

More or less the safest choice This is what I do - though not the rightmost lane because of constant merging. Few people cut in front of me - because our lane is generally going slow.

Seriously, all this follow distance and speed I was commenting on was w.r.t. Green when traveling hundreds of miles on freeway. They don't apply to all situations in the city.

FSD needs to develop different logic for overtaking quickly in the city etc. Very different scenario than what Green was describing. Green wanted to go at 100 mph and be able to essentially tailgate.