FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[EVNow]

Is 11.4.x worse because with the release (or the map update) we lost the extra map metadata we had before the release?

Slowly as the car downloads extra metadata, the drives get better … ?

 

[JHCCAZ]

Is 11.4.x worse because with the release (or the map update) we lost the extra map metadata we had before the release?

Slowly as the car downloads extra metadata, the drives get better … ?

That doesn't match what happened to me today. 11.4.2 started out as good or better than 11.3.6, which had indeed improved over time due to reao-time map hints or whatever. But after about a week of use now, suddenly today it lost the improvements.

So any hypothetical "reset" did not coincide with the 11.4.2 release. OTOH, it may coincide with the latest North American Maps release - I can't tell since I don't know when I actually got that, only that I have it as of this afternoon when I checked.

 

[PVia]

Interesting...I'm still on 11.3.6 which performed admirably for the first week I had it (last week) but has been misbehaving quite a bit these last few days.

 

[MARKM3]

Interesting...I'm still on 11.3.6 which performed admirably for the first week I had it (last week) but has been misbehaving quite a bit these last few days.

Same

 

[sleepydoc]

in general with past releases I've seen improvements in performance after the first week or two. I could never be sure if it was real or imagined but my assumption was it was due to improvements in the neural net or mapping data.

The issues I had a few days ago were on a route I haven't driven in a while so I can't say if they were present when I first got 4.2 but they're definitely new from 3.6 (and previous releases)

 

[jabloomf1230]

in general with past releases I've seen improvements in performance after the first week or two. I could never be sure if it was real or imagined but my assumption was it was due to improvements in the neural net or mapping data.

The issues I had a few days ago were on a route I haven't driven in a while so I can't say if they were present when I first got 4.2 but they're definitely new from 3.6 (and previous releases)

I think it's first, a matter of sample size. The more drives that are made on a given firmware, the more unique circumstances that are encountered by a driver. If the first drive is great, the next drives are likely to be worse. If the first drive is terrible, subsequent drives will probably be better. Eventually, this all averages out if enough drives are made.

There's also a bias that is related to human perception, as FSDb mistakes are remembered more clearly than error free drives. The more drives that are done, the more mistakes that are experienced, so it seems like that a specific FSDb version is always getting worse as time goes on.

 

[EVNow]

I think it's first, a matter of sample size. The more drives that are made on a given firmware, the more unique circumstances that are encountered by a driver. If the first drive is great, the next drives are likely to be worse. If the first drive is terrible, subsequent drives will probably be better. Eventually, this all averages out if enough drives are made.

There's also a bias that is related to human perception, as FSDb mistakes are remembered more clearly than error free drives. The more drives that are done, the more mistakes that are experienced, so it seems like that a specific FSDb version is always getting worse as time goes on.

While these are true in general - there are specific cases of how FSD handles particular intersections - or detailed logging which helps get rid of the "feeling".

 

[old pilot]

Did a 80 mile round trip to Nashville today and was surprised at how well it went. The traffic was light to moderate, no rain or road closures. The car made correct lane selections, although I didn't see the need for some changes however they were not incorrect nor did they cause a problem. The car did not try to get into turning lanes just because they were there nor did it use the turn signal for no reason, that was the strangest part. The car made the left turn out of my driveway (which in the past it did not do) and drove the entire trip except for one disconnect I chose to do to change the route slightly at the end of the trip. The return trip was equally good.
I must admit I do not understand why it works so well sometimes. I notices others here saying the same thing, is there something going on that we are not aware of? Version 2023.7.10 FSD 11.4.2 nav. data ending in 515

 

[heapmaster]

Someone in SF that has FSD should run it on the same path as the Waymo that got stuck that Elon commented on, I would like to see how FSD handles the same intersection and travel path thats blocked.

 

[kabin]

Maybe others have witnessed this but first time for me. The UI message said something to the effect of slowing for emergency lights. We were on the freeway with no emergency lights or emergency vehicles in sight.

 

[uscbucsfan]

Maybe others have witnessed this but first time for me. The UI message said something to the effect of slowing for emergency lights. We were on the freeway with no emergency lights or emergency vehicles in sight.

It's happened with reflections before. They intentionally made it really sensitive.

 

[Mike1080i]

Maybe others have witnessed this but first time for me. The UI message said something to the effect of slowing for emergency lights. We were on the freeway with no emergency lights or emergency vehicles in sight.

I saw that message my first time last week. I was on an interstate at night, approaching a bridge as patrol car with it's blue lights on (not flashing) was crossing over the interstate.

 

[JHCCAZ]

I think it's first, a matter of sample size. The more drives that are made on a given firmware, the more unique circumstances that are encountered by a driver. If the first drive is great, the next drives are likely to be worse. If the first drive is terrible, subsequent drives will probably be better. Eventually, this all averages out if enough drives are made.

I understand completely what you're saying, and in general I would strongly agree with these points of perception vs statistics. It's a topic that comes up quite often in troubleshooting meetings of Fab manufacturing yields, test limits and so on. You have to be disciplined in the application of statistical process control (SPC) principles to avoid over-reaction or under-reaction.

However, I tend to drive relatively simple and highly repeatable routes 7 days a week, at consistent times but with some variations due to workday vs weekend traffic. To me, there is no question that I experienced step changes while on 11.3.6. From one set of consistent behaviors every single day for weeks, then a clear shift to a different set of consistent behaviors, that largely persisted through the 11.4.2 release, until a couple of days ago.

I'll admit that I did not log these in a notebook but I discussed them regularly with my son and posted some of them here. All of these are non-highway suburban style driving.

(Please understand that I'm not generally an anti-FSD complainer. This capability is very important to me, but its often-amusing and sometimes scary foibles are part of the development and testing regime.)

The most reliably trackable misbehavior category is that of lane-change attempts. There are very repeatable problem spots where the car consistently signals an unneeded / unwanted change.

• Some of these are in the sub-category of transient indecision, easy to cancel or self-canceling but confusing to everyone. These tend to happen immediately before a traffic light.
  • The self-canceling attempts seem to be related to the appearance of turn lanes, causing ego indecision even though the route proceeds straight through. It's especially ironic that these signals occur while still planted in the through lane - whereas FSD is very late to properly signal a real shift into a real turn lane.
  • The non-self-canceling attempts are mostly the car trying to shift from one through lane to the next, initiated immediately before the light. I can cancel, but if there's no traffic and I let it go, it will smoothly change right through the traffic light intersection.
• Another sub-category is the persistent unwanted lane change, out of the lane you need to be in, less than a mile ahead of an upcoming turn. In these, the car will fight you if you cancel the change, trying a second or third time until it realizes what's needed at the "In 1000 feet" announcement point. If you let it go in the absence of traffic, it will indeed shift out and soon back in to the needed lane.

Tracking of the behaviors listed above is how I justify that it's not just day -to-day statistical variation. I became very enthused to find that 11.3.6 was able to show clear step-change improvements in behavior, especially when it suddenly dropped the predictable and stubborn "get out of the needed before a turn attempts. The same problem every day, then no problem. Then 11.4.2 came along with a few unrelated improvements and regressions, but no regressions at these particular trouble spots -- until a couple of days ago.

I wish I'd been paying attention to the NA map version updare, but I wasn't, so I can't confirm or deny that correlation.

I'll keep going and see whether or when the regressions get fixed again in a convincing step-change fashion. Not sure but reasonably optimistic.

 

[Pdubs]

Yea it’s in the right spot and follows along appropriately. I have not seen this message before 11.4.x

mine was happening for a couple of weeks before 11.4.x, while I was on 11.3.6

 

[EVNow]

Another sub-category is the persistent unwanted lane change, out of the lane you need to be in, less than a mile ahead of an upcoming turn. In these, the car will fight you if you cancel the change, trying a second or third time until it realizes what's needed at the "In 1000 feet" announcement point. If you let it go in the absence of traffic, it will indeed shift out and soon back in to the needed lane.

These are headscratchers... I've detailed this a couple of times too.

Only thing I can come up with is that the car was trying to overtake slow traffic that's usually present at intersections. In your case is the traffic heavy in those places where it turned away from where it needs to be ?

 

[JHCCAZ]

These are headscratchers... I've detailed this a couple of times too.

Only thing I can come up with is that the car was trying to overtake slow traffic that's usually present at intersections. In your case is the traffic heavy in those places where it turned away from where it needs to be ?

No, in these places it does it whether there's traffic or not. In the most egregious place, it's already in the leftmost of three lanes, less than a mile from a double left turn lane that it needs to be in one of, yet it insists on signaling and changing towards the middle lane. I was so happy when it got fixed about a month ago, stayed fixed on 11.4.2, but now it's back.

 

[willow_hiller]

Someone in SF that has FSD should run it on the same path as the Waymo that got stuck that Elon commented on, I would like to see how FSD handles the same intersection and travel path thats blocked.

There's a very wholesome YouTube channel that does this in general (not for that specific intersection), if you haven't seen it yet:

I imagine it might be hard to get FSD Beta to take the same routes as Waymo.

 

[heapmaster]

Maybe others have witnessed this but first time for me. The UI message said something to the effect of slowing for emergency lights. We were on the freeway with no emergency lights or emergency vehicles in sight.

I have seen that for police cars on the side of the freeway, and also there was some random car with blue steady lights on the bottom of their car that it said the same thing for

 

[heapmaster]

There's a very wholesome YouTube channel that does this in general (not for that specific intersection), if you haven't seen it yet:

I imagine it might be hard to get FSD Beta to take the same routes as Waymo.

Ya I have seen that one, I was hoping the same intersection to see how it handled the construction right in the middle of the road blocking the path.

 

[aronth5]

I noticed the same thing this afternoon. wild lane selection errors in places that had been perfect for the last several releases.

Today I had unusual problems that I've never seen before on routes I've taken 100+ times.
And yet a couple of new improvements mixed in. For example two highways merges where FSD enabled the blinker and moved to the faster lane "before" it reached the end of the merge lane.