FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[tmoz]

Sigh. That’s not what I’m asking for, and I have a feeling you know that. I’m asking for updates outside of Elon’s Twitter, primarily. This is not how you properly beta test software, especially one that has so much complexity and real world interaction. We have no idea, for example, why some “civilians” (influencers) got 11.4.6 but the rollout has seemingly halted. Was an issue found? When can we expect the next beta? We have no answers.

Also, sending feedback after a disengagement is a relatively recent thing- and not a replacement for proper bug reports. But I don’t even know since Tesla literally hasn’t told us how our voice notes are even used. If I had to guess they just get aggregated somehow?

I have constant problems with my map and lane selection as well as a turn that dangerously jerks the wheel into the other lane every time. I’ve reported it across the last few betas using voice notes. Still not fixed, and I have no way of knowing its status.

As far as what’s “owed”- I paid $10K for something that isn’t stable yet.

You need to start your own YouTube channel and post videos of you almost dying with that wheel jerk. You'll get thousands of views and Tesla will take notice. We need more "Chuck Cook"s. You'd think Tesla would have hired a few people to be their "Chuck Cook" in their arsenal of data collectors, but they may rely on simulations more than actual driving. It would be interesting to work at Tesla on this project for a year or half year to get a feel for what they are dealing with.

 

[JulienW]

Yes it seems that 11.4.x is a failed project and they have moved on… The wait continues.

I don't see how they can "move on" to 11.5.x until they get a stable 11.4.x since the same issues would still exist. There is no since in adding new "improvements" until you conquer the 11.4.x dilemma. I bet that it is all about integrating HW4 into the HW3 code. Tesla is likely optimizing for HW4 but it regresses for HW3 and vise versa when they try the other. So they are probably caught in a near Catch-22 trying to balance both HWs and make mutual improvements in both without regressions in one or the other.

Meanwhile we are stuck waiting.

 

[uscbucsfan]

I don't see how they can "move on" to 11.5.x until they get a stable 22.4.x since the same issues exist. No since in adding new "improvements" until you conquer the 11.4.x dilemma. I bet that it is all about integrating HW4 into the HW3 code. Tesla is likely optimizing for HW4 but it regresses for HW3 and vise versa. So they are probably caught in a near Catch-22 trying to balance both HWs and make mutual improvements in both without regressions in one.

Meanwhile we are stuck waiting.

I wouldn't be shocked if we started seeing 2 versions of FSD, not saying that's the issue they are running into, but it makes sense.

 

[D Good]

I wonder when there will be an obvious difference in hw3 vs hw4 performance? I think it’s a pretty awful idea for them to try and keep supporting both platforms. It’s pure nightmare fuel from a software engineer’s standpoint.

They either need to give people with hw3 their money back or give them a retrofit of some sort to hw4.
Take a look:

 

[spacecoin]

I wonder when there will be an obvious difference in hw3 vs hw4 performance? I think it’s a pretty awful idea for them to try and keep supporting both platforms. It’s pure nightmare fuel from a software engineer’s standpoint.

Agree.

They either need to give people with hw3 their money back or give them a retrofit of some sort to hw4.

The problem is that a retrofit is too expensive, and 95% of FSD sales are on HW3.

 

[Mardak]

I wonder when there will be an obvious difference in hw3 vs hw4 performance?

Based on the video, it seems like LED traffic light flickering is much reduced, so even existing training/behavior should work better on HW4 as they'll no longer appears like a temporarily off light. Other parts of the video show off the higher resolution potential, and practically based on Tesla removing the dedicated 1.2MP narrow/far camera and covering it with the 5MP main camera, that presumably means we effectively have those far cameras for all directions now with a rough estimate of 2x detection distance all around helping with cross traffic.

Unclear if the HW4 behavior in 11.4.x is "just" downsampling to behave like the old cameras or if there's dedicated neural networks already?

 

[Mardak]

Maybe 11.x direct downsampling won't work for HW4 rear camera as it seems to have a much wider field of view?

Notice how HW4 can see above the tree directly behind but more importantly rear cross traffic if you follow the ends of painted parking lines.

 

[scottf200]

Maybe 11.x direct downsampling won't work for HW4 rear camera as it seems to have a much wider field of view?
Notice how HW4 can see above the tree directly behind but more importantly rear cross traffic if you follow the ends of painted parking lines.

Look on the left and right parts of the screen.

 

[tivoboy]

I wonder when there will be an obvious difference in hw3 vs hw4 performance? I think it’s a pretty awful idea for them to try and keep supporting both platforms. It’s pure nightmare fuel from a software engineer’s standpoint.

They either need to give people with hw3 their money back or give them a retrofit of some sort to hw4.
Take a look:

I’m closer to 100% certain now that my 2018 LR M3 with HW3 will never have L3 let alone L4 capability, if even FSD/Robotaxi. I am also nearly 100% certain that some company is going to re-imburse me for what was paid for FSD plus some amount or credit me full FSD on some future model of car purchased.

Great video. That highway sign, the configurable one with the Emergency/Caution messages on it, which is TOTALLY ILLEGIBLE on HW3 pretty much sold it for me. No way no how DOT allows a HW3 Tesla to be approved for autonomous driving with that failure.

The RED STREET light flickering is also probably indicative of why my car wanted to go through red lights so often.

 

[tivoboy]

Um. The, "Unprotected Left Turn" guy from Florida, to the point where a bunch of Tesla engineers showed up on his doorstep and snarfed data for a while. But he's a special case.

Frankly, I think they over focused on UPL guy and that situation, WHICH IS A VALID SITUATION, and less on overall situations that would have had a broader applicability.

Squeaky wheel gets the grease I guess. Do I have that idiom right? (ESL)

 

[uscbucsfan]

Frankly, I think they over focused on UPL guy and that situation, WHICH IS A VALID SITUATION, and less on overall situations that would have had a broader applicability.

Squeaky wheel gets the grease I guess. Do I have that idiom right? (ESL)

It's funny, because they overfit it and then completely broke it with v11.

 

[kpanda17]

Is this post for 11.3 or 11.4?

 

[AlanSubie4Life]

It's funny, because they overfit it and then completely broke it with v11.

Umm...if by overfit there is an implication that it "works" (not clear what you meant), it has never worked, as has been extremely carefully documented here. They never got to better than 90% success rate! And basically it has only ever worked reasonably reliably when there is no traffic (in either direction) to deal with! So there was nothing to break with v11.

So any oscillation down to 30% success rate vs. 80% success rate is just noise, presumably caused by alterations in the hard-coded behaviors and varying traffic conditions (though it's often less clear what exactly is the reason for improvements and regressions).

 

[uscbucsfan]

Umm...it has never worked, as has been extremely carefully documented here. They never got to better than 90% success rate! And basically it has only ever worked reasonably reliably when there is no traffic (in either direction) to deal with!

So any oscillation down to 30% success rate vs. 80% success rate is just noise, presumably caused by alterations in the hard-coded behaviors (though it's often less clear what exactly is the reason for improvements and regressions).

90%, yes at like 10.69 it was pretty consistent, but as soon as 11 came out it was back to a coin flip of potentially killing him or not.

 

[AlanSubie4Life]

90%, yes at like 10.69 it was pretty consistent, but as soon as 11 came out it was back to a coin flip of potentially killing him or not.

I'd take a close look at traffic conditions and their impact on the success rate for any given iteration. It's never been good and the coin flip thing existed in 10.69, pretty clearly - there were safety interventions and I don't think there's much reason to think the success rate was much different in 10.69 vs. what we have now.

Of course there's no way to know for sure. But in the end it's not that important because 50/50 vs. 90/10 is pretty much the same. It's all coin flips.

I don't really buy the "overfit" hypothesis because some of the skills required for this turn really did improve performance in other areas. The problem is that it's nowhere near good enough with those skill improvements (they're not good enough), and there's a huge number of other skills which the car needs to have which it doesn't, which are not addressed by this turn.

 

[kabin]

Yes it seems that 11.4.x is a failed project and they have moved on… The wait continues.

Gotta wonder if they are gun shy - 3 strikes and they are out.

 

[kabin]

I don't really buy the "overfit" hypothesis because some of the skills required for this turn really did improve performance in other areas. The problem is that it's nowhere near good enough with those skill improvements (they're not good enough), and there's a huge number of other skills which the car needs to have which it doesn't, which are not addressed by this turn.

In some ways that darn UPL overfit for my drives with the infamous turn right (into traffic) at intersections when initiating intersection left turns. I think they implemented turn right logic to better land in the median area. And if I recall correctly, the damn creep hack began from UPL improvement attempts. If so UPL was heavily overfit for my drives.

 

[AlanSubie4Life]

In some ways that darn UPL overfit for my drives with the infamous turn right (into traffic) at intersections when initiating intersection left turns. I think they implemented turn right logic to better land in the median area. And if I recall correctly, the damn creep hack began from UPL improvement attempts. If so UPL was heavily overfit for my drives.

It’s true that both of these things occurred as a result.

The turn right is just an example of a skill which is not good enough (you only do this as a human when you need to - which is pretty infrequent).

The creep is definitely an essential skill (humans do it to obtain visibility). It’s just not nearly good enough. In a way the car should perform a creep way better than a human; it would go straight to the final limit with no actual creep - I’m not sure why it doesn’t do that better and inches forward - why does it take so long to figure it out, should not it take less than a second to figure out the exact right spot? But anyway, it would be disconcerting to a human to do it really quickly.

So it’s poorly implemented and also has a poor human-machine interface.

I don’t think either of these are really over fit - though the “right before left” I can see the argument, for sure.

 

[D Good]

Forget UPLs, UPRs are a consistent fail in certain situations as well. All down to terrible creeping behavior. In some situations it will make the right turn into the wrong lane.

Driving is hard.

 

[fasteddie7]

https://twitter.com/x/status/1688951180561653760

interesting look at the view between HW3 and 4. Makes me wonder if 3 will ever be able to read signs, especially the illuminated construction ones.