FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[MP3Mike]

Yesterday I did a bunch of two-lane country road driving, and it did very good slowing down for the corners, and no phantom braking.

I just did ~120 miles 99.9% on FSD today using the average profile. (Mostly highway) The only "phantom braking" was it slowing down more for curves than I would have. (But the other cars on the road slowed down a similar amount as FSD Beta did.)

And I was very happy with the performance. It does "zipper merge" when it is an on-ramp lane, but if the lane is also an off-ramp it merges "properly" before the lane exits the highway. It leaves about twice as much distance to the lead vehicle as other people, but I thought it was really good. It tends to stay in the "passing lane" most of the time, but if the lane is clear in front of me and someone comes up behind it moves right to let them pass and then goes back to the left. (Before I knew it did this I was just about to use the turn signal stock to make it go to the right, but before I managed to push it, it signaled and moved over on it's own. So it moved exactly when I would have.) The other thing I couldn't 100% confirm, but I swear if the person in front of me started tail-gating the person in front of them it slowed down and left more room between us, which is exactly what I would do.

City street turn performance is still a little "scaredy cat", if they improve that it will be great.

On the other hand my insurance company dinged it a lot for hard braking, but it feels a lot better to me, so it must be very close to the threshold.

 

[Knightshade]

I think it is pretty clear Autosteer on City Streets has not yet been delivered since it cannot yet reliably Autosteer on City Streets.

Autopark couldn't reliably even notice a spot existed to park in for years after it was considered delivered so your standards might be a bit high.

Plus, at least for me, prior to 11.6-which has been the first BIG step back I've seen in a while- city streets worked more often than not (at L2). I'm speaking functionally, not your "I don't like how it stops/gos" bit.

 

[AlanSubie4Life]

On the other hand my insurance company dinged it a lot for hard braking, but it feels a lot better to me,

With enough time you can get used to anything.

I'm speaking functionally, not your "I don't like how it stops/gos" bit.

It’s not a “bit!” The stopping reliably smoothly (note: it has many smooth and successful stops) is worse than the going.

 

[EVNow]

I've stopped reading posts from people who are constantly raving about how amazing it is or are constantly ranting about what junk it is. It's a product with a mixed bag of experiences, and people who recognize that get my attention.

You mean, you don't like simpleton generalizations ? Me too.

Like the below, perhaps ?

That YouTube drive demonstrated FSDb's real strength, which is lane keeping in even the most demanding circumstances. Its excessive caution is perhaps the source of its greatest weakness. I hope it won't take HW4 to get it to be confident enough in its decision-making to achieve a useful level of self-driving. Highways are pretty much there because that's primarily lane keeping.

 

[kabin]

Lol, love it at 37 seconds when it surges forward towards an intersection, oblivious of the obviously jaywalking pedestrian that it needs to stop for. Then slams the brakes on to a stop (0.6g, 18 frames, 12mph, 1.5x speedup). After the intersection, it jerks the wheel back and forth wildly, swerving back and forth in a lane that is clearly marked such that lane changes are not available, with a bus lane to the right. Then it blithely changes lanes into the extremely clearly marked bus lane and carries on for at least a block or so (I stopped watching).

That indicates (to me at least) it is unusable in that area. But maybe I'm nuts.

Sadly, no disengagement reports produced.

Me too. I quit watching after the first 3 or 4 FAILS within the first 2 mins as I thought it was tongue in cheek.

On the other hand, as long as one doesn't mind the nonhuman behaviors, upsetting cars on the roadway, breaking laws, avoiding near misses, then its absolutely !

 

[Mardak]

the treatment of stop signs and such is so bad and jerky

This reminded me of one of the most egregious issues I had with 10.x: stop signs after steep downhills. Before it would suddenly stop at the crest then jerky stops along the way down sometimes even visualizing a vehicle for the painted STOP road making accompanied by swerving and emergency braking.

I'm guessing one particular intersection of this type has autolabeled training data from video sent back from our vehicles as 11.3.6 sometimes even visualizes the street below before the cameras should be able to get a clear view.

Are you saying all your stop sign experiences are equally bad? If not, then I think there's hope that additional neural network training on similar problematic intersections should hopefully result in smoother experiences in general. Part of the intent of this approach is that there's generalization from these similar situations to others.

 

[AlanSubie4Life]

Are you saying all your stop sign experiences are equally bad?

To be clear I said “stop signs and such.” The stop signs aren’t so much jerky (except for the scrubbing wheel) as they are painfully slow (as they have been for time immemorial, they just took it up one notch for 11.3.x).

Usually going slow enough for stop signs that the regen oscillation is not induced.

 

[AlanSubie4Life]

Yesterday I did a bunch of two-lane country road driving, and it did very good slowing down for the corners

I did CA-78 westbound from Santa Ysabel today for about three minutes and then turned it off. It was not confidence inspiring. No massive jerks or anything. Just a little wide to the outside on corners, and a little rough and lacking smoothness for speed changes. Also late on speed changes, not natural flow. I took over and drove smoothly and properly through the corners following traffic ahead.

Passenger was unhappy at the brief engagement as this was a violation of our agreement. I would argue that this wasn’t city streets, and wasn’t freeway, so fell in a gray area.

Maybe sometime I will get to test it.

 

[FloridaJohn]

Technically anybody who bought FSD after ~March 2019, and has fsdb, has already gotten everything promised during the purchase, since the only 2 "undelivered" features when they redefinied what they were selling were stoplight/stop signs which has been available for years now....and city streets, which is fsdb and available to everybody in NA who has FSD (for certain values of everybody)

The pre 3/19 buyers are still owed quite a bit feature-wise though that I agree they're unlikely to ever get on current cars-- the good news for Tesla is they're a relatively small # they'll need to refund, and they only paid ~3k for it.

As a 2018 purchaser of my Model 3 (and FSD) I am substantially less concerned about the “good news for Tesla,” and much more concerned about the bad news for me.

 

[Frunkenstein]

The 2022.45.15 build seems to be brutal on efficiency for me. Checked the tires, weather has been fine, nothing out of the ordinary for AC, but the efficiency has tanked to where there is about a half inch of space between actual and projected. Prior to this update all has been pretty accurate and lined up closely in the graph. I’m at a loss on this one.

 

[Ramphex]

Have not seen a single Tesla in Philippines. Surprisingly, I see a decent amount of Fords…. Wonder why they buy that garbage here considering it’s an import vehicle for them.

 

[black knight 3]

The 2022.45.15 build seems to be brutal on efficiency for me. Checked the tires, weather has been fine, nothing out of the ordinary for AC, but the efficiency has tanked to where there is about a half inch of space between actual and projected. Prior to this update all has been pretty accurate and lined up closely in the graph. I’m at a loss on this one.

not sure what is causing that. I'm also experiencing the same thing.

 

[black knight 3]

FSD beta 11.3.3 was running Ok for me but a recent upgrade to 11.3.6 was regressive in a major way. I 'm not sure if it was because I brought the car in to service center per tesla recommendation and service bulletin to disable radar: "In 2021 - 2022 Model S vehicles running firmware version 2022.20.9 or later, Autopilot and Active Safety Features that were using front radar sensor data transitioned to using camera data (Tesla Vision), making the front radar sensor unnecessary.” Anyways, I'm not sure if it's 11.3.6 installation or the actual disabling of the radar sensor but everything has regressed in a major way. It's super hesitant and jerky in turns and has on multiple occasions slowed down to almost a stop right in the middle of a turn. Also, at a roundabout that it's never had an issue with before, it sees an oncoming car and enters into the intersection just ahead of it which is Ok if it would commit and speed up but the car actually slows down to a stop. It sees the car and yet still jumps out in front and then realizes it may have made a mistake and suddenly gives and stops right in front of the oncoming car. I have no idea what is going on with this but it seems to be suggest a computational latency where it is sluggish in deciding what to do.

 

[mytes]

FSD beta 11.3.3 was running Ok for me but a recent upgrade to 11.3.6 was regressive in a major way. I 'm not sure if it was because I brought the car in to service center per tesla recommendation and service bulletin to disable radar: "In 2021 - 2022 Model S vehicles running firmware version 2022.20.9 or later, Autopilot and Active Safety Features that were using front radar sensor data transitioned to using camera data (Tesla Vision), making the front radar sensor unnecessary.” Anyways, I'm not sure if it's 11.3.6 installation or the actual disabling of the radar sensor but everything has regressed in a major way. It's super hesitant and jerky in turns and has on multiple occasions slowed down to almost a stop right in the middle of a turn. Also, at a roundabout that it's never had an issue with before, it sees an oncoming car and enters into the intersection just ahead of it which is Ok if it would commit and speed up but the car actually slows down to a stop. It sees the car and yet still jumps out in front and then realizes it may have made a mistake and suddenly gives and stops right in front of the oncoming car. I have no idea what is going on with this but it seems to be suggest a computational latency where it is sluggish in deciding what to do.

That is strange, I also have a 21 S and have not heard of any notice requiring disconnection by service of the radar. And why would it need disconnecting as it would simply just not be used by the system. I have not also noticed the SDD beta problem you mention except as sometimes it seems to happen with any of the other models (slowing or stopping in front of an oncoming car. And radar I think was used mainly on the highway

 

[Dewg]

Yeah, FSD Beta would get nauseous and vomit too...

 

[Knightshade]

As a 2018 purchaser of my Model 3 (and FSD) I am substantially less concerned about the “good news for Tesla,” and much more concerned about the bad news for me.

I mean, I'm also a 2018 Model 3 and FSD buyer- so I agree it'd be bad news when they finally admit I'm never getting L4 (or likely even L3 highway, which was all I wanted when I originally bought, and still think THAT is possible with current HW but Tesla has been highly resistant to conditional autonomy).

On the other hand I can consider I'll have gotten the HW3 upgrade, and the current FSD features including what (before 11.3..6 which is garbage) is a pretty decent L2 in almost all situations system for "free" when they refund me my 3k plus interest, so there's a bit of a silver lining at least.... and there's always the chance they offer such owners a one-time free FSD transfer to a new vehicle on newer HW or something like that as well- which effectively cost them $0 instead of 3k but would represent a ~15k savings to such owners (assuming the new HW vehicle finally gets capabilities worth upgrade for of course)

 

[JulienW]

..... so I agree it'd be bad news when they finally admit I'm never getting L4 (or likely even L3 highway, which was all I wanted when I originally bought, and still think THAT is possible with current HW but Tesla has been highly resistant to conditional autonomy)......

I bet Tesla never admits or announces that we won't be getting at least L3. They can always be "working" towards it. Even if we get a HW4 or HW5 roboxaxi specific car Tesla can just say HW3 requires more time and they are working on it.

EDIT: I always said I would be happy with a L3 system. But even if the driving abilities reach L3 Tesla will probably stay L2 to avoid the legal liability of becoming the driver.

 

[boonedocks]

I bet Tesla never admits or announces that we won't be getting at least L3. They can always be "working" towards it. Even if we get a HW4 or HW5 roboxaxi specific car Tesla can just say HW3 requires more time and they are working on it.

EDIT: I always said I would be happy with a L3 system. But even if the driving abilities reach L3 Tesla will probably stay L2 to avoid the legal liability of becoming the driver.

Has there ever been a survey on TMC of how many of us would be 100% okay if @tesla would just give us L3 on expressways? That’s all I have ever wanted too and others manufacturers have it and we (Tesla owners) seem to see it as so far away, if at all, from ever happening for us

 

[Knightshade]

I bet Tesla never admits or announces that we won't be getting at least L3. They can always be "working" towards it. Even if we get a HW4 or HW5 roboxaxi specific car Tesla can just say HW3 requires more time and they are working on it.

Eventually it'll have been long enough that a legit law firm will see a big class action payday- at which point it'll come down to how badly Tesla wishes to avoid opening themselves up to the discovery process where the lawyers might get access to internal emails and other docs showing Tesla knew they couldn't deliver >L2 quite some time ago... versus just settling via things like full FSD refunds and/or transfer offers, in exchange for admitting no wrongdoing and not giving external access to internal FSD discussions.

Honestly the case I'm kinda shocked nobody has filed yet is actually for the POST 3/19 buyers who bought while it still said "by end of year" for city streets in 2019 and 2020.... those guys have a slam dunk case, just as the folks who bought EAP in late 2016 did when it said "by end of year" for the EAP features that weren't delivered until later (and those guys did indeed get a settlement from Tesla over it)

 

[JulienW]

Has there ever been a survey on TMC of how many of us would be 100% okay if @tesla would just give us L3 on expressways? That’s all I have ever wanted too and others manufacturers have it and we (Tesla owners) seem to see it as so far away, if at all, from ever happening for us

Tesla could improve and remove the hands on requirement and still stay L2. So you could have a L3 equivalent on the highway while remaining on L2 and you will still be the responsible party. Kinda like a faux L3 with all of the features with non of the liability.