FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[Ramphex]

11.4.2 report: An attempt at running a red light yesterday.

So, there's this one intersection that, when commuting to and fro work, FSD-b has had a periodic habit, when it's first in line at the red light, to make an attempt at running the red light. It didn't happen to do that with 10.69, but, like I said, it didn't do it all the time. But three times in four months or so, and only there, is enough to make that behavior stick in anybody's mind.

So, when commuting, that particular intersection was crossed going from south to north. Fine.

Yesterday was making a pizza run starting from a not-usual voting location and, as it happened, was going from west to east on the same road that was crossed during commuting where the fun occurred. Red light; car came to a halt, and, as it happened, was first in line.

And the blame FSD-b started to run the red light. Hit the brakes, reported the incident. As noted, this time was crossing the intersection west to east.

This is just weird. Now that I think of it, I think that this might be the only intersection where FSD-b has tried to run a light. And, across the different versions, in two different directions.

Some kind of bizarre map error? There's nothing particularly weird about this intersection. The roads are at right angles. It's very nearly on the flat, with a slight rise from south to north. There's no buildings particularly blocking the view. But FSD-b tries to commit suicide there. There's four lights hung around the intersection, with but one visible in each direction. But, still.

Aren’t we at the point where the Junk should be relying on its Vision superpower and not the map data?

 

[Tronguy]

Aren’t we at the point where the Junk should be relying on its Vision superpower and not the map data?

Well.. But the Vision is doing fine at other stoplights that FSD-b has been stopping at. On local roads, bigger local roads, and all that. Note that after seeing those mentioned attempts at running a red light in previous versions of the Beta, it's not like I don't have my foot poised near the brake pedal at any light; it's a Beta, after all, and I did see that run-the-red-light stuff before.

And it's not like RADAR, ultrasonic sensors, or the lack thereof, has anything to do with red, green, or yellow lights: That's been vision since the very start. Don't think those in-the-air lights get to move around much.

Just bizarre that it's seemingly been just this one intersection. Other people have reported attempts to run red lights in the year-or-so that I've been hanging out on the FSD-b threads. Now I'm beginning to wonder: Is it some kind of bizarro map based problem?

Heck, on 10.69 the car has previously navigated its way through some five (or was it six?) way light-controlled junctions without throwing a spaz. This plain vanilla two-roads-cross-each-other is just weird. What, a local black hole or curse?

 

[JulienW]

This isn't an ambiguous pronoun reference.

I don't know what it is at all so why did you disagree with my post? I need to know what it refers to before I could treat it with ambiguity. I don't know what is coming so please help me understand instead of disagreeing and explain what is coming (the it), when is it coming and to who.

 

[DelPhonic1]

11.3.6 is running red lights, whereas before in the past year and a half, running a red light was extremely rare for beta. Maybe its an 'added to neural net' thing, and the neural net has to catch up, then be magnificently better. We're waiting on that magnificently better right now.

On the contrary, there were 3 intersections in Burbank where beta always failed-- every single time. Now with 11.3.6, it can go thru those same 3 intersections nicely and easily.

 

[EVNow]

Aren’t we at the point where the Junk should be relying on its Vision superpower and not the map data?

Well.. But the Vision is doing fine at other stoplights that FSD-b has been stopping at. On local roads, bigger local roads, and all that. Note that after seeing those mentioned attempts at running a red light in previous versions of the Beta, it's not like I don't have my foot poised near the brake pedal at any light; it's a Beta, after all, and I did see that run-the-red-light stuff before.

Does the visualization show traffic lights (and red light) ? If so, its not a vision problem.

Curt showed similar issue.

I think it is some other (planning related) issue.

 

[DelPhonic1]

I subbed to try FSD 11.3.6 over the past few weeks in my area with one 300 mile road trip. It’s really neat beta software that does make road trip highway driving nicer.

For my daily drive currently, it’s just too expensive and doesn’t add enough value. Too many interventions in my local town with poor lane selection, bad lane changes prior to navigation route (will change left then right). The steering is way too jerky at times. My road by the house with speed limit signs posted 45 but yet the car decides to default down to 25 instead of 45? Weird. Im noticing the same speed limit issues with FSD as I was with standard autopilot. Why are so many highways in my area not mapped with the correct speed limit? It’s ridiculous imo. The speeds haven’t changed ever on these roads?

I’ve had several very bad safety related interventions to where my car will try to pull out in front of someone nearly causing a t-bone / rear end event. I have to slam on the brakes last minute, because I give the car the benefit of the doubt just assuming for sure it isn’t about to pull out in front of the car. On my commute on a narrow two lane road I always get phantom braking when cars are coming in the opposite lane.

There are times where the same route FSD generally does pretty well at, will downright suck - giving me the impression of not wanting to use it.

The same route can give me many different reasons for disengagement causing what I consider an intermittent defect which nullify overall product enjoyment and usage for me.

I figured I’d give it a few more months so I can see for myself the software progression and from there judge what I call the quality of life improvement. If my disengagements and constant speed limit bug reporting isn’t fixed, then I’ll know general autonomy is still way more than a year out. When it works, and I get those non intervention routes, it’s amazing. Unfortunately those are more of the exception in my area.

FSD Beta progress is a moving target. Making assumptions as to value today is a fools errand lol.

 

[DelPhonic1]

Looks from TeslaFi like several people with 2023.12.10 (11.3.6) have been updated to 2023.12.11!

So, anybody know if this is a new 11.4.2, or maybe it's 11.4.3? Even possible it's 11.3.7, which would be rather disappointing. It's almost certain that it's another FSD beta version.

Thats weird, according to Spotify, 2023.12.11 is not FSD beta. Maybe those folks opted out?

 

[Mardak]

Looks from TeslaFi like several people with 2023.12.10 (11.3.6) have been updated to 2023.12.11!

Across TeslaFi and Teslascope, vehicles getting the update are indeed coming from 2023.12.10 which includes FSD Beta 11.3.6 on seemingly regular vehicle software branch. Most of those vehicles so far getting this latest update wouldn't have been able to even activate FSD Beta due to software restricting based on region and hardware. My guess is that there could be fixes related to FSD Beta shadow mode or bugs from integrating FSD Beta into the regular software or merging in some important/urgent changes that were added to 2023.12.5.4 and 2023.12.9.1 dot releases after the release of 2023.12.10.

If there were actual FSD Beta functionality change, it would still likely go through employees and pre-Safety-Score groups first.

 

[Supcom]

Does the visualization show traffic lights (and red light) ? If so, its not a vision problem.

Curt showed similar issue.

I think it is some other (planning related) issue.

Possibly a flicker rate issue with the specific LED lights used in the traffic light. If the flicker rate is close to the camera frame rate, it could be seen as a flashing red instead of solid red. FSD might then stop, then attempt to go when the intersection looks clear.

Might even seem random if the sync between the camera and the light need to be just right.

 

[EVNow]

Possibly a flicker rate issue with the specific LED lights used in the traffic light. If the flicker rate is close to the camera frame rate, it could be seen as a flashing red instead of solid red. FSD might then stop, then attempt to go when the intersection looks clear.

Might even seem random if the sync between the camera and the light need to be just right.

True - but then, won't the visualization show it as flickering ?

In Curt's video, it was solid red.

ps : It makes sense that somehow FSD thinks its like a stop sign situation ...

 

[Mardak]

I think a full vector map accurate to +- 1 foot would be a large download: every node would need global coordinates and relationships to one another

To be clear, OpenStreetMap type of data already has GPS coordinates with 7th decimal place precision, e.g., top right-edge node of Giga Texas 30.2257056, -97.6168843, and that precision of coordinates is 11.1mm / 0.44" between each 0.0000001 difference in latitude. Whereas Google Maps satellite view for that same corner of the building is -97.616802 -- a difference of 0.0000823º or roughly 7.9m / 26' at that latitude.

I don't know which one is more accurate to the true location, but indeed FSD Beta doesn't need the accuracy to be down to 1'. In fact, many 4-way stop signs in OSM are logically placed at the intersection node of the 2 road lines crossing, where even if you assume the accuracy of the roads is good, clearly the stop sign is not actually physically in the middle of the intersection and yet FSD Beta can easily deal with this say 20' inaccuracy.

Basically, FSD Beta doesn't require high-accuracy and high-precision map data as Vision allows it to successfully navigate in many more scenarios without the additional problem of maintaining and delivering of the data.

 

[EVNow]

Basically, FSD Beta doesn't require high-accuracy and high-precision map data as Vision allows it to successfully navigate in many more scenarios without the additional problem of maintaining and delivering of the data.

They just need to change the logic where they stop at the creep limit and wait for zero mph before proceeding - instead of stopping first at the stop sign and then again at the creep limit.

Its interesting that FSD has always had smoother stopping at traffic lights than stop signs. So, they know how to figure out where the stop line is and come to a smooth stop. Its just that stop sign + stop line + creep limit logic is now too convoluted resulting in weird, unnatural behavior.

 

[Supcom]

They just need to change the logic where they stop at the creep limit and wait for zero mph before proceeding - instead of stopping first at the stop sign and then again at the creep limit.

I wonder if all states permit this. In Texas, the transportation code states that you can stop at a point where you have good visibility, not necessarily at the stop sign or a stop line.

 

[stealthyc]

I've experienced a lot more running of red lights in the last couple versions. It generally seems to be an issue of not being able to tell which light corresponds with which lane. There is a particular intersection near me where if I'm waiting at a red left turn arrow and the light for the straight lane turns green, the car just goes. Another intersection has a railroad crossing light in between two stop lights and being not lit the car seems to interpret it as a stop light that is off for my lane, so even though there are two red lights, it will come to a brief stop and then proceed onward. Another intersection has the road coming at an angle to the stop lights and the car just pretends they are not even there until the very last second. Another intersection where the road branches off at an angle with a red/green arrow light has the car just 100% ignoring that light. This issue is fairly concerning to me, as knowing which light corresponds to which lane is a very fundamental aspect of driving.

Additionally the car almost 100% of the time stops at flashing yellow lights now.

There are myriad other issues I encounter requiring intervention of one sort or another, and as is my luck all of these things are very near my house and on just about every common route I take. It would be neat to live in an area where the beta works well. For me, I can barely go anywhere without some guaranteed interventions required.

 

[clydeiii]

Repeat after me: vision is hard. To truly understand what its looking at, car might require something on the order of AGI. (Functional word: 'truly'. It can fake it 90% of the time without AGI I'm sure.)

 

[JHCCAZ]

I wonder if all states permit this. In Texas, the transportation code states that you can stop at a point where you have good visibility, not necessarily at the stop sign or a stop line.

I agree that this would be superior logically but may not be supported by law in all regions.

For myself, I have to say that I've always stopped at the seemingly most appropriate place (if there's no painted stop line), regardless of the actual stop sign placement. I didn't even think about it; I just treated the stop sign as an indication of the right-of-way rule at that intersection, while not assigning any particular significance to where it was actually planted in the ground.

 

[uscbucsfan]

My car ran a single red light twice this weekend and then went through about 10 successful only to run another.

Every time the screen said, Stopping for red light, showed a sold red light, then changed to waiting for turn and started going.

Something is up. I've seen at least 10 videos of 11.4.X running red lights.

 

[powertoold]

Basically, FSD Beta doesn't require high-accuracy and high-precision map data as Vision allows it to successfully navigate in many more scenarios without the additional problem of maintaining and delivering of the data.

I agree with you on that. I was mostly referring to the road and parking lot geometries predicted on the visualization. Some people think it's a downloaded HD map because it seems the car is predicting geometries it can't possibly see.

I also think it's "superhuman" in how accurate it is, but I don't think it's a downloaded map.

 

[Tronguy]

Does the visualization show traffic lights (and red light) ? If so, its not a vision problem.

Curt showed similar issue.

I think it is some other (planning related) issue.

Good point. The location is about a 15-minute drive from home base. I'll maybe give it a stab tomorrow from both directions that have failed and keep an eye on the screen. In between not getting killed.

 

[JHCCAZ]

I agree with you on that. I was mostly referring to the road and parking lot geometries predicted on the visualization. Some people think it's a downloaded HD map because it seems the car is predicting geometries it can't possibly see.

I also think it's "superhuman" in how accurate it is, but I don't think it's a downloaded map.

I think there's little question now, that the car is downloading and using various map hints and details.

I wouldn't say that it's an "HD map" and definitely don't want to get into a side argument about what that term meananin this context. I don't think the important improvements have much to do with high-resolution spatial coordinates, but rather additional lane and environment details. On the other hand, I would agree that this isn't the only mechanism for the FSD to infer and visualize certain details. Some of that capability comes from NN training of course.

But if you're saying the car is not downloading any helpful information (beyond its builtl-in general map) during the routing calculation - well then I'd have to disagree with that. I think it's become fairly clear that this is happening and it's mostly helpful - but occasionally detrimental, if the added details aren't quite right.

The main argument that would make me retract that assertion, would be that FSD in the car is now building its own individualized route experience and acting upon that across various drives. That may happen someday, but pretty much everyone says it's impossible with the present in-car NN architecture.