FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[fasteddie7]

Hope not. Last mobile service I had, the tech pulled it out.

It’s the new Phoenix one in all HW4 S/X

 

[Nvbob]

Got 11.4.3 this afternoon. First drive to grocery store the car on FSDj the car decided to go through a stop sign, a red traffic signal, and not stick to the right side of the street on an unmarked road. When we were on 11.3.6, the display would have a "virtual" set of lines down the middle of the road which the car would stay to the right of it.

I remain . . . optimistic

 

[Ramphex]

Got 11.4.3 this afternoon. First drive to grocery store the car on FSDj the car decided to go through a stop sign, a red traffic signal, and not stick to the right side of the street on an unmarked road. When we were on 11.3.6, the display would have a "virtual" set of lines down the middle of the road which the car would stay to the right of it.

I remain . . . optimistic

I do appreciate the appropriate use of the J

 

[jebinc]

Got 11.4.3 this afternoon. First drive to grocery store the car on FSDj the car decided to go through a stop sign, a red traffic signal, and not stick to the right side of the street on an unmarked road. When we were on 11.3.6, the display would have a "virtual" set of lines down the middle of the road which the car would stay to the right of it.

I remain . . . optimistic

As expected…

 

[Knightshade]

It’s the new Phoenix one in all HW4 S/X

Radar only on S/X- guess the heard Ramphex saying it needs more help than 3/Y!

 

[totoro722]

HW4 tear down shows the radar units are indeed present. Wonder if this will be used to improve FSD or be implemented to somehow act as an addition sensor.

I am guessing HW4 is not complete. Future releases will utilize the radar. There may be additional camera placements too. Just a guess.

 

[Ramphex]

I am guessing HW4 is not complete. Future releases will utilize the radar. There may be additional camera placements too. Just a guess.

Is it………




HW4.0 (Beta)?

 

[jebinc]

I am guessing HW4 is not complete. Future releases will utilize the radar. There may be additional camera placements too. Just a guess.

So, the current cars delivering now with “HW4” are bastards? Meaning v2 of HW4 will have camera location and camera count changes? WTF, Tesla….

 

[jebinc]

Is it………




HW4.0 (Beta)?

HW4.b (for bastard)!

 

[D Good]

I am very much losing faith in FSD. Just had the worst day of drives on FSDb (or FSDj or whatever) since I got the beta almost 2 years ago. 11.4.2 is godawful. Horrible. It’s not smooth, it can almost never pick the correct lane. Give me back 11.3.6 and I’d be happy again. Don’t know why anyone is calling .4.2 better then .3.6.

This failure of a release is the beginning of the end for FSD I feel. We have been driving for so long now with no real progress.

 

[jebinc]

I am very much soling faith in FSD. Just had the worst day of drives on FSDb (or FSDj or whatever) since I got the beta almost 2 years ago. 11.4.2 is godawful. Horrible. It’s not smooth, it can almost never pick the correct lane. Give me back 11.3.6 and I’d be happy again. Don’t know why anyone is calling .4.2 better then .3.6.

This failure of a release is the beginning of the end for FSD I feel. We have been driving for so long now with no real progress.

Wait until you try 11.4.3…. And yes, it’s FSD(j)….

 

[EVNow]

Give me back 11.3.6 and I’d be happy again. Don’t know why anyone is calling .4.2 better then .3.6.

It seems people in CA are happier with 4.2 and many of us outside CA are not.

 

[Dewg]

Map is now 2022.44, had a route issue on the way home. Turned onto Main St, and a few hundred feet on the street the guidance said to turn onto Main St. The map showed me on Main with the guidance having me go straight. Not sure why the voice prompt said to turn onto the street I was already on. Onscreen map and visualization was correct, but nav route was wrong and FSD Beta started to change lanes so I disengaged and reported.

Perhaps there is an odd interaction between route data, map data, and FSD Beta.

This is the first time I've had route issues, but I was on 2022.22 map data for quite some time.

 

[rlsd]

Map is now 2022.44, had a route issue on the way home. Turned onto Main St, and a few hundred feet on the street the guidance said to turn onto Main St. The map showed me on Main with the guidance having me go straight. Not sure why the voice prompt said to turn onto the street I was already on. Onscreen map and visualization was correct, but nav route was wrong and FSD Beta started to change lanes so I disengaged and reported.

Perhaps there is an odd interaction between route data, map data, and FSD Beta.

This is the first time I've had route issues, but I was on 2022.22 map data for quite some time.

Yes. I had similar map issues. When exiting from freeway, FSD exited to an earlier exit but navigation said exiting to the next exit.

 

[Ramphex]

Yes. I had similar map issues. When exiting from freeway, FSD exited to an earlier exit but navigation said exiting to the next exit.

You’d imagine with all of us little guinea pigs running around the country Tesla would have their own mapping by now.

 

[Scott Chapman]

what the fudge?! I got 11.4.3 yesterday. What gives?!

 

[Rikster]

11.4.3 must be going fairly wide now, as I just got the notification to install it on my 2022 MSLR. It doesn't seem like much has changed, but I'll be interested to see if there are any further improvements. I'm one of those people who thought 11.3.6 was a bit better than 11.4.2 overall...

 

[fasteddie7]

For those who know, what actually gets updated with these updates? In my short time using FSDb I noticed that silent or map updates change the behavior on the roads, so is it just a ton of driving data to help make decisions/pathfinding better?

 

[JHCCAZ]

For those who know, what actually gets updated with these updates? In my short time using FSDb I noticed that silent or map updates change the behavior on the roads, so is it just a ton of driving data to help make decisions/pathfinding better?

You're correct about this, but it's a relatively new thing - maybe in planning and development for a long time but relatively new in wide deployment and effect. The main source is user verygreen = greentheonly on Twitter, a well-known guy who likes to hack into the software to see what's going on in there.

See the discussion starting with posts in the Elon Tweets thread, quotrd below. This started with a prior discussion about whether it was possible for the software release code, or at least parameters that could affect its operation, could be altered by Tesla without pushing a whole new release. It seems the answer is basically no except for the revelation about real-ime map details:

there are some limited settings that could be persisted. There are actually two (three, but third is only for dev cars) sets, some ap-related stuff in ap firmwares, some ap related stuff on infotainment most of which (but not all) you can control in the settings UI.

And then there are maps. I was exploring mapping recently and discovered that not only Tesla has those pretty frequent incremental updates, but when you have a route set - it asks the mothership and receives a route outline with very detailed info including where various intersections, crosswalks, stop signs, traffic control devices, speed limits, lanes, ... are. Every time you use "navigate to" function (and a bunch of that data is fleet-collected). As you can imagine updates in that sort of thing can have big impact on performance of AP without any code changes.

Thanks for that great information. What you just described about maps - route-specific data that is downloaded in near real time - is potentially very significant, and fairly surprising (to me) in a number of ways:

First, because most users agree that a great number of today's FSD foibles seem to stem from mapping errors or inadequacies.

Second, because the map database and updates have been kind of mysterious and often are assumed to be chronically out of date.

Third, because there's always a background debate smoldering here, about Tesla's supposed lack of interest or dependence on detailed maps for its real time driving performance.

Fourth, because I for one had assumed that drive-time data connectivity was basically irrelevant to Tesla driving operations, things like media streaming voice command processing, data collection uploads and so on are all things that don't really affect actual driving, so lack of connectivity is a temporary inconvenience.

But what you just revealed is something that could make a real difference in FSD capability, depending on whether the data connection is available. It begs a few further questions (answers for which may be unknown now, but if you have any insight please comment):

1. Is this drive time server download a new thing, or something that has been added recently? Is it applicable only to FSD beta, or has it been around for legacy NoA as well?
2. What happens if the call to the Mothership server fails during the route planning or route execution? Is there a timeout, or does it keep trying as the drive proceeds along the route? Upon failure, does the information devolve to the in-car stored map database (with presumably coarser and/or very stale information), or does it remember data from the most recent drive on or near that route?
3. Do you think this map details feature will be an integral enabler of things like parking lot mapping for Actual Smart Summon and presumably required robotaxi pickup/dropoff operation?

Finally, is it your sense that this discovered capability is indeed highly significant for the upcoming development of FSD, or am I extrapolating too far from what you explained? Thanks so much for whatever additional commentary you can provide.

this map server was added relatively long time ago, I just only recently looked into it. This is what provides alternate routes, parking lot outlines and so on. Features keep getting added. It's in use no matter if you are on the beta or not. The "suspension adjustments for rough roads" is also fed from here.
If the call to the server fails (as in there's no connection) you just get the "offline navigation" icon and none of this additional info. No alt-routes, not display of slowdowns on route on the actual route outline.
There is some route caching built in that's server controlled too (and on several levels, it also caches google maps stuff from your recent destinations an such).

This is definitely an important part of the whole package and allows them to minimize staledness of map data and get a lot of extra details relevant for this drive without overloadign the car with lots of (potentially very quickly spoiling) data too.

Think of it like waze. Once you drive soem with it, driving without it is unthinkable even if you are very familiar with the route and don't need the actual directions, all the other information it provides is still important.

 

[D Good]

The solution is for FSD to just use Apple Maps. It’s way more accurate and provides significantly better routing than the idiotic Tesla nav. Tesla frequently prefers unprotected lefts over protected turns to save one minute on the drive. Apple Maps also has much more accurate and up to date lane data. Tesla uses Google maps somewhat I think because I can replicate much of the route failures there. Seems like the tables have turned since Apple Maps was first released.