FSD Beta v11.x

The next big milestone for FSD is 11. It is a significant upgrade and fundamental changes to several parts of the FSD stack including totally new way to train the perception NN.

From AI day and Lex Fridman interview we have a good sense of what might be included.

- Object permanence both temporal and spatial
- Moving from “bag of points” to objects in NN
- Creating a 3D vector representation of the environment all in NN
- Planner optimization using NN / Monte Carlo Tree Search (MCTS)
- Change from processed images to “photon count” / raw image
- Change from single image perception to surround video
- Merging of city, highway and parking lot stacks a.k.a. Single Stack

Lex Fridman Interview of Elon. Starting with FSD related topics.

Here is a detailed explanation of Beta 11 in "layman's language" by James Douma, interview done after Lex Podcast.

Here is the AI Day explanation by in 4 parts.

Deep Understanding Tesla FSD Part 1: HydraNet

From Theory to Reality, Analyze the Evolution of Tesla Full Self-Driving

saneryee-studio.medium.com

Here is a useful blog post asking a few questions to Tesla about AI day. The useful part comes in comparison of Tesla's methods with Waymo and others (detailed papers linked).

Twelve Questions about Tesla Autopilot Technologies Exposed at 2021 AI Day

On Aug. 19 2021, Telsa held an AI day , exposing more technologies about Autopilot , from perception, planning, data labeling, simulation…

yuhuang-63908.medium.com

 

[powertoold]

I am getting tired of all these “major improvements“ that are supposed to come with yet another release.

My car wanted to drive into the left lane at unprotected left turn. A turn that it has done flawlessly numerous times. Today, however, it made this stupid mistake. I don’t know why…

The other day the car waited for a red light only to start accelerating suddenly while light was still red. Turned out the car started looking at another light across the street which turned green.

Tesla is in the home stretch of fsd.

Get all the networks into 4D vector world.

Gather as many variations of collisions and situations as possible.

Turn the human video labelers into driving planner analysts.

Train the whole stack end-to-end, rewarding the car for following the "correct" way of driving or dealing with situations.

We may not be training the final stack directly, but it's all just part of the process.

Even when it's all said and done, fsd will still not be perfect, but the imperfections will be looped back into the training stack.

 

[sleepydoc]

@Ramphex @jebinc @EVNow

Dan Burkland has posted here on TMC and has a youtube channel. I’ve found his videos to be pretty balanced and accurate compared to my experiences. Unfortunately he doesn’t have an X or an S so it won’t prove/disprove your hypothesis but it’s a pretty accurate picture of how FSD works around Minneapolis.

https://www.youtube.com/@danburkland/videos

 

[EVNow]

Tesla is in the home stretch of fsd.

Not according to disengagement metrics. Tesla is just getting started.

I know some say disengagement rate (which is an industry standard metric) doesn't matter for Tesla ... since this is an extraordinary claim, the burden of proof is on them. And of course what's the other metric to use instead ?

 

[sleepydoc]

Not according to disengagement metrics. Tesla is just getting started.

I know some say disengagement rate (which is an industry standard metric) doesn't matter for Tesla ... since this is an extraordinary claim, the burden of proof is on them. And of course what's the other metric to use instead ?

FSDb is a love/hate, frustrating feature. It’s cool and impressive to be able to let my car drive me home. Improvements have been frustratingly slow at times. When I look back over the last year and a half I see huge improvements. Even so, I still have disengagements that seem basic and trivial and it’s frustrating that they haven’t been fixed yet.

So where are we really with FSD? I don’t know. I don’t think anyone really knows, not even Elon. In spite of the progress they still have a lot of work to do but I can say it’s become much more relaxing to use and the ‘wife acceptance factor’ has improved dramatically. As I posted earlier, FSD drove me all the way home from Grand Forks to Minneapolis and it drives me home from work every day, usually with no interventions so it’s definitely useful for me.

 

[powertoold]

Not according to disengagement metrics. Tesla is just getting started.

I know some say disengagement rate (which is an industry standard metric) doesn't matter for Tesla ... since this is an extraordinary claim, the burden of proof is on them. And of course what's the other metric to use instead ?

I don't place a large emphasis on disengagements as a measure of progress.

The reason is that disengagements can be placed into categories.

Let's say that FSD is perfect everywhere except it always runs red lights. That means you're disengaging every 1-5 minutes. Then red lights get fixed 100%. Now you're disengaging never.

What I'm looking at is, based on this approach and progress in this approach, are there any fatal flaws?

 

[Knightshade]

I don't place a large emphasis on disengagements as a measure of progress.

The reason is that disengagements can be placed into categories.

Let's say that FSD is perfect everywhere except it always runs red lights. That means you're disengaging every 1-5 minutes. Then red lights get fixed 100%. Now you're disengaging never.

What I'm looking at is, based on this approach and progress in this approach, are there any fatal flaws?

<NickFuryVoice>
Running right lights can be fatal
</NickFuryVoice>

 

[Porkalletta]

Oh - that is a NHTSA thing. I don't think Tesla is getting rid off this anytime soon. It has reduced a lot in 11.x, though ...

BTW, interestingly the nag frequency increases when its raining.

And every 30 seconds when it’s dark vs. 3 mins during day. Night makes sense as that’s when you would be more likely to fall asleep.

 

[jebinc]

<NickFuryVoice>
Running right lights can be fatal
</NickFuryVoice>

And so can attempts to pass a loading school bus at speed….

 

[Porkalletta]

That is the model used by SOME software. It's not the model used by ALL software.

If you buy an OEM PC with an OEM version of windows for example, that window license is tied to THAT computer.

Same with OEM Office, and many other examples that are tied to 1 specific piece of hardware- much like FSD is tied to 1 specific car.

You mean that the OEM software is free with the computer or do you pay for it separately?

 

[jebinc]

No doubt thats the reason Tesla and SpaceX are always in top 5 most desired companies to work for (among engineers)

That’s a fair point, however that feeling may not be uniform across all Tesla teams. “One size does not fit all”. Take, for example, the Optimus team vs the FSD team. I suspect morale is not higher in the FSD team. The prior head of overall AI quit last year - reportedly because he was tired of Elon’s antics and demands. We also have that one disgruntled FSD engineer that when public on how Elon faked that early FSD video, when in fact the car crashed into a fence. Who knows….

 

[EVNow]

You mean that the OEM software is free with the computer or do you pay for it separately?

You may have to pay for OEM office separately.

 

[EVNow]

That’s a fair point, however that feeling may not be uniform across all Tesla teams. “One size does not fit all”. Take, for example, the Optimus team vs the FSD team. I suspect morale is not higher in the FSD team. The prior head of overall AI quit last year - reportedly because he was tired of Elon’s antics and demands. We also have that one disgruntled FSD engineer that when public on how Elon faked that early FSD video, when in fact the car crashed into a fence. Who knows….

No doubt the engineers will get burnt out soon in Tesla. I'm surprised Carpathy lasted so long. But engineers still want to work in Tesla and SpaceX because the work is challenging and is great on the resume.

Heck - there are a TON of other jobs in the valley that will equally burn one out .... without the prestige attached to the job.

 

[jebinc]

No doubt the engineers will get burnt out soon in Tesla. I'm surprised Carpathy lasted so long. But engineers still want to work in Tesla and SpaceX because the work is challenging and is great on the resume.

Heck - there are a TON of other jobs in the valley that will equally burn one out .... without the prestige attached to the job.

Do you know anyone who works there?

 

[EVNow]

I don't place a large emphasis on disengagements as a measure of progress.

The reason is that disengagements can be placed into categories.

What I'm looking at is, based on this approach and progress in this approach, are there any fatal flaws?

The correct way to use the disengagement metric would be to categorize them according to severity and only compare the safety ones to accident rates. For Tesla this would be a problem given the volume of disengagements - and ofcourse they don't publish any stats on that for us to figure out anything too.

Let's say that FSD is perfect everywhere except it always runs red lights. That means you're disengaging every 1-5 minutes. Then red lights get fixed 100%. Now you're disengaging never.

But thats not how it works in real-life. There would be a "long tail" of problems with very large number of reasons for disengagement and they have to solve each of those and make sure none of the earlier "solved" ones regress. That is why the industry experience is that it becomes exponentially more difficult to decrease the disengagement rate - but Elon thinks it becomes exponentially easier.

 

[Knightshade]

You mean that the OEM software is free with the computer or do you pay for it separately?

It definitely not "free"

OEM Windows is going to be built into the cost of the BOM for the system- the OEM paid something for it, they're going to mark up the PC to account for that-- further, they'll often offer paid upgrades for example a system that default comes with the Home edition might offer you an option to pay a specific additional amount to get OEM Pro loaded instead.

Likewise OEM Office is going to be a separate fee.

The fee will be lower than the retail price-- because you're not getting a retail version.

The way Tesla sells FSD is precisely like OEM Microsoft (and other vendors) software-- it's tied to THAT hardware.

Unlike Microsoft they've not chosen to ALSO offer a more expensive, retail, "portable" version.

 

[powertoold]

But thats not how it works in real-life. There would be a "long tail" of problems with very large number of reasons for disengagement and they have to solve each of those and make sure none of the earlier "solved" ones regress. That is why the industry experience is that it becomes exponentially more difficult to decrease the disengagement rate - but Elon thinks it becomes exponentially easier.

I'm in the exponentially easier camp, once the proper foundation / approach is achieved. FSD progress has been riddled with local minima. We'll see once 11.4.1 is released.

The only possible "fatal flaw" I can think of right now is when to overtake cars vs not, but this decision has been improving a lot with 11.3.6. I've only had 1 mistake so far, whereas in the past, it'd do it more often. The great thing with Tesla is they will get all the edge cases related to this decision, because it's very obvious in hindsight.

 

[Artful Dodger]

My car wanted to drive into the left lane at unprotected left turn. A turn that it has done flawlessly numerous times. Today, however, it made this stupid mistake. I don’t know why…

It's likely because FSD still has mulitple neural nets making the final driving control decision. Elon had this back to say about FSD v11.3 on Jan 14, 2023:

Elon Musk on Twitter: "@teslaownersSV @Teslarati @JohnnaCrider1 Many small things. We’re starting to make use of neural nets for vehicle navigation & control, not just vision." / Twitter​

Telsa is still in the process of cutting over all its legacy C++ code into neural nets, and then making those NNs play nice with eachother. Expect it to have some wrinkles, they're working on it.

This is a consequence of the ongoing effort to create a "unified vector space" for FSD (see #348,760) IE: with 11.3.x FSD creates two vector models of the world, one for static objects, and another for moving objects. It's easy to imagine how any miscategorization of a moving object into the static space could result in sudden lurchs and abrupt manoeuvers. Elon explains the potential errors that could arise in this discussion:

(41:45) FSD Beta Development "Unified vector space" | TSVOC on Youtube (Jun 22, 2022)

Again, they're working on it. Tesla will add 10x compute power to its training cluster in 2023, and plans a further 10x increase in 2024. They'll get there (did I mention it takes time?)

Bottom line is IT'S STILL BETA, YOU ARE THE DRIVER. and you are responsible, regardless. If however, you think you are NOT able to properly supervise the beta features, please turn off FSD beta and drive manually (see above).

Cheers!

 

[aronth5]

That’s a fair point, however that feeling may not be uniform across all Tesla teams. “One size does not fit all”. Take, for example, the Optimus team vs the FSD team. I suspect morale is not higher in the FSD team. The prior head of overall AI quit last year - reportedly because he was tired of Elon’s antics and demands. We also have that one disgruntled FSD engineer that when public on how Elon faked that early FSD video, when in fact the car crashed into a fence. Who knows….

Speculating on morale in Tesla's development teams has no value IMO.

 

[VanFriscia]

It definitely not "free"

OEM Windows is going to be built into the cost of the BOM for the system- the OEM paid something for it, they're going to mark up the PC to account for that-- further, they'll often offer paid upgrades for example a system that default comes with the Home edition might offer you an option to pay a specific additional amount to get OEM Pro loaded instead.

Likewise OEM Office is going to be a separate fee.

The fee will be lower than the retail price-- because you're not getting a retail version.

The way Tesla sells FSD is precisely like OEM Microsoft (and other vendors) software-- it's tied to THAT hardware.

Unlike Microsoft they've not chosen to ALSO offer a more expensive, retail, "portable" version.

I agree 100% with everything you said. I still think it sucks that FSD is non-transferrable . Pretty sure it's because the software clearly isn't finished by any stretch of the imagination. Once they have an actual, finished product to sell that's no longer in beta, then this hardware-locked approach might make more logical sense. As it stands, Tesla has a lot of customers feeling burned on their FSD purchase, and others hesitant to upgrade their vehicles because they don't want to pay for FSD again.

 

[LowlyOilBurner]

I agree 100% with everything you said. I still think it sucks that FSD is non-transferrable . Pretty sure it's because the software clearly isn't finished by any stretch of the imagination. Once they have an actual, finished product to sell that's no longer in beta, then this hardware-locked approach might make more logical sense. As it stands, Tesla has a lot of customers feeling burned on their FSD purchase, and others hesitant to upgrade their vehicles because they don't want to pay for FSD again.

It sucks it’s not transferable, for sure. I’d even be ok with paying a “transfer” fee, say a couple of grand or something. I’ve bought “FSD” twice, and the next go around, I’ll subscribe. I do find value in FSDb, and use it 98-99% of the time.