FSD v12.x (end to end AI)

[Daniel in SD]

What's the metric?

90% success. I don’t remember if that meant 90% without interventions or if there was a more strict criteria. I don’t think it ever achieved 90% without interventions with V11. I lost a few beers.

 

[Mardak]

The V12 work is focused on addressing driving on secondary streets, where accident rates roughly triple. I imagine the miles per accident figure will drop significantly unless V12 has solid accident avoidance capabilities.

Practically, FSD Beta hasn't really been used that much on city streets, so most of the recent miles come from highway driving with 11.x and some increases with wider release population. I would guess FSD Beta is around 15M miles/month off highways, so across a fleet of ~450k, that's about 1 mile a day per vehicle. So the near-term effect of V12 will likely be minimal on these safety numbers unless it's so comfortable that it drives up usage on secondary streets.

Tesla did update total Autopilot miles back in January 2023 when releasing 2022 Safety numbers with a mention of "billions of miles of real-world data from our global fleet – of which more than 9 billion have been driven with Autopilot engaged," and I believe the previous milestone was 3B in February 2020, so perhaps the global fleet is closer to a 3B annual Autopilot rate now. The current annualized rate of FSD Beta mileage seems to be around 0.9B, so FSD Beta performance should be moving the overall safety numbers at least for highway driving.

But as we've seen in 12.1.2 videos, the previous 11.x single stack is used for freeways for now perhaps suggesting 12.x isn't ready to take on that driving yet. When 12.x is safer on highways and if Tesla makes this end-to-end driving available to all with Basic Autopilot, we should be able to see significant changes on the overall safety report.

 

[JB47394]

So the near-term effect of V12 will likely be minimal on these safety numbers unless it's so comfortable that it drives up usage on secondary streets.

That was the effect that I was focusing on. If folks start using FSD on secondary roads, the numbers should start to fall as they put their cars in harm's way.

 

[AlanSubie4Life]

What's the metric?

Detailed extensively in 10.69 thread.

90% success. I don’t remember if that meant 90% without interventions or if there was a more strict criteria. I don’t think it ever achieved 90% without interventions with V11. I lost a few beers.

Same metric as before.

No interventions by the driver or other drivers. (Covers creeping at the wrong time or too far, cutting it too close, not accelerating up to speed and forcing traffic to go around or slow down, etc.). Any honking from other drivers is disqualifying.

Missing large easy opportunities (more than five-second gaps).

So basically just normal driving. No mistakes, no weirdness, no interventions.

Using the median is weirdness and very silly, but that’s allowed.

Anyway no change from prior rules in 10.69 thread. That covers minimum number of attempts too.

 

[willow_hiller]

Unfortunately, FSD training hasn't been TSLA's strong suit.

What do you mean by that?

To date, the only output we've seen from Tesla's video data training efforts is the birds-eye-view representation of the world. And perception has always been the strongest piece of FSD.

 

[Daniel in SD]

Detailed extensively in 10.69 thread.

Same metric as before.

No interventions by the driver or other drivers. (Covers creeping at the wrong time or too far, cutting it too close, not accelerating up to speed and forcing traffic to go around or slow down, etc.). Any honking from other drivers is disqualifying.

Missing large easy opportunities (more than five-second gaps).

So basically just normal driving. No mistakes, no weirdness, no interventions.

Using the median is weirdness and very silly, but that’s allowed.

Anyway no change from prior rules in 10.69 thread. That covers minimum number of attempts too.

90% success (March of Nines!) not 100% success though, right?
I can't believe it's been a year and a half since 10.69!

https://twitter.com/x/status/1561429523899482112

(Elon apparently didn't watch the video. LOL)

 

[zoomer0056]

The V12 work is focused on addressing driving on secondary streets

We can only say that the Youtubers are testing here. Speaking for Tesla is pure speculation.

 

[TK211X]

There was chatter that V12 was supposed to start going actually public with a slow roll starting this weekend. Been pretty quiet so far.

 

[Scott Chapman]

There was chatter that V12 was supposed to start going actually public with a slow roll starting this weekend. Been pretty quiet so far.

Probably stuck at a 4-way stop or maybe a rotary

 

[Todd Burch]

There was chatter that V12 was supposed to start going actually public with a slow roll starting this weekend. Been pretty quiet so far.

There was? Only things I heard was:

1. From Elon at earnings call: Starting to roll out in a few weeks.

2. From Omar: Thinks Tesla said they might be opening it up to a few additional public testers this weekend.

Where did you hear going public starting this weekend? Unless you're referring to (2) above.

 

[Supcom]

There was? Only things I heard was:

1. From Elon at earnings call: Starting to roll out in a few weeks.

2. From Omar: Thinks Tesla said they might be opening it up to a few additional public testers this weekend.

Where did you hear going public starting this weekend? Unless you're referring to (2) above.

Wishful thinking, perhaps.

The OG youtuber crowd must be livid that Omar is getting all the V12 views.

 

[sleepydoc]

Ah yes, our wager. In hindsight it was obvious that traditional programming would never be able to negotiate the arc of Chuck’s UPL. There’s just no way to calculate the future positions of all the vehicles with C code!
I will bet a beer that Chuck’s first version of FSD V12 will finally achieve 90% performance. V12 will be the start of the March of Nines!

Is that a Pabst Blue Ribbon, Schlitz, or Miller Highlife?

 

[sleepydoc]

Wishful thinking, perhaps.

The OG youtuber crowd must be livid that Omar is getting all the V12 views.

Well, if they’d do a better job of sucking up maybe they’d get earlier access!

 

[kabin]

What do you mean by that?

To date, the only output we've seen from Tesla's video data training efforts is the birds-eye-view representation of the world. And perception has always been the strongest piece of FSD.

Most of us see and experience it behind the wheel. They add a new feature and we get regressions. And an almost endless number of 'edge cases' that are not resolved over any period of time and data training set.

I think Tesla has acknowledged training challenges and shortcomings. And again, v12 success will likely be even more dependent on optimal training if/when it lacks heuristic sanity checks.

 

[FSDtester#1]

There was chatter that V12 was supposed to start going actually public with a slow roll starting this weekend. Been pretty quiet so far.

It comes tonight at 1 am your local time !!

Except for @jebinc ... Elon just texted me and said he is pushing V12 to him asap. He wants him to be able to drive out of his neighborhood and make it out the gate with no disengagements..

 

[AlanSubie4Life]

90% success (March of Nines!) not 100% success though, right?
I can't believe it's been a year and a half since 10.69!

https://twitter.com/x/status/1561429523899482112

(Elon apparently didn't watch the video. LOL)

Yep 90%. This has never been accomplished with any prior FSD version.

It is an honor to do this again and I do hope I finally have to pay for a beer.

There were some caveats about how much traffic as well (seems silly to give it a pass with extremely light traffic where it is just sailing through every time). Easiest thing would be to just do the results of all attempts on the first version, since even if Chuck does a light traffic video he will follow up with one in heavier traffic (usually). Usually he does not do 10 attempts in one video (our minimum number IIRC) but want to avoid the chance of a trivial 10 attempts being successful, and somehow labeling that a success. I think this is what we did last time.

Anyway once v12 easily accomplishes this hurdle, we can take bets on the second 9. 99 out of 100 attempts would be truly impressive (though quite far from where it needs to be!).

 

[Daniel in SD]

Yep 90%.

It is an honor to do this again and I do hope I finally have to pay for a beer.

There were some caveats about how much traffic as well (seems silly to give it a pass with extremely light traffic where it is just sailing through every time). Easiest thing would be to just do the results of all attempts on the first version, since even if Chuck does a light traffic video he will follow up with one in heavier traffic (usually). Usually he does not do 10 attempts in one video (our minimum number IIRC) but want to avoid the chance of a trivial 10 attempts being successful, and somehow labeling that a success.

Anyway once v12 easily accomplishes this hurdle, we can take bets on the second 9.

I'm looking forward to watching 1000 successful turns when they get the third nine. I wonder what Chuck's manual driving success rate is?

 

[AlanSubie4Life]

I wonder what Chuck's manual driving success rate is?

He's probably done it hundreds of times (including of course his attempts while supervising FSD which also count).

Would be nice to know how many accidents there have been there. Low hundreds of attempts a day, probably, meaning a little less than 100k attempts per year.

Would 1 accident out of 1 million (unsupervised) attempts on this specific turn be good enough? I'm not sure how I feel about that. I guess it depends on the failure mode.

 

[Daniel in SD]

He's probably done it hundreds of times (including of course his attempts while supervising FSD which also count).

Would be nice to know how many accidents there have been there. Low hundreds of attempts a day, probably, meaning a little less than 100k attempts per year.

Would 1 accident out of 1 million (unsupervised) attempts on this specific turn be good enough? I'm not sure how I feel about that. I guess it depends on the failure mode.

Chuck has said he's not aware of any collision at the UPL. Of course our wager isn't the same as collision rate. You could get one 9 for collisions just YOLOing it like this:

Yeah, it's really dependent on failure mode. A side impact to the drivers side at 60mph would be near certain death so 1 in a million of that happening would be bad.

 

[AlanSubie4Life]

In hindsight it was obvious that traditional programming would never be able to negotiate the arc of Chuck’s UPL. There’s just no way to calculate the future positions of all the vehicles with C code!

True! This is an impossible problem for traditional programming. Complex physics calculations like that have never been implemented in code. The NNs will just know and sense when it is the right time to go, based on millions of video clips. Much better than having a concrete, quantifiable idea of how much margin is required. Just trust the AI, it's flawless; we can see that now with the generative AI revolution.

Of course our wager isn't the same as collision rate

True. It's possible Chuck has been honked at before or caused other vehicles to slow (not counting his FSD failures). So maybe 1 out of 100k would be good enough.

just YOLOing it like this:

Eva looked both ways. Totally in control.

The more I think about v12 NNs handling this problem based on millions of video clips, the more terrified I am. Maybe they have a physics engine backstop for trajectories and collision prediction. Could be separate from the planner.