FSD v12.x (end to end AI)

[Mardak]

Tesla can (as near as matters) instantly disable any FSD feature on the cars, so a NHTSA recall wouldn't prevent sales for any length of time

Right, practically Tesla has ways to mitigate the effects of recalls preventing new vehicle sales, so that's something for them to balance how much effort, etc. Now that FSD Capability is no longer "Coming Soon," there have been factory software such as 2023.20.100.1 that included 11.3.6, so presumably Tesla is planning on getting 11.4.x and eventually 12.x merged to the factory versions. So relative to say Mercedes with potential recall affecting new sales (nationwide? with today's announcement $2500/yr subscription), the impact is still limited to how many people would buy cars with Drive Pilot vs all new Tesla vehicles that can get FSD Capability.

NHTSA probably also focuses its attention on Tesla's higher potential impact too where even if the actual recall "only" prevents sales of new vehicles, they probably know Tesla will do a fleet-wide update to improve the safety even for existing vehicles. So given their experience in regulating Tesla's driver assistance features, it seems likely they'll continue to do so for end-to-end and more automation.

 

[Mardak]

The only localization thing I can think of off hand that would really make any difference within those (in the US anyway) is where it is, or is not, ok to go right on red

From the CVPR presentations and V12 livestream, there were examples of prompting end-to-end to change behavior such as "change lane to the right." There could potentially be "system" prompts not from the "user" that Tesla uses to enforce this right-on-red behavior, e.g., "You're in an area that disallows right turn on red at any intersection." This should at least cover cases where FSD already knows how to handle this type of behavior because there are existing intersections that allows and disallows the maneuver.

Potentially driving on the left side could be generally understood even without prompting based on other visual cues as well as general GPS/navigation data. But if not, a similar prompt for "You're in a left-hand traffic area" could work too.

 

[DrChaos]

Tesla can (as near as matters) instantly disable any FSD feature on the cars, so a NHTSA recall wouldn't prevent sales for any length of time.

With a deterministic robotics human written policy, perhaps.

But if its really trained end to end only with video input and human behavior as the training signal, there is no way to programmatically "disable any feature" instantly. It's like disabling a behavior on a non-verbal orangutan driver who has learned by watching humans drive. A human is better at zero shot learning having been linguistically told a concept. Humans understand the meaning of words, the concept it attaches to and what that may mean in driving practice.

Disabling a behavior in end-to-end ML policy nets requires a significant retrain---but even then given the huge datasets, even finding and re-labeling the inputs which exhibit this behavior as going from desirable to undesirable is also very difficult, and that's a prerequisite for the retrain. How would a ML scientist do that? General solution is probably recreating the 300,000 line of C code robotics based policy that you dumped before and simulating/assimilating its internal state from observed data in parallel and using those parameters and behavior and its flags to determine which policy rule you violated when. That's also a huge effort. You don't have the computational limitations of needing to run on-board in real time, but you add on the necessity to assimilate from observed sensors (generalized Kalman filter type ideas) into internal state variables which is also hard.

 

[DrChaos]

Localisation has to do with culture among other things. People drive differently in NYC, SF, Milan, Nairobi, Shanghai and Tokyo. So retrain model for every market then using a completely different training set? Doesn't sound very scalable to me.

actually for L2+++ that is the scalable approach compared to trying to hand code those. FSD doesnt even try to work outside North America. Getting training data for those regions from existing cars and balancing the train set for their networks to emphasize those regions (probably not exclusively but biased) is easier than hand-rewriting policy code.

It will seem like progress, but may be regress for actually getting to safe L4.

 

[mongo]

With a deterministic robotics human written policy, perhaps.

But if its really trained end to end only with video input and human behavior as the training signal, there is no way to programmatically "disable any feature" instantly. It's like disabling a behavior on a non-verbal orangutan driver who has learned by watching humans drive. A human is better at zero shot learning having been linguistically told a concept. Humans understand the meaning of words, the concept it attaches to and what that may mean in driving practice.

Disabling a behavior in end-to-end ML policy nets requires a significant retrain---but even then given the huge datasets, even finding and re-labeling the inputs which exhibit this behavior as going from desirable to undesirable is also very difficult, and that's a prerequisite for the retrain. How would a ML scientist do that? General solution is probably recreating the 300,000 line of C code robotics based policy that you dumped before and simulating/assimilating its internal state from observed data in parallel and using those parameters and behavior and its flags to determine which policy rule you violated when. That's also a huge effort. You don't have the computational limitations of needing to run on-board in real time, but you add on the necessity to assimilate from observed sensors (generalized Kalman filter type ideas) into internal state variables which is also hard.

NHTSA: we don't like v12.6.8
Tesla: rolls back to v12.6.7, later releases v12.6.9
Worst case, disables FSD until v12.6.9 .Point is sales don't get stopped.

 

[Knightshade]

With a deterministic robotics human written policy, perhaps.

To be clear the recall was about Tesla writing static code to allow the option of doing illegal rolling stops at stop signs.

If instead they'd trained end to end on people always stopping completely at stop signs it would've never had an issue.

So your one example doesn't require a retrain fix at all unless they trained it completely wrong the first time because that problem would never exist.

 

[DrChaos]

NHTSA: we don't like v12.6.8
Tesla: rolls back to v12.6.7, later releases v12.6.9
Worst case, disables FSD until v12.6.9 .Point is sales don't get stopped.

2029: NHTSA we dont like V14-18 behavior. Stop doing X,Y,Z.

Tesla: uh, that's our entire software architecture that we've developed over 6 years. We don't know how to stop specifically X and Y and Z since drivers do that.

NHTSA: tough *sugar*

Tesla: does anyone know how to get V13 built back on our current platform?

Engineers: the people who worked on it haven't worked here for 4 years. That old build isn't compatible with our current car software anyway and the hardware sensors are different now anyway so it wouldn't work.

NHTSA: turn it all off then

Tesla: *sugar*

 

[DrChaos]

To be clear the recall was about Tesla writing static code to allow the option of doing illegal rolling stops at stop signs.

If instead they'd trained end to end on people always stopping completely at stop signs it would've never had an issue.

The problem is that they tuned the behavior from observations on real data and people don't stop completely at stop signs: NHTSA forced them to do something other than what the emprical data said. This problem will grow with more end-to-end training.

So your one example doesn't require a retrain fix at all unless they trained it completely wrong the first time because that problem would never exist.

Thats the point, when "training it wrong" was the natural thing to do when that is training from observations.

 

[Knightshade]

The problem is that they tuned the behavior from observations on real data and people don't stop completely at stop signs:

Right.... the equivalent is they trained on people literally breaking the law on video

Seems unlikely they'd repeat that mistake.

 

[Mardak]

To be clear the recall was about Tesla writing static code to allow the option of doing illegal rolling stops at stop signs

It'll be interesting if there's some frequency threshold of the illegal behavior that deserves a recall. With the static code, presumably the rolling stops happened 100% of the time in a given setup that perhaps NHTSA was able to reproduce consistently in their own testing. How much end-to-end training of desired behavior will satisfy NHTSA? Given that Tesla has pushed back before, I wouldn't be surprised if Tesla tries to find a more reasonable limit with end-to-end.

What if rolling stops happened less than half of the time? 10%? 1%? What if the stop isn't at the stop line as legally required as people often roll past to avoid the awkward stop-go-stop/check behavior?

 

[DrChaos]

Right.... the equivalent is they trained on people literally breaking the law on video

Seems unlikely they'd repeat that mistake.

Here's the problem. You have 10^8 video clips and ego telemetry. You used this for FSD 15 end to end. Training took 5 months and $50 million of electricity and occupied $1B of semiconductors for that time.

Which of them has somebody "literally breaking the law"? Which law? All of them. Go.

 

[KArnold]

Even with E2E, couldn't they leave FSD speed controls as they are today and hardcode full stops at stop signs/red lights?

I'm not coming up with much else that the masses do every day but would be a problem with the NHTSA on a working FSD hypothetical.

 

[EVNow]

With end-to-end training on curated human driving examples, theoretically the car should behave much more human-like somewhat similar to how ChatGPT not only used text from humans but also human feedback to bias its responses even more

If Chat GPT made an error once in 100 responses, it would be a non issue. But if FSD made one error every 100 miles or one every 100 lane changes - it would be a big deal.

All disengagements are obviously not critical - but to get a good count of miles per disengagement, they have to analyze all the disengagements. A tall order. Or may be they use statistically significant sampling.

Either way I find it interesting that internally disengagement rates are used so much internally but never talked about externally - even though it's the industry standard.

 

[uscbucsfan]

If Chat GPT made an error once in 100 responses, it would be a non issue. But if FSD made one error every 100 miles or one every 100 lane changes - it would be a big deal.

All disengagements are obviously not critical - but to get a good count of miles per disengagement, they have to analyze all the disengagements. A tall order. Or may be they use statistically significant sampling.

Either way I find it interesting that internally disengagement rates are used so much internally but never talked about externally - even though it's the industry standard.

The current disengagement rate is bad and you can see this in the community sourced tracking.

They will talk about it when it's something respectable.

 

[Krash]

…The only localization thing I can think of off hand that would really make any difference within those (in the US anyway) is where it is, or is not, ok to go right on red…

Allowance of U turns varies by region.

 

[DrChaos]

Even with E2E, couldn't they leave FSD speed controls as they are today and hardcode full stops at stop signs/red lights?

No. The E2E model creates its own policy diffused implicitly across neural networks. We don't know where it is any more than someone can look into your grey goo on your head and change a specific behavior. There's no known way to "Put that stuff in". All we get is video in and control outputs out.

 

[Kablooie]

From the description Elon gave on the livestream this is exactly the opposite of what they'd have to do.

it doesn't know "you stop at a stop sign which is an octagonal red thing because the law says you do that and you have to stop in X way legally"

Instead it's shown millions of clips of good drivers following the law, and copies what they did in the same situation. It doesn't know what a "stop sign" is, let alone the laws around them.


So if a law changes that'd impact driving they'd need to retrain the system with lots of clips of good drivers obeying the new law.

That DOES raise the fun point that if you can think of a situation where there ARE no such clips until after the law is passed because nobody drives that way until after the law is passed they'd be unable to fix it until well after said law was passed... but I'm not sure I can think of an example off hand of a law they might change where there's NO existing footage of correct behavior somewhere?

California has such a law right now. This year a driver must change lanes if possible when encountering a bicyclist in your lane.

 

[Kablooie]

With a deterministic robotics human written policy, perhaps.

But if its really trained end to end only with video input and human behavior as the training signal, there is no way to programmatically "disable any feature" instantly. It's like disabling a behavior on a non-verbal orangutan driver who has learned by watching humans drive. A human is better at zero shot learning having been linguistically told a concept. Humans understand the meaning of words, the concept it attaches to and what that may mean in driving practice.

Disabling a behavior in end-to-end ML policy nets requires a significant retrain---but even then given the huge datasets, even finding and re-labeling the inputs which exhibit this behavior as going from desirable to undesirable is also very difficult, and that's a prerequisite for the retrain. How would a ML scientist do that? General solution is probably recreating the 300,000 line of C code robotics based policy that you dumped before and simulating/assimilating its internal state from observed data in parallel and using those parameters and behavior and its flags to determine which policy rule you violated when. That's also a huge effort. You don't have the computational limitations of needing to run on-board in real time, but you add on the necessity to assimilate from observed sensors (generalized Kalman filter type ideas) into internal state variables which is also hard.

Alternatively the AI could be trained to understand and interpret human language. Not an easy task to add to the self driving system, I’m sure, but it would allow the system to be modified if needed.

 

[spacecoin]

Either way I find it interesting that internally disengagement rates are used so much internally but never talked about externally - even though it's the industry standard.

The only reason they're not talking about this KPI publicly is because it's more or less flat for three years and at a level that's about 4 orders of magnitude away from autonomy...

 

[willow_hiller]

No. The E2E model creates its own policy diffused implicitly across neural networks. We don't know where it is any more than someone can look into your grey goo on your head and change a specific behavior. There's no known way to "Put that stuff in". All we get is video in and control outputs out.

That's like saying: "There's no way to prompt ChatGPT. It just regurgitates the text it was trained on, and we have no way of influencing what text it generates next."

Of course there's a way to "put that stuff in." How else does navigation work? It needs to be able to be prompted to go straight, turn left, turn right, etc. Just like those navigation inputs can be added, so too can SD map information.