FSD v12.x (end to end AI)

[Mardak]

he stated that the combination of the existing NNs and the C++ code was slower than an all NN based solution

Yes, there are many potential bottlenecks for parallel compute such as gathering results from multiple processors whether from NPU, GPU or CPU with some waiting on inter-chip communication to pass on to the next sequential step. The neural network scheduling visualization actually shows the CPUs are the least used but maybe it's because existing C++ control filled it up but not shown because it hasn't been neural networks.

Overall, Tesla was able to significantly speed things up with end-to-end even though NPUs were already close to full utilization with 11.x, so they must have found some optimizations/simplifications to additionally process control related neural network parameters while still showing visualizations.

 

[DrChaos]

But really, my views has nothing to do with Optimus or Tesla. It has everything to do with the advancement of NN models and their implementation in robotics.

I can totally see general purpose robots being used for specialized use in the next couple years just from how quick things are accelerating from academia.

I can definitely see a private company with huge resources being able to build a ML factory to create a unified & general agent, with unlimited compute, large scale data gathering and testing, simulator 1.0 (game engine based), simulator 2.0 (neural based), human labelers, auto labeling, hd mapping, etc.

Basically, utilize all the tools built for SDC for robotic.

I think it's more realistic that the Tesla robot hardware platform becomes a target of other entities software for robotics experimentation and research---Tesla isn't going to outperform the rest of industry and academia on new ideas, but the people who work on ML aren't good at understanding low cost mechanical engineering and manufacturing where Tesla has an advantage.

Tesla would be better off to make Optimus a platform that others can target software to, down to actuator and sensor level.

 

[stopcrazypp]

Speaking on the whole general robotics topic.
I actually think the whole Optimus thing will be a HUGE success compared to "FSD" (which i would refer to as a scam).
Obviously not in the time-line that Elon states.

But really, my views has nothing to do with Optimus or Tesla. It has everything to do with the advancement of NN models and their implementation in robotics.

I can totally see general purpose robots being used for specialized use in the next couple years just from how quick things are accelerating from academia.

I can definitely see a private company with huge resources being able to build a ML factory to create a unified & general agent, with unlimited compute, large scale data gathering and testing, simulator 1.0 (game engine based), simulator 2.0 (neural based), human labelers, auto labeling, hd mapping, etc.

Basically, utilize all the tools built for SDC for robotic.

https://twitter.com/x/status/1742605699174236663

https://twitter.com/x/status/1742603121682153852

There was a similar thing that happened with robot quadripeds. Boston Dynamics basically had a monopoly on the most capable ones, with their least expensive model, Spot released at $75,000.

I remember watching a video that pointed out after US academia created open source versions (MIT's Mini Cheetah in 2019, and Stanford's Pupper in 2021), the Chinese startups started to release a flood of them (of course all claiming they independently developed everything). Now you can buy fairly inexpensive ones that are mass manufactured (you can get one for less than $500 on AliExpress).

A similar thing can still happen for humanoid robots.

 

[AlanSubie4Life]

just looking at the car display showing 0mph for however long

Case in point look at latest FSD v11 release and sense how this is done. It is very poorly implemented! It applies friction brakes from 6-2mph. Then releases them and painfully and slowly regens to 0mph.

I don’t think this is optimal, nor is it compliant with local customs (meaning it is not what drivers behind expect).

They should a) start regen at the exact right point - this is trivial for human; our physics engine can just sense the right time quite easily b) regen with max regen to 2mph and c) use friction brakes from 2mph to 0mph until the display shows 0mph and d) start moving again when clear.

Anyway, it's all messed up, it's one of the worst things about FSD beta, and it seems different than what they do at traffic lights (though with 11.4.9 I think they may have broken that again too with a return of oscillatory stopping behavior).

 

[DanCar]

Case in point look at latest FSD v11 release and sense how this is done. It is very poorly implemented! It applies friction brakes from 6-2mph. Then releases them and painfully and slowly regens to 0mph.
...

Oddly enough, I noticed something similar with a waymo ride I took. My preference would be that both cars slow down in a slower smoother fashion. What is the point of racing to a red light?

 

[Mardak]

They should a) start regen at the exact right point - this is trivial for human; our physics engine can just sense the right time quite easily b) regen with max regen to 2mph and c) use friction brakes from 2mph to 0mph until the display shows 0mph and d) start moving again when clear.

Do we know if FSD Beta has ever differentiated between regen vs friction brakes? Even if not, potentially end-to-end would end up copying the common behaviors of human drivers if the training data already generally does what you describe. Although given the range of vehicles from Model 3 to Cybertruck potentially towing, I wonder if there needs to be special control to handle differences or if 12.x could realize it's slowing down too much or not enough to smoothly adjust dynamically.

 

[AlanSubie4Life]

Do we know if FSD Beta has ever differentiated between regen vs friction brakes?

I don’t think so. They just have a desired target deceleration and they do it.

This needs to be fixed. Don’t need to differentiate. Just need the correct profile.

It is way worse than most human drivers. No one slows down this way.

There should not be anything special needed for other vehicles. Just slow down correctly and use the brakes if necessary to match the desired profile.

FSD should take into account slope though - just like a human - and allow longer/shorter stopping distances. It needs to do this for safety reasons anyway (emergency stops take longer on a downhill).

 

[kpanda17]

Code chat
Most of is are currently on 44.30.7/11.4.9
The Tesla insiders testing FSD v12 are on 30.10
Like we saw in December, Tesla moved us from the train with v12, to 44
Imagine we get 30.10 or 11 with FSD 12 soon
Or they take us out of 30 and to a new train, sticking with 11.4.9
Crazy code steering

 

[DrChaos]

Case in point look at latest FSD v11 release and sense how this is done. It is very poorly implemented! It applies friction brakes from 6-2mph. Then releases them and painfully and slowly regens to 0mph.

This is probably a pro-safety measure to have more predicted time available before coming close to an oncoming car, under the likely supposition that distance and velocity estimates will have some error. It's not as fast or good as people.

 

[AlanSubie4Life]

This is probably a pro-safety measure to have more predicted time available before coming close to an oncoming car, under the likely supposition that distance and velocity estimates will have some error. It's not as fast or good as people.

It is possible. But it happens when there are no vehicles present.

 

[MP3Mike]

Just slow down correctly and use the brakes if necessary to match the desired profile.

You act like there is only one "correct" way... I think FSDb currently does pretty good.

For example:

They should a) start regen at the exact right point - this is trivial for human; our physics engine can just sense the right time quite easily b) regen with max regen to 2mph and c) use friction brakes from 2mph to 0mph until the display shows 0mph and d) start moving again when clear.

I don't touch the brake pedal at all, I regen all the way to 0 if I am driving myself. (Just like FSDb does.)

 

[enemji]

Case in point look at latest FSD v11 release and sense how this is done. It is very poorly implemented! It applies friction brakes from 6-2mph. Then releases them and painfully and slowly regens to 0mph.

I have not seen that yet

 

[AlanSubie4Life]

I have not seen that yet

Keep an eye out! You’ll see it.

My particular test was at night, no other vehicles, unmarked residential streets.

 

[AlanSubie4Life]

You act like there is only one "correct" way... I think FSDb currently does pretty good.

This is just quantifiable. You know that perfectly well. There’s huge effort put into this stuff for passenger comfort for planes, etc.

There is clearly a better and worse way to obtain a good stop. There definitely are ways that are more correct than others. And FSD doesn’t do it right now (it was borderline acceptable on the last release).

Note in the past my complaints have been validated by Tesla specifically acting to address said behavior, in the release notes.

The beatings (of FSD Beta) will continue until it improves.

I’ll say when it improves, and I’ll say when it regresses, and will provide detailed description of the exact behaviors. Sometimes with video if I have time.

 

[archae86]

I’ll say when it improves, and I’ll say when it regresses, and will provide detailed description of the exact behaviors. Sometimes with video if I have time.

Here is a respectful criticism and suggestion. Rather than devote so much energy to GoPro video, that you soon upgrade by equipping your vehicle with an aftermarket add-on CAN bus connector, a suitable OBD2 to Bluetooth adapter, and some device running the Scan My Tesla (for Android) or the slightly differently named iOS version.

Once you have that working, you need to adjust the displayed parameter to include Max Regen (as reported in real-time) and estimated brake temperatures.

Then you need to widen your application scope to include low temperature and meaningful downgrades (say 1000 feet elevation loss in 4 mitesl).

If you do that and pay attention, you'll learn a lot you don't currently know about the reality of situational maximum regen, and will stop saying silly things like "it should just go to max regen for ....). My max regen varies all the way down from the absolute configuration max of 85 kW down easily to below 20 kW, and with somewhat less common circumstances as low at 3 to 5 kW.

I sincerely hope you'll upgrade your observations, criticisms and findings along these lines--or some other method to obtain similar data.

Why brake temps? In my opinion, there is a world of difference between my brakes rising 1C as the result of a light touch near the end of an otherwise successful decleeration, vs. rising 100C because really significant braking happened giving large energy waste and brake wear.

 

[AlanSubie4Life]

Then you need to widen your application scope to include low temperature and meaningful downgrades (say 1000 feet elevation loss in 4 mitesl)

Yeah I am well aware of regen limitations due to past regen use. San Diego is actually extremely hilly due to its unique geography. See it all the time.

If you do that and pay attention, you'll learn a lot you don't currently know about the reality of situational maximum regen, and will stop saying silly things like "it should just go to max regen for ....). My max regen varies all the way down from the absolute configuration max of 85 kW down easily to below 20 kW, and with somewhat less common circumstances as low at 3 to 5 kW.

I will learn nothing from such a setup.

Specifically under the exact same circumstances on the same drive I can come to normal stops without using the brakes at all (and with the blended regen and friction braking option turned off of course - not that it matters, because I do not care whether friction brakes are used except for the final 2mph-0mph where it makes things WAY faster to use them). The point is that I can make smooth stops.

You don’t need to know anything about CAN bus data to be able to tell when regen is limited (as long as you have blending off). It’s very obvious indeed. Both in the obvious lack of braking power and the slight additional noise the car makes as it operates the motors inefficiently to warm things up (similar but not identical to supercharger preconditioning).

In my opinion, there is a world of difference between my brakes rising 1C as the result of a light touch near the end of an otherwise successful decleeration, v

I have no concerns about my brakes. I think you’ve entirely missed my point. It is irrelevant whether the car uses brakes or not. But obviously to maintain good comfort which we are used to it should not be necessary in most circumstances.

Except for 2mph and below where it is probably essential for quick stops. Which is where it is not being used. It’s backa**wards.

 

[archae86]

I have no concerns about my brakes. I think you’ve entirely missed my point.

Golly, you have no interest in whether your detailed commentary seriously misinforms people who take it literally. I had thought better of you. I'll desist on this matter. I can see you have zero interest in my point.

 

[AlanSubie4Life]

Golly, you have no interest in whether your detailed commentary seriously misinforms people who take it literally. I had thought better of you. I'll desist on this matter. I can see you have zero interest in my point.

Yes. I don’t have zero interest in the point of having detailed instrumentation of car parameters, I just know it won’t be useful in this particular situation:

with somewhat less common circumstances as low at 3 to 5 kW

FWIW a Model 3 traveling 10mph stopping in 2 seconds on a level surface (faster than what is happening here) would require 9kW of stopping power on average. 6kW is probably enough (3 seconds). Anyway it is just to kind of ballpark what is required, but it doesn't really matter since I actually think friction brakes are required to solve this problem.

I am sorry I have potentially misled. This should be taken to the v11 thread anyway, where I think the issues have been more clearly described.

I just do my best to describe the issues I see. Premature friction braking is just a symptom of the issue, not the issue. It’s fairly common for people to misinterpret the complaint as excessive brake use. But it is not the primary issue, as I said. It's all about the profile, and it doesn't matter how it is accomplished.

Can discuss in v11 thread. Hopefully v12 makes progress on this - that is the only way it is even related.

 

[drtimhill]

There’s huge effort put into this stuff for passenger comfort for planes, etc.

Er, when was the last time you were on a plane as it landed???

 

[drtimhill]

There is clearly a better and worse way to obtain a good stop. There definitely are ways that are more correct than others. And FSD doesn’t do it right now (it was borderline acceptable on the last release).

Note in the past my complaints have been validated by Tesla specifically acting to address said behavior, in the release notes.

The beatings (of FSD Beta) will continue until it improves.

So, are we really at a point where the biggest issue with FSDb is how comfortably it slows?