FSD v12.x (end to end AI)

[fasteddie7]

Why are they continuing to update the v11 branch when the v12 branch won’t use any of the v11 coding?

Based on the last few releases, “update” is a strong word. Bugfixes, maybe, but there have been very few changes from version to version for me (HW4 Model S) and most changes seem to come as map data changes daily without an actual firmware update, other than vision parking and some QoL infotainment updates built into the releases. Elon did just say 11.4.7 has a bunch of AI features so maybe there’s some training data in there that 12 will use,

 

[goRt]

Why are they continuing to update the v11 branch when the v12 branch won’t use any of the v11 coding?

Good point, the RoW demands the new code hase

 

[spacecoin]

Why are they continuing to update the v11 branch when the v12 branch won’t use any of the v11 coding?

Because it's 6-12 months out?

 

[Acisplat]

Why are they continuing to update the v11 branch when the v12 branch won’t use any of the v11 coding?

It's going to be a while before FSD v12 and they probably want to make sure everyone with FSD v11 is running the safest version possible until then

 

[DrChaos]

My theory is that good drivers all drive the same way and bad drivers drive badly in different ways from one another, different bad habits and behaviors. There is more diversity for bad drivers. So the data points for good drivers are all clustered close together, while bad drivers are scattered all over the place. Its easier to draw a box around the good drivers.

That's a very good self-supervised heuristic, I like it.

Now the problem is "draw a box around the good drivers". As in literally---if you have no idea about the structure of the internal representations because it's all been trained end-to-end, how do you make a metric on something to cluster them in a useful way? The drivers won't be all going through exactly the same intersections at the same times of day and seasons with identical conditions. And in high dimensions (as this will be) the nearest neighbor is almost as far as a random neighbor (a property of high dimensional spaces).

 

[Mullermn]

If you want to know why most CEOs hide behind PR teams and marketing and don’t do live streamed pre-release product demos while simultaneously answering questions from the public (aside from the fact that they wouldn’t know what they were talking about), I present you a 25 post debate on whether Musk should be considered a laughing stock because he used the word ‘photons’.

 

[InDaClub]

This is a hyperbolic statement that means absolutely nothing. Literally all AV NN are trained using videos.

CVPR23 best paper from June, E2E autonomous driving using just cameras. The world didn't change then, cameras didn't have a paradigm shift.

Planning-oriented Autonomous Driving using E2E NN.

GitHub - OpenDriveLab/UniAD: [CVPR 2023 Best Paper] Planning-oriented Autonomous Driving

[CVPR 2023 Best Paper] Planning-oriented Autonomous Driving - OpenDriveLab/UniAD

github.com

[2212.10156] Planning-oriented Autonomous Driving

arxiv.org

Tesla is the leader in this space and have been working on autonomy for many years. An in-depth presentation can be viewed in 'Tesla Autonomy Day' from 2019. This is an excellent presentation. Enjoy.

6,339,183 views Streamed live on Apr 22, 2019

 

[Bitdepth]

Tesla is the leader in this space and have been working on autonomy for many years. An in-depth presentation can be viewed in 'Tesla Autonomy Day' from 2019. This is an excellent presentation. Enjoy.

6,339,183 views Streamed live on Apr 22, 2019

1. Not E2E
2. Path prediction was C++ code and not NN based.
3. They’ve since abandoned lots of the works in that video following various rewrites.
4. Tesla is not leader in this space. Watch the AI day video to discover whose works they are going off of. It’s Google and Facebook.

 

[zoomer0056]

twitter

X

 

[Bitdepth]

X

What’s X?

 

[InDaClub]

2. Path prediction was C++ code and not NN based.

You clearly didn't watch the video.

How large are the Facebook and Google fleets?

 

[Bitdepth]

You clearly didn't watch the video.

You clearly do not follow the field and AV as a whole. It wasn't until last year that Tesla moved their planner from physics-based numerical optimization search function to a Neural Planner.

How large are the Facebook and Google fleets?

Thats a stupid question. NN architectures are domain agnostic.

 

[powertoold]

I must admit as a heuristics-based programmer, I have little idea how V12 is architected or works.

If what Elon says is true (V12 is being tested internationally), and his livestreamed drive is what he says it is (end-to-end approached), my mind is blown.

When I think about nothing but NNs, things like GPT4 come to mind. But GPT4 has a tendency to imagine things or make things up (hallucinate), and it's not great at being accurate or precise, which are important for self-driving.

However, one of the main reasons why GPT4 hallucinates is because the input parameters (human prompts) are very very low data and sparse inputs.

Why end-to-end FSD is less likely to hallucinate is that the input parameter (video) is a high bitrate constant stream of data. This means that the NN is constantly providing an ever-evolving output, leading to higher quality and accurate predictions.

 

[Knightshade]

Because it's 6-12 months out?

Also because it might not be better.

Right now they THINK it will be much better, but the fact it's still far from public release tells us that's not true today.

It's possible that's ONLY because they're compute constrained on training and some months in the future it'll get better than 11.x generally and we'll get a public release and then updates from there.

It's also possible they run into some limitation or local maximum they did not expect (as they've done many times before when they thought they had "the solution") and they have to instead leave 11.x as the public system until they come up with the next "the" solution idea. Or possible they find they can't dumb the computer-on-car needed enough to get >L2 on HW3 after all but maybe can on HW4 eventually.

Nobody, including Tesla, knows the answers to these today though.

Nobody, including Tesla, WILL know how much HW, and what type of SW, solves generalized self driving L4 or better until they actually solve it

 

[keeehlan]

Still on 11.4.4.

Not even getting a software update, let alone FSD.

Weren’t we all supposed to be on a single build by now?

 

[uscbucsfan]

Weren’t we all supposed to be on a single build by now?

No. Nothing was communicated about a single build by X date and even if it was, dates and timelines aren't Elon's strong suit as discussed in the other thread.

 

[JB47394]

How would you ever safety validate an AI system that is basically "we don't know how but it works like magic".

The same can be said of human drivers. We test them, assume that they will operate reliably, then send them on their way. It is when they go off and do something that we call "stupid" that we find out that they had a bug. "Yes, you're supposed to check your blind spot every time"

I don't mean to be dismissive of the problem. I also want to be able to look at a visualization of a neural net system and walk away thinking "Oh, so that's how it knows when it's safe to cross a bike lane to make a turn". I think that's not going to happen for several years. Longer, if people are faked out by nonsense about sentience and consciousness in neural networks. First we'll go down the road of "it's magic" because companies will be able to make money from that. Later, people will have spent enough time taking these things apart that they can begin to figure out exactly how they work, how to precisely control them, etc. It will start in universities and slowly spread into the commercial sector.

I think he's being aspirational again, maybe there's a bit of AI on top of mostly V11 coding.

Where would you draw the line between the two? I can't imagine where a clean break could be placed between a neural network and heuristics when it comes to translating from perception to control. I assume that the remaining traditional code is essentially device driver stuff to convert the neural network outputs into directives to steering, acceleration, signals, and so on.

 

[Mullermn]

First we'll go down the road of "it's magic" because companies will be able to make money from that.

I can’t see there being any chance of it getting certified in Europe without being able to evidence its intent so the driver can intervene if it’s wrong. Bear in mind NOA is not even permitted to make lane changes autonomously here, it has to request the change and have the driver actively confirm it first.

 

[DrChaos]

This is a hyperbolic statement that means absolutely nothing. Literally all AV NN are trained using videos.

CVPR23 best paper from June, E2E autonomous driving using just cameras. The world didn't change then, cameras didn't have a paradigm shift.

Planning-oriented Autonomous Driving using E2E NN.

GitHub - OpenDriveLab/UniAD: [CVPR 2023 Best Paper] Planning-oriented Autonomous Driving

[CVPR 2023 Best Paper] Planning-oriented Autonomous Driving - OpenDriveLab/UniAD

github.com

[2212.10156] Planning-oriented Autonomous Driving

arxiv.org

What "Changed the World" the most is top level research from large entirely PRC teams is the norm. That organization (SenseTime Research) has hundreds of very good ML papers, much more than Tesla.

 

[DrChaos]

You clearly do not follow the field and AV as a whole. It wasn't until last year that Tesla moved their planner from physics-based numerical optimization search function to a Neural Planner.

What is not clear at all is if the neural planner is a distillation of results from an offline (train-time) physics-based numerical optimization done at higher-resolution/computational load, or if the planner is trained only from empirical observations?

The second is implied by Elon but I have major concerns about controllability and safety if you don't have substantial train inputs from the first. And that needs an internal perception vector space that we understand as physics (i.e. the conventional approach). Empirical inputs from drivers can be a good signal about "human sociology" of driving, as we don't fully obey rules rigidly either but interact implicitly with other cars driving given the situation.

The fact of position, momentum velocity, angles etc we know from Newton is a greatly important physical prior that we should be building into the nets---a fair amount of nnet research for some problems is precisely devoted to favoring representations which maintain appropriate physical invariants/laws that we build in.