FSD v12.x (end to end AI)

[Eno Deb]

As the CEO of the only tech company in the world with an ADAS working on city streets in an entire country, Elon’s credibility is very high.

That’s not marketing. City Streets ADAS is SHIPPING FUNCTIONALITY WAY ahead of everyone else.

That could mean that they are technologically ahead, or just that they are more reckless in releasing unfinished functionality ...

Measured by Musk's promises, what they released simply doesn't work. Here's a quote I ran across earlier today:

" The car will be able to find you in a parking lot, pick you up, take you all the way to your destination without an intervention this year. I'm certain of that. That is not a question mark. It will be essentially safe to fall asleep and wake up at their destination towards the end of next year."

That was in February 2019 ...

 

[Mardak]

Since the approach is dependent on video training, will overfitting be an issue?

Compared to traditional control logic? Sure, the neural network could mispredict a behavior because it thought the current situation was like something else. Karpathy gave an example in some presentation that without intentional diversity of lane lines, the neural network would likely predict lanes keep going straight because a random sampling of all lines will likely be straight. This overfitting is caused by the neural network picking up on signals that aren't actually related to the actual curvature of lines.

But as they get to rarer edge cases, Tesla will need more and more data to get the same results and progress will slow down

Depends on the type of edge case and if some generalized understanding would have covered it. For example a potential edge case for a double parked car requiring maneuvering around, and a corner case of a similar situation except there's oncoming traffic, so it's not safe to maneuver around. Would generally knowing not to drive into oncoming traffic be a strong enough signal to prevent maneuvering around the parked vehicle? Even if the control network ended up at the correct behavior, it might not be smooth, so additional training specifically for the corner cases could make things better and potentially collected in an automated fashion.

In the US, different areas have different road infrastructure, different driving behaviors, different traffic rules. Won't be hard to truly generalize one NN to handle all of those differences?

Why wouldn't a single neural network be able to handle the differences? If you're thinking the size is insufficient to effectively learn the differences from the training data, then Tesla can add more layers at the cost of additional compute and latency. At least right now, it sounds like there's still plenty of space to expand with it theoretically able to run at 50fps. However, increasing the size of the neural networks probably makes it more susceptible to overfitting as the weights in the additional layers want to "do something" and could incorrectly detect patterns that aren't meaningful.

 

[Mardak]

My hunch is that Tesla was looking to expand FSD Beta globally and noticed a lot of bugs from their internal testers. These would be complex to fix while also not regressing the existing driving behavior with extensive testing automation and result in significantly growing the lines of code from 300k+ resulting in poor maintainability.

Switching to end-to-end control requires collecting a lot of high quality data that also acts as a test cases which presumably will be grouped by the type of behavior that is intended to be achieved/fixed. Yes, regressions can still happen, but this no longer requires digging into the code to debug what's going on for each scenario and instead "just" collect more data.

 

[Mardak]

I wonder if Ashok Elluswamy's "two weeks" for releasing v12 neural networks will be accurate? Although might be tricky for people to notice unless maybe watching the download size and/or install time of updates in the next 2 weeks? Presumably Tesla will want to collect a lot of data to evaluate in shadow mode, so maybe if we see a large push of some new software version, it could be the one?

In the next two weeks we're going to release the shadow mode release where we're gonna run this network in the background and then check when, for example in this case, we would have wanted to go but then the driver would not have gone. Then we can get the data back to the labelers, and the labelers say who was correct. That helps. Like we should not have gone, so you don't have to even have to get the intervention; you can just passively observe "what we want to do" and "this is what happened in reality."​

 

[EVNow]

I’m skeptical about what 99% end-to-end means. I think some of the planning is now done by NN instead of code … how much is anyone’s guess.

 

[goRt]

In fact, he refers to a few countries.. And these countries have test mules in all the major cities.

View attachment 968480

Tesla Hired a Lot of ADAS Test Operators in Europe, and We All Know What It Means

Tesla has opened many ADAS test operator positions throughout Europe, hinting at the Full Self-Driving program beginning testing in full force

www.autoevolution.com

BTW, NFI what GA means.

"a few" - not many

GA - General Availability - as in everyone with a licence can access it

RoW - Rest of the World - where we have a stack rushed out 7 years ago after the falling out with MI

 

[AlanSubie4Life]

Silicon chip models that mimic the real thing? It's in our cars..

Not sure how you can say this. Seems very far from clear to me and notably for example our cars have no ability to learn (not saying they should, but it is a notable massive difference in the two entities).

Just an example. In any case I don’t think structurally or functionally there is a lot of evidence that a NN computer is really that similar at all to a human brain.

It’s tempting when you see these networks perform certain tasks way better/faster than humans to conclude that they must be pretty close structurally, but I don’t think that follows.

 

[Bladerskb]

I’m skeptical about what 99% end-to-end means. I think some of the planning is now done by NN instead of code … how much is anyone’s guess.

The stuff coming out of Elon Musk's team in the last couple of days is seriously wacky.

It's cool that you're not falling for the nonsense either. Seems like Musk told his crew to hype up the livestream event, and it's pretty obvious.

Honestly, what they're saying doesn't make much sense and is full of wrong info. Who cares if they're only using 100 watts? It's kind of pointless and just sounds like the latest elon-babble ("useful AI compute per Watt").

Bringing up 36 FPS as if it's a big deal? Other companies with better compute and cameras run their networks way faster.

Saying "sub-human photon-to-control latency" literally makes zero sense. They're really going all-in on the Elon-babble.

'pure vision', 'vision only', 'photon', 'real world ai', 'pure ai', 'useful AI compute per Watt', are all terms coined by Elon which make absolutely zero sense.

Saying they're the only ones doing 'real-world AI inference on int8 quantization' is weird. Like, what's 'real-world AI inference' even supposed to mean?

A bunch of companies using Nvidia chips are doing int8 quantization and they are doing it at scale. Just Xpeng and NIO by themselves have hundreds of thousands of Nvidia cars. Quantized models are so common nowadays. Don't forget about Mobileye either.

Using 8 low resolution cameras with multiple blind-spots is not a flex and certainly nothing you should brag about.
All of the above are either none-sense or actual negatives. I guess with the target audience, making logical sense doesn't matter.

https://twitter.com/x/status/1695516856315150735

 

[diplomat33]

The stuff coming out of Elon Musk's team in the last couple of days is seriously wacky.

It's cool that you're not falling for the nonsense either. Seems like Musk told his crew to hype up the livestream event, and it's pretty obvious.

Honestly, what they're saying doesn't make much sense and is full of wrong info. Who cares if they're only using 100 watts? It's kind of pointless and just sounds like the latest elon-babble ("useful AI compute per Watt").

Bringing up 36 FPS as if it's a big deal? Other companies with better compute and cameras run their networks way faster.

Saying "sub-human photon-to-control latency" literally makes zero sense. They're really going all-in on the Elon-babble.

'pure vision', 'vision only', 'photon', 'real world ai', 'pure ai', 'useful AI compute per Watt', are all terms coined by Elon which make absolutely zero sense.

Saying they're the only ones doing 'real-world AI inference on int8 quantization' is weird. Like, what's 'real-world AI inference' even supposed to mean?

A bunch of companies using Nvidia chips are doing int8 quantization and they are doing it at scale. Just Xpeng and NIO by themselves have hundreds of thousands of Nvidia cars. Quantized models are so common nowadays. Don't forget about Mobileye either.

Using 8 low resolution cameras with multiple blind-spots is not a flex and certainly nothing you should brag about.
All of the above are either none-sense or actual negatives. I guess with the target audience, making logical sense doesn't matter.

https://twitter.com/x/status/1695516856315150735

I think my favorite Elonism is when he said "photons are different". That is pseudoscience nonsense. Maybe he was trying to say that they can't take the V12 trained on HW3's lower res cameras and simply put it on HW4 cars because HW4 uses higher res cameras, hence they need to retrain on higher res cameras for V12 to work right on HW4 cars. But saying "photons are different" is dumb.

https://twitter.com/x/status/1695395665491193917

 

[cliff harris]

Its amazing that this thread is so packed with machine learning geniuses, who find the time on their gold plated yachts to educate us about how Tesla is doing it all wrong. Such a joy to read.

 

[Buckminster]

I think my favorite Elonism is when he said "photons are different". That is pseudoscience nonsense. Maybe he was trying to say that they can't take the V12 trained on HW3's lower res cameras and simply put it on HW4 cars because HW4 uses higher res cameras, hence they need to retrain on higher res cameras for V12 to work right on HW4 cars. But saying "photons are different" is dumb.

https://twitter.com/x/status/1695395665491193917

When you run out of things to have a go at….

 

[hooty]

Demo was a complete disaster. A huge safety disengagement with multiple traffic infractions in just acouple miles of simple driving.
But to the prominent Tesla Faithful it was a declaration of victory.
Just shows how delusional and completely ignorant they are.
Elon could poop out a pile of dung and these Tesla fans would go on a frenzy and lap it up.
It never ceases to amaze me.

https://twitter.com/x/status/1695382425092714713

https://twitter.com/x/status/1695261635479314434

https://twitter.com/x/status/1695317132513599718

https://twitter.com/x/status/1695380463479574663

https://twitter.com/x/status/1695285253181698415

https://twitter.com/R6Alex/status/1695310876918595843

The sad one of the lot is Chuck Cook. Over the last 12 months, he has transformed into one of the starry eyed fanbois. I reckon the social media dollar signs were just too enticing. His latest unprotected left turn video was yet another fail, so that was nice

 

[willow_hiller]

Honestly, what they're saying doesn't make much sense and is full of wrong info. Who cares if they're only using 100 watts? It's kind of pointless and just sounds like the latest elon-babble ("useful AI compute per Watt").

Saying "sub-human photon-to-control latency" literally makes zero sense. They're really going all-in on the Elon-babble.

These are both really simple and straightforward. Do you really not understand them?

Tesla has opted for a low-power, low-profile compute board so that it doesn't take trunk space or spend an appreciable part of an EV's energy. I know you prefer companies that fill the entire trunk with computers and don't care about how much power they draw, but that doesn't invalidate the usefulness of working within constraints.

And let me break down the second part for you. Sub-human = shorter time interval than perceived by people. Photon-to-control = camera inputs in, driving outputs out.

The only explanation for you I can come up with comes from Upton Sinclair: "It is difficult to get a man to understand something, when his salary depends on his not understanding it."

 

[Mardak]

Maybe he was trying to say that they can't take the V12 trained on HW3's lower res cameras and simply put it on HW4 cars because HW4 uses higher res cameras

HW4 cameras are not just higher resolution. They have different filters that block certain photons where before there were clear filters for half of the subpixels/channels. Additionally, the new cameras seem to be more sensitive especially noticeable at nighttime, so it's able to detect more photons even though green get filtered.

All of these differences in photon counts between HW3 and HW4 can affect existing neural network predictions. Sometimes it can be good such as HW4 sampling rate seems to more consistently detect photons from LED traffic signals/signs, but reusing neural networks that don't realize what used to be "any color" photons are now only "green" could be confusing.

 

[diplomat33]

HW4 cameras are not just higher resolution. They have different filters that block certain photons where before there were clear filters for half of the subpixels/channels. Additionally, the new cameras seem to be more sensitive especially noticeable at nighttime, so it's able to detect more photons even though green get filtered.

All of these differences in photon counts between HW3 and HW4 can affect existing neural network predictions. Sometimes it can be good such as HW4 sampling rate seems to more consistently detect photons from LED traffic signals/signs, but reusing neural networks that don't realize what used to be "any color" photons are now only "green" could be confusing.

Yes. The photon count may different, the way the photons are filtered may be different. For those reasons, V12 will need to be retrained for HW4 but the photons are not different. That was a poor choice of words on Elon's part. Just a nitpick maybe on my part.

 

[Captkerosene]

I went to bed Friday as a master of the universe. Today, I'm nothing but a bag of meat. Thanks Elon.

This Sunday, I'm thinking that maybe people should start reading Iain Banks instead of going to church for their spiritual guidance.

 

[Max Spaghetti]

If the system is told that driving on the road is good, why does it have to be told that driving off a cliff is bad?

So, if the system only knows what it has been taught by video, if it has never seen a video of a car going over a cliff (with a negative attached), it won't know that a cliff is bad. It only knows that a road is good, and a cliff is undefined. Presumably undefined means bad. I would be keen to learn more about this.

It also was slow to respond and/or acknowledge the yellow to red stop light in the neighborhood.

So, it's been taught by videos of real drivers. So we wouldn't expect it to react to a light turning yellow any quicker than a human would. If a human takes 1.2 seconds to respond to a yellow light, then that's how long the NN will take, even if the NN can process the yellow in a milli-second (what was it? sub-human photon to control?)

How do we go from here to "better than human"?

 

[animorph]

I'm sure Tesla's AI team, and other researchers as well, have developed their own dialect for describing their work. Elon will pick that up. Engineers can sound pretty obscure at times. In the case of photons, it's not hard to see that HW3 and HW4 will present different video data to the NN input, hence the "photons" are different.

 

[hooty]

I wonder what happens with HW2 cars, since they “have all the hardware necessary for full self driving”, per Elon Musk 2016.
Will be interesting to see if Tesla can reach an average of 1 disengagement per 10,000 city miles, for starters, with this rewrite & that data is independently assessed

 

[Supcom]

So, it's been taught by videos of real drivers. So we wouldn't expect it to react to a light turning yellow any quicker than a human would. If a human takes 1.2 seconds to respond to a yellow light, then that's how long the NN will take, even if the NN can process the yellow in a milli-second (what was it? sub-human photon to control?)

How do we go from here to "better than human"?

I believe the standard is to be safer than humans, not necessarily faster in every situation. In your example, yellow light timing is set with human response times in mind. So, if the car reacts in the same time as a human, it should have no difficulty stopping before the light turns red.

However, if it is desired to respond faster, then Tesla could feed simulation video to the training system with improved reaction times. But this may not be safer if there is a tailgating human with typical human reaction times.