FSD v12.x (end to end AI)

[Todd Burch]

I read your sun test idea but still don't think the camera sees or computer behind it process as far as human eye can see especially in low light conditions. Example : when you drive in non-lit country roads at night the pillar cameras tell you they are blocked. I can still see the side of the road and trees etc with the ambient light available but the Tesla camera think it is blocked and visualization show me a pitch black picture for the road side.

Not saying vision only can't ever work. All I'm saying is with the current cameras we have onboard our HW4 vehicles the capability is still far from reaching or exceeding human eyes.

There's no way the HW3 or HW4 cameras see anywhere near as well as the human eye. You are absolutely right and I 100% agree with you. I am not claiming that and there's tons of scientific evidence that proves that few cameras in existence can approach the capabilities of the human eye.

What I'm saying is that driving can be done safely with cameras that knowingly have less capability than the human eye. USAF drone pilots regularly fly multi-million dollar UAVs using cameras that have much lower acuity than the human eye, for example.

 

[AlanSubie4Life]

For L4 performance, you have to be able to accurately assess distance and velocity reliably.

For red lights, or yellow lights turning to red lights, I've MANY times seen the car react well in advance of the time required to come to a nice relaxed stop (I'm not saying it came to a relaxed stop (the car doesn't know how to stop), it just reacted with AMPLE margin (then completely f'ed up the profile)). Other times, I have seen the car fail to react for a long time, then suddenly react, slamming on the brakes, etc.

There's no way for us to determine why this is. One possibility is that the reliability of detection is poor (I don't think that's the case for the red light case - I think it is planning - but it is possible the vehicle did not properly perceive the red light on occasion due to lighting, etc.).

Anyway, for the cross traffic case on Chuck's turn, it's very clear the cameras have ample range to perform the turn reliably at least 9/10 times. We've seen it make the right go/no-go decision in recent videos 100% of the time (resulting in 3/7 success rate). Based on many examples of the car's response, they can detect at at least around 150 meters (including some estimate of velocity presumably), sometimes more. But it's really unclear how reliably the cameras and perception system can detect position and velocity at this range.

It's possible that reliability/noise becomes an issue when we get to the 99/100 or 999/1000 level. It seems extremely unlikely to prevent getting to the first 9. You can still have one fatal t-bone collision out of 10 attempts to achieve the first 9 (I guess it has to be the last one)! (I hope Chuck continues to be extremely careful - it's quite a bit more hazardous to supervise the system than it is to drive the turn yourself, as I'm sure we all know from experiencing this.)

I have no idea how we could assess this, because we can't know what happened on any particular failure.

The better the visual system & the perception system, the lower the rate of failure will be. It needs to be really low. We have no idea what is the failure rate currently.

 

[OldManCan]

There's no way the HW3 or HW4 cameras see anywhere near as well as the human eye. You are absolutely right and I 100% agree with you. I am not claiming that and there's tons of scientific evidence that proves that few cameras in existence can approach the capabilities of the human eye.

What I'm saying is that driving can be done safely with cameras that knowingly have less capability than the human eye. USAF drone pilots regularly fly multi-million dollar UAVs using cameras that have much lower acuity than the human eye, for example.

I really want you to be right and my 2023 MYLR to reach L4/Robotaxi without any need for retrofit. I really want Tesla/Elon to be right on this. I just feel the rate of progress is not fast enough.

 

[Todd Burch]

EDIT - I see your post above seems like we were typing about same time. Your points are valid however the issue is what is possible and not possible with what we have as of today in our cars. I for one will be very happy to be proven wrong if Tesla can pull off L4/Robotaxi with the HW4 vision only as that would mean I don't have to replace my car which otherwise I like very much!

Elon and Ashok have seen 12.4 and early 12.5 versions. Elon accelerated RT plans right around the time he would have been seeing 12.4 or early 12.5 results. I'll just say this: I don't think that's a coincidence.

 

[Todd Burch]

I really want you to be right and my 2023 MYLR to reach L4/Robotaxi without any need for retrofit. I really want Tesla/Elon to be right on this. I just feel the rate of progress is not fast enough.

The rate sucked for quite awhile as the team was persuing hard-coded control code. That was never going to work. But they've now seen the light. I would classify the improvement from 11.4.9 to 12.3 to be huge.

 

[JulienW]

......Elon accelerated RT plans right around the time he would have been seeing 12.4 or early 12.5 results.......

.....or could be after seeing the 1st qt results.

 

[AlanSubie4Life]

Elon and Ashok have seen 12.4 and early 12.5 versions. Elon accelerated RT plans right around the time he would have been seeing 12.4 or early 12.5 results. I'll just say this: I don't think that's a coincidence.

Bookmarking this one so we can revisit in a few months!

 

[OldManCan]

Elon and Ashok have seen 12.4 and early 12.5 versions. Elon accelerated RT plans right around the time he would have been seeing 12.4 or early 12.5 results. I'll just say this: I don't think that's a coincidence.

There are numerous threads bashing Elon so I don't mean to add more on it but allow me to say this much. Elon is saying or not saying things based on how he wants to shape the Tesla narrative and stock ticker curve. I think many of us have learned to take things with a grain of salt when it comes from his mouth or X posts.

 

[Todd Burch]

There are numerous threads bashing Elon so I don't mean to add more on it but allow me to say this much. Elon is saying or not saying things based on how he wants to shape the Tesla narrative and stock ticker curve. I think many of us have learned to take things with a grain of salt when it comes from his mouth or X posts.

You're fairly new here. If you think Elon's concerned about the TSLA stock price, you definitely haven't done your homework. As someone who has owned TSLA since the IPO, I have "enjoyed" the fruits of Elon's stock pumping.

Like the time he said the TSLA price was too high. Or said that without autonomy, Tesla is worth nothing.

Ask me how I know. I've had many 6-figure down days as a stockholder as a result of Elon's "significant interest" in propping up the TSLA stock price.

But I'll tell you one thing, if there's one thing Elon doesn't care much about, it's short-term stock price swings.

 

[kabin]

Elon and Ashok have seen 12.4 and early 12.5 versions. Elon accelerated RT plans right around the time he would have been seeing 12.4 or early 12.5 results. I'll just say this: I don't think that's a coincidence.

FSD has a history of early legendary performance. The only issue is when the rubber hits the road.

 

[uscbucsfan]

The rate sucked for quite awhile as the team was persuing hard-coded control code. That was never going to work. But they've now seen the light. I would classify the improvement from 11.4.9 to 12.3 to be huge.

I was hoping that the move to V12 would result in really fast new releases, but so far that hasn't been the case. Now we have at times waited as much as previous version longest, yet, but it hasn't been a real update every 2 weeks...only minor bug fixes, which were frequent on previous versions as well.

I think there's substantive proof that Elon isn't happy with the sales or direction of the company right now and potentially the stock price (as he wants more voting percentage and his comp package).

Until we see what the next real update is like vs. 12.3 in 12.4, it's all a guess at this point.

 

[rlsd]

I have a hunch that Tesla will add 8 more cameras for redundancy for robotaxi. That's why Elon said 8/8.

 

[JB47394]

Until we see what the next real update is like vs. 12.3 in 12.4, it's all a guess at this point.

Three is a pattern. I'll be waiting to see 12.3, 12.4, and 12.5 before I start to form an opinion about progress moving forward.

 

[uscbucsfan]

Three is a pattern. I'll be waiting to see 12.3, 12.4, and 12.5 before I start to form an opinion about progress moving forward.

Maybe, but 2 certainly will give us insight.

Odd that it is taking so long for 12.4, when Elon spoke about how great it was before 12.3 went wide. How much "cleaning up" needs to be done on each version before it's released? (obviously rhetorical).

I think the assumption was that releases, without as much hard coding, would produce and deploy rapidly, probably our/my mistake with the interpretation.

 

[JulienW]

A little back on topic. To me form 10.2 - 11.4 FSD was a fun CHALLENGE. I almost always used FSD but it was like "video game" type "work/stress" and many times would kinda dread driving in challenging urban areas. No doubt for me V12 is a "game changer" and still a fun experience but NO longer a challenge or stressful. Also instead of going to the gym, home, work, etc. directly I now almost always put in waypoints to drive through different areas for "fun".

I have also been VERY skeptical that our cars will ever be L4. Now with V12 I have softened a little but still over all skeptical. I feel the camera locations preclude L4 but I can almost see a Vision L4 Tesla robotaxi with the addition of a few cameras and faster more powerful HW5 processor. Still needs 4 cross traffic cameras for full safety and will also need a front bumper camera too. Can't be blind in the front bumper area and risk a "Cruze" type accident where a person is in the front bumper and dragged under the car because it can't see them.

I'm still happy with our cars getting as good as they have and can defiantly see a reliable point to point L2 for us within a year.

 

[Mardak]

The best way to do that is to be refining one version (12.4) while you’re still earlier in the process for the next version (12.5).

I would guess 12.4 is fine-tuning from 12.3 failures whereas 12.3 was mostly refining from 11.x failures, and these earlier 12.0-12.4 were built on some base foundation understanding that needed to go through various iterations of training and fixing regressions before a new version like 12.4 is ready for customers.

Potentially 12.5 differs in that base understanding has been retrained with more data, improved architecture and for longer with available compute. The same type of fine-tuning needed before 12.3 was widely released would need to happen with this new foundation to avoid regressions for customers, but Autopilot team can already see where it's much improved even if still weeks out from release.

 

[JB47394]

I think the assumption was that releases, without as much hard coding, would produce and deploy rapidly, probably our/my mistake with the interpretation.

They still have development tasks, only now they're curation of training data, ordering of training data, tuning depth and width of network layers, the consequences of each change, understanding what approaches converge on good behavior or bad, and so on. They have a research project of a safety critical system on production hardware. I do not envy them.

 

[cyborgLIS]

Removing humans from the equation won't be a reality for another 50 years, except perhaps for limited "autonomous-only" freeway lanes. There's no practical way to skip the intermediate step, unfortunately.

Yes, it's a long ways off and you can't skip the intermediate human+robot driver step. Though I imagine it could be done within 20 to 30 years. There are probably not many cars on the road older than that at any given time. I imagine it would require just a few years to draft and begin implementation of a protocol for inter-operation of autonomous vehicles. The government could mandate that all new vehicles incorporate the system. The system would add a modest cost to the car, but consumers could be incentivized with credits (similar to our current EV consumer incentives), or buy a used car in their price range. In 30 years the number of running cars without the system would be minimal. Those cars would need to be prohibited from being driven so the switch could be made to autonomous only roads.

Yes, this is a big problem, and a reason I wish Tesla had designed more forward-compatibility into its compute hardware, so that e.g. a HW5 computer could be a drop-in replacement for HW3 or HW4. That would allow them to put more focus on a single newer compute platform, although it would still need to be adaptive to a range of sensor suites. It's also why I vastly prefer Lucid's sensor approach to Tesla's, in terms of forward-thinking. (Not that Lucid doesn't have other tremendous challenges.)

The thing is there will always be advancements in hardware (better cameras, more cameras, new sensors, better placement of sensors), And it doesn't seem possible to future-proof the neural nets. Anytime a new sensor is added or enhanced, the NN will need to be rebuilt. For example, let's say Tesla adds 3 forward facing cameras near the front bumper to enhance cross-traffic detection. Under the previous approach, only the NN responsible for perception needs to be retrained. The layer responsible for making decisions based on output from the perception layer would not need any adjustments. It would simply have a more detailed map of its environment with which it could make better decisions. With E2E, is there even a separate perception layer anymore? As I understand, there is one continuous NN that would need to be retrained with a brand new set of training data featuring the new sensors. And that new data is not going to be of identical scenarios if you are re-capturing real world footage. Which means the new NN might introduce regressions for some situations.

Furthermore, since we are using machine learning, would adding sensors even help? The current placement of the cameras approximates what the human driver can see from the driver's seat. You could add 3 front bumper cameras, but since the human driver cannot see information from that vantage point, whatever information is picked up by those cameras wouldn't influence a human driver's decisions, and therefore wouldn't influence the behavior demonstrated in the training footage. Though I suppose this could be addresses with simulated footage.

 

[willow_hiller]

Just a PSA for anyone who hasn't checked their Software tab in a few days:

I received 2024.3.20 (FSD v12.3.5) a few days ago, and then never got any app notifications about 2024.3.25 (FSD v12.3.6). Just checked the software tab in my car, and not only was 2024.3.25 available, but it was already downloaded and my car was prompting to install immediately or schedule an installation.

 

[Ben W]

I'm just going to have to disagree with you there on multiple counts for multiple reasons. Adding lidar does NOT reduce the compute requirements, because you STILL need camera vision and the corresponding neural networks to do things that lidar is completely and utterly useless for--Reading signs, identifying and determining the color of stoplights, lane markings, etc. So with Tesla's approach, you get all the info you need from the single set of cameras. With Lucid's approach, you need those same neural networks to read signs, identify stoplight colors, etc...but then you have to do all the lidar processing, the radar processing, you need to localize and fuse that information into one coherent picture of the world around you, etc.

The size of the input does not determine the required size of the network. Look at AlphaGo. Tiny input (just a few hundred bits), but a massive network required to process it. Part of the heavy lifting Tesla's network currently does is to effectively construct the lidar point-cloud from the visual inputs. (Not literally, but roughly informationally equivalent.) Feeding the lidar data in directly completely removes the need for the rest of the network to do that part of the information-processing job, so the overall network become much simpler for the same quality final result, despite more bits being fed in. A tiny network is sufficient to read stoplight colors from an image. The difficult part is fusing all the inputs into the 3d map. (Obviously the "control" part of the network remains roughly the same, in terms of what sorts of processing it needs to do, and of course with E2E all these "parts" are entangled together.)

(I do point cloud processing of billions of 3D points from precision measurement hardware for a living, so have a little bit of experience in this area).

I'm willing to bet the incremental cost of 5 radars, N lidars, and the processing sufficient to do the above for an autonomous driving application is going to be significantly north of $1000.

The major cost component is one RoboSense lidar unit (not 'N'), with an apparent list price of around $1900. Lucid undoubtedly gets it at a volume discount, and the price should decrease rapidly over time. I believe the rest of the sensors are much cheaper. Even if it's currently a $5k premium over Tesla's sensor array (for a much more "premium" car), it's a fairly minor factor in Lucid's overall financial situation, and again the sensor costs should decrease rapidly over time and with economies of scale.

Yes, of course Lucid is losing money on each car for a whole host of reasons--that's why I said part of. One of Rawlinson's biggest mistakes was that he wanted to get back at Musk so tried to out-Tesla the car. Fine, if you disregard the cost of production you can make a car that outperforms a Tesla. But ultimately you have to make money on the thing, and Rawlinson suffered from the same flaw that most CEOs who try to start a car company suffer from: The inability to recognize that making a profit on the car is key to survival.

It's impossible to make a profit from day one. A multi-year period of substantial losses is expected for any startup in the EV space. Tesla didn't turn its first full-year profit until 2020, twelve years after delivery of its first cars. Lucid's first deliveries were in 2021, so if they're profitable by 2032 they will be ahead of Tesla by that metric.

But the proof is in the pudding. Can't wait to see Lucid's autonomous driving solution in a few years and compare it to Tesla's. That is, if the Saudis haven't decided to stop funding the black hole they're throwing their money into by that point.

Yes, that's the question. (Disagreed though that it's a black hole; Lucid's cars are some of the most compelling out there. Faraday Future, on the other hand...)