Investor Engineering Discussions

[mongo]

I think basic AP is probably fine with current HW-- because it's mostly relying on the 3 forward cams that actually benefit from the wipers (and redundancy)

I've only had basic AP tell me it had to disable itself for weather ONCE--- and it was in truly horrific rain that I pulled over for 10 min for it to pass because even humans should probably not have driven in it.


It's NoA and FSDb that fail in weather humans can easily safely drive in (or even without any weather) because it's the fender, rear, and b-pillars that get blocked/dirtied/sun blinded with no redundancy or recourse (to say nothing of an actual HW failure- which while very rare would be non-zero in a huge robotaxi fleet).


Now- you could easily get to L3 with those problems- because you have a backup human on board.

But L4 and L5 you simply can't... because if suddenly one of the side cams is out and you're not already in the lane next to the shoulder-or there is no shoulder- the car can no longer "fail safely" without a human by knowing it's clear to pull off the road and park until someone comes to help.

I suppose you could add some really bizarre "find some place you CAN safely pull over with whatever cams still are working so like only make rights and right lane changes if a left cam is out" type code that might involve going long distances out of your way but even that I can think of many situations that'd be insufficient due to lanes ending, one way streets, etc...

If only one side camera is lost (B is more critical than repeater), I think you are still ok. The car knows if there were obstacles to the side, before loss of signal. The front, rear, and remaining side cameras will detect objects entering that blind spot. As long as there is not an object smaller than the loss of coverage specifically matching speeds, it seems doable.

Plus the option of pulling over to side where both caneras are still active (highway). Non-ideal but it happens: hazards and slow to a stop.

Loss of coverage:

 

[Knightshade]

If that were the case, why do you lose lane change entirely if any single side cam is "blocked" (by rain, sun, or dirt)?

Also the car still is pretty garbage at having any memory of where anything was the moment it can no longer see it--- that certainly CAN be improved without more or better or self cleaning cameras- but it might not be improvable enough without more compute. Single stack highway once we see it might give some insight there, but even the FSDb stack still forgets stuff exists after seconds for the most part.

Lack of memory is also relevant to the existing camera placement for getting in/out of parking since the cameras can not see close to the car especially forward... this is at least part of why new cars without USS lost several parking related features--- they say they'll bring them back to parity eventually but they said that about removing radar and we're still at follow 2 and top speed 85 over a year later, so still no parity... it's possible memory can eventually get us close to parity here someday-- but not today--- and even then it'll be inferior to having a real low-bumper-fisheye parking cam like most others use, because it'll still miss things that get near the car WHILE parked then stop in the blind areas like say a small animal, child or their toy, etc (esp. if you don't leave sentry on 24/7).

 

[mongo]

If that were the case, why do you lose lane change entirely if any single side cam is "blocked" (by rain, sun, or dirt)?

Also the car still is pretty garbage at having any memory of where anything was the moment it can no longer see it--- that certainly CAN be improved without more or better or self cleaning cameras- but it might not be improvable enough without more compute. Single stack highway once we see it might give some insight there, but even the FSDb stack still forgets stuff exists after seconds for the most part.

Lack of memory is also relevant to the existing camera placement for getting in/out of parking since the cameras can not see close to the car especially forward... this is at least part of why new cars without USS lost several parking related features--- they say they'll bring them back to parity eventually but they said that about removing radar and we're still at follow 2 and top speed 85 over a year later, so still no parity... it's possible memory can eventually get us close to parity here someday-- but not today--- and even then it'll be inferior to having a real low-bumper-fisheye parking cam like most others use, because it'll still miss things that get near the car WHILE parked then stop in the blind areas like say a small animal, child or their toy, etc (esp. if you don't leave sentry on 24/7).

I feel we're drifting...
Current SW != best possible SW nor human interpreted camera view, remove the compute constraint.

Peripheral topics:
Current SW has no reason to allow functionality with any HW issues
85 is the highest speed limit in the US
USS provides zero coverage for items between the front and rear wheels nor short ones directly adjacent

Cybertruck prototype has a front bumper camera.

 

[Ogre]

If only one side camera is lost (B is more critical than repeater), I think you are still ok. The car knows if there were obstacles to the side, before loss of signal. The front, rear, and remaining side cameras will detect objects entering that blind spot. As long as there is not an object smaller than the loss of coverage specifically matching speeds, it seems doable.

Plus the option of pulling over to side where both caneras are still active (highway). Non-ideal but it happens: hazards and slow to a stop.

At some point, the car needs to identify that it’s sick and either limp “home” to where it can get serviced and refuse to leave until it is, or if it’s bad enough, just pull over and phone home for help. Maybe it could ask it’s passengers for help?

This is why I think FSD can be 100% well before RoboTaxi is possible. RoboTaxi requires is a lot more complex behaviors and judgement than just getting your personal car from A-B.

 

[Knightshade]

I feel we're drifting...
Current SW != best possible SW nor human interpreted camera view, remove the compute constraint.

Ok. But that still leaves things the car possibly can not do at L4 or especially L5.... (L4 you can technically get around say the weather limits by setting the ODD to "only clear weather or at most light rain" for example).

SW isn't magic- if a needed cam is blinded or lacks the physical ability to see in a specific spot you can't AI your way around that.

Peripheral topics:
Current SW has no reason to allow functionality with any HW issues

You've suggested no reason future SW would either. If it can't know for sure if it's safe to change lanes due to camera issues it's not going to change lanes, now or in the future.

I agree you can made some educated guesses SOME of the time with future software though but I doubt they'll enable something in an L4 or L5 vehicle that works "some" of the time yet is critical to safety.

85 is the highest speed limit in the US

Ok. But radar cars can still be set to 90.

And a follow distance of 1, which also can't be done on vision cars over a year after Tesla promised they'd "soon" reach parity.

USS provides zero coverage for items between the front and rear wheels nor short ones directly adjacent

And yet far more coverage than vision with current cameras can. See again Tesla disabling multiple features in cars without the USS...and again promising parity "soon"

Cybertruck prototype has a front bumper camera.

Which seems to confirm everything I'm saying? That current HW is insufficient, and among those reasons is the lack of something that can see forward and low.

 

[Discoducky]

@Knightshade and others, thank you so much for the conversation and the points made.

I've read through them and I think it might be best to rank them as I see the issues.

1. AI in optimal conditions
2. AI in poor weather conditions
3. HW in poor weather conditions
4. HW when damaged
5. HW - Not enough compute, not enough memory...etc
6. HW - Not enough detail, resolution
7. HW - Not enough vision overlap/blind spot

All of these are solved with AI and the current sensor set.

The reason that the AP/FSD computer was updated was to put the entire stack into the hands of Tesla. It is the same reason Dojo is coming online vs nVidia GPU clusters.

AI in optimal conditions needs to be solved first and then the rest will come.

Individual concerns:

FSD AI and HW concerns	Feedback
It really is though.	I don't think so as I've stated that a human could remotely operate the vehicle with the current sensor suite and why is below
There's no redundant cameras for most of the 360 view, there's blind spots close to the car with current # and placement, and there's no ability to clear most of the cameras in bad weather. Auto lane change disables itself if there's a spot of dirt on one of the side cameras, or if there's too much sun glare on one of them. NoA and FSDb shut down in even moderate rain here, let alone heavy rain (can't speak to snow, we rarely get it and don't drive in it when we do- but certainly that's not true of elsewhere).	There does not need to be redundancy of sensors, just a percentage of overlap to stitch the perceived environment with high confidence. The blind spot in front of the car is not blind to the AI. I've written about this before and explained how the car can remember what was in the space that is occluded and watch what enters that space while the car is parked. There are no other blind spots that I'm aware of. AI is not currently resilient to sensors that are not optimally sensing the environment. This can be solved with AI as training data can be used to improve this behavior. There has been very little done to improve this to date. Solving optimal weather AI is the top priority. Tesla will focus on this once it becomes a priority.
Jury is still out on issue of b pillars being as far back as they are for obstructed intersections (Chuck Cook has a nice video of this where he has external cams at the b-pillar and front fender locations to show this issue-- I know I've personally had BOTH situations where it just never went because it couldn't see well enough at the creep limit- and cases where it DID go and should not have because the creep limit wasn't quite far enough but it thought it was)	The overlap in vision should be sufficient. The distance of vision for high speed oncoming traffic might be at its very limit however with the current AI. However, when I watch the video feed, as a human, I can tell, so I assume that AI can be trained to also see at a further limit.
The weather aspect BTW is one place I strongly disagree with removing radar... I get that it's a low res radar, and eliminating it for MOST uses is probably a net benefit.... but in bad weather it was still better than just vision- and better radar is pretty cheap now too.	Sensor fusion ends up causing local maxima and does not allow for better decision making. Elon has talked about this at length. The local max is that you end up having to either trust one sensor or another when they completely disagree. Trying to solve the sensor fusion problem introduces complexity giving better results. You can't just choose when to use one or the other. You let the AI do that, but there in lies the complexity that doesn't result in better results for a trained NN. It worked for heuristics however, but not AI.

 

[Knightshade]

I apologize for this as I generally find your posts, and especially in this area, incredibly helpful and thoughtful.

But honestly that reply reads like handwaving away every concern with the magic of AI.

AI can't see through a camera occluded by weather or dirt. No matter how good it is. Doubly so when the resolution of those cameras is already quite low to start with, and it's relying on stitching all the relatively low res camera input together to understand the world around it- consider what one being unable to see does to those already quite low res voxel maps it's building.

So to be clear- I don't think a human could drive with JUST the camera inputs either-- not at L5 reliability anyway. As I say there's a big difference between a human that does NOT have a bunch of water or dirt in their eyes, and can turn their head as needed if a specific angle is a problem, when compared to fixed cameras that both weather and dirt can take out of useful commission.

I think current sensors can easily do L3.... and can certainly do L4 with some ODD restrictions around things like bad weather.

BTW your "the car can remember what was in the space that is occluded and watch what enters that space while the car is parked."

First- this is only true if the car intends to run sentry mode 24/7/365... and the power drain would be higher than sentry is now, because it'd need to be running additional NNs to build an understanding of what is hidden (and keep track of what moves in/out of blind areas).... and even then could miss things approaching low to the ground like small animals- which could get into the blind spot without any camera on the parked car being capable of seeing it get there.

BTW it's not just one spot directly in front- it's all around very close to the car, though it's by far the largest directly in front, and then also to the sides in front of the fender cams.

Here's a picture from @verygreen where he illustrates the actual blind area(s)

Also, your claim of "The reason that the AP/FSD computer was updated was to put the entire stack into the hands of Tesla"

Unless I'm misunderstanding you- that just ain't so (or certainly not at all the only reason)

Elon was pretty clear the FSD computer was created because HW2.x lacked the power to do the job (just as 2.0 did even though tesla originally claimed each at time of intro did have enough to do the job)

We know now for a FACT they couldn't do the job because they lacked the processing power to handle all 8 cameras at their full frame rates and much of Teslas changes to the design in the last year have been exactly around more video rather than single frame analysis.

IIRC the math was the cameras capture at 36 fps (except the rear at 30) for 282 fps total needing to be processed... and HW2.x was maxed out at 200 fps, so it would've needed nearly 50% more power just to keep up with the cameras- let alone have the power to run all the other NNs and code HW3 is running today.... and HW3 has been near capacity on compute for quite some time itself-- so much so they had to throw the whole original idea of redundancy out the window some time ago and inefficiently split code between the nodes.


Which brings us to another reason HW3 can't be L5--- it can't survive a single node crashing because there's no redundancy there either now. (You could arguably get to L4, again with restrictive ODD and having "fail safe" code running on both nodes at least, possibly- but it'd be pretty inefficient and require running whatever NNs are needed to perceive and fail safely on both nodes at all times and it's unclear they have the spare compute to do that)


It's possible HW4 fixes that by actually being powerful enough to run everything in a single node and having a second as backup.... but it's also possible that's also not enough--- because again nobody.... not me, not you, not Tesla, knows what "enough" is for compute until they have a working L4/L5 system.

 

[Discoducky]

I apologize for this as I generally find your posts, and especially in this area, incredibly helpful and thoughtful. As I do yours!

But honestly that reply reads like handwaving away every concern with the magic of AI. - AI is essentially magic as the manual math required to work out how large trained NNs output inference would take a human 100s of years.

AI can't see through a camera occluded by weather or dirt. - It can, no lens is perfectly clean. Have you ever seen the movie "Real Genius"? There is some amount of coverage that makes it impossible to see, but some amount of debris is normal and some amount is fine. AI tends to impress for things like this. Transformer language models actually work like this where you take a sentence and remove a word and the model predicts the word that is missing. Same with an image, you remove the portion that is occluded and the model predicts what is missing. If that thing is of interest, the model will track it.

No matter how good it is. Doubly so when the resolution of those cameras is already quite low to start with - Why is it low exactly?

, and it's relying on stitching all the relatively low res camera input together to understand the world around it- consider what one being unable to see does to those already quite low res voxel maps it's building. - It doesn't seem low to me at all. It seems like it is doing quite well. I see how Optimus is building incredibly detailed maps of close up objects with the same NNs and HW.

So to be clear- I don't think a human could drive with JUST the camera inputs either-- not at L5 reliability anyway. - Why? The data I'm looking at seems incredibly detailed. Remember, the computer is able to think in increments of ~20ms. Way faster than a human can react.

BTW your "the car can remember what was in the space that is occluded and watch what enters that space while the car is parked."

First- this is only true if the car intends to run sentry mode 24/7/365... and the power drain would be higher than sentry is now, because it'd need to be running additional NNs to build an understanding of what is hidden (and keep track of what moves in/out of blind areas).... and even then could miss things approaching low to the ground like small animals- which could get into the blind spot without any camera on the parked car being capable of seeing it get there. - Easily solvable, as the car would then just back up a bit before proceeding forward. Easy solutions for a bit of blind spots in certain situations.

BTW it's not just one spot directly in front- it's all around very close to the car, though it's by far the largest directly in front, and then also to the sides in front of the fender cams.

Here's a picture from @verygreen where he illustrates the actual blind area(s)

View attachment 877377



Also, your claim of "The reason that the AP/FSD computer was updated was to put the entire stack into the hands of Tesla"

Unless I'm misunderstanding you- that just ain't so (or certainly not at all the only reason)

Elon was pretty clear the FSD computer was created because HW2.x lacked the power to do the job (just as 2.0 did even though tesla originally claimed each at time of intro did have enough to do the job)

We know now for a FACT they couldn't do the job because they lacked the processing power to handle all 8 cameras at their full frame rates and much of Teslas changes to the design in the last year have been exactly around more video rather than single frame analysis.

IIRC the math was the cameras capture at 36 fps (except the rear at 30) for 282 fps total needing to be processed... and HW2.x was maxed out at 200 fps, so it would've needed nearly 50% more power just to keep up with the cameras- let alone have the power to run all the other NNs and code HW3 is running today.... and HW3 has been near capacity on compute for quite some time itself-- so much so they had to throw the whole original idea of redundancy out the window some time ago and inefficiently split code between the nodes. - The root of the issue was that nVidia did NOT want to pursue Tesla's path to autonomy. Tesla needed to write its own stack.

Which brings us to another reason HW3 can't be L5--- it can't survive a single node crashing because there's no redundancy there either now. - There are two TRIP inference chips per FSD onboard computer. Each run the same equations in lock step which means it is full redundancy. Failure of one TRIP chip is handled by the other bringing the vehicle to safe harbor. The system is designed for L5.

(You could arguably get to L4, again with restrictive ODD and having "fail safe" code running on both nodes at least, possibly- but it'd be pretty inefficient and require running whatever NNs are needed to perceive and fail safely on both nodes at all times and it's unclear they have the spare compute to do that)- They have way more than needed. They fill the compute cycles with other test code, because it is wasted cycles if they didn't.


It's possible HW4 fixes that by actually being powerful enough to run everything in a single node and having a second as backup.... but it's also possible that's also not enough--- because again nobody.... not me, not you, not Tesla, knows what "enough" is for compute until they have a working L4/L5 system. - Moving to HW4 TRIP chip is more about moving to a more efficient and higher throughput fab, than needing more compute.

Going to inline mode...hope that is ok. I took out some of your text, but I kept what seemed best. I'm enjoying this discussion as I see the world from an AI perspective and I've seen such amazing things that trained AI models can accomplish so I'm a bit bullish.

I run my view of TMC in dark, so writing in green (my fav color)...

 

[heltok]

BTW it's not just one spot directly in front- it's all around very close to the car, though it's by far the largest directly in front, and then also to the sides in front of the fender cams.

Here's a picture from @verygreen where he illustrates the actual blind area(s)

View attachment 877377

I sometimes rent non-Tesla cars. Last car I rented had a very high front my blindspot area was definitely bigger than the FSD blindspot. While driving over a hill I was pretty much blind and just had to pray that I wouldn't hit a cat or child or that any car coming into the road wouldn't hit me.

What FSD has that somewhat helps in those situations is memory. If it has been a few meters away a few seconds ago it might at least understand where any static objects are. As for dynamic objects hopefully it will see them entering its blindspot. From a parked situation it sucks. Maybe just reversing out is the solution as the rear camera is mounted a lot lower.

 

[Knightshade]

Going to inline mode...hope that is ok. I took out some of your text, but I kept what seemed best. I'm enjoying this discussion as I see the world from an AI perspective and I've seen such amazing things that trained AI models can accomplish so I'm a bit bullish.

I run my view of TMC in dark, so writing in green (my fav color)...

FWIW I agree AI can do amazing amazing things... but I think that does tend to induce the idea it can do ANY amazing thing, and I'm not sure I agree with that.


Regarding redundancy--I think you are conflating 2 different things?

There's 2 nodes but they are absolutely not redundant and haven't been in years because there isn't enough compute on either node to run the system... They only ran a full copy of the full stack on each node for about 6ish months back in late 2019/early 2020 and haven't been able to since as compute demands increased beyond the capacity of a single node.

There's also 2 NPUs per node- but last I saw those were being used together for more compute, not as redundancy for each other (in fact most writeups with any detail I can find specifically cite the combined throughput of the two)... but even if they were running independently, that doesn't make the whole node redundant because they can't fit all the code in a single node (even WITH 2 NPUs) so again- not redundant.

There BEING two entire nodes was explicitly for redundancy per the presentation where HW3 was revealed- but they ran out of compute roughly mid-2020 in a single node so that has been long gone.

This has been pretty widely discussed- for years- so I'm unsure why you still think they "have plenty"?

If they had plenty they would not be node-splitting code, as that causes a large hit to performance because the system was never designed for that and you take a big performance hit crossing between nodes. Instead they'd be running the prod stack in one node, and all the test/campaign code in node B. They can't because the prod stack doesn't have nearly enough compute for it and hasn't since at least mid-2020.

do keep in mind that cross-node multitasking is much-much-much harder than local node, so IMO going for that trouble signifies how badly they need the extra compute

As to the rest, as I say, it largely reads like "AI does amazing things, so clearly it'll keep doing even more amazing ones that solve all these issues" which still reads as magical thinking to me.

I'm not remotely an expert on AI or ML- want to be clear on that. So I do think at least SOME of what you suggest is solvable with AI... I'm not convinced all of it is, but....

But I have read papers on for example some of the really amazing stuff they can do cleaning up things like "cameras that get raindrops on them" so I can understand why that may seem solvable... but for one, those papers are generally using significantly higher resolution cameras to start with, so they have a LOT more information to use to clean up the image... and for another they're not typically ALSO trying to use the image to, in real time, determine distance, speed, etc in mono (which is also solvable of course but adds a lot more compute and complexity in addition to the image cleanup and it ALL has to happen virtually instantly in real time)....and speaking of real time, the cleanup generally takes a LONG time (relative to the time a car has to understand what it is seeing)- one paper I recall to do this on images roughly the resolution of Teslas cameras would take almost 1 full second per frame of added processing- which is massively too much delay to be useful for driving even if you had the spare compute available to do it- which HW3 does not. Those papers are a few years old now, doubtless they've improved, and video prob. improves further- but I'm not aware of it having improved so much it wouldn't add too much processing time for driving at speed exclusively on vision, again if the spare compute to do it existed.


Which again points to, at the very least, the compute HW being woefully insufficient for this task if that's your fix. And possibly the cameras being too low res to provide enough non-obscured data per frame to get fully useful info out of even if you do throw tons of compute at it (and cleaning up higher res cameras from dirt/rain/etc would take even more compute to render in a useful amount of time.

They are ALREADY out of compute, fully using BOTH nodes for a single stack of it with no redundancy. At L2 that while pretty good still needs significant human supervision and can't see well enough in moderately bad weather let alone really bad weather...or if there's dirt on one camera....or if there's sun in the "eye" of one camera. So even if we buy into "AI can fix everything- eventually- even if it's stuff they haven't fixed yet years later and seen no sign they're fixing so far because this stuff still turns itself off in even moderate rain" it can't do that with the existing HW3.


As to why HW3 was developed- again, we know, for a fact, HW2.x was incapable of processing all the cameras at full frame rates...by a significant margin.

I don't disagree there were OTHER reasons and benefits for making their own-- but the need to actually be able to process every frame would have to be one of them. They made HW3 because neither HW2.0 or HW2.5 could not do the job--- even though they repeatedly claimed each could when they introduced THAT.

Eventually they will admit the same about HW3 from every bit of behind the scenes data folks like Green have shown us, plus everything I've read about what is required for some of the AI magic you're basing your conclusions on.



Moving on to the cameras.... HW does fail.... (I've had a side camera replaced for this in fact under warranty). AI can't fix a camera no long existing. A redundant camera could though. It would also make handling obscured vision from dirt/water/sun MASSIVELY easier to deal with because you'd have a second view- and also significantly simplify distance and speed judgement with stereo. There's SOME overlap on the side/rear stuff, but there's also pretty large blind spots if you remove one entirely. This is one I think would be entirely unneeded for L3, but would be for L5.



Lastly- addressing your suggestion of solving being unable to see objects low/in front when parked by simply backing up a bit first.... it's a good idea at first glance but there's a major flaw with that idea.

How does FSD park?

It pulls in backward.

Which often means it will be physically impossible to back up from parked later because there's a wall, another car, a concrete bar, etc behind the vehicle- leaving no room to do so.

Now, I don't want to make this a THINK ABOUT THE STRAY CATS CRAWLING UNDER CARS thing... there's still any number of production cars that can't see this stuff in this situation, regardless of a human being there or not- so while I don't think this is a deal breaker to the car driving and just accepting the same sort of risk current cars do, I also don't think it's fully solvable with current HW unless they add a parking cam (as they appear to have done on the prototype cybertruck for example).

 

[MC3OZ]

We need more water than rain can provide: refilling rivers with desalination

Why? We believe that water should be unconditionally abundant. In the face of extended droughts, aspiring for greater usage efficiency is not, by itself, a sufficiently robust solution. The Colorad…

caseyhandmer.wordpress.com

Desalination still looks like a major market opportunity for solar and maybe batteries. This article from physicist/engineer Dr Handmer shows a sketch up of what large-scale desal with reverse osmosis could look like for the American Southwest with some comparisons for other regions.

Here is an alternative form of wave powered desalination which is more environmentally friendly:-

If we think about it, some aspects of the wave powered design could be adapted to a solar based design,

In many countries solar powered desalination can be a seasonal load backing off in winter and perhaps only running for 6-9 months of the year depending on the location.

Batteries are probably needed, as plants may take a while to stop and start.

The additional reason why we need desalination is to provide water to make Hydrogen/Ammonia.

Making hydrogen/ammonia can be another seasonal load, as could hydroponics, which uses less water.

Mounting some bi-facial solar panels vertically can improve the amount of solar generation in early morning/late afternoon.

Finally agrivolatics can reduce the amount of water needed to grow crops and provide good shade for livestock.

Put it all together, some of the seasonal variation and time of day variation of solar can be solved by clever deployment and smart use of controlled loads.
These controlled loads can be used to make water and can also be used to make even more useful products like hydrogen/ammonia or food with water.

Back off some of these controlled loads off in winter, and the solar overbuild helps reduce seasonal variation.

 

[JRP3]

Desalination still looks like a major market opportunity for solar and maybe batteries. This article from physicist/engineer Dr Handmer shows a sketch up of what large-scale desal with reverse osmosis could look like for the American Southwest with some comparisons for other regions.

I still have concerns with the unintended consequences of the highly salinated discharge that results.

 

[bkp_duke]

I still have concerns with the unintended consequences of the highly salinated discharge that results.

It's not an issue if dumped deep at see and in a channel current.

The reason a recent project in the LA area was voted against / halted is because they didn't want the expense to build a deep and long pipeline for these brine dumps in a current channel (which would effectively wash them out and dilute them immediately).

 

[Gigapress]

I still have concerns with the unintended consequences of the highly salinated discharge that results.

The solution to pollution is dilution. As long as it’s not highly concentrated and dumped in one spot it should be fine.

Alternatively the brine could be boiled or evaporated down to just solid precipitates, have some useful elements like lithium mined from that, and the rest put into landfill containment.

 

[Discoducky]

FWIW I agree AI can do amazing amazing things... but I think that does tend to induce the idea it can do ANY amazing thing, and I'm not sure I agree with that.


Regarding redundancy--I think you are conflating 2 different things?

There's 2 nodes but they are absolutely not redundant and haven't been in years because there isn't enough compute on either node to run the system... They only ran a full copy of the full stack on each node for about 6ish months back in late 2019/early 2020 and haven't been able to since as compute demands increased beyond the capacity of a single node.

There's also 2 NPUs per node- but last I saw those were being used together for more compute, not as redundancy for each other (in fact most writeups with any detail I can find specifically cite the combined throughput of the two)... but even if they were running independently, that doesn't make the whole node redundant because they can't fit all the code in a single node (even WITH 2 NPUs) so again- not redundant.

There BEING two entire nodes was explicitly for redundancy per the presentation where HW3 was revealed- but they ran out of compute roughly mid-2020 in a single node so that has been long gone.

This has been pretty widely discussed- for years- so I'm unsure why you still think they "have plenty"?

If they had plenty they would not be node-splitting code, as that causes a large hit to performance because the system was never designed for that and you take a big performance hit crossing between nodes. Instead they'd be running the prod stack in one node, and all the test/campaign code in node B. They can't because the prod stack doesn't have nearly enough compute for it and hasn't since at least mid-2020.





As to the rest, as I say, it largely reads like "AI does amazing things, so clearly it'll keep doing even more amazing ones that solve all these issues" which still reads as magical thinking to me.

I'm not remotely an expert on AI or ML- want to be clear on that. So I do think at least SOME of what you suggest is solvable with AI... I'm not convinced all of it is, but....

But I have read papers on for example some of the really amazing stuff they can do cleaning up things like "cameras that get raindrops on them" so I can understand why that may seem solvable... but for one, those papers are generally using significantly higher resolution cameras to start with, so they have a LOT more information to use to clean up the image... and for another they're not typically ALSO trying to use the image to, in real time, determine distance, speed, etc in mono (which is also solvable of course but adds a lot more compute and complexity in addition to the image cleanup and it ALL has to happen virtually instantly in real time)....and speaking of real time, the cleanup generally takes a LONG time (relative to the time a car has to understand what it is seeing)- one paper I recall to do this on images roughly the resolution of Teslas cameras would take almost 1 full second per frame of added processing- which is massively too much delay to be useful for driving even if you had the spare compute available to do it- which HW3 does not. Those papers are a few years old now, doubtless they've improved, and video prob. improves further- but I'm not aware of it having improved so much it wouldn't add too much processing time for driving at speed exclusively on vision, again if the spare compute to do it existed.


Which again points to, at the very least, the compute HW being woefully insufficient for this task if that's your fix. And possibly the cameras being too low res to provide enough non-obscured data per frame to get fully useful info out of even if you do throw tons of compute at it (and cleaning up higher res cameras from dirt/rain/etc would take even more compute to render in a useful amount of time.

They are ALREADY out of compute, fully using BOTH nodes for a single stack of it with no redundancy. At L2 that while pretty good still needs significant human supervision and can't see well enough in moderately bad weather let alone really bad weather...or if there's dirt on one camera....or if there's sun in the "eye" of one camera. So even if we buy into "AI can fix everything- eventually- even if it's stuff they haven't fixed yet years later and seen no sign they're fixing so far because this stuff still turns itself off in even moderate rain" it can't do that with the existing HW3.


As to why HW3 was developed- again, we know, for a fact, HW2.x was incapable of processing all the cameras at full frame rates...by a significant margin.

I don't disagree there were OTHER reasons and benefits for making their own-- but the need to actually be able to process every frame would have to be one of them. They made HW3 because neither HW2.0 or HW2.5 could not do the job--- even though they repeatedly claimed each could when they introduced THAT.

Eventually they will admit the same about HW3 from every bit of behind the scenes data folks like Green have shown us, plus everything I've read about what is required for some of the AI magic you're basing your conclusions on.



Moving on to the cameras.... HW does fail.... (I've had a side camera replaced for this in fact under warranty). AI can't fix a camera no long existing. A redundant camera could though. It would also make handling obscured vision from dirt/water/sun MASSIVELY easier to deal with because you'd have a second view- and also significantly simplify distance and speed judgement with stereo. There's SOME overlap on the side/rear stuff, but there's also pretty large blind spots if you remove one entirely. This is one I think would be entirely unneeded for L3, but would be for L5.



Lastly- addressing your suggestion of solving being unable to see objects low/in front when parked by simply backing up a bit first.... it's a good idea at first glance but there's a major flaw with that idea.

How does FSD park?

It pulls in backward.

Which often means it will be physically impossible to back up from parked later because there's a wall, another car, a concrete bar, etc behind the vehicle- leaving no room to do so.

Now, I don't want to make this a THINK ABOUT THE STRAY CATS CRAWLING UNDER CARS thing... there's still any number of production cars that can't see this stuff in this situation, regardless of a human being there or not- so while I don't think this is a deal breaker to the car driving and just accepting the same sort of risk current cars do, I also don't think it's fully solvable with current HW unless they add a parking cam (as they appear to have done on the prototype cybertruck for example).

It sounds like you are basing your conclusions on Green's insights? or is there someone else? Have you heard James Douma talk about it?

I get you are skeptical and that is fine. I've managed this type of engineering and come to different conclusions. The thing is, we may never know or we might. Time will tell! Will be interesting to see the cameras and placement on the production Semi on Thursday!

To wrap up my points, I think the current HW is sufficient for L5. Tesla may go to more compute and higher resolution cameras as the course of technology is evolving, but it is not necessary.

As a data point, I put hydrophobic coating on my windshield (no other cameras however) and drove surface streets in heavy rain. FSD worked fine and I had no issues vs perfect conditions. It seems it is currently keying off of the windshield wipers to 'detect' poor conditions.

Edit: I'll add one other data point. Large AI language models are trained on 10 to 100k of GPU/ASIC cores and takes up to a month to converge. The trained model it spits out is run on just one core and it can run 10k inference predictions a second. Maybe this will help you understand the scale of AI better.

 

[winfield100]

I-------snip-------
As a data point, I put hydrophobic coating on my windshield (no other cameras however) and drove surface streets in heavy rain. FSD worked fine and I had no issues vs perfect conditions. It seems it is currently keying off of the windshield wipers to 'detect' poor conditions.

------snip-------
@Discoducky
RainEx or similar? (have a spray bottle of same in garage)...

 

[Discoducky]

------snip-------
@Discoducky
RainEx or similar? (have a spray bottle of same in garage)...

Yep, just original RainX, super simple.

 

[Knightshade]

It sounds like you are basing your conclusions on Green's insights?

Regarding compute? Yes because it's direct evidence of what is actually being used and the fact the design of the system takes a major performance hit for having to do it that way (thus they wouldn't be doing it that way if they had any other choice)

or is there someone else? Have you heard James Douma talk about it?

All I've heard Douma say is some handwaving "optimzations will fix it all" stuff.

TBH I don't put much faith in Douma at this point--I thought I'd mentioned this earlier? I've also seen him tell us how lidar wasn't needed, but radar was actually really important for bad weather. Then a few weeks later, because Tesla announced they were dropping radar, tell us how radar isn't needed and dropping it made total sense.

He also came up in the Elon FSD Beta tweet thread with multiple folks pointing out his lack of any real world experience with the stuff Tesla is actually doing

0 Machine Learning Experience
0 Computer Vision Experience
0 Autonomous Driving Experience

Having looked into it, he appears to be in some industry discussion groups related to AI, and once coauthored a paper about semantic matching, but I can't find much he's ever actually been educated or trained or produced professionally anything that debunks any of that- he seems to be just a "regular" coder with a strong INTEREST in ML/AI, despite Dave Lee constantly calling him an "expert" on these topics when having him on his show.... so while I'm sure he's more familiar with the state of the industry than the average person, he doesn't seem to be someone who is much of an authority on this.... see again his quick backtracking on radar once Tesla switched gears on it.

I'm totally open to being corrected here- but then a lot of folks in that other thread need that too if it's the case.

I get you are skeptical and that is fine. I've managed this type of engineering and come to different conclusions.

Which is also fine. It's entirely possible you're 100% right and I'm 100% wrong-- I just see among the little public evidence we do have that it all points to that second one- time will indeed tell.

The thing is, we may never know or we might. Time will tell! Will be interesting to see the cameras and placement on the production Semi on Thursday!

Well, I think we'll EVENTUALLY know.... I don't see any way we're say 10 years from now, Tesla is making millions and millions of cars, and still promising 2016 cars came with everything for L5 FSD, don't need upgraded HW, and it's still not delivered. I think in that timeframe they'd have to either deliver, or admit they can't and refund the buyers.

As to Semi- it will be interesting, and potentially informative... though I expect there'll be those who use the many ways the semi is different from existing cars (size, shape, intended length of trailer, and the fact it's a commercial vehicle among others) to dismiss the significance of any changes in # and placement of cameras we do see.

If there's NO changes that would be pretty shocking though.

As a data point, I put hydrophobic coating on my windshield (no other cameras however) and drove surface streets in heavy rain. FSD worked fine and I had no issues vs perfect conditions. It seems it is currently keying off of the windshield wipers to 'detect' poor conditions.

This one I can debunk with direct experience, as can many I'm sure.... I've had FSDb degrade simply from a single side camera being "blocked"--- and for at least 3 different reasons at different times....water, dirt, and direct sunlight.

Likewise even just NoA do things like disable lane changes because a single side cam was blocked- from all 3 of the same causes.

I WILL say FSDb seems to be somewhat more tollerant before it gives up.... I've had NoA do it in light rain... FSDb usually needs at least "moderate" rain.... only basic single-lane AP appears to keep working up into heavy rain reliably- probably exactly because it only needs to use the front behind-the-windshield cams most easily kept clear of rain/dirt/etc.

Edit: I'll add one other data point. Large AI language models are trained on 10 to 100k of GPU/ASIC cores and takes up to a month to converge. The trained model it spits out is run on just one core and it can run 10k inference predictions a second. Maybe this will help you understand the scale of AI better.

I was gonna say well, not ONE core since they're having to split compute because they ran out... but it's more they can't fit all the different NNs (plus the non-NN code) they need to make the overall system work in 1 core.

And the fact that there's entire functions the current system completely lacks but they need to add for L5 tells you there just ain't remotely enough compute to fit in HW3, let alone ONE NODE of the two.

See the CA DMV stuff for example, where they explain FSDb lacks an OEDR capable of higher than L2, and they have no intent of ever adding one to that code. That whatever they develop above L2 will be a different, future, thing.

 

[mongo]

Regarding redundancy, this is not an aircraft that is inherently unstable/ needs active control. A stopped car has only other cars to worry about.

As such, the car already has many single points of failure.
Single steering rack
Single battery pack
Single motor (for RWD cars).
No true redundancy on any suspension components

As long as the FSD system can detect its failure (interleave check frames/ swap chips), the failure mode is no worse (and arguably better) than, say, a rear differential failure.
If one TRIP is still good, it could reduce focus to the cameras needed to pull over. Either by removing two side and two front or reducing frane rates in the less critical directions.

 

[Discoducky]

Regarding compute? Yes because it's direct evidence of what is actually being used and the fact the design of the system takes a major performance hit for having to do it that way (thus they wouldn't be doing it that way if they had any other choice)




All I've heard Douma say is some handwaving "optimzations will fix it all" stuff.

TBH I don't put much faith in Douma at this point--I thought I'd mentioned this earlier? I've also seen him tell us how lidar wasn't needed, but radar was actually really important for bad weather. Then a few weeks later, because Tesla announced they were dropping radar, tell us how radar isn't needed and dropping it made total sense.

He also came up in the Elon FSD Beta tweet thread with multiple folks pointing out his lack of any real world experience with the stuff Tesla is actually doing




Having looked into it, he appears to be in some industry discussion groups related to AI, and once coauthored a paper about semantic matching, but I can't find much he's ever actually been educated or trained or produced professionally anything that debunks any of that- he seems to be just a "regular" coder with a strong INTEREST in ML/AI, despite Dave Lee constantly calling him an "expert" on these topics when having him on his show.... so while I'm sure he's more familiar with the state of the industry than the average person, he doesn't seem to be someone who is much of an authority on this.... see again his quick backtracking on radar once Tesla switched gears on it.

I'm totally open to being corrected here- but then a lot of folks in that other thread need that too if it's the case.




Which is also fine. It's entirely possible you're 100% right and I'm 100% wrong-- I just see among the little public evidence we do have that it all points to that second one- time will indeed tell.



Well, I think we'll EVENTUALLY know.... I don't see any way we're say 10 years from now, Tesla is making millions and millions of cars, and still promising 2016 cars came with everything for L5 FSD, don't need upgraded HW, and it's still not delivered. I think in that timeframe they'd have to either deliver, or admit they can't and refund the buyers.

As to Semi- it will be interesting, and potentially informative... though I expect there'll be those who use the many ways the semi is different from existing cars (size, shape, intended length of trailer, and the fact it's a commercial vehicle among others) to dismiss the significance of any changes in # and placement of cameras we do see.

If there's NO changes that would be pretty shocking though.





This one I can debunk with direct experience, as can many I'm sure.... I've had FSDb degrade simply from a single side camera being "blocked"--- and for at least 3 different reasons at different times....water, dirt, and direct sunlight.

Likewise even just NoA do things like disable lane changes because a single side cam was blocked- from all 3 of the same causes.

I WILL say FSDb seems to be somewhat more tollerant before it gives up.... I've had NoA do it in light rain... FSDb usually needs at least "moderate" rain.... only basic single-lane AP appears to keep working up into heavy rain reliably- probably exactly because it only needs to use the front behind-the-windshield cams most easily kept clear of rain/dirt/etc.




I was gonna say well, not ONE core since they're having to split compute because they ran out... but it's more they can't fit all the different NNs (plus the non-NN code) they need to make the overall system work in 1 core.

And the fact that there's entire functions the current system completely lacks but they need to add for L5 tells you there just ain't remotely enough compute to fit in HW3, let alone ONE NODE of the two.

See the CA DMV stuff for example, where they explain FSDb lacks an OEDR capable of higher than L2, and they have no intent of ever adding one to that code. That whatever they develop above L2 will be a different, future, thing.

Is there deterministic engineering evidence to support these statements? I'd love to see it. This thread is for engineering stuff, and so far, I've seen no engineering evidence to support the conclusion that the current system HW is not capable of L5.

I've only seen Green produce data that is assumed to come to these conclusions.

Happy to continue the conversation provided focus on specific engineering deficiencies.

The CA DMV stuff, found here, page 26, shows that Tesla continues to work on L3+ features

"Please note that Tesla’s development of true autonomous features (SAE Levels 3+) will follow our iterative process(development, validation, early release, etc.) and any such features will not be released to the general public until we have fully validated them and received any required regulatory permits or approvals."

And I'm happy to provide a video of my 2017 X working, without any material difference from perfect conditions, in rain with FSD without the windshield wipers as the hydrophobic coating allows the water to bead and thus not completely blind/obscure the forward camera.

NOTE: I had the 2021 Model 3 driver fender camera replaced a few weeks ago and the car still reports that it is functioning poorly from time to time in perfect conditions. So, this demonstrates that there are separate issues which others might be experiencing that could be causing the system to not function as designed.