-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
Preventive eMMC replacement on MCU1
- Thread starter widodh
- Start date
Well that was pretty lazy of the SC.
supratachophobia: one of the FPGAs is the gateway, and it's programming is volatile. Its config is stored in one of the two SD cards.
supratachophobia: one of the FPGAs is the gateway, and it's programming is volatile. Its config is stored in one of the two SD cards.
Akikiki
A'-Lo-HA ! y'all
@DaveBC, is correct with this statement. I have read and know who they are of two creditable and knowledgeable 3rd party fixers that have studied the eMMC controller and reported this very information One lives in NL and the other in Wash state. I won't give their names in the clear. But if you invest some time reading Unofficial Tesla Tech
edit. And I see one of those to really smart guys I was referring to had already stepped up and explained. I wasn't expecting him to do that when I posted way on a post two weeks back.
Last edited:
What are the chances it's actually the FPGA and they aren't blowing smoke because there is a shortage of daughter boards?
dark cloud
Active Member
That's a good question: Is Tesla ordering new Tegra boards new with the 64GB chip? Or are they refurbing them, essentially doing the same as the 3rd party repairers, resoldering in the new chip, in which case they should have plenty.
What would you guys estimate the cost to Tesla for the new daughter board: $100?
Akikiki
A'-Lo-HA ! y'all
New or refurbished? I don't think that info has leaked yet. But it could be both. I think though that this big bunch of boards that have been deployed separately from MCUs in the last three months, were all new. How/why? Well, they have been desperate for hardware. These' not been enough non-installed MCU1's to keep even 1 at an SC ready to go in cars. And in June I know of a backlog of 70 cars waiting for remanu MCU1. Suddenly late July/Aug we see standalone daughterboards? Those could not have been removed from MCUs to repair and send out. I'd heard a vague rumor in late June - early July that Tesla order a big batch of daughterboards from China. And without going through nVida to get them. We know the Tegra board was an off the shelf modded board from nVida. So, logic or maybe legally they still owned its rights. I guess it took Tesla a while to get rights and permissions and a source to make an out of production board again but this time in China and not the original owner of the board.
But let's not rule out that Tesla would simply not also use the old ones and fix them.
But let's not rule out that Tesla would simply not also use the old ones and fix them.
LuckyLuke
Model S P90DL
I had one in my hands a few days ago. They are definately refurbished. @dark cloud @Akikiki
- it was visible that the chip was replaced (from the side I could see it was not perfectly sinked in),
- there was a extra sticker near the emmc chip,
- there were dust remains/scratches on the heatsink,
- there were glue remains on part of the pcb where normally a sticker would be,
- the board still had a old-style (large) spansion/nor chip, which would never be used on a new board.
Don't know what the symptoms are. The gateway FPGA is on the mainboard.
The Tegra 3 chip on the daughtercard is actually custom nVidia silicon.
Symptoms are the standard: random reboots of IC and MCU, *very* long reboot times, sometimes requires multiple reboot attempts after 5+ min of waiting, browser non-functional, streaming/USB playback errors.
Akikiki
A'-Lo-HA ! y'all
@rooter, Can you help us here, please. Can you help us understand what is happening with these "*very* long reboot times, sometimes requires multiple reboot attempts after 5+ min of waiting," ??
Is the car/the system trying to boot during this long pause?
Is there a sign or signal that the Tegra chip has given up trying to read code in and begin operation?
Thanks.
It's the eMMC. It has an internal controller which mediates access to the flash cells, and this controller is the weak area of the chip.
I've pulled multiple images of every partition in failing chips, and can not get the same sha256sum from multiple reads of the same part. This tells me that it's not the memory cells which are failing, but the controller itself.
The Tegra3 (CPU) is patiently waiting for a response from the eMMC and this causes the delays. Fortunately Tesla built in a long timeout, so it can be more forgiving of failures, rather than just crapping out and giving the equivalent of the BSoD.
I've pulled multiple images of every partition in failing chips, and can not get the same sha256sum from multiple reads of the same part. This tells me that it's not the memory cells which are failing, but the controller itself.
The Tegra3 (CPU) is patiently waiting for a response from the eMMC and this causes the delays. Fortunately Tesla built in a long timeout, so it can be more forgiving of failures, rather than just crapping out and giving the equivalent of the BSoD.
I just upgraded to 20.40.9.2 yesterday, and today I got a message on the display "center display storage device degraded" with an advisement to schedule a service appointment. I hadn't had a blank screen yet.
dark cloud
Active Member
Any of the following?
- reboots
- pixalation of touchscreen graphics
- connectivity dropouts
- map graphics slow
- voice control doesn't function
- streaming pausing/ loss of connectivity
- rebooting after playing video games
Had a couple of unscheduled reboots in the past couple of weeks, but nothing major.
I have had all but "rebooting after playing video games" for about a year now. I dont' get reboot after playing games, mainly b/c I don't play games
How long before it's non functional?
dark cloud
Active Member
How long before it's non functional?
That is the 64 thousand-million byte question.
last visit to SC, I told them how tired I was of it rebooting, not responding, not on when i get in the car for several minutes (and then reboot), screen corruption, etc...They said HW is not reporting any issues, but they updated FW. That was Wednesday. The Saturday prior to that, I had just updated to latest FW, and I have selected to get updates "earlier". So what are the chances there was another FW update between Saturday and Wednesday? I think we can assume none. So the question is, is it the same FW?
If yes - reflashing the same FW should absolutely not fix anything. If it does, there is a different and bigger issue.
If no - then do they have a special FW for those who have more issues with MCU?
Frankly, I wouldn't give this one second thought before, but now after seeing a pattern of behavior with Tesla, I have to wonder.
If yes - reflashing the same FW should absolutely not fix anything. If it does, there is a different and bigger issue.
If no - then do they have a special FW for those who have more issues with MCU?
Frankly, I wouldn't give this one second thought before, but now after seeing a pattern of behavior with Tesla, I have to wonder.
Akikiki
A'-Lo-HA ! y'all
@mymagiccarpet,
I agree with you. There is a pattern. And its not the first time. This one seems a bit more organized or explicit than we have seen earlier this year. I believe earlier this year it was no thought out, not organized, just reactionary, it was denial and forced to recognize a problem. But that's another story.
Now, let's briefly talk about what we are seeing right now. This is not denial. This also is not refusing to fix. They are fixing some. I think, its triage - worse first. Really worse first - dying first. And they don't really want to admit to an owner that his is dying, they don't want to use those words. I think, the plan (this part of the year) the plan is stall and do those first that really need it because they are near total failure. And if you (not you specifically, but you in general) are seeing problems with your car, they are resisting not refusing. They are biding their time. Hoping/waiting on more parts to be available.
I think its a supply problem. I think there's not enough MCU2 and eMMCs available to do everyone that wants it. In early summer there were dozens of S/Xs down waiting on remanufactured MCU1. This was before we saw daughter boards. Then two months later, there were daughter boards. And some people pre-emptively went in and got their daughter board replaced before there were serious problems. And simply because there was a rush - relative to just two months earlier, all the next couple of months of supply of daughter boards were used up in late July/Aug and now, they are having to ration them. Rationing what's available, but scrutinizing what can wait against what is imminently about to fail. But they don't want to talk about it.
Do you (again, not you specifically) think they are just sitting around and watching this mess get worse? No. I believe quietly there's folks working to get more MCU2 build/bought/shipped and out to SCs' And get more daughter boards bought and out to SCs. I think TonyT identified that some old daughter boards from 8 GB MCU1's are being refurbished and are making their way out to SCs and into cars. That hints that Tesla is using every trick they can to get more functioning daughter boards and MCU1s out to the SCs.
It also suggests to me that they are sensitive to the complains about the cost now of MCU1 replacement versus daughter board swap and the costs to the customer. Stop and think, if there's old daughter boards being refurbished, that means they are opening MCU1 up, removing the daughter board, using that board as a core and putting the new 64 GB eMMC on them and making them available without putting them back in the 4-5X more expensive MCU1. (Well, we know its a remanufactured MCU but instead of sending that daughter board back out to and SC where it probably cost the customer $400 or less they could wrap that refurbished board around a remanufactured MCU1 and charge (unnecessarily most time) 4-5X more for the replacement. Do we appreciate that when Tesla replaces a daughter board vice a MCU1, it takes more labor, but the labor fee to the customer is just about the same price? When they just replace the daughter board, they must stop, open the MCU, swap the board and reassemble the MCU. That's a lot more time than just taking a remanufactured MCU1 off the shelf and putting the old on in the "back to the factory box".
Okay, I have rambled on far to long. I think there's a pattern. And I think its going to change rapidly, soon, just as soon as my guess that there's a supply problem not a denial problem clears with more parts available. But then - what do I know? Just my two cents.
I agree with you. There is a pattern. And its not the first time. This one seems a bit more organized or explicit than we have seen earlier this year. I believe earlier this year it was no thought out, not organized, just reactionary, it was denial and forced to recognize a problem. But that's another story.
Now, let's briefly talk about what we are seeing right now. This is not denial. This also is not refusing to fix. They are fixing some. I think, its triage - worse first. Really worse first - dying first. And they don't really want to admit to an owner that his is dying, they don't want to use those words. I think, the plan (this part of the year) the plan is stall and do those first that really need it because they are near total failure. And if you (not you specifically, but you in general) are seeing problems with your car, they are resisting not refusing. They are biding their time. Hoping/waiting on more parts to be available.
I think its a supply problem. I think there's not enough MCU2 and eMMCs available to do everyone that wants it. In early summer there were dozens of S/Xs down waiting on remanufactured MCU1. This was before we saw daughter boards. Then two months later, there were daughter boards. And some people pre-emptively went in and got their daughter board replaced before there were serious problems. And simply because there was a rush - relative to just two months earlier, all the next couple of months of supply of daughter boards were used up in late July/Aug and now, they are having to ration them. Rationing what's available, but scrutinizing what can wait against what is imminently about to fail. But they don't want to talk about it.
Do you (again, not you specifically) think they are just sitting around and watching this mess get worse? No. I believe quietly there's folks working to get more MCU2 build/bought/shipped and out to SCs' And get more daughter boards bought and out to SCs. I think TonyT identified that some old daughter boards from 8 GB MCU1's are being refurbished and are making their way out to SCs and into cars. That hints that Tesla is using every trick they can to get more functioning daughter boards and MCU1s out to the SCs.
It also suggests to me that they are sensitive to the complains about the cost now of MCU1 replacement versus daughter board swap and the costs to the customer. Stop and think, if there's old daughter boards being refurbished, that means they are opening MCU1 up, removing the daughter board, using that board as a core and putting the new 64 GB eMMC on them and making them available without putting them back in the 4-5X more expensive MCU1. (Well, we know its a remanufactured MCU but instead of sending that daughter board back out to and SC where it probably cost the customer $400 or less they could wrap that refurbished board around a remanufactured MCU1 and charge (unnecessarily most time) 4-5X more for the replacement. Do we appreciate that when Tesla replaces a daughter board vice a MCU1, it takes more labor, but the labor fee to the customer is just about the same price? When they just replace the daughter board, they must stop, open the MCU, swap the board and reassemble the MCU. That's a lot more time than just taking a remanufactured MCU1 off the shelf and putting the old on in the "back to the factory box".
Okay, I have rambled on far to long. I think there's a pattern. And I think its going to change rapidly, soon, just as soon as my guess that there's a supply problem not a denial problem clears with more parts available. But then - what do I know? Just my two cents.
Similar threads
