Preventive eMMC replacement on MCU1

[Togg]

Interesting. In the future I think the best will be for Tesla to have 3 eMMC chips configured in a what's know as a RAID array, where the chips vote and thus check each other's work, and such that if any one fails the other two keep working. When a failure does occur the car won't fail, and the failed chip can be easily replaced, and the other 2 working chips can update it. This is not rocket science, it's standard stuff available to Linux.

Possibly even better would be to have two redundant computers, but that is much more complex. BTW, full redundancy is the sort of thing that is done onboard spacecraft where a failure might have very bad consequences. I wouldn't be surprised to see high end car companies offer fully redundant electronics as an option someday.

Also the life of an eMMC chip depends on how much it's used. If you drive a lot, wear will probably be more pronounced, and thus it will fail sooner. It would be interesting to see if something called wear leveling is being used, as I suspect it is. Also clearly different chips might have somewhat different life spans, so your results may vary. And temperature plays a role too. Another good reason for multiple, redundant, eMMC chips in something as dangerous as a car.

 

[.jg.]

A simple software mod might be to allow the filesystem(s) on the eMMC to be backed up to a suitably partitioned USB disk - and then to allow an MCU with a dead eMMC to use partitions from a USB disk instead.

 

[HankLloydRight]

what's know as a RAID array, where the chips vote and thus check each other's work,

That's not how RAID works. If you have three eMMC chips in a RAID array, they'll all likely fail near the same time since the writes are spread over all drives in a RAID array. So that's not really a solution.

The solution is not writing useless OS logs to the eMMC in the first place, so the chips outlast the cars they're in.

I think the best will be for Tesla to have 3 eMMC chips configured in a what's know as a RAID array,

Three drives in a RAID 5 configuration "as they say" is one drive+one bit away from total array loss. So it's only one single bit more reliable than just having one drive by itself.

 

[widodh]

Interesting. In the future I think the best will be for Tesla to have 3 eMMC chips configured in a what's know as a RAID array, where the chips vote and thus check each other's work, and such that if any one fails the other two keep working. When a failure does occur the car won't fail, and the failed chip can be easily replaced, and the other 2 working chips can update it. This is not rocket science, it's standard stuff available to Linux.

Possibly even better would be to have two redundant computers, but that is much more complex. BTW, full redundancy is the sort of thing that is done onboard spacecraft where a failure might have very bad consequences. I wouldn't be surprised to see high end car companies offer fully redundant electronics as an option someday.

Also the life of an eMMC chip depends on how much it's used. If you drive a lot, wear will probably be more pronounced, and thus it will fail sooner. It would be interesting to see if something called wear leveling is being used, as I suspect it is. Also clearly different chips might have somewhat different life spans, so your results may vary. And temperature plays a role too. Another good reason for multiple, redundant, eMMC chips in something as dangerous as a car.

RAID won't work as all three chips will endure the same writes and thus wear evenly. That's the problem with RAID on NAND memory.

Tesla just has to write less. Many components in the car are a SPoF (Single Point of Failure) and that's normal, not everything in cars has to be redundant.

It's just that this piece of eMMC memory seems like the Achilles heel of Model S/X with MCU1.

It will take a few weeks before I attempt the replacement, but I promise I'll do a proper write-up of the replacement.

 

[Togg]

RAID won't work as all three chips will endure the same writes and thus wear evenly. That's the problem with RAID on NAND memory.

True, but it's also extremely unlikely that they will fail together at the same time.

 

[HankLloydRight]

True, but it's also extremely unlikely that they will fail together at the same time.

No, not at the same time, but just ask anyone who's lost a NAND RAID array because they couldn't replace the failed drive fast enough before another drive failed.

And like I said, once one drive fails, you just need a one-bit failure in the remaining drives to lose the array. And in your 3 eMMC RAID scenario, you would need some sort of alerting mechanism to let you know that one of the eMMC chips failed, since the volumes would still be valid, the OS wouldn't know there was a drive failure. So you'd need some out-of-band BMC to raise an alert that an eMMC chip died (I don't know if the MCU has a BMC). Then you'd have to find time to dis-assemble the dash, take out the MCU, and replace all the chips (it would be foolish to only replace one of the three drives), so you're back to cloning the partitions, just like you have to do now with one drive. So what does a three chip RAID array get you? It gets you exactly one-bit of additional reliability.

Seems like there are better solutions.

 

[MP3Mike]

I'd like to understand (for Model 3 which I hope wouldn't have the issue but probably would) how Tesla can call the car a million mile car with a known defect

I don't think they said the car was a million mile car they said the chassis and driveunit where designed for a million miles, and that the next generation of batteries would be as well.

We don't know if they sized and speced the eMMC in the Model 3 to last approximately or not for the usage pattern.

 

[Msjulie]

I don't think they said the car was a million mile car they said the chassis and driveunit where designed for a million miles, and that the next generation of batteries would be as well.

We don't know if they sized and speced the eMMC in the Model 3 to last approximately or not for the usage pattern.

When I had my roadster, the thing folks worried about was (or seemed) pretty similar - read/write to a chip not best suited for long duration access w/no easy replaceable part.

I can't say about Android, but I know iOS has made significant steps (more needed perhaps) to reduce the amount of logging the OS and more importantly, the apps do - because the storage does have a finite life even for a device replaced in a handful of years. App developers are supposed to 'know better' I guess.

It's well known that the Linux kernel and systems can log a ton of noise so it seems a bit of a design flaw (or just lack of attention?) to either not disable some of that logging, send it to /dev/null or equivalent ram disk etc or write it to a medium more easily replaced in time..

I hope there is a difference in the M3 systems though I will say that the distinction between chassis + driveunit and 'car' seems a bit misleading.. the car is a paperweight w/o that computer, least the S/X are..

 

[widodh]

When I had my roadster, the thing folks worried about was (or seemed) pretty similar - read/write to a chip not best suited for long duration access w/no easy replaceable part.

I can't say about Android, but I know iOS has made significant steps (more needed perhaps) to reduce the amount of logging the OS and more importantly, the apps do - because the storage does have a finite life even for a device replaced in a handful of years. App developers are supposed to 'know better' I guess.

It's well known that the Linux kernel and systems can log a ton of noise so it seems a bit of a design flaw (or just lack of attention?) to either not disable some of that logging, send it to /dev/null or equivalent ram disk etc or write it to a medium more easily replaced in time..

I hope there is a difference in the M3 systems though I will say that the distinction between chassis + driveunit and 'car' seems a bit misleading.. the car is a paperweight w/o that computer, least the S/X are..

Linux can be very ‘silent’ with minimal changes to the configuration.

Afaik Tesla runs a modified version of Ubuntu on their MCU and that is just a matter of sending syslog to /dev/null indeed.

MCU2 of S and X run on the same HW and probably software as 3 does.

Time will tell.

The MCU1 issues seem like a lack of attention.

 

[KLuuppo]

I'm actually in the process of doing this right now on a previously swapped MCU. I have one running S that definitely has emmc near its end (screen black for a while, no connectivity, tesla diag said corruption and need new mcu). This MCU I am working with now was replaced (and given back to the owner, who gave it to me for experimenting) due to bad emmc.

Anyway, I've had the emmc desoldered and reballed, but reading it out didn't go smoothly. Tried with the suggested allsocket adapter and another chinese adapter as well, but can't reach the data. It's just retrying or timing out depending which adapter I try. I suppose there are options to bypass the mmc-controller on the chip, but they seem to be +2k EUR. I'm waiting for a reply from a data recovery specialist if he can suggest anything.

Now, when I'm going to try this on the live car, I'm probably trying to get the dump without desoldering, as that removes one possibility of ruining the chip.

 

[StarLog]

I

Now, when I'm going to try this on the live car, I'm probably trying to get the dump without desoldering, as that removes one possibility of ruining the chip.

What process do you use to get the data dump without desoldering.?

 

[Maarten]

Most interesting thread for me since a long time @widodh !
Keep us posted!

 

[KLuuppo]

What process do you use to get the data dump without desoldering.?

I'm not sure about linking to other forums, but googling for "Tapping In to the eMMC" should yield the thread. Have not tried yet, so can't confirm.

 

[rxom]

Hi everybody,

the same just hammeded for my Cousin un France.
I"m considerinf too to do the same , and find someone to copy the eMMc for (i'm haven"t got enouth skills with linux to do it)

I've got one supid question here : do you thing we can use the car without the touchscreen ?

 

[Krash]

You can drive the car without the touchscreen. If the MCU runs, I think you can still use the app to control some HVAC, charging and media controls, depending on where it was left. But generally anything that requires a touch is gone.

You can even drive the car without the MCU, but then no app control of anything.

 

[rxom]

ok thanks !

another dumb question : If we replace MCU with touchscreen with another one like we can find on ebay
(for waiting to repair the original with new eMCC):
is this plug and play or do we have to do something so that the MCU recognise the car?

 

[MP3Mike]

ok thanks !

another dumb question : If we replace MCU with touchscreen with another one like we can find on ebay
(for waiting to repair the original with new eMCC):
is this plug and play or do we have to do something so that the MCU recognise the car?

The MCU has to be programmed to match the car. (You have to copy the eMMC data from the old MCU to the new one. (Or have some way to rebuild it from a blank image.)

 

[rxom]

ok thanks !

so, if the eMMC of your car is dead and you have a MCU replacement you have buy on the net : how could we do this ? (i guess tesla doesn't have to use the old dead eMCC when they change the MCU)

 

[MP3Mike]

so, if the eMMC of your car is dead and you have a MCU replacement you have buy on the net : how could we do this ? (i guess tesla doesn't have to use the old dead eMCC when they change the MCU)

If you have a totally dead MCU I don't know if someone like @wk057 or @Ingineer can program a MCU to match your car or not. Obviously Tesla has the tools to do it.

 

[rxom]

I read somewhere thare there is a removal card on the MCU : maybe we only have to switch this card : take the one with the car on the old MCU and replace it on the donor MCU ?

Someone here can confirm/invalidate this ?