Preventive eMMC replacement on MCU1

[David99]

Just uploaded a fixed version of the MCU removal tutorial.
Mods: please remove my previous post with the old link

 

[tar.sor]

Did someone read EMMC using easy jtag? I need to know where is reset pins located. Because without reset EMMC died after 3 days, but I have p3 dump.

 

[Akikiki]

I'm tellin' you. Better not. You'll get shut out by those who matter. Notice that Luke isn't sharing any details?

I'm the author and I'm still down. Got some help from an old friend for a bit, but not lately.

My articles are 'down for some reason'? Read my note more carefully. Somebody saved my article in case -I- took it down? Be honest; you saved it case it got taken down by outside forces. Don't bite the hand that fed you.

There's an easy way to recover hard-to-read chips which was in one of my posts over there. Research recovering flash data.

rooter, I can't do this work. Simply don't have the skillset. You are impressive. I have read your wiki. Wow. A lot of detail, takes a lot of comprehension to understand and put that to words. Wow.

Question sort of. All this discussion about Swissbit chips and 16's 32's even 64's and being better than the 8's in the cars already. I read Broker's challenge to keep cars running and as they increase in use, he's going to need to do them again and maybe again. So, these chips are better. But what's best? What's the best - high endurance - long lasting - will be a long long time before they fail chip - model, brand. Yes its going to be more expensive. Even if 10 times more expensive than the Swissbit 16 at $25.00 would be better than doing it all again - later.

Any idea where/what the very best one is?

thanks

 

[rooter]

Thanks Akikiki, I do put alot of effort into the wiki, although I am extremely limited in time, as we all are. I generally make improvements every weekend.

I've seen evidence that the primary failure mode of the Hynix is not necessarily with the cells 'wearing out', but with the controller circuitry, as chips I've tried to read gave anomalies on all partitions. And it is a heat-related failure. It's just a cheap chip.

The reason I originally chose the Swissbit is it is an industrial-grade chip. As I say in the article, I searched and searched for an 'endurance-grade' chip but this was the very best I could find.

I am beginning to suspicion that this 'endurance-grade' that we see on SD cards simply means they are using the dual channels of the chip controller, and dividing the chip into two halves like RAID1. (mirroring) This means that your chances of recovering the data are doubled, the chip capacity is halved, and coincidentally the speed is increased as the non-busy channel is chosen next.

The Swissbit chip as-is is likely to outlast other electronics in the car, but you can buy it also set to dual-channel mode. I don't remember what the secret code word is for this mode, but it's just a bit set.

 

[Akikiki]

More wow. Nice work on the evidence and studying what's happening. And I believe you when you say you can't find a better chip. If you can't find, I bet it doesn't exist yet. Maybe someone will see its needed. But since you are already confident the Swissbit will outlast other electronics in the car, I guess it not needed.

I really admire what you are doing. You are under appreciated. So are the 3rd party fixers that are helping the community.

 

[EV-Fixme]

@rooter has done a great job providing lots of information and expertise on this issue. I have started to think similar if a separate component of the chip is going bad. We refer to this as a single chip, but there is really much more inside. We have seen reports of the chip being disassembled and new controller put in to read the content with @widodh recovery. as @whitex has mentioned the failure rate of eMMC chips is on a bell curve. We have seen chips that went well beyond the 3000 writes. We have seen chips with heavy corruption on chips much less that 3000 writes, sometimes around 1000-2000. Only time will tell how much better this upgrade is than the original. But we do know that in several years 3000 writes can happen so having a chip that has much more writes is expected to last longer. The pSLC mode we use gives it 20k write which we like a lot more than 3000, plus have double the space so there is many more write cycles due to the chip's wear leveling. We will be continuing to use the Swissbit until we or someone finds a even better chip. And we expect the current chip to outlast the car, however we would not hesitate to spend a few more dollars to offer a even better upgrade.

Unfortunately at this time we don't have to expertise to identify the exact failure and simply identify the chip as less then reliable enough to use and need a replacement.

Anytime we discover improvements we incorporate into the fix.

 

[ahkahn]

@rooter has done a great job providing lots of information and expertise on this issue. I have started to think similar if a separate component of the chip is going bad. We refer to this as a single chip, but there is really much more inside. We have seen reports of the chip being disassembled and new controller put in to read the content with @widodh recovery. as @whitex has mentioned the failure rate of eMMC chips is on a bell curve. We have seen chips that went well beyond the 3000 writes. We have seen chips with heavy corruption on chips much less that 3000 writes, sometimes around 1000-2000. Only time will tell how much better this upgrade is than the original. But we do know that in several years 3000 writes can happen so having a chip that has much more writes is expected to last longer. The pSLC mode we use gives it 20k write which we like a lot more than 3000, plus have double the space so there is many more write cycles due to the chip's wear leveling. We will be continuing to use the Swissbit until we or someone finds a even better chip. And we expect the current chip to outlast the car, however we would not hesitate to spend a few more dollars to offer a even better upgrade.

Unfortunately at this time we don't have to expertise to identify the exact failure and simply identify the chip as less then reliable enough to use and need a replacement.

Anytime we discover improvements we incorporate into the fix.

I'm wondering if that is what the problem is/was with my chip... 1182 cycles, but slow as molasses and crashing.

 

[whitex]

I am beginning to suspicion that this 'endurance-grade' that we see on SD cards simply means they are using the dual channels of the chip controller, and dividing the chip into two halves like RAID1. (mirroring) This means that your chances of recovering the data are doubled, the chip capacity is halved, and coincidentally the speed is increased as the non-busy channel is chosen next.

The Swissbit chip as-is is likely to outlast other electronics in the car, but you can buy it also set to dual-channel mode. I don't remember what the secret code word is for this mode, but it's just a bit set.

While an interesting theory, the performance data does not support it, at least not for the Swissbit pSLC case. RAID1 configuration would mean you get twice the sustained read performance and the same (or slightly slower) write performance, but when you compare the specs for pSLC vs. MLC modes of Swissbit, that is not he case - write speeds get most of the performance boost, while read is faster but not as dramatically as writes.

It is a lot more likely they they actually switch to using MLC (Multi Level Cell) cells in a pseudo-SLC (Single Level Cell) mode, where:

• 0b00 (no charge) means 0b0
• 0b01, 0b10 and 0b11 (any charge) mean 0b1

Then they just write only 0b00 and 0b11 levels, which can be accomplished a lot faster as it is faster to charge a capacitor to max Q instead of carefully try to aim at some set level between 0 and max (overshooting less than max would mean having to erase/discharge and rewrite from scratch, no such thing as overshooting SLC, so charging can happen at a faster rate). Reads get the benefit of a simplified analog comparator which can settle much quicker to tell charged vs. not charged, than no charge vs. 1/3 charge vs. 2/3 charge vs. full charge (the 4 levels of MLC). This would be consistent with the published performance numbers, and even provides a plausible theory of why the factory pSCL chips (EM-26 series) are faster than reconfigured MLC chips (EM-20) - the factory ones can have write and read analog circuitry optimized to only read and write 2 levels vs. 4 per cell.

Full disclaimer, this is purely an educated guess based on public data - I have no insider information from Swissbit.

 

[whitex]

Also, the main benefit of using MLC cells for only 2 levels (pseudo SLC), is that it would increase it's robustness to data decay. If a cell charge in MLC cell decays, if it's used in MLC mode 0b11 will start reading 0b10 or even 0b01, end eventually 0b00. With pseudo-SLC, the value in the cell is preserved even the charge has decayed even most of the way, since any charge means 1 - only complete charge loss will corrupt the cell value read in pSLC mode. Perhaps that is why they called pSLC region a "reliable region".

 

[simonb]

Has anyone figured out how to read the proprietary usage/wear/health data out of the Hynix chip? Tesla can read it, so there must be a way.

 

[jcaspar]

I just pulled my MCU for eMMC replacement. I was planning on not replacing the MCU till I got the chip replacement back as I won't be driving the car. Have car on a charger but the dash screen is constantly on. Is there a way to turn off power to this? (Pull another fuse, disconnect 12V?) Would rather avoid dealing with the MCU connectors till I get the chip back. Thanks.

 

[KArnold]

Pull the fireman's loop on the HV, pull the negative cable on the 12v. Put 12v on trickle charger. The car is most definitely not powering anything

I'm expecting my EMMC back tomorrow.

 

[ahkahn]

Yeah, I forgot to pull everything and no battery tender, so I'm guessing I'll also be investing in a new 12v battery.

 

[whitex]

I just pulled my MCU for eMMC replacement. I was planning on not replacing the MCU till I got the chip replacement back as I won't be driving the car. Have car on a charger but the dash screen is constantly on. Is there a way to turn off power to this? (Pull another fuse, disconnect 12V?) Would rather avoid dealing with the MCU connectors till I get the chip back. Thanks.

If you shut off the car (via the big MCU screen) before pulling MCU fuse, the IC should stay off. Not sure which fuse is for the IC, but that would be your next step.

All that said, without the MCU physically connected, your 12V battery will be done in just couple to few hours. If you have no plans on connecting the MCU or supplementing the 12V battery with a tender, might as well disconnect it before it discharges completely, and if you do that, might as well disconnect the fireman loop to shut off the HV to hibernate the car (no need to keep the car plugged in either in that case). Once you get your MCU fixed and back in, you will need a 12V battery that is not completely dead to get things going again - so if it's dead already, you will need an external charge to recharge it, so plan ahead.

 

[jcaspar]

Thanks for the advice. I so far have pulled the #49 fuse which powers the dash screen and have it on a charger. eMMC went out today.

 

[Trebek1762]

Just reinstall your MCU without the tegra and you won’t have to worry about the 12v. It’s just a bunch of plugs and then reinstall the pulled fuse.

 

[Newscutter]

For those that have had MCU failures, this may be of interest. There's at least *some* possibility that the cars could be recalled for free (or reimbursed) replacements of the original eMMC chips. If you've replaced an MCU, you might be well-advised to add your complaints to the NHTSA database as they're only reporting 11 (!) complaints.
https://www.cnbc.com/2020/06/24/us-opens-probe-into-tesla-model-s-after-touchscreen-failure-reports.html

 

[Aggmeister2010]

For those that have had MCU failures, this may be of interest. There's at least *some* possibility that the cars could be recalled for free (or reimbursed) replacements of the original eMMC chips. If you've replaced an MCU, you might be well-advised to add your complaints to the NHTSA database as they're only reporting 11 (!) complaints.
https://www.cnbc.com/2020/06/24/us-opens-probe-into-tesla-model-s-after-touchscreen-failure-reports.html

I'm confused... the article says that investigators are targeting vehicles through 2015, but then mentions that the processors in question are used through 2018?

I wonder if this will come down to the fact that when the screen dies, you lose your back-up camera, which the vehicles are regulated to have.

 

[T.R.T.e.s.l.a.]

For those that have had MCU failures, this may be of interest. There's at least *some* possibility that the cars could be recalled for free (or reimbursed) replacements of the original eMMC chips. If you've replaced an MCU, you might be well-advised to add your complaints to the NHTSA database as they're only reporting 11 (!) complaints.
https://www.cnbc.com/2020/06/24/us-opens-probe-into-tesla-model-s-after-touchscreen-failure-reports.html

Only 11 complains??
Man, if at least 1/4 of all those affect had complained, i guess Tesla would have already been forced to recall all the MCU 1 cars for an Emmc replacement

 

[scaesare]

I don't see a lot of risk in attempting to fix a failed ECU. Soldering the eMMC would be about it. Copying the data off the old eMMC to the new one shouldn't be a big issue. Caveat: I've been a Linux tech since 2002.

Actually, copying data off a failed ECU is a BIG issue.