Preventive eMMC replacement on MCU1

[KLuuppo]

I guess I should report my proceedings as well, even though there haven't been much success.

Desoldering, I preheated the board to 200C and then used hot air at 380C and the chip came off easily in ~10sec. The chip was readable with allsocket usb reader.

Ddrescue recovered ~97% of the data, and I assumed the lost part would be on the worn out partitions of the chip. I tried two times to dd partitions 1-3 to a new chip and soldering it on, but the thing stayed dead.

Now that I had a bit more time to see to it, it seems that the bad sectors are spread all around the disk. Unsquashfs couldn't unpack the partitions. It tells me that p1 is of unsupported version and "failed to read fragment table" on p2. I suppose these images could be obtained from IC board. However, this is not my car and it's going to Service center on Friday so probably have to admit defeat on this one as well. If anyone has intact FW images (and willing to share) I could still try flashing them on.

My spare Tegra is now with Allsocket to be studied.

 

[3mp_kwh]

No expert, here, but I also remember lower temps to "re-ball". Not saying I trust everyone on the internet, but if a number are using just over 300C I'd be concerned about 390. This limit should be on a Hynix spec sheet, or tech doc someplace? Speaking of tech docs:

Since Phil left the building, I was able to get a chip pic as I was putting things back together. I think I'm seeing H26M42003GMR, which at least has Google hits. The MCU was not out of the dash for this pic. It was taken with cell-phone, from the upper right side of the cage.

If among thermal limit details, anyone knows the skinny on viable readers for this chip, etc., it would be good to know what to expect?? Otherwise, hope would be it operates same as ...42001FMR, and can similarly be replaced by Swissbit.

[EDIT: it would be good to be reminded, by those posting, if the chips we're trying to read came out of dead/black MCU's. This could give the 'Preventative' crowd a little more hope. Thx]

 

[scaesare]

Replacing the eMMC of my MCU (September 2013) didn't go that well this weekend.

My MCU has the Hynix H26M42001FMR 8GB chip and after removing the chip from the MCU it wouldn't read.

There could be two reasons for this:

1: The H26M42001FMR chip needs a special reader/programmer. I doubt this
2: The eMMC died during the desoldering. The guy doing this for me took special car with the chip by desoldering at a maximum of 390C.

We wrote a new image with firmware 2019.20.2.1 (You need to know how to do this) on a new 64GB Swissbit eMMC and soldered that on the MCU.

The MCU booted with the new image as it's responding to Ethernet commands and the network came online. The MCU is being send back to me today and I'll put it back in the car tomorrow.

The homebrew Image is however lacking:

- VPN certificates for Tesla
- Some VIN specific information

We can hopefully recover this from the old eMMC with a different programmer, but otherwise a few features of the car like the app won't work anymore.

I'll update tomorrow with more information once the MCU is running again in the car.

View attachment 452514 View attachment 452515 View attachment 452516 View attachment 452517

Excellent!

 

[scaesare]

I guess I should report my proceedings as well, even though there haven't been much success.

Desoldering, I preheated the board to 200C and then used hot air at 380C and the chip came off easily in ~10sec. The chip was readable with allsocket usb reader.

Ddrescue recovered ~97% of the data, and I assumed the lost part would be on the worn out partitions of the chip. I tried two times to dd partitions 1-3 to a new chip and soldering it on, but the thing stayed dead.

Now that I had a bit more time to see to it, it seems that the bad sectors are spread all around the disk. Unsquashfs couldn't unpack the partitions. It tells me that p1 is of unsupported version and "failed to read fragment table" on p2. I suppose these images could be obtained from IC board. However, this is not my car and it's going to Service center on Friday so probably have to admit defeat on this one as well. If anyone has intact FW images (and willing to share) I could still try flashing them on.

My spare Tegra is now with Allsocket to be studied.

A couple of thoughts:

- Try ddrescue and let it do multiple passes on the partitions

- Once you have as much data as ddrescue can retrieve, make a copy of the partitions & fsck them (or at least p3 /var)

- Mount file system(s) locally via loopback

- See if that prvides enough recovered/fixed filesystem to grab your car-specific data out of /var

-If the above doesn't work, you can try fsck'ing p3 on the chip directly.

 

[widodh]

I guess I should report my proceedings as well, even though there haven't been much success.

Desoldering, I preheated the board to 200C and then used hot air at 380C and the chip came off easily in ~10sec. The chip was readable with allsocket usb reader.

Ddrescue recovered ~97% of the data, and I assumed the lost part would be on the worn out partitions of the chip. I tried two times to dd partitions 1-3 to a new chip and soldering it on, but the thing stayed dead.

Now that I had a bit more time to see to it, it seems that the bad sectors are spread all around the disk. Unsquashfs couldn't unpack the partitions. It tells me that p1 is of unsupported version and "failed to read fragment table" on p2. I suppose these images could be obtained from IC board. However, this is not my car and it's going to Service center on Friday so probably have to admit defeat on this one as well. If anyone has intact FW images (and willing to share) I could still try flashing them on.

My spare Tegra is now with Allsocket to be studied.

p1 and p2 are not very important, you can use the generic Tesla image.

The /var/etc files containing the VPN certificates are the ones you really want. They are on p3 and you want that.

No expert, here, but I also remember lower temps to "re-ball". Not saying I trust everyone on the internet, but if a number are using just over 300C I'd be concerned about 390. This limit should be on a Hynix spec sheet, or tech doc someplace? Speaking of tech docs:

I'm just not sure. I trust the guy who did this work for me.

He had another 42001 chip he was able to read flawlessly, so it's still unsure why this is the case.

The 42001 chip seems to be present in 2013 and early 2014 cars. Onwards Tesla switched to a different chip.

Since Phil left the building, I was able to get a chip pic as I was putting things back together. I think I'm seeing H26M42003GMR, which at least has Google hits. The MCU was not out of the dash for this pic. It was taken with cell-phone, from the upper right side of the cage.

If among thermal limit details, anyone knows the skinny on viable readers for this chip, etc., it would be good to know what to expect?? Otherwise, hope would be it operates same as ...42001FMR, and can similarly be replaced by Swissbit.

I now have a Swissbit 64GB on my MCU. Installing the MCU tomorrow

 

[scaesare]

p1 and p2 are not very important, you can use the generic Tesla image.

Minor clarification on this point: you can use an obtained copy of the images provided they match the last one active on your car. The crypto signatures on the active eMMC partition and the Spansion processor (also on the daughter card) have to match.

 

[scaesare]

Well, this is.... "interesting"...

On a lark I took my troublesome previously-unreadable Hynix eMMC chip and put it in my programmer, and tossed the whole thing in the freezer for a few hours.

Once it was good and chilly, I plugged it in and quickly tried to run a data extraction. Lo-and-behold... after an initial minute or so of no activity, I started getting data read:

After about 10 minutes or so, it stalled where you see it here at 9.84%. It edged up to 9.85% after another 5 minutes or so while composing this. Most notable are no read errors reported. And my extract filesize is non-zero this time.

So... one of a few possible things:

1) The cold trick actually works, and once the chip warmed back up some, my extract stalled.

2) This new reader (DS300) actually is compatible with this chip, and I simply threw the towel in too early when it seemed to stall out of the gate as my previous SD-card reader did, and chilling the chip has nothing to do with it.

3) The chip is only somewhat readable, and regardless of either #1 or #2 above being the case, I'm only going to get a small portion of it.


Whichever the case... I'm going to let this bake a while as long as read-errors are reported as zero (or remain low), and the percent rescued continues to climb. Once that progress seems to stop, I'll toss it back in the freezer for a bit, and then try to resume... ddrescue keeps a map file of it's progress, so I think I should be able to pick up where I left off....

Maybe there's hope yet...

On Edit: After 20 more minutes it edged up to 9.86%...

Edit #2: After almost another half, hour, up to 9.87%! At this rate it's estimating 108 days, lol... I'm going to bed.

 

[KLuuppo]

Maybe try running the usb cable to freezer and extract while cold?

 

[brkaus]

There are also various freeze sprays for component testing.

 

[scaesare]

After running over night:

 

[T.R.T.e.s.l.a.]

After running over night:

View attachment 452938

Man.... that is sloooow...

 

[scaesare]

Maybe try running the usb cable to freezer and extract while cold?

I've thought about exactly that.

Might have to order me a long USB3 cable.

 

[scaesare]

There are also various freeze sprays for component testing.

Yeah, unfortunately with the clamshell socket these use, I can't get it directly on the chip during operation. I'm not sure how much effective cooling would get to the chip itself during operation witht he socket closed.

I left the socket open while in the freezer, and let the whole unit cold-soak.

I did spend some time thinking about what hose adapters I'd need to turn my beer brewing heat exchanger and wort pump in to a chilled mineral oil bath to immerse the entire thing in...

 

[scaesare]

Man.... that is sloooow...

True. But it's faster than "unreadable", hence never.

Honestly, if this takes a week or two, I won't care. It's not going to prevent me from getting the car back on the raod at this point.

It seems to go in fits and starts... when it was chilled, it ran up to almost 10% in the first 20 mins or so. Then it took almost an hour to advance another 0.02%. After 7 hours over night, it advanced another ~4%.

So the cold may ave helped accelerate the first portion... or it just may be that some sections of the chip are more toasted than others.

Again, as long as it's making progress, I'm inclined to let it go. If it stalls, I'll freeze it again and restart.

 

[widodh]

Today I received my MCU and put it back into the car.

First a picture of the Swissbit eMMC on the MCU:



If you look closely you can see the Swissbit 64GB eMMC on there.

My cars are parked in a public parking garage and when I arrived the car wouldn't respond to the key. It was dead. I suspected the 12V.

I removed the nosecone and used my S100D to Jump Start the 12V, that worked!



While the 12V was charging I could unlock and open the doors.

After about 10 minutes the MCU as in there and it was time to put the 12V fuse back in so the MCU would be powered. After 1 minute both screens came on and the car was starting to get back to life!



And about 2 minutes later everything started to work again:



Voila! There we go.

I plugged the car into one of the 22kW chargers in the garage so that it could charge the HV and 12V battery while I disconnected it from my S100D. In the meantime I also started to assemble the dashboard.

After letting it charge for 30 minutes I took it out for a drive and it just works! It charges, drives, etc.

All settings were lost as we needed to rebuild a new firmware image for this car. I had to set the driver profiles, navigation, favorite radio stations, etc.

 

[scaesare]

Today I received my MCU and put it back into the car.

First a picture of the Swissbit eMMC on the MCU:

View attachment 452950

If you look closely you can see the Swissbit 64GB eMMC on there.

My cars are parked in a public parking garage and then I arrived with the MCU the car wouldn't respond anymore to the key. It was dead. I suspected the 12V.

I removed the nosecone and used my S100D to Jump Start the 12V, that worked!

View attachment 452951

While the 12V was charging I could open the unlock and open the doors and start to get the MCU back in the car.

After about 10 minutes the MCU as in there and it was time to put the 12V fuse back in so the MCU would be powered. After 1 minute both screens came on and the car was starting to get back to life!

View attachment 452952

And about 2 minutes later everything started to work again:

View attachment 452953

Voila! There we go.

I plugged the car into one of the 22kW chargers in the garage so that it could charge the HV and 12V battery while I disconnected it from my S100D. In the meantime I also started to assemble the dashboard.

After letting it charge for 30 minutes I took it out for a drive and it just works! It charges, drives, etc.

All settings were lost as we needed to rebuild a new firmware image for this car. I had to set the driver profiles, navigation, favorite radio stations, etc.

YESSS!!!

Congrats @widodh , great news!

There's a chance I'll be doing similar this evening...

 

[T.R.T.e.s.l.a.]

Today I received my MCU and put it back into the car.

First a picture of the Swissbit eMMC on the MCU:

View attachment 452950

If you look closely you can see the Swissbit 64GB eMMC on there.

My cars are parked in a public parking garage and when I arrived the car wouldn't respond to the key. It was dead. I suspected the 12V.

I removed the nosecone and used my S100D to Jump Start the 12V, that worked!

View attachment 452951

While the 12V was charging I could unlock and open the doors.

After about 10 minutes the MCU as in there and it was time to put the 12V fuse back in so the MCU would be powered. After 1 minute both screens came on and the car was starting to get back to life!

View attachment 452952

And about 2 minutes later everything started to work again:

View attachment 452953

Voila! There we go.

I plugged the car into one of the 22kW chargers in the garage so that it could charge the HV and 12V battery while I disconnected it from my S100D. In the meantime I also started to assemble the dashboard.

After letting it charge for 30 minutes I took it out for a drive and it just works! It charges, drives, etc.

All settings were lost as we needed to rebuild a new firmware image for this car. I had to set the driver profiles, navigation, favorite radio stations, etc.

AWESOME!! Congrats!

One of these days i may make a trip to NL, so you can help me change the EMMC on my S85

 

[T.R.T.e.s.l.a.]

YESSS!!!

Congrats @widodh , great news!

There's a chance I'll be doing similar this evening...

Fingers crossed!

 

[3mp_kwh]

Congrats, widowh!

We wrote a new image with firmware 2019.20.2.1 (You need to know how to do this) on a new 64GB Swissbit eMMC and soldered that on the MCU.

Need to know how to A) Go find someone with basic LINUX skills, or B) Bone up and DIY, because it's like learning DOS?

The homebrew Image is however lacking:
- VPN certificates for Tesla
- Some VIN specific information

What was the fix?

 

[widodh]

Congrats, widowh!


Need to know how to A) Go find someone with basic LINUX skills, or B) Bone up and DIY, because it's like learning DOS?

You need Linux skills indeed to do this. It's like a very Advanced version of DOS. Will take a bit of time to learn though

What was the fix?

The VIN information is all read by the firmware on boot from a 'gateway' inside the car. It keeps the ODO, VIN and other vehicle specific information.