Successful data recovery of broken eMMC chip MCU1

[widodh]

For more information about what happens when the eMMC chip dies, see: Consolidated eMMC Thread (MCU repair)

My story was that last summer (September 2019) the eMMC chip of my Model S from 2013 failed. It has 260.000km on the ODO and the chip failed before I was able to replace it.

I did put on a new chip (64GB Swissbit), but I lost my (Open)VPN certificates.

Through the Service Centers I tried to get Tesla to perform a re-install of my car, but they refused. I wrote to their European HQ a couple of times, they refused. The only thing they would do is replace my MCU and only charge for the parts.

As I refused to pay the ~EUR 2000 for a new MCU I send my failed eMMC chip to Poland: Multi-Com.eu » Cell Phone Service, Repair, Unlock - Software, Tools, Devices

In my own equipment this chip wouldn't respond anymore, but these guys did an amazing job!

At first they said: The chip is toast.

Then they came up with a solution: Rrder exactly the same chip and by reading the NAND memory directly using ECC corrections using the new donor chip they might have a shot at it.

My chip needed to be cooled to -28C while they were reading data from it.

Reading was slow; super, super slow. It took 8 weeks to get the bits of the chip and to restructure the data. But they did it!

This morning I received the files I needed:

- car.crt
- car.key
- ta.key

They charged me a flat-fee of EUR 250 for this service. They say it took them way to much time to make a profit on it, so I paid them something in addition. It was still way cheaper then replacing the whole MCU.

So should you be in the same boat as I was: Use these guys from Poland!

 

[amiral_sub]

unbelievable ! I'm very impressed.

 

[widodh]

unbelievable ! I'm very impressed.

Yeah, me too! I honestly didn't expect this.

But when I received the e-mail this morning with "openvpn.zip" as an attachment containing the files I needed I was very happy.

 

[T.R.T.e.s.l.a.]

That is great Wido, and thanks for letting us know about those guys in Poland.
But i always thought you had successfully saved all the data from the emmc you replaced on your S85.

 

[rooter]

Nice, Wido. Good idea to cool it. One thing I've noticed about failed chips is you don't just get read errors on partition 3. (var) You often see them on all other parts too. This implies that it is the support circuitry to blame, which wouldn't surprise me. (yaay, Hynix)

It's not clear why vpn certs were necessary, unless you want Tesla to continue to reach the car. (Assuming you have the same SIM card)

Also you might ask Multi-Com for birthday, vin, and gateway.cfg files.

 

[SoCal Buzz]

Nice job...! What was the status of your car while you waited to get the data files? Completely down?

 

[widodh]

That is great Wido, and thanks for letting us know about those guys in Poland.
But i always thought you had successfully saved all the data from the emmc you replaced on your S85.

I kept it silent as I was trying to get Tesla to help me. I didn't want them to read on TMC that I replaced my eMMC and lost my certs.

Nice, Wido. Good idea to cool it. One thing I've noticed about failed chips is you don't just get read errors on partition 3. (var) You often see them on all other parts too. This implies that it is the support circuitry to blame, which wouldn't surprise me. (yaay, Hynix)

It's not clear why vpn certs were necessary, unless you want Tesla to continue to reach the car. (Assuming you have the same SIM card)

Also you might ask Multi-Com for birthday, vin, and gateway.cfg files.

I want the VPN so that firmware updates work, Spotify works, the Tesla app, etc, etc. My wife drives this car.

Nice job...! What was the status of your car while you waited to get the data files? Completely down?

It was working. We put in a new eMMC with the proper firmware. So the car would drive, charge, everything. Remote access and firmware updates weren't working though.

 

[T.R.T.e.s.l.a.]

I kept it silent as I was trying to get Tesla to help me. I didn't want them to read on TMC that I replaced my eMMC and lost my certs.


I want the VPN so that firmware updates work, Spotify works, the Tesla app, etc, etc. My wife drives this car.


It was working. We put in a new eMMC with the proper firmware. So the car would drive, charge, everything. Remote access and firmware updates weren't working though.

Yeah, that's understandable.

Regarding firmware updates it's probably better that you just leave it as it is... haha.
What version do you have installed currently?

 

[whitex]

I kept it silent as I was trying to get Tesla to help me. I didn't want them to read on TMC that I replaced my eMMC and lost my certs.


I want the VPN so that firmware updates work, Spotify works, the Tesla app, etc, etc. My wife drives this car.


It was working. We put in a new eMMC with the proper firmware. So the car would drive, charge, everything. Remote access and firmware updates weren't working though.

Out of curiosity, have you tried using your other car's certs/keys? I wonder if Tesla bothers to check whether the cert matches the car and/or how many simultaneous sessions there are (if only the latter, a donor dead MCU could be cheaper than new MCU too).

 

[widodh]

Out of curiosity, have you tried using your other car's certs/keys? I wonder if Tesla bothers to check whether the cert matches the car and/or how many simultaneous sessions there are (if only the latter, a donor dead MCU could be cheaper than new MCU too).

No, I haven't. I also heard that it doesn't work. They strictly check this. The car also signs it's HTTP request using certificates.

 

[whitex]

No, I haven't. I also heard that it doesn't work. They strictly check this. The car also signs it's HTTP request using certificates.

The 3 files you recovered look like standard openvpn configuration files (public cert, private key, pre-shared HMAC key used to add an additional layer of tls authentication to each packet). How does HTTP browser interact with those files?

 

[dannycamps]

Perhaps compare the files you received to a set from another car and see what the differences are.

 

[widodh]

The 3 files you recovered look like standard openvpn configuration files (public cert, private key, pre-shared HMAC key used to add an additional layer of tls authentication to each packet). How does HTTP browser interact with those files?

If you check the running processes you'll see that it's the hermes proxy doing this:

/opt/hermes/hermes_proxy --log-level=info --ca=/etc/ca.crt --cert=/etc/openvpn/car.crt --key=/etc/openvpn/car.key --api-server-host=api-prd.vn.tesla.services --unix-socket-buffer=200000 --http-signature-pub-keys=/etc/carserver-pub-keys/production.json

When it sends HTTP requests to Tesla's servers it seems to sign requests with this so that the HTTP server can verify who is doing the call.

So without these certificate files you can't connect to OpenVPN and if you are connected your VIN is encoded in all the requests you do.

 

[LaurS]

I plan to start taking things apart at the end of next week. I am worried that the eMMC will get damaged during removal from the board. I found this video where it show data being read while the eMMC is still on the board, only using 4 connection points:

minute1:04. I assume two wires are power an two data. Does anybody know what communication device they used and what connection points? Is it worth trying to figure this out or just go with removal and reballing? I am in Portland OR area, any recommended shop to do the eMMC removal, reball and new eMMC install? Thank you.

 

[squall333]

 

[squall333]

Based on this image it looks like possible

CLK
dD

and two pins that arent on this maybe a 1.8v thats located from somewhere else?

 

[rooter]

squall333 you need to credit that image. It is not yours.

And the guys who made the above video know absolutely nothing about static, its effects on electronics, nor preventative measures.

BTW, I now have permission to publish the all-powerful Fusee Gelee hack adapted to Tesla by Sid. In theory this should make possible downgrading firmware.

But don't expect it to be easy. TonyT et al learned how to replace the eMMC, root, and pull images from my articles. Maybe at some point they'll be able to offer firmware downgrade.

All I have to do now is figure it out...

 

[squall333]

@rooter its not common internet etiquette to credit reference images posted freely on the internet. Taking credit for images you didnt take is another story

 

[rooter]

@rooter its not common internet etiquette to credit reference images posted freely on the internet. Taking credit for images you didnt take is another story

Typical priggish NJ attitude.

I'll do it for you then. That picture was taken by ce2078 and it is his reverse-engineering depicted.

In open-source we all stand on the shoulders of others and give credit for their science, as we hope others give credit for our work. A squall333 wouldn't know this but we share to help others, and as others share our work we deserve credit as the basis.

To quote George Costanza, "We are supposed to be having a CIVILIZATION here people!"

 

[Slexs]

Based on this image it looks like possible

CLK
dD

and two pins that arent on this maybe a 1.8v thats located from somewhere else?

Im a bit confused when you say pin dD, what pin are you referring to here?
Im in the process of planning emmc swap out for my own car, whst pins are actually connected/used to ie. AllSocket to get the data out?