Preventive eMMC replacement on MCU1

[scaesare]

Good news / bad news:

BAD: I screwed up my scheduling and missed my appointment to have my new chip soldered back on to my MCU daughtercard.


GOOD:

EVEN GOODER:

 

[KLuuppo]

@scaesare congratulations, man! Almost there! How did you get the data out in the end?

I'm just a tiny bit ahead:



@LuckyLuke helped me out by making an intact image with stock fw and keys extracted from my dump. Soldering took still two attempts (my two cents: soak both the board and chip in copious amounts of flux, make sure that your board is not on the edge of preheater).

 

[T.R.T.e.s.l.a.]

So, can any of you guys do a step by step guide on how to actually swap the emmc and put it all working again?
I mean a simple one for all of us uneducated on these matters.

 

[widodh]

So, can any of you guys do a step by step guide on how to actually swap the emmc and put it all working again?
I mean a simple one for all of us uneducated on these matters.

I'm thinking about a blogpost, but such a post takes time, a lot of time to write.

 

[T.R.T.e.s.l.a.]

I'm thinking about a blogpost, but such a post takes time, a lot of time to write.

I understand. Do take your time.
In the meantime i would be a lot more interested to just be able to send my MCU to the shop you've sent yours in NL for them to swap the eMMC for the new one and have all the data transferred to the new chip.

 

[itDied]

@scaesare : That's the goodiest news I have heard all day So nice to hear that your perseverence paid off!

@widodh : We could make this a collaborative effort right?

 

[widodh]

@scaesare : That's the goodiest news I have heard all day So nice to hear that your perseverence paid off!

@widodh : We could make this a collaborative effort right?

Yes, we could A Git repository and writing a markdown document should already be enough.

I'll start with this in Github.

 

[scaesare]

@scaesare congratulations, man! Almost there! How did you get the data out in the end?

I'm just a tiny bit ahead:

View attachment 453279

@LuckyLuke helped me out by making an intact image with stock fw and keys extracted from my dump. Soldering took still two attempts (my two cents: soak both the board and chip in copious amounts of flux, make sure that your board is not on the edge of preheater).

Thanks...

So my previous entire-volume extract I had left left running finally died around 18% or so with a whole bunch of read errors. I restarted it using the existing map, and was still generating tons of read errors. So back in the freezer it went.

After chilling down, I plugged it back in (sandwiched between ice packs, lol), and noticed that I was getting block devices for the individual partitions /dev/sdb1-4), whereas before I was just getting two volume block devices (sdb & sdc).

So I immediately started an extract on p3, as that's where /var (and the important stuff) lives. It rapidly got to about 20%, and then died. I power-cycled my reader to force it to re-detect the block devices. They showed up again and I re-started the extract using the existing map and the appropriate switches to re-try the skipped parts and and re-open the file on error. Boom, got 100% recovered in the space of a few minutes.

I copied off the original extract, mounted a copy via loopback, and no errors that I can see.

I started on p4 last night and it was at about 20% this morning, where it had stalled. Power-cycled the programmer again, re-started, and sure enough it got past that point and started recovering more. We'll see where it's at when I get home.

So... it appears that chilling the chip can help. It also appears that power-cycling the reader, also seems to help.

Also, ddrescue is valuable here... this would probably not work well with a straight dd.

And congrats on having your car going.... I'll have to try and catch up!! lol

 

[scaesare]

Yes, we could A Git repository and writing a markdown document should already be enough.

I'll start with this in Github.

Cool.. I'm in for participation... I've got some stuff collected already that I've shared with some folks.

 

[milleron]

Does degradation of the MMC occur only when the car is in operation? Can it occur when it's sitting in the garage overnight charging? What's the most important factor, miles on the odometer or age of the car?

 

[scaesare]

Does degradation of the MMC occur only when the car is in operation? Can it occur when it's sitting in the garage overnight charging? What's the most important factor, miles on the odometer or age of the car?

The biggest factor by far is writes.

Unfortunately Tesla as left the OS logging turned up and writing to the filesystem on the flash. Logging to tempfs (in RAM) for non-critical events would have largely prevented this.

Just sitting in your garage shouldn't do much.

 

[scaesare]

Oh and:

 

[widodh]

Does degradation of the MMC occur only when the car is in operation? Can it occur when it's sitting in the garage overnight charging? What's the most important factor, miles on the odometer or age of the car?

The MCU is also active (although the screen is off) when charging. While being active it writes to the local eMMC of the MCU.

The eMMC in my Model S from September 2013 did not wear out yet. Even though there is 250.000km on the odometer.

Other Model S/X from 2014 with less KM on the odo did wear out. So I'm not exactly sure what might happen. It will happen eventually though.

My settings:
- Energy saving On
- Always connected Off
- Passive Entry Off
- Auto Present door handles Off
- Never used TeslaFi or anything else

It could be that this helped my S to live longer because the MCU was active less.

 

[S-Pain]

EVEN GOODER:

View attachment 453210


EVEN MORE GOODERER YET:

caesare@HP-Pavillion-Linux:/mnt/var/etc/openvpn$ sudo more car.key
Bag Attributes
    localKeyID: 24 36 21 2D REDACTED REDACTED 42 CC 2A EC CB 7F 82
    friendlyName: 5YJSA1REDACTED
Key Attributes: <No Attributes>
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAs89trJf9ONJBOt0FR5abj/pcHJE+N1AgWwBdFzPclAl/ZlQe
0haXCZ5PrlaNhRCYJZ1qH9OoJ6/FjNJMr9K/I7Hz0gdzYlx0qLkM5n9O4iond55i
zSaVEUj664nBCN77y+ppuqDomv90R+3+8IwEebYycvR3BImRKmg+U4dI8bTW+Gti
39Peby+CHyX+qBpVLmh6s8Zo75aXH0XcOtwT7hvszQp7ll3+yJodMbpASmZLN94f
E6dRnRxreK0D1TXMmdZrdxD1IyR+ziAF1+XWy88dFfDFGcaZPGMM8eZXmtdCuHYy
m/NE1bj909wpSuGd1p7s2qeHDkee3O5gxOxe6wIDAQABAoIBAG0zJz8vWvi8MCNI
SEVERAL LINES
REDACTED BUT
IT'S ALL THERE
7DRPof1ZvycCByTDmTY4j0j4MY4MpvxI6/q8zoPqSZZkpZtzROOHY2PjDJxU9tFG
+fsTXbdCZMcbA+BJt2Ibpv1LsDF+XKBasqn4M4SRDZf+af8YAPXnqqJp9cRQbq3+
cqMo4QKBgQCe/mSDdQaELD1Mt3ORB8BsX4A/QRUSirLKuFxn6vhHPiXKSIXhvFdn
Wxozr3JpVNbGyEpGB1P/1bl55xSMKATlxN7BCPFjMt5tToKWRtbSkaYvxCuEf8Bh
qzUU0dlhp3uf8W1wHCHp8dFPVr3bCmuArHB1gecx4JpDUTx4IOpYzQ==
-----END RSA PRIVATE KEY-----

Yay now you've weakened the security of your private key. I hope it’s specific to your car and can be revoked and a new private key created. I believe they use this for the OpenVPN connection back to Tesla’s network and probably other things too. Now that most of the private key is public the remainder can probably be figured out pretty quickly.

 

[MP3Mike]

Does degradation of the MMC occur only when the car is in operation? Can it occur when it's sitting in the garage overnight charging? What's the most important factor, miles on the odometer or age of the car?

The degradation occurs anytime the MCU is "awake" and active. So one thing that can lead to a quick demise is having third party apps continually polling the car and not allowing it to sleep.

The most important factor is uptime for the MCU. Distance doesn't matter as much because driving 100k miles at an average of 60MPH would degrade it less than driving 100k miles at an average of 30 MPH. (So Model Ss used as a taxi would likely hit this much earlier as they tend to drive slower and be parked waiting for a fair with the MCU active.)

Same hold true for charging, charging at a higher rate would allow the charging to complete sooner which would allow the MCU to go to sleep.

 

[S-Pain]

But sentry mode keeps the car awake too.. so turning off sentry mode. I had my car for about a year with mcu 1 before it failed and had to be replaced. Tesla really needs to fix this even out of warranty. It’s unacceptable.

 

[Flyboyn222pa]

Just curious here. I have a very early 2013 model S with 133,000 miles. I was wondering about autopilot implementation and the eMMC. Since the early cars don’t have autopilot, and mine doesn’t even have tech package, does that limit the amount of logging Tesla is doing and this increase the life of the MCU? Additionally my early car has a 4g LTE MCU. Does this mean it was replaced or can you upgrade to 4g without replacing the MCU?

 

[Plainfittsu]

Hi have been watching for a while and have been inspired to follow , have spent a month collecting parts and researching as much as possible.

Long story short,
Have removed THE Chip, despite best efforts, heating from below to 200C, had to turn heat gun up to 300C as nothing was happening even with plenty of the fancy flux, chip popped off the board looked in good shape.

In to the Allsocket DS3000 no Vcc , blue Data and power on showing. Freeze spray and a couple of hours in freezer no Joy.
(thinking too much time at lower heat or too much pressure easing it off)

Have a couple of Swissbit chips looking for a program and a Tesla out of love

Hows the Github plan coming along? would love to help but not a software person and after 38 years in electrical industry it would seam not much good at electronics.

Was talking to LuckyLuc on the Salvaged car thread he may be interested in the Github plan

 

[MP3Mike]

Just curious here. I have a very early 2013 model S with 133,000 miles. I was wondering about autopilot implementation and the eMMC. Since the early cars don’t have autopilot, and mine doesn’t even have tech package, does that limit the amount of logging Tesla is doing and this increase the life of the MCU? Additionally my early car has a 4g LTE MCU. Does this mean it was replaced or can you upgrade to 4g without replacing the MCU?

No and you can easily upgrade to 4G.

 

[widodh]

Hi have been watching for a while and have been inspired to follow , have spent a month collecting parts and researching as much as possible.

Long story short,
Have removed THE Chip, despite best efforts, heating from below to 200C, had to turn heat gun up to 300C as nothing was happening even with plenty of the fancy flux, chip popped off the board looked in good shape.

In to the Allsocket DS3000 no Vcc , blue Data and power on showing. Freeze spray and a couple of hours in freezer no Joy.
(thinking too much time at lower heat or too much pressure easing it off)

Have a couple of Swissbit chips looking for a program and a Tesla out of love

Hows the Github plan coming along? would love to help but not a software person and after 38 years in electrical industry it would seam not much good at electronics.

Was talking to LuckyLuc on the Salvaged car thread he may be interested in the Github plan

I am writing up some things to share my experience.

Without the original data you are lacking some certificates to have the car talk to Tesla through a VPN. That’s a problem which is not easy to fix.