Wiki Consolidated eMMC Thread (MCU repair) (Black Center Screen)

[jiten87]

Hello

Tesla is saying that my MCU1 2014 S60 is ineligible for an infotainment upgrade. Has anyone successfully done the $2500 upgrade on a similar build? Thanks!

Edit: Some more information - The center screen on the model S in blank and wont turn on, reached out to Tesla and inquired about the infotainment upgrade and they said it will only go from MCU1 to MCU1

 

[DaveBC]

Hello

Tesla is saying that my MCU1 2014 S60 is ineligible for an infotainment upgrade. Has anyone successfully done the $2500 upgrade on a similar build? Thanks!

Edit: Some more information - The center screen on the model S in blank and wont turn on, reached out to Tesla and inquired about the infotainment upgrade and they said it will only go from MCU1 to MCU1

Currently the software on MCU2 only works on AP HW2+ cars.
Described here Tesla confirms infotainment system upgradeability from MCU1 to MCU2 for $2500

 

[JakeM]

Hey all. First post.
Can someone that does repairs for the eMMC and MCU in general reach out to me please? I tried to message but I’m not yet allowed.

Dec 2013 P85 with symptoms of failing eMMC and a huge bubble on MCU screen.

I’m looking for suggestions and costs.

thanks!!

Jake
[email protected]

 

[EV-Fixme]

This weekend we replaced a 2018 mcu1 chip, one of the last mcu1 cars. The chip was about 50%-60% used up in write cycles, but already the chip was pretty slow and feeling worn out and starting to show some of our listed signs of failure.

Don't wait too long to upgrade if your planning to keep your car. Many of our old chips we get are really bad and sometimes require much more expensive recovery for the key files.

 

[EV-Fixme]

Today we recovered a chip with about 2100 writes, corruption on all partitions. Another with about 2200 writes, also heavy corruption. Both were mcu's that failed to boot.

a P85 2013 came a long way for preventive emmc and screen fix. I'm glad they came, they chip was ultra slow, owner didn't realize how bad it was since they only had the car for a bit over a year. I can see black screens happening any day now. Another note, car got 1 bar of att when arrived, and left with new emmc with full 5 bars of att 3G. Unfortunately sims are restricted again and I was not able to upgrade the car to LTE.

Waiting to get a few extra years may save $20-$30, but the risk is lost of files and needing a Tesla replacement. I've heard rumors of Tesla supplying 16 gig upgrades now. To debunk this I'm willing to remove the MCU from a car that has claimed to have a 16 gig chip from Tesla for free so we can find out if there is any truth to this.

 

[whitex]

Today we recovered a chip with about 2100 writes, corruption on all partitions. Another with about 2200 writes, also heavy corruption. Both were mcu's that failed to boot.

a P85 2013 came a long way for preventive emmc and screen fix. I'm glad they came, they chip was ultra slow, owner didn't realize how bad it was since they only had the car for a bit over a year. I can see black screens happening any day now. Another note, car got 1 bar of att when arrived, and left with new emmc with full 5 bars of att 3G. Unfortunately sims are restricted again and I was not able to upgrade the car to LTE.

Waiting to get a few extra years may save $20-$30, but the risk is lost of files and needing a Tesla replacement. I've heard rumors of Tesla supplying 16 gig upgrades now. To debunk this I'm willing to remove the MCU from a car that has claimed to have a 16 gig chip from Tesla for free so we can find out if there is any truth to this.

The rated number of writes is a typical number assuming typical usage conditions. This means that some number of chips will be better, other worse, but the majority should make it to the rated number of writes if operated in typical conditions (think typical bell curve distribution). If chips gets hotter or colder than typical during its life, the wear occurs faster. There is even some relationship of when you write cold and read hot, or vice versa (I don't remember which one which one is worse, but when I was talking with some experts on flash chips a little while back this was also mentioned as a factor). Then there is an issue of decay, where blocks were written a long time ago and never refreshed, so even if residing in moderately worn cell, potentially written in less than optimal conditions, could decay faster than expected. Back in the day when Tesla was designing it, there were some refresh protocols available to help mitigate that, but not for all chips, always proprietary, and often causing major performance hickups (like locking the chip up for up to 1 second if the block refresh was triggered) - we don't know whether Tesla implemented any of such protocols (or even whether it was available in the Hynix firmware on those parts). Note, those refresh algorithms always required host software participation, if for nothing else but to track which cell was written how long ago (flash chips do not have a reliable real time clock to be able to tell this). This also means that if everything is powered off, no way to refresh. One of the worst case scenarios for emmc flash retention auto manufacturers consider is parking a powered down car in a desert for few months - cycling of hot/cold, combined with no power, is the perfect storm for retention failure.

Bottom line, no surprise that you find chips at only 70% rated writes which are failing. Cars are subject to some extreme temperatures during normal usage.

 

[EV-Fixme]

The rated number of writes is a typical number assuming typical usage conditions. This means that some number of chips will be better, other worse, but the majority should make it to the rated number of writes if operated in typical conditions (think typical bell curve distribution). If chips gets hotter or colder than typical during its life, the wear occurs faster. There is even some relationship of when you write cold and read hot, or vice versa (I don't remember which one which one is worse, but when I was talking with some experts on flash chips a little while back this was also mentioned as a factor). Then there is an issue of decay, where blocks were written a long time ago and never refreshed, so even if residing in moderately worn cell, potentially written in less than optimal conditions, could decay faster than expected. Back in the day when Tesla was designing it, there were some refresh protocols available to help mitigate that, but not for all chips, always proprietary, and often causing major performance hickups (like locking the chip up for up to 1 second if the block refresh was triggered) - we don't know whether Tesla implemented any of such protocols (or even whether it was available in the Hynix firmware on those parts). Note, those refresh algorithms always required host software participation, if for nothing else but to track which cell was written how long ago (flash chips do not have a reliable real time clock to be able to tell this). This also means that if everything is powered off, no way to refresh. One of the worst case scenarios for emmc flash retention auto manufacturers consider is parking a powered down car in a desert for few months - cycling of hot/cold, combined with no power, is the perfect storm for retention failure.

Bottom line, no surprise that you find chips at only 70% rated writes which are failing. Cars are subject to some extreme temperatures during normal usage.

Thanks I was thinking something like that but had not found info yet. I think cold is worse, I remember reading that the chips work best once they are warmed up. Too hot also hurts it. I'm finding things are for sure in a bell curve, some of the old (7 year) emmcs that are working are in generally pretty hot climates, at the same time, many in the same areas are failing after 3 years.

Also I tried a few chips that were initially still readable after removal from Tegra, now several months later most of them are not readable either after sitting around unpowered. No really huge temp swings as they are sitting in my house. Still waiting till I have enough to see if "will eMMC's blend"

Even more reason to use big chips. And replace early.

 

[Coburnsc]

How much it it to send the MCU out to repair the EMMC? SC said they got mine back up after this last update. They wanted 2k to replace my MCU. Even though they just replaced the screen less than a year ago.

 

[emulsifide]

Thanks I was thinking something like that but had not found info yet. I think cold is worse, I remember reading that the chips work best once they are warmed up. Too hot also hurts it. I'm finding things are for sure in a bell curve, some of the old (7 year) emmcs that are working are in generally pretty hot climates, at the same time, many in the same areas are failing after 3 years.

Also I tried a few chips that were initially still readable after removal from Tegra, now several months later most of them are not readable either after sitting around unpowered. No really huge temp swings as they are sitting in my house. Still waiting till I have enough to see if "will eMMC's blend"

Even more reason to use big chips. And replace early.

To add to the conversation, here's a JEDEC presentation on how temperature affects data retention in solid state hard drives:

https://www.jedec.org/sites/default/files/Alvin_Cox [Compatibility Mode]_0.pdf

Page 27 has a neat chart that shows retention of data drops as the powered-off temperature rises. As long as the chip is actively powered and at reasonably warm temps though, retention is at its highest. I'm not sure how this correlates to eMMC, but I can only imagine it's similar.


To shift gears for a minute, I'm very interested in pulling the trigger on doing a preventative eMMC replacement with you, but I have concerns that I have not been able to find an answer for.

I'm assuming the eMMC contains a lot of private data (GPS locations, personal identifiers/tokens with Tesla, etc.) that I don't want getting into the wrong hands. Granted, I can do a master reset to wipe the data, but I'm also assuming that just zeros out the partition table and the data is still available to pull using ddrescure or any other standard data recovery method once you have physical connectivity to the chip.

What is your stance on data privacy? Again, I want to send my board to you for preventative repair, but I need written commitment that my data is treated as confidential, only pulled to copy directly to the new chip, that all backups are deleted after the job is finished, and that you agree not to distribute my data in any form what-so-ever.

I'd also like to see a statement on any modifications you're making to the operating system since I know Electrified Garage advertises they root the OS and enable remote access (they don't say if it's just SSH or something else). I don't wish for anyone other than Tesla and myself to have access to my car remotely.

 

[EV-Fixme]

To add to the conversation, here's a JEDEC presentation on how temperature affects data retention in solid state hard drives:

https://www.jedec.org/sites/default/files/Alvin_Cox [Compatibility Mode]_0.pdf

Page 27 has a neat chart that shows retention of data drops as the powered-off temperature rises. As long as the chip is actively powered and at reasonably warm temps though, retention is at its highest. I'm not sure how this correlates to eMMC, but I can only imagine it's similar.


To shift gears for a minute, I'm very interested in pulling the trigger on doing a preventative eMMC replacement with you, but I have concerns that I have not been able to find an answer for.

I'm assuming the eMMC contains a lot of private data (GPS locations, personal identifiers/tokens with Tesla, etc.) that I don't want getting into the wrong hands. Granted, I can do a master reset to wipe the data, but I'm also assuming that just zeros out the partition table and the data is still available to pull using ddrescure or any other standard data recovery method once you have physical connectivity to the chip.

What is your stance on data privacy? Again, I want to send my board to you for preventative repair, but I need written commitment that my data is treated as confidential, only pulled to copy directly to the new chip, that all backups are deleted after the job is finished, and that you agree not to distribute my data in any form what-so-ever.

I'd also like to see a statement on any modifications you're making to the operating system since I know Electrified Garage advertises they root the OS and enable remote access (they don't say if it's just SSH or something else). I don't wish for anyone other than Tesla and myself to have access to my car remotely.

Good information.

• I do not modify or support root.
• I provide you an additional copy of your data for your safe keeping.
• I'm collecting all the old chips make a "will it blend" video in the future. If you want I can send your original chip back to you. Generally the data gets destroyed when removing as chips are close to end of life. Preventive may be different and last longer.
• I do hang on the data for a short while to wait you have your board back (mail issues... Rare but just never know) I only store the data local, and remove after a short time after confirmation of success. I do not retain data long after.
• I do not share your personal data with anyone else.

I will pm you with additional information.

 

[EV-Fixme]

No promises but I may do additional fun videos like shorting out the emmc, running high voltage through them to see if the "burn or explode" I do have a little destructive side so once I have time I do want to make some "interesting" videos.

 

[randompersonx]

I've got a December 2014 P85D (with Ludicrous upgrade).
My MCU1 just failed, for the second time. Fortunately I have the Extended Service Agreement.

The previous time it was replaced I think in 2018. I was pretty disappointed at the time that I was made to wait several weeks, and it was replaced with another MCU1 with 3G modem. Hoping this time they at least put in an LTE modem.

I was wondering though, since it seems the MCU didn't even make it 2 years this time, is it perhaps wise to just replace the eMMC very early on to avoid troubles in the future when the warranty is finally up? My warranty still goes out for a while, so I will probably roll the dice until closer to the warranty is up, personally -- but might be a warning for others since I really didn't expect it to fail again so quickly.

 

[EV-Fixme]

I've got a December 2014 P85D (with Ludicrous upgrade).
My MCU1 just failed, for the second time. Fortunately I have the Extended Service Agreement.

The previous time it was replaced I think in 2018. I was pretty disappointed at the time that I was made to wait several weeks, and it was replaced with another MCU1 with 3G modem. Hoping this time they at least put in an LTE modem.

I was wondering though, since it seems the MCU didn't even make it 2 years this time, is it perhaps wise to just replace the eMMC very early on to avoid troubles in the future when the warranty is finally up? My warranty still goes out for a while, so I will probably roll the dice until closer to the warranty is up, personally -- but might be a warning for others since I really didn't expect it to fail again so quickly.

This is on the short side for sure. I would have love to see how many write cycles were on that chip to get as a data point. I do recommend anyone not in warranty and planing to keep their car to upgrade to be safe. It's much easier and cheaper to do preventive. Some who decide to upgrade after has spend much additional dollars and time to get additional specialty recovery of the certs and somehow get through to Tesla to get supplied the certs, most have been in countries where laws help the owner much more. Hope you get LTE this time, really suprised you didnt last time.

 

[EV-Fixme]

I've been hearing a lot of people tell us that the Service Centers can to a MCU replacement till the get inventory, and it's weeks out. If that really is the case people are really going to need help. If you have removed your chip to send to us, or any good MCU repair companies, feel free to use this thread to post a location you can help with removal of Tegra to send out, or post if you need someone to help you with the removal. Friends don't let friends be stuck with dead MCU's.

Also added to top post

 

[idoco]

Last year our 2013 MS developed bubbles in the screen. The screen was replaced under warranty.

Would the eMMC have been replaced as part of the repair? Trying to deciding whether to replace the eMMC since the MS goes out of warranty next year.

-------------------------
Replaced MCU touchscreen due to internal failure.
Correction: Replace MCU Touchscreen Panel Model S
Parts Replaced or Added
Part Quantity
SX-MCU SUB-ASY, SHIP. OGS DSPLY(1084333-00-C)

 

[EV-Fixme]

Last year our 2013 MS developed bubbles in the screen. The screen was replaced under warranty.

Would the eMMC have been replaced as part of the repair? Trying to deciding whether to replace the eMMC since the MS goes out of warranty next year.

-------------------------
Replaced MCU touchscreen due to internal failure.
Correction: Replace MCU Touchscreen Panel Model S
Parts Replaced or Added
Part Quantity
SX-MCU SUB-ASY, SHIP. OGS DSPLY(1084333-00-C)

That is display only, actually current one that is supposed to not yellow or leak, so should be a good one, time will tell. I would wait till warranty is out, then replace the emmc the day after. We can help you.

 

[GoBlue88]

My 2nd MCU1 died in early March (first one was replaced under warranty in March 2018 due to the de-lamination issue) right before COVID lockdowns started, which was unfortunate timing. I had an appointment with the Carlsbad service center which they delayed. Poster @Akikiki put me in touch with @TonyT and I spoke to him about it. In the meantime, the service center said they had a replacement MCU1 and would swap it out for $1500 plus labor so I went with that. However when I got the car back I was only charged $170 because instead of swapping it out they just re-wrote the chip. “Great!” I thought and let Tony know. Tony predicted that it would die again and encouraged me to upgrade to the 32gb anyway. It being the start of the COVID lockdown, I decided to do a wait and see.

Fast forward exactly 2 months and the thing was dead again. I contacted the service center and they texted me back and told me that they could re-write it again but it would definitely need an MCU swap out eventually (for $1500 plus labor). Problem is they also told me that MCU’s are on back order with no ETA. SO, I contacted Tony again and visited him yesterday. Fortunately I live about an hour’s drive away.

Let me tell you - watching Tony work to tear apart my dash is a thing to behold. He made it look so easy and had my MCU out in about 30 minutes. Amazing. Also got the good news that despite the 2nd killing, the certs are still accessible, so I now look forward to a new upgraded eMMC in the next few days.

Few things of interest:
- As some others have reported, my turn signals are not working at all. Probably pissed off a lot of people on my drive up to Tony’s and back, and will again when I drive up there to get everything put back together.
- The car still supercharges normally. With the tegra board out, I was still able to supercharge normally (to the 70% I had set before the eMMC death) on the way back home. AC charging on my NEMA 14-50 at home is at 2 amps and extremely slow, apparently as expected.
- When Tony pulled the MCU, we discovered that it had an early 2018 manufacture date. So my original 4-year-old MCU was swapped for a new one when they replaced it under warranty due to the de-lamination, and this new one failed in 2 years. So it would seem that paying $1500 for another MCU would not be so prudent, especially for what will likely be a refurb.

Thanks to Akikiki for pointing me to Tony and thanks especially to Tony for doing this for us! I really learned a lot about my 6-year-old Model S both reading through this thread today and watching Tony’s masterful work yesterday.

 

[MP3Mike]

So it would seem that paying $1500 for another MCU would not be so prudent, especially for what will likely be a refurb.

It should come with a 4-year warranty. But it still isn't worth it. What I think would be worth it is the $2,500 option to upgrade to MCU2, but that still isn't available for everyone yet.

 

[Akikiki]

@GoBlue88, Thank you for that thank you. I'm pleased to be of help. That was a great write up. And I appreciate you pointing out Tony's skill and work. I think we are not fully appreciating what Tony is doing to help the community. Aside from performing the work to get many back on the road right away, its doing a better job that we are currently getting from Tesla. *And cost less in the end. Tony is using size wise, the same size chip that's in the MCU2. Tesla is still putting the same size 8GB chip in for an eMMC replacement. @TonyT, standup and take a bow while we folks at home give you a standing ovation for your work. YOU DA MAN ! We surely appreciate you.

Let's remember, we use the MCU interchangeably as failed when its the eMMC that failed. I'm sure there have been MCUs that have failed with more than one serious problem - example screen failed and eMMC failed. Not all replaced MCUs are failed MCUs, it could be replaced because the screen or the Tegra board just to name two components. However even having said this, it does not appear of late that Tesla is breaking the MCU down to swap components. It looks like they are removing the MCU, putting another in, and sending the MCU back to someplace for work. But they were breaking the MCU down to swap the center screen. Did mine. I was there.

GoBlue88, let's take a look at what happened to your car. Your 4 year old MCU was replaced because the center screen had a problem. In 2018. Mine too in my 2017 S was replaced about the same time. At the time it was replaced, the mobile tech told me they were using new screens because they had a shortage of refurbished. He said, the refurbished ones are actually better quality, because they were addressing the things then that were happening to the new ones. He worked in my garage and I watched (and had to be pushed away from time to time) while he worked. He didn't swap MCU's which was easier. (I've pulled my own MCU recently. I now understand how pulling the MCU and tearing it down are different.) He used my table and tore down the MCU, removed the center screen and put a new one in. And he missed something during the reassembly, so when he realized it, I got to watch him take it apart a second time. Cool. Why was this important? I now, know I would rather he swapped the MCU, because it would have included a fresh eMMC with fresh copies of the certs and fresh copy of firmware. That's not how it went. He only swapped the screen, and left the Tegra board attached to the mainboard. Then back together.

I point all this out because, there's a very good chance that you did not get a new eMMC back when your center screen was replaced. They likely removed your MCU, tore it down, replaced the center screen and back together. So, you had the same eMMC and same eMMC future problems that you recently are dealing with.

I think, in case anyone cares, that getting a replacement MCU in the SC now is getting a fresh (just like the old one) eMMC along with a center screen, maybe LTE upgrade and fresh copy of certs/firmware and maybe some other stuff. And I think that a new/fresh eMMC is a reallllllly good thing for a Tesla. Where there was an opportunity two years ago (maybe more) when they swapped the screen, we could also have received a new eMMC if the entire MCU assembly was replaced - but they were not doing it then.

My three cents and allowing for inflation.

 

[EV-Fixme]

@Akikiki @GoBlue88 Since I checked and it's ok, I will share a little extra info so we can have a very nice study of this case. First the MCU absolutely appears to be replaced in 2018 and I believe the eMMC was new with the MCU as the chip was only 50% used up. I did see very high levels of corruption after retrieving all the data and correcting the best we can. This recovery may need a factory reset incase the data was not corrected well enough. We wont know till we complete the restoration.

We are very grateful to have the support from the community and look forward to offering much more support/repairs in the future. Our success is only because of all the owners willing to believe in us and give us the chance. A huge thanks to TMC and all the wonderful feedback we have received, and insightful information.

@MP3Mike I believe warranty is only 4 years with purchased MCU, replaced under warranty just continues with car warranty unless Tesla decides to goodwill.

Another interesting MCU I recovered recently was actually a 2019 MCU with only around 500 writes. Again heavy levels of corruption, and all important files obtained. Talking with the owner we decided to replace the chip anyway since the corruption was so bad even though the chip was fairly new. As @whitex pointed out, the 3000 writes is a bell curve, since we see a good amount of eMMC's last 3000 writes or more, and we also see eMMC's lasting much less. Now all these low eMMC's can probably supply many more years of service while giving the owners massive headaches and anxiety. I do look forward to hearing from people as time goes on how well the 3rd party eMMC's hold up.

Now as general the most common "MCU failures" are screens and eMMC. In very rare instances has other components failed on MCU's and often appear to be external factors, such as operator error (should not have done what that did) or accident (car crash)

eMMC is by far the most issues caused and resolved repair. eMMC's will fail, every single one. The goal is to have good performing ones that last a reasonable time.

That's my 4 cents.