Preventive eMMC replacement on MCU1

[GlmnAlyAirCar]

Are you sure you will get the MCU2? Your car is a pre facelift - right (with the old Nosecone)?

What does that have to do with anything? Infotainment Upgrade

 

[badgirl]

Found the NHTSA investigation under action number PE20010. I guess I will also submit a complaint and reference that action number? Here is the actual NHTSA write up:

NHTSA Action Number: PE20010

Components ELECTRICAL SYSTEM, BACK OVER PREVENTION

Opened From: June 22, 2020–Present

Summary

The Office of Defects Investigation (ODI) has received eleven (11) complaints (VOQs) alleging failures of the touchscreen/Media Control Unit (MCU) in 2013 through 2015 Tesla Model S vehicles. The reports have been received over the past 13 months at service intervals ranging from 3.9 to 6.3 years.

The display control unit subassembly of the subject MCU is an NVIDIA Tegra 3 processor with an integrated 8GB eMMC NAND flash memory device. EMMC NAND flash devices have a finite lifespan based upon the number of program/erase (P/E) cycles. The subject MCU allegedly fails prematurely due to memory wear-out of the eMMC NAND flash. Tesla used the same MCU with the Tegra 3 processor in approximately 159 thousand 2012-2018 Model S and 2016-2018 Model X vehicles built by Tesla through early-2018.

MCU failures resulting from eMMC memory wear-out are likely to occur after periods of progressively degraded performance (e.g., longer power-up times, more frequent touchscreen resets, intermittent loss of cellular connectivity, loss of navigation). Final MCU failure results in loss of audible and visual features provided by the touchscreen, such as infotainment, navigation, and web browsing. This includes loss of rear camera image display when reverse gear is selected, resulting in reduced rear visibility when backing. Other effects of MCU failure include climate control defaulting to Auto mode and limits on battery charging current and maximum state of charge when recharging. MCU failure does not affect vehicle control systems (e.g., braking, steering, speed control) or supplemental restraint systems.

A Preliminary Evaluation has been opened to assess the scope, frequency, and safety consequences of the alleged defect.

The following VOQ numbers were referenced in the defect petition: 11317875, 11317499, 11315374, 11312177, 11288980, 11281197, 11279959, 11257209, 11232852, 11207819, 11302674

2014 TESLA MODEL S

Hi Everyone,
I'm new to posting on the forum so I'm limited.

I started reading this tread after my 2015 70D (91,100miles, out of warranty) displayed a black screen when I came out to the car after shopping. SC gave me an appointment July 21st. They told me on the phone to use hand signals since not blinkers, no backup camera, no climate control, had to manually get it to charge (good thing I had turned scheduled changing off). I took it to SC 8am saturday morning, they told me they seem this happen with another car and it was the firmware. After being in the waiting room 2 hrs they told me they need to keep my car and its possible might need to replace MCU and it cost $2500 depending on what version MCU I have and a $195 diagnosis fee . I told him first I have MCU 1, that on a forum someone posted an invoice for $1300, plus this is a known issue and that tesla is under investigation (showed him article) and my car is due for NY safety inspection which it would fail. They waived the fee and called at 4:30 pm to say my car was ready to pickup. They reinstalled the software.

After reading all 43 pages, various youtube videos and multiple forums I think my car is ticking away at a complete failure. That said I don't feel confident in taking apart my dash to pull the MCU. I did purchase Swissbit 64GB (arriving today) from Mouser. I don't know Linux or how to solder. Is there anyone in the NY area or Northern VA (I travel there for work) that can do this at a resumable cost?

Thank you, for all the informative posts, I do appreciate everyones input.

 

[S100D]

@TonyT, I have a few questions. I just did the Infotainment a week ago, and now I have a eMMC with higher capacity. My questions are:

1. Do you know the number of Gigs for the MCU2 eMMc?
2. If the MCU2 eMMC fails, do you have the capability to replace it and move all of the data to the replaced eMMC? I know that it has a 2 year and 25,000 warranty. I plan to keep the car longer that the MCU2 warranty.

I believe that you are replacing the current MCU1 eMMCs with a 64 Gig eMMC and I am curious if Tesla upgraded to a 64 Gig eMMC.

Thanks!

 

[Hummer75]

What does that have to do with anything? Infotainment Upgrade

The rumor has it - that it's only on faceliftet models - that's why I ask.

 

[Chaserr]

Hi Everyone,
I'm new to posting on the forum so I'm limited.

I started reading this tread after my 2015 70D (91,100miles, out of warranty) displayed a black screen when I came out to the car after shopping. SC gave me an appointment July 21st. They told me on the phone to use hand signals since not blinkers, no backup camera, no climate control, had to manually get it to charge (good thing I had turned scheduled changing off). I took it to SC 8am saturday morning, they told me they seem this happen with another car and it was the firmware. After being in the waiting room 2 hrs they told me they need to keep my car and its possible might need to replace MCU and it cost $2500 depending on what version MCU I have and a $195 diagnosis fee . I told him first I have MCU 1, that on a forum someone posted an invoice for $1300, plus this is a known issue and that tesla is under investigation (showed him article) and my car is due for NY safety inspection which it would fail. They waived the fee and called at 4:30 pm to say my car was ready to pickup. They reinstalled the software.

After reading all 43 pages, various youtube videos and multiple forums I think my car is ticking away at a complete failure. That said I don't feel confident in taking apart my dash to pull the MCU. I did purchase Swissbit 64GB (arriving today) from Mouser. I don't know Linux or how to solder. Is there anyone in the NY area or Northern VA (I travel there for work) that can do this at a resumable cost?

Thank you, for all the informative posts, I do appreciate everyones input.

Don't forget to file with the NHTSA! Tesla has to report your complaint to them, and if they advised you how to deal with failed turn signals they knew it was a serious safety complaint. The NHTSA only had 11 reports when they started the recall process so Tesla hasn't been reporting these to them ever, help them out before they get fined for failure to report! If Tesla claims all you lose is air conditioning it won't be recalled, but things like turn signals have to be recalled. And if they recall, you get your money back from the repair and the new one has more than 2 years warranty.

 

[Trebek1762]

Hi Everyone,
I'm new to posting on the forum so I'm limited.

I started reading this tread after my 2015 70D (91,100miles, out of warranty) displayed a black screen when I came out to the car after shopping. SC gave me an appointment July 21st. They told me on the phone to use hand signals since not blinkers, no backup camera, no climate control, had to manually get it to charge (good thing I had turned scheduled changing off). I took it to SC 8am saturday morning, they told me they seem this happen with another car and it was the firmware. After being in the waiting room 2 hrs they told me they need to keep my car and its possible might need to replace MCU and it cost $2500 depending on what version MCU I have and a $195 diagnosis fee . I told him first I have MCU 1, that on a forum someone posted an invoice for $1300, plus this is a known issue and that tesla is under investigation (showed him article) and my car is due for NY safety inspection which it would fail. They waived the fee and called at 4:30 pm to say my car was ready to pickup. They reinstalled the software.

After reading all 43 pages, various youtube videos and multiple forums I think my car is ticking away at a complete failure. That said I don't feel confident in taking apart my dash to pull the MCU. I did purchase Swissbit 64GB (arriving today) from Mouser. I don't know Linux or how to solder. Is there anyone in the NY area or Northern VA (I travel there for work) that can do this at a resumable cost?

Thank you, for all the informative posts, I do appreciate everyones input.

Try contacting appleguru. I know he's out in the northeast.

 

[badgirl]

Don't forget to file with the NHTSA! Tesla has to report your complaint to them, and if they advised you how to deal with failed turn signals they knew it was a serious safety complaint. The NHTSA only had 11 reports when they started the recall process so Tesla hasn't been reporting these to them ever, help them out before they get fined for failure to report! If Tesla claims all you lose is air conditioning it won't be recalled, but things like turn signals have to be recalled. And if they recall, you get your money back from the repair and the new one has more than 2 years warranty.

I did file a complaint with NHTSA on Monday.

 

[badgirl]

Try contacting appleguru. I know he's out in the northeast.

I did contact appleguru and my tegra is on the way to him.
I did take apart my car (reinstalled MCU) to move it back into the garage. I'm not planning on driving it unless it's an emergency.

 

[Genedr]

MCU2 is what they said. Yes nosecone version, pre-facelift. For @2500 it better be MCU2. (An intel upgrade for TegraAP1.0) is what’s been ordered. Not MCU2????

 

[Akikiki]

@Genedr, Sounds right. Sounds like MCU2. I think you are going to be pleased.

 

[Genedr]

@Genedr, Sounds right. Sounds like MCU2. I think you are going to be pleased.

Thanks...will let the thread know how it goes end of month.

 

[HankLloydRight]

Just curious, will the car drive with the MCU removed?
I know other features might not work (A/C, charging, etc), but does it drive?

 

[tar.sor]

Just curious, will the car drive with the MCU removed?
I know other features might not work (A/C, charging, etc), but does it drive?

Yes, you can drive it without memory, but you can't use turn signals, and AC charging will takes a lot of time.

 

[Trebek1762]

Just curious, will the car drive with the MCU removed?
I know other features might not work (A/C, charging, etc), but does it drive?

You can drive with the Tegra removed from the MCU. I don't know if you can drive without the MCU.

 

[tar.sor]

You can drive with the Tegra removed from the MCU. I don't know if you can drive without the MCU.

You cant drive without MCU. Car runs on Gateway, after remoeving tegra board gateway still work abd allow to start the car and drive it.

 

[CR3P]

@rooter has done a great job providing lots of information and expertise on this issue. I have started to think similar if a separate component of the chip is going bad. We refer to this as a single chip, but there is really much more inside. We have seen reports of the chip being disassembled and new controller put in to read the content with @widodh recovery. as @whitex has mentioned the failure rate of eMMC chips is on a bell curve. We have seen chips that went well beyond the 3000 writes. We have seen chips with heavy corruption on chips much less that 3000 writes, sometimes around 1000-2000. Only time will tell how much better this upgrade is than the original. But we do know that in several years 3000 writes can happen so having a chip that has much more writes is expected to last longer. The pSLC mode we use gives it 20k write which we like a lot more than 3000, plus have double the space so there is many more write cycles due to the chip's wear leveling. We will be continuing to use the Swissbit until we or someone finds a even better chip. And we expect the current chip to outlast the car, however we would not hesitate to spend a few more dollars to offer a even better upgrade.

Unfortunately at this time we don't have to expertise to identify the exact failure and simply identify the chip as less then reliable enough to use and need a replacement.

Anytime we discover improvements we incorporate into the fix.

I have investigated this error under (scanning electron)microscope and probing. The problem is related to an OTP register inside the controller of the chip. The register fault varies a lot from chip to chip, but is definitively defective due to a silicon design flaw, it seems to be all "GMRA" chips, but failure rate depends on the production date. This also explains the variation in failure rate. When the memory area is near fully occupied, the corrupted OTP register will make the chip loose vital information, and write cycles are not performed, but no data is overwritten, a few garbage bytes (64) are sometimes written though. The crazy logging going on is not really the root cause of the chips going bad, but much more the percentage of occupied cells for data and then time. The number of erase/read/write cycles are way less destructive.

The good news is that almost all of the bad chips can be read out. I do that professionally, and I know of two other companies that does the same. With my method, it takes about an hour to read out the vital key and certificates.

 

[MP3Mike]

I have investigated this error under (scanning electron)microscope and probing. The problem is related to an OTP register inside the controller of the chip. The register fault varies a lot from chip to chip, but is definitively defective due to a silicon design flaw, it seems to be all "GMRA" chips, but failure rate depends on the production date.
<snip>
The crazy logging going on is not really the root cause of the chips going bad

I wonder what Tesla's agreement with chip supplier is. Will they be able to recover any of the repair costs from them?

Also, does this mean it is likely that the chips currently being used in refurbished MCU1s don't suffer from this issue?

 

[CR3P]

I wonder what Tesla's agreement with chip supplier is. Will they be able to recover any of the repair costs from them?

Also, does this mean it is likely that the chips currently being used in refurbished MCU1s don't suffer from this issue?

I guess Tesla have a manufacturing contract with Nvidia (or a supplier to Nvidia), but I seriously doubt that any supplier would recover any repair costs, due to the age of the contract/hardware. Also the chip manufacturer most likely would argue that the way Nvidia/Tesla use the chip is not as recommended (although not the reason for failure).

As fas as I know, all EU Tesla MCU1s are simply replaced with the same revision B or C of the Tegra board, and thus the very same emmc chip, with the same design flaw. Guess this is a good reason for the revised limited warranty (from 4 to 2 years).

 

[badgirl]

They will be with the same Nvidia chips that are the cause of the failure.
I had my Nvidia chip (consumer grade) replaced with a Swissbit 64gb chip (industrial grade) so it should last 10-20 years.

 

[MP3Mike]

They will be with the same Nvidia chips that are the cause of the failure.

The eMMC chip is not a Nvidia chip. (If I recall correctly it is a Hynix chip.)