MCU1: Black center screen / mcu failure caused by a faulty navigation MicroSD card

[LuckyLuke]

Hi all. Just a informative / warning message from me

There is now a new, until now largely unknown, reason for getting a full black center screen on a first generation MCU1.0 found on cars up to and including 2015. This report does not cover the newer 1.5 MCU found in 2016 and newer cars, however a similar problem could occur in that system as well.

In the past weeks I have seen some cars where the EMMC was already replaced before (most by 3rd party, but also one that was replaced by Tesla).
The problem: the main screen went black again, and a error message pops up in the Instrument Cluster "center screen not functional". Exactly like you would see when the EMMC chip fails.
Some cars had been to Tesla Service Center already for this problem. At the Tesla Service Center they usually diagnosed it as "MCU needs to be completely replaced" and one customer was even told the problem was caused by "third party EMMC repair". Lucky for these customers they contacted me to find out what the problem was exactly.

The car owners all reported that the problems started with navigation issues, slow calculating routes, nav icon keeps spinning, new maps trying to download over wifi, etc. After scroll-wheel rebooting the MCU it stays black. Appearing almost exactly when EMMC chip is failing.

I have investigated these cases and they are all related to a failing navigation maps MicroSD card (this 16GB MicroSD card is located on the MCU1.0 mainboard). Apparently when the navigation card dies this can cause the entire system to crash and not even start anymore (not even when power cycling the mcu).
Even when the MicroSD card was completely removed or replaced the system would still not start and will still give a black screen. Huh!?
This was all on 2013-2015 MCU1.0 cars, newer MCU1.5 cars from 2016 and up have replaced the MicroSD card with a internal/soldered EMMC chip for the maps.

Anyway: a dead/corrupt MicroSD card can prevent the system from starting up completely. The Tegra computer will boot in what that is called "Recovery" mode and it will also write some changes to the boot environment stored on the NOR chip which is located on the Tegra module (this Spansion NOR chip contains the bootloader, environment and kernel images). This change prevents the Tegra from booting to normal mode. Wut?!

To fix this problem you will need to do several things:
1) Replace the MicroSD card with a new one (16GB recommended, but can be larger as well)
2) Fix the environment on the NOR chip on the Tegra module (mtd7 partition, swap boot bank, fix fail count)
3) Partition the new MicroSD card with the correct partitions (can use nav-sdcard-format.sh for that)
4) Install the correct navigation maps on the MicroSD card (EU maps are about 8GB in size)
Step 1 can be done easy by swapping the card in the MCU (accessible from the side of the MCU housing)
Step 2 is way more complex and only possible when you have full read/write access to the NOR chip. Most repair shops that deal with EMMC repairs will be able to do this.
Steps 3&4 can be done on a (Linux) pc if needed, but can also be done on the MCU itself (need root access for that).

If you go to Tesla with this problem they will most certainly tell you that "The entire MCU needs to be replaced", or even throw in a "Problem caused by 3rd party repair" excuse. All that for a cheap MicroSD card...

Considering these cars are getting older, the chances are increasing that more owners with MCU1.0 cars will eventually face a failing navigation MicroSD card.
Conclusion: If you see reports on black screens then it could be a navigation MicroSD issue. Even on cars that already had the EMMC chip replaced (by Tesla or by third party).

If you face any MCU1 related issues and don't want to go to Tesla feel free to PM me for help.

 

[ThomasD]

Could this issue affect newer cars 4 to 5 years in the future

 

[RuudL]

Hi all. Just a informative / warning message from me

There is now a new, until now largely unknown, reason for getting a full black center screen on a first generation MCU1.0 found on cars up to and including 2015. This report does not cover the newer 1.5 MCU found in 2016 and newer cars, however a similar problem could occur in that system as well.

In the past weeks I have seen some cars where the EMMC was already replaced before (most by 3rd party, but also one that was replaced by Tesla).
The problem: the main screen went black again, and a error message pops up in the Instrument Cluster "center screen not functional". Exactly like you would see when the EMMC chip fails.
Some cars had been to Tesla Service Center already for this problem. At the Tesla Service Center they usually diagnosed it as "MCU needs to be completely replaced" and one customer was even told the problem was caused by "third party EMMC repair". Lucky for these customers they contacted me to find out what the problem was exactly.

The car owners all reported that the problems started with navigation issues, slow calculating routes, nav icon keeps spinning, new maps trying to download over wifi, etc. After scroll-wheel rebooting the MCU it stays black. Appearing almost exactly when EMMC chip is failing.

I have investigated these cases and they are all related to a failing navigation maps MicroSD card (this 16GB MicroSD card is located on the MCU1.0 mainboard). Apparently when the navigation card dies this can cause the entire system to crash and not even start anymore (not even when power cycling the mcu).
Even when the MicroSD card was completely removed or replaced the system would still not start and will still give a black screen. Huh!?
This was all on 2013-2015 MCU1.0 cars, newer MCU1.5 cars from 2016 and up have replaced the MicroSD card with a internal/soldered EMMC chip for the maps.

Anyway: a dead/corrupt MicroSD card can prevent the system from starting up completely. The Tegra computer will boot in what that is called "Recovery" mode and it will also write some changes to the boot environment stored on the NOR chip which is located on the Tegra module (this Spansion NOR chip contains the bootloader, environment and kernel images). This change prevents the Tegra from booting to normal mode. Wut?!

To fix this problem you will need to do several things:
1) Replace the MicroSD card with a new one (16GB recommended, but can be larger as well)
2) Fix the environment on the NOR chip on the Tegra module (mtd7 partition, swap boot bank, fix fail count)
3) Partition the new MicroSD card with the correct partitions (can use nav-sdcard-format.sh for that)
4) Install the correct navigation maps on the MicroSD card (EU maps are about 8GB in size)
Step 1 can be done easy by swapping the card in the MCU (accessible from the side of the MCU housing)
Step 2 is way more complex and only possible when you have full read/write access to the NOR chip. Most repair shops that deal with EMMC repairs will be able to do this.
Steps 3&4 can be done on a (Linux) pc if needed, but can also be done on the MCU itself (need root access for that).

If you go to Tesla with this problem they will most certainly tell you that "The entire MCU needs to be replaced", or even throw in a "Problem caused by 3rd party repair" excuse. All that for a cheap MicroSD card...

Considering these cars are getting older, the chances are increasing that more owners with MCU1.0 cars will eventually face a failing navigation MicroSD card.
Conclusion: If you see reports on black screens then it could be a navigation MicroSD issue. Even on cars that already had the EMMC chip replaced (by Tesla or by third party).

If you face any MCU1 related issues and don't want to go to Tesla feel free to PM me for help.

I had this problem this week and it was fixed @LuckyLuke very quick. (Thanks again pall)
The issue with my MS was spinning navigation often for two days. 2 times a scroll wheel reset fixed it but the third reset gave a black screen.
I had the eMMC fixed at LuckyLuke in December and it seemed the same issue. So I contacted him and he was thinking it could be the Nav-SD that was corrupt. And it was. Luckily (his name says it all) he found the above solutions.

 

[EV-Fixme]

Could this issue affect newer cars 4 to 5 years in the future

A similar issue could occur.

 

[Droschke]

If the faulty navigation MicroSD card is the root cause of the MCU1 failures then what is the NHTSA's ordered eMMC recall suppose to mean? In other words, isn't the recall trying to fix the wrong part (the eMMC alone)?

 

[MP3Mike]

If the faulty navigation MicroSD card is the root cause of the MCU1 failures then what is the NHTSA's ordered eMMC recall suppose to mean? In other words, isn't the recall trying to fix the wrong part (the eMMC alone)?

There is no single root cause for all MCU failures. Multiple things can fail and give the same symptoms. It seems the most common three failures are:

1. eMMC (By far the most common)
2. FPGA
3. Navigation SD card

Other failures include things like the parrot module (BT/WiFi), 3G board, LTE board, GPS unit. (Though those don't tend to leave you with a blank screen.)

 

[dark cloud]

 

[EV-Fixme]

There is no single root cause for all MCU failures. Multiple things can fail and give the same symptoms. It seems the most common three failures are:

1. eMMC (By far the most common)
2. FPGA
3. Navigation SD card

Other failures include things like the parrot module (BT/WiFi), 3G board, LTE board, GPS unit. (Though those don't tend to leave you with a blank screen.)

Touchscreen input controller can also fail, causing loss of touch control, all the other failures are much more rare but can happen.

 

[slainla]

I haven't had the EMMC chip replaced yet and my SD card has been bad for a couple of months. The navigation stopped working and it kept asking for map updates. I did have a black screen once with an error on the instrument cluster error saying center display unavailable. The OTA update supposedly fixed it but I do get some freezes here and there, but not multiple reboots.

Tesla quoted about $220 to change it out for me but I think I can do it myself. However, I'm not sure whether Tesla needs to partition and program it remotely.

Does anyone know if the SD card can be prepped on a PC?

 

[LuckyLuke]

It can. But if you current card is still somewhat alive why not just make a 1:1 bitcopy of it to a new card? Then you don't need to bother with partitioning etc.

 

[slainla]

Hmm, I'll try that out then. Do you think it'll work even if the maps no longer work?

I remember the first time it happened, Tesla kept sending updates to force the map update. Then after two days they said the SD card was corrupted and they did a remote repartition as a courtesy. Maps worked for 1-2 days then died again and I gave up after that.

 

[EV-Fixme]

Hmm, I'll try that out then. Do you think it'll work even if the maps no longer work?

I remember the first time it happened, Tesla kept sending updates to force the map update. Then after two days they said the SD card was corrupted and they did a remote repartition as a courtesy. Maps worked for 1-2 days then died again and I gave up after that.

We will have preformated SD cards available soon.

 

[LuckyLuke]

Hmm, I'll try that out then. Do you think it'll work even if the maps no longer work?

I remember the first time it happened, Tesla kept sending updates to force the map update. Then after two days they said the SD card was corrupted and they did a remote repartition as a courtesy. Maps worked for 1-2 days then died again and I gave up after that.

If you have a Linux system its fairly easy to test if the copied data is integer or not, the maps are in sqfs format so you can use a tool like 7z to test the integrity of the partition. But else you could just partition a new card, put it in and wait for it to download the maps automatically. Or ask Tesla to push the map update to your car. It's a few clicks in their system only.

 

[slainla]

Thanks guys, I'll wait or try it out.

And it sounds like any previously Tesla formatted SD card should work on other MS/MX? Just seeing if salvage yards would be an option for others.

 

[EV-Fixme]

MX have a eMMC containing maps no SD. Older software maps card may not be partitioned correctly and need Tesla help.

 

[dmdz]

Is there a way to self-diagnose which has failed/differences from the traditional emmc failure? I've got my car booked in for service next week but after finding this post my symptoms are consistent with SD-card failure so im less confidant its going to get fixed under the recall. Car was asking for new maps to download from wifi, after a scroll wheel reboot its sat with a black screen stuck on the reverse camera image with the "centre screen not functional" message. Previously ive had increasingly longer route recalculations as well as random freezing/reboots with no messages about emmc health.

Car is booked in with Tesla but by the sounds of it if its the nav SD card then I'm on my own/facing a new MCU. If my suspicions are true and its not covered by the recall... what SD card would you recommend/could a remote service be facilitated? Based in the UK.

Thanks in advance.

 

[LuckyLuke]

Self diagnose can be done by taking the MicroSD card out and testing the card on a computer. If it is faulty replace with a 16GB or 32GB card, and ask Tesla service to initialize/partition the card and push a map update. (both can be done over the air)
If you want I can provide you a binary image which you can pre-load on the card, PM me if you need that.
For removing the MCU see for instance this video:

Card is located in a small slot in the MCU housing, please note that there is some factory yellow kapton tape over the slot&card, remove the tape carefully with some sharp knife and/or tweezers.

 

[Virus90]

Does my Facelift Model S from 06/2016 with original MCU (MCU1) also have the MicroSD card with navigation data?

Because I also had some issues regarding navigation system in the past few days/weeks. It often doesn't load and only shows the message that the navigation system is initializing.

So can I take the original MicroSD out of my MCU, copy the data onto a new 32GB MicroSD, stick it in my MCU and the problems should be solved?

 

[LuckyLuke]

Facelift / 2016+ cars have the newer MCU1 (sometimes referred to as MCU1.5). This model MCU has a secondary 16GB emmc chip on the MCU mainboard where the navigation maps are stored. I have not seen any of these go faulty yet. Also on-board emmc is somewhat more reliable than a separate consumer grade microsd card. So I would not worry too much for these models.

 

[DannyDrive]

Facelift / 2016+ cars have the newer MCU1 (sometimes referred to as MCU1.5). This model MCU has a secondary 16GB emmc chip on the MCU mainboard where the navigation maps are stored. I have not seen any of these go faulty yet. Also on-board emmc is somewhat more reliable than a separate consumer grade microsd card. So I would not worry too much for these models.

<<

Self diagnose can be done by taking the MicroSD card out and testing the card on a computer. If it is faulty replace with a 16GB or 32GB card, and ask Tesla service to initialize/partition the card and push a map update. (both can be done over the air)
If you want I can provide you a binary image which you can pre-load on the card, PM me if you need that.
For removing the MCU see for instance this video:

Card is located in a small slot in the MCU housing, please note that there is some factory yellow kapton tape over the slot&card, remove the tape carefully with some sharp knife and/or tweezers.
View attachment 655634

I was looking for some help for the MicroSD card. I am considering to change the original 16GB to a 32GB MicroSD Card as my MCU is disassembled already.

1. I have checked the reading and writing speed of the original 16GB Card and its very slow as its a old MicroSD card. Does a faster 32GB card effect the speed of the MCU mapping (tiles) and route calculation, do you have any idea about it? Or is MCU always using the data from the the second onboard 16GB internal/soldered EMMC chip?

2. I have the MCU1 or MCU1.5 in my 2016 Model S with the second onboard 16GB internal/soldered EMMC chip for the maps. I tested the MCU by taking out the 16GB MicroSD card and reboot (reset) the MCU. The map tiles are still working in same speed, but the route calculation is not working without the MicroSD card "after a reboot of the MCU" (before a reboot routing is working like MicroSD Card is inside). This looks to me like there is a kind of indinteraction between the MicroSD, card MCU and the second onboard 16GB internal/soldered EMMC chip.
- I am curious about why the MCU require the MicroSD card after a reboot instead to use directly the second onboard 16GB internal/soldered EMMC chip?
- Do you know details about how exactly the MCU is interacting with the MicroSD card and the the second onboard 16GB internal/soldered EMMC chip?
- If the MCU need to read often data from the 16GB MicroSD card, then a new 32GB faster MicroSD card would be not bad right?

3. Do you know which type of partitions are fine to run the new 32GB Card? EXT3, EXT4 or squashsf?

It's a kind of weird as the my original 16GB Card is divided into 2 partitions. 1st is squashfs 2nd is Ext3 format and roughly 8GB per partition.
- Do you also have 2 partitions?

LuckyLuke can you pm me your binary image please?