MCU fails for the second time

[JakeP]

@JakeP Thanks for the detailed writeup on your resolution. Question: was it required to first revive the black MCU using the double forced firmware update, prior to upgrading the eMMC? Or, could you have had the eMMC fixed regardless?

I’ll have to let @tedsk answer that question for an accurate answer, but my recollection was that his job was made easier having a working MCU, as retrieving the data from the eMMC was possible. Even if it hadn’t been revived successfully, I was still going to send it to him for sure.

 

[aaron0k]

I still ended up spending less than half the $2400 that Tesla wanted me to pay to replace the MCU. And now with the bigger and more robust chip, I should be protected from future issues caused by excessive logging to the eMMC!

While I'm happy you found a reasonable resolution; it's also deeply disconcerting the hoops you had to jump through.

It seems the majority of MCU's will fail within the lifetime of the car... while Elon can tweet about how ‘Brake pads on a Tesla literally never need to be replaced for lifetime of the car.'... I don't see any tweets about how you'll have to spend thousands to replace a $25 solid-state chip due to "wear".

This is an embarrassment for Tesla...

 

[JakeP]

While I'm happy you found a reasonable resolution; it's also deeply disconcerting the hoops you had to jump through.

It seems the majority of MCU's will fail within the lifetime of the car... while Elon can tweet about how ‘Brake pads on a Tesla literally never need to be replaced for lifetime of the car.'... I don't see any tweets about how you'll have to spend thousands to replace a $25 solid-state chip due to "wear".

This is an embarrassment for Tesla...

Absolutely agree. Tesla should be providing this fix at no cost to the owner, and including the MCU in the drivetrain warranty window.

And don’t get me started on the brake pads comment...

 

[MP3Mike]

and including the MCU in the drivetrain warranty window

There is no "drivetrain" warranty. There is a driveunit, and battery, warranty...

 

[LuckyLuke]

4) After reading several threads here and on the DIYElectricCars forum, I thought that upgrading to a 64GB was ideal, and that we should leave 32GB unpartitioned. Turns out I may have misunderstood some of those threads, as the 64GB chip was only suggested for MCU2 and newer. So Tedsk and I decided on a 16GB chip instead for my first gen MCU, leaving 8GB unpartitioned. Thanks @verygreen for input here.

Background: a few months ago the MCU (on my 2015 S) went black for the first time, Tesla did the double firmware reinstall and it worked again for some weeks. But it went black again so emmc replacement was inevitable.

So yesterday my friend (who has a phone repair shop and does frequent bga work) volunteered to help me replace the emmc chip. I got the 64GB Swissbit (like mentioned here).
We tested the mcu (without lcd) on a bench setup/psu before the removal of the Hynix emmc and it worked fine, we could ping the MCU ip and ssh into it (was already rooted).
Chip removal went smooth, as was the reading&writing of the old&new chip (RT809h programmer). We dit not modify the image in any way, since it was already rooted and booting to a certain degree.
New Swissbit chip placement went smooth, so we were very hopeful that it would be a succes. But no cigar sadly, the MCU would not boot .

At first attempt the power pulled was oscillating between 6 and 10 Watts (reboot loop?), we tried to reflow: still the same. Then we removed the Swissbit chip, put it in the programmer, and luckily it read just fine and verified 1:1 with the dump we had.
Then we tried to reball the chip but that was an ordeal. My friend only had a general .5mm stencil instead of a specific bga153 grid. Took a lot of time to clean things up.
Anyway we ran out of time and flowed it on the pcb and bench tested again: Now it pulled 10W constantly now, but still it wouldn't boot. We ordered the 153 stencil so we can reball properly. Will try that in a few days when package arrives.
We assumed the failure was because of the sloppy reball job, but now I'm reading your post I am starting to doubt if I purchased the wrong emmc chip?

I hope someone (@verygreen ?) can comment?

Also: is there any way (like leds or serial) to see on what stage the boot of the MCU fails?

 

[Piotr12]

My MCU from 18 May is black, I have to pickup my Tesla to SC,the car is unused and I am waiting for my visit on July 11( waiting time one month) I never had so bad car like Tesla , MCU died after 3.5 years and 100 thousand km

 

[eutechmaster]

I hope someone (@verygreen ?) can comment?

Also: is there any way (like leds or serial) to see on what stage the boot of the MCU fails?

Hello,

i have too problem with MCU they black , and i can confirm that my mcu now was boot with this swissbitchip.

 

[eutechmaster]

also when you put full copy using dd command , then try resize last 4 partition to 15-32 gb using gparted (sudo apt install gparted) and leave 32 gb unmapped .

 

[Piotr12]

Now just waiting for new MCU, I dont know why but they not return the old one, i think if I will pay for new one I will keep The old

 

[MP3Mike]

Now just waiting for new MCU, I dont know why but they not return the old one, i think if I will pay for new one I will keep The old

Under the law you have to request to keep the old parts before they do the service. But even with that you normal have to pester Tesla to honor the law/your request.

 

[Gixx1300R]

Under the law you have to request to keep the old parts before they do the service. But even with that you normal have to pester Tesla to honor the law/your request.

That depends on the law in your state. a simple email to the service center along with a link to the state statutes is normally enough.

 

[Piotr12]

New MCU costs in Germany SC about 3600 EUR. :-(

 

[P100D_Me]

New MCU costs in Germany SC about 3600 EUR

Utter insanity, but I suspect it'll be the same story for most countries outside the US (not that it is much better there anyway).

 

[Piotr12]

All done, but car still on SC I will pickup after next week, the cost was 3600 EUR. :-( they replace MCU , antenna,rear camera,BLUETOOTH, LTE, I Hope now will be work for longer time

 

[LuckyLuke]

At first attempt the power pulled was oscillating between 6 and 10 Watts (reboot loop?), we tried to reflow: still the same. Then we removed the Swissbit chip, put it in the programmer, and luckily it read just fine and verified 1:1 with the dump we had.
Then we tried to reball the chip but that was an ordeal. My friend only had a general .5mm stencil instead of a specific bga153 grid. Took a lot of time to clean things up.
Anyway we ran out of time and flowed it on the pcb and bench tested again: Now it pulled 10W constantly now, but still it wouldn't boot. We ordered the 153 stencil so we can reball properly. Will try that in a few days when package arrives.
We assumed the failure was because of the sloppy reball job, but now I'm reading your post I am starting to doubt if I purchased the wrong emmc chip?

It was the flash data which was too corrupt, the chip was soldered on correctly but the contents of the squashfs partition was too corrupt to boot. That's why it was rebooting every few seconds. Removed the chip, reprogrammed with corrected data and now it works perfect.

Also: is there any way (like leds or serial) to see on what stage the boot of the MCU fails?

To answer my own question: there are some serial leds on the main mcu pcb, which tell you if cid / gw are on or not. Also you can check tegra serial by going to recovery mode from the gw (google tesla gateway internals how to do this).
In my case recovery mode was of no use to fix my corrupted squashfs partition, since I had a emmc v5 chip on it, and recovery kernel is so old/minimal it would not recognise it. I guess when you replace with a Hynix (or modify the ext csd version number) you can get recovery to work fine as well.

 

[rickrickrick]

Trying to find the right thread for help, hope it's here. Short version is that I am being told that my 2015 S60 MCU is dead and that it needs to be replaced.

I asked for the cause of MCU failure and was told that the engineering team says that the electonic multi-media chip is at 80% of its expected life expectancy which caused the MCU to shut down.

Advice? What options do I have?

Edit to add: I've had the car just under four years but I've driven over 53,000 miles so Tesla is stating that I am responsible for all costs because the car is out of warranty. It's not driveable in its current condition.

 

[rickrickrick]

@JakeP thank you for your documentation, it's incredibly helpful as I'm going through this thread again.

For all, are there options other than to pay Tesla to replace the MCU (knowing that is a systematic problem with the unit) or having an expert like @tedsk replace the memory (which is not scaleable)? This just seems crazy.

 

[sorka]

Trying to find the right thread for help, hope it's here. Short version is that I am being told that my 2015 S60 MCU is dead and that it needs to be replaced.

I asked for the cause of MCU failure and was told that the engineering team says that the electonic multi-media chip is at 80% of its expected life expectancy which caused the MCU to shut down.

Advice? What options do I have?

Edit to add: I've had the car just under four years but I've driven over 53,000 miles so Tesla is stating that I am responsible for all costs because the car is out of warranty. It's not driveable in its current condition.

Make the case to them that your MCU was already having issues before 50k miles. When they finally die due to the inevitable eemc^2 failure they've usually been having issues for a while before they die for good.

 

[rickrickrick]

Real time update:

• Escalating to request Tesla replace MCU at no cost.
• If I pay for the replacement it would be a new 2nd generation MCU, I was told they've had a retrofit to cross install in to 1st generation MCU cars for about a month.
• The warranty period for the new MCU would be one year.
• Tesla wants to charge ~$250 on top of the cost of the new MCU to give me my old MCU.

 

[MP3Mike]

The warranty period for the new MCU would be one year.

That doesn't agree with the warranty information on their web site:

The Tesla Parts, Body & Paint Repair Limited Warranty begins on the purchase date of the part(s), and coverage extends for a period of 12 months. Specific categories of parts have unique warranty coverage periods:

• Sheet metal: Limited lifetime
• Drive Unit: 4 years or 50,000 miles, whichever comes first
• Vehicle High Voltage Battery: 4 years or 50,000 miles, whichever comes first
• Wall Connectors: 4 years
• Touchscreen and microcontroller unit: 4 years

Tesla wants to charge ~$250 on top of the cost of the new MCU to give me my old MCU.

That is a reasonable core charge. (Someone else has reported they were told ~$700.)