Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

MCU fails for the second time

Sorry I may have misread, if you flash is gone (known to go bad on earlier cars or older MCUs) than your MCU is toast, its embeded. As others have said, rooted Tesla's and their support networks have disabled many logging features in an effort to preserve flash life. Hell, I have a salvage MCU off a 2014 with 12000 miles and my flash started acting up recently.
 

verygreen

Curious member
Jan 16, 2017
3,031
11,707
TN
Not really, from the photos of the CID that I have seen, the flash appears soldered on to the logic board. If that is what failed, even if you replaced it, you would likely be SOL, as you wouldn't have the raw firmware/bootloader on them to actually boot your system (unless you took the replacement chips from known good board and installed them on the same pads they came from).
Actually there are multiple flash devices in there. Bootloader and kernel/ramdisk are stored on a different flash that than the one that is failing due to all the logging.
Granted, there's still no firmware in the new one you'd put, but it's relatively easy to put one in.
 

appleguru

Active Member
Mar 15, 2017
1,121
1,949
MA, US
Actually there are multiple flash devices in there. Bootloader and kernel/ramdisk are stored on a different flash that than the one that is failing due to all the logging.
Granted, there's still no firmware in the new one you'd put, but it's relatively easy to put one in.

Interesting. When the emmc fails, what symptoms does the CID present? Does it not boot at all? Backlights come on but no UI? Boots mainline, but can't write or mount the emmc log partition? (I would presume it would throw a "contact Tesla" type error in this case but otherwise function...)

Sounds like they made some boneheaded logging decisions and/or hardware choices for emmc (or the emmc didn't/doesn't have a good enough controller or enough "unused" memory to do proper flash maintenence).
 

verygreen

Curious member
Jan 16, 2017
3,031
11,707
TN
Interesting. When the emmc fails, what symptoms does the CID present? Does it not boot at all? Backlights come on but no UI? Boots mainline, but can't write or mount the emmc log partition? (I would presume it would throw a "contact Tesla" type error in this case but otherwise function...)

Sounds like they made some boneheaded logging decisions and/or hardware choices for emmc (or the emmc didn't/doesn't have a good enough controller or enough "unused" memory to do proper flash maintenence).
I suspect the symptoms are "cid shows black screen", but also depends on emmc failure mode - i.e. could the existing data be read or not.
If it could be read then probably actual gui or parts of it can come up.

Car does gobs of logging, just syslog alone is super chatty (and flushes to emmc every 5 seconds), I have 2.5M syslog file and it rotated 4 hours ago (I drove ~10 miles in that time). previous 12 hours is 1.6M gzipped (and nobody was driving it at the time I imagine). - this is lkely the biggest offender.
The updater thing also writes a file for every time it does anything (like ask the server if there's an update, or check some internal stats and so on).
In addition to that every firmware upgrade is a full partition overwrite + more.
Also the cid never sleep it seems (at least not when you have the always connected option) so it keeps being chatty even when the car is not in use.

I have a script to convert /var/log/ to tmpfs, but the log rotate is then messed and their logrotate is too old to have "renamecopy", I guess I should just build my own copy and use that because storing rotated logs in tmpfs is stupid.
 
  • Like
  • Informative
Reactions: Chaserr and MikeBur

appleguru

Active Member
Mar 15, 2017
1,121
1,949
MA, US
I suspect the symptoms are "cid shows black screen", but also depends on emmc failure mode - i.e. could the existing data be read or not.
If it could be read then probably actual gui or parts of it can come up.

Car does gobs of logging, just syslog alone is super chatty (and flushes to emmc every 5 seconds), I have 2.5M syslog file and it rotated 4 hours ago (I drove ~10 miles in that time). previous 12 hours is 1.6M gzipped (and nobody was driving it at the time I imagine). - this is lkely the biggest offender.
The updater thing also writes a file for every time it does anything (like ask the server if there's an update, or check some internal stats and so on).
In addition to that every firmware upgrade is a full partition overwrite + more.
Also the cid never sleep it seems (at least not when you have the always connected option) so it keeps being chatty even when the car is not in use.

I have a script to convert /var/log/ to tmpfs, but the log rotate is then messed and their logrotate is too old to have "renamecopy", I guess I should just build my own copy and use that because storing rotated logs in tmpfs is stupid.

(!). Does the CID's kernel support overlayfs? Can you change the logging partition to disposable storage, like (one of) the unit's SD cards perhaps? This would undoubtedly fry your SD card relatively quickly (though an industrial card should last much longer), but at least only costs a few dollars to replace... Or even better, use a USB stick for logging. Then it can be very easily accessed/replaced... and you have quick access to the logs if you want (not sure how useful/human readable they are...)
 
  • Like
Reactions: andyrebele
It can be done, but I think it is surface mounted so you would need a re-work station and some skills. On the other hand the screen could probably be taken off and put on a MCU with the bubbling/leaky problem with ease.

Out of curiosity how long ago was your MCU replaced, and what is your total mileage? I would still push to get them to cover it, since in all likelihood you got a refurb MCU that already had a significantly worn emmc/flash module. (You could always PM JonMc here on TMC.)



We have seen a number of cars with this problem, and @Ingineer has helped a few people transfer their data to a salvaged MCU vs. paying Tesla to replace it.

They said it was replaced the first time back in 2013.
 

verygreen

Curious member
Jan 16, 2017
3,031
11,707
TN
(!). Does the CID's kernel support overlayfs? Can you change the logging partition to disposable storage, like (one of) the unit's SD cards perhaps? This would undoubtedly fry your SD card relatively quickly (though an industrial card should last much longer), but at least only costs a few dollars to replace... Or even better, use a USB stick for logging. Then it can be very easily accessed/replaced... and you have quick access to the logs if you want (not sure how useful/human readable they are...)
Yeah, I thought about overlayfs, but they did not build it. I probably can build my own module for it, but I think that does not work with logrotate anyway because overlayfs passes through underlying fsid and it changes depending on where the file is actually located, and we do not want the rotation to put stuff there either, might be worth experimenting some more I guess.

The problem with using usb/sdcards is they are all powered down when the cid enters this semi-sleep state when the screens are off.
Probably can make them stay awake all the time too, I guess.

BTW IC also has similar logging situation - i.e. syslog running full blast to internal partition, but those don't die as easily - probably because IC is actually off for long periods of time.
 

sorka

Well-Known Member
Feb 28, 2015
10,001
8,143
Merced, CA
I escalated one level above the service center person I was dealing with, and asked, but they said it could not be covered under goodwill because it was out of warranty and my car is high mileage.

My co-workers 2012 MCU was JUST replaced long out of warranty with 100K miles on it due to the bubble issue. It wasn't even not working but just a minor cosmetic issue and they STILL replaced it under "good will".
 
  • Love
Reactions: SeminoleFSU
My co-workers 2012 MCU was JUST replaced long out of warranty with 100K miles on it due to the bubble issue. It wasn't even not working but just a minor cosmetic issue and they STILL replaced it under "good will".
They might be giving these out now quietly. I fought for months on mine and ended up doing it myself. This was over a year ago though
 
Probably replacing a bad screen with one from a salvage MCU. (Before Tesla would replace just the screen.)

I replaced the entire MCU, due to a stupid error on my part... the intent was to replace the screen only. Once I messed up the MCU (read, cat knocked it over as i was closing it back up) WK had to help me from there on :(
 
  • Like
Reactions: MP3Mike
Tesla just pushed out a software update and I got the notification on my phone so I went out to the car a little later and my 17" MCU is dead. It was working just fine before that. I'm at 155k miles. There's no way I'm paying for this out of pocket. They bricked their own CPU firmware so there's just no way I'm shelling out $3k for the repair. Absolutely no way. They want me to drop the car off at SC...
 
  • Informative
Reactions: davidc18

appleguru

Active Member
Mar 15, 2017
1,121
1,949
MA, US
Tesla just pushed out a software update and I got the notification on my phone so I went out to the car a little later and my 17" MCU is dead. It was working just fine before that. I'm at 155k miles. There's no way I'm paying for this out of pocket. They bricked their own CPU firmware so there's just no way I'm shelling out $3k for the repair. Absolutely no way. They want me to drop the car off at SC...

Did you actually install the update? Or just get a notification that an update was ready to install?
 

MP3Mike

Well-Known Member
Feb 1, 2016
20,433
51,885
Oregon
Tesla just pushed out a software update and I got the notification on my phone so I went out to the car a little later and my 17" MCU is dead. It was working just fine before that. I'm at 155k miles. There's no way I'm paying for this out of pocket. They bricked their own CPU firmware so there's just no way I'm shelling out $3k for the repair. Absolutely no way. They want me to drop the car off at SC...

It is always possible that the update had nothing to do with the MCU failing. And like @appleguru said above, unless you started the install, the upgrade wouldn't have done anything to cause it to fail yet. (All that is done before hand is the updated is downloaded, unpacked, and validated. After that is done you get the notice that it is available to install.)
 

verygreen

Curious member
Jan 16, 2017
3,031
11,707
TN
unless you started the install, the upgrade wouldn't have done anything to cause it to fail yet
This is not true.

The process is roughly like this:
1. mothership tells the car it needs to update
2. the car downloads the ~830M update and writes it in place to the "offline partition" (also with lots of logging)
3. after some checks on the downloaded image, a UI dialog is shown (at this point everything is downloaded already, this is for AP1 and no AP, for AP2 there's an extra step)
4. After you agree to install it proceeds to update IC and gateway.

So if your emmc was on its last legs, you might imagine extra 850+M write might topple it over, though even with no update it would have failed rather soon.
 

About Us

Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.

Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


SUPPORT TMC
Top