Sorry I may have misread, if you flash is gone (known to go bad on earlier cars or older MCUs) than your MCU is toast, its embeded. As others have said, rooted Tesla's and their support networks have disabled many logging features in an effort to preserve flash life. Hell, I have a salvage MCU off a 2014 with 12000 miles and my flash started acting up recently.
-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
MCU fails for the second time
- Thread starter andyrebele
- Start date
-
- Tags
- mcu Model S touch screen
Actually there are multiple flash devices in there. Bootloader and kernel/ramdisk are stored on a different flash that than the one that is failing due to all the logging.
Granted, there's still no firmware in the new one you'd put, but it's relatively easy to put one in.
Interesting. When the emmc fails, what symptoms does the CID present? Does it not boot at all? Backlights come on but no UI? Boots mainline, but can't write or mount the emmc log partition? (I would presume it would throw a "contact Tesla" type error in this case but otherwise function...)
Sounds like they made some boneheaded logging decisions and/or hardware choices for emmc (or the emmc didn't/doesn't have a good enough controller or enough "unused" memory to do proper flash maintenence).
I suspect the symptoms are "cid shows black screen", but also depends on emmc failure mode - i.e. could the existing data be read or not.
If it could be read then probably actual gui or parts of it can come up.
Car does gobs of logging, just syslog alone is super chatty (and flushes to emmc every 5 seconds), I have 2.5M syslog file and it rotated 4 hours ago (I drove ~10 miles in that time). previous 12 hours is 1.6M gzipped (and nobody was driving it at the time I imagine). - this is lkely the biggest offender.
The updater thing also writes a file for every time it does anything (like ask the server if there's an update, or check some internal stats and so on).
In addition to that every firmware upgrade is a full partition overwrite + more.
Also the cid never sleep it seems (at least not when you have the always connected option) so it keeps being chatty even when the car is not in use.
I have a script to convert /var/log/ to tmpfs, but the log rotate is then messed and their logrotate is too old to have "renamecopy", I guess I should just build my own copy and use that because storing rotated logs in tmpfs is stupid.
(!). Does the CID's kernel support overlayfs? Can you change the logging partition to disposable storage, like (one of) the unit's SD cards perhaps? This would undoubtedly fry your SD card relatively quickly (though an industrial card should last much longer), but at least only costs a few dollars to replace... Or even better, use a USB stick for logging. Then it can be very easily accessed/replaced... and you have quick access to the logs if you want (not sure how useful/human readable they are...)
Yeah, I thought about overlayfs, but they did not build it. I probably can build my own module for it, but I think that does not work with logrotate anyway because overlayfs passes through underlying fsid and it changes depending on where the file is actually located, and we do not want the rotation to put stuff there either, might be worth experimenting some more I guess.
The problem with using usb/sdcards is they are all powered down when the cid enters this semi-sleep state when the screens are off.
Probably can make them stay awake all the time too, I guess.
BTW IC also has similar logging situation - i.e. syslog running full blast to internal partition, but those don't die as easily - probably because IC is actually off for long periods of time.
My co-workers 2012 MCU was JUST replaced long out of warranty with 100K miles on it due to the bubble issue. It wasn't even not working but just a minor cosmetic issue and they STILL replaced it under "good will".
They might be giving these out now quietly. I fought for months on mine and ended up doing it myself. This was over a year ago though
Probably replacing a bad screen with one from a salvage MCU. (Before Tesla would replace just the screen.)
I replaced the entire MCU, due to a stupid error on my part... the intent was to replace the screen only. Once I messed up the MCU (read, cat knocked it over as i was closing it back up) WK had to help me from there on
Tesla just pushed out a software update and I got the notification on my phone so I went out to the car a little later and my 17" MCU is dead. It was working just fine before that. I'm at 155k miles. There's no way I'm paying for this out of pocket. They bricked their own CPU firmware so there's just no way I'm shelling out $3k for the repair. Absolutely no way. They want me to drop the car off at SC...
Did you actually install the update? Or just get a notification that an update was ready to install?
It is always possible that the update had nothing to do with the MCU failing. And like @appleguru said above, unless you started the install, the upgrade wouldn't have done anything to cause it to fail yet. (All that is done before hand is the updated is downloaded, unpacked, and validated. After that is done you get the notice that it is available to install.)
It very well coulda pushed his flash memory over the edge, thus killing his MCU... totally within the realm.
This is not true.
The process is roughly like this:
1. mothership tells the car it needs to update
2. the car downloads the ~830M update and writes it in place to the "offline partition" (also with lots of logging)
3. after some checks on the downloaded image, a UI dialog is shown (at this point everything is downloaded already, this is for AP1 and no AP, for AP2 there's an extra step)
4. After you agree to install it proceeds to update IC and gateway.
So if your emmc was on its last legs, you might imagine extra 850+M write might topple it over, though even with no update it would have failed rather soon.
I can't wait to see what happens with all these model 3 cars which rely on everything for the 15 inch screen. That will be a doozy.
Well, Model S/X have only 8G EMMC, where at model 3 has 64G, that alone would make it a lot more robust in light of overwrites even if the rate of writes is otherwise the same.
Similar threads
