eMMC flash failure - Models S/X

[Bmac]

UPDATE: Tesla Cars Have A Memory Problem That May Cost You A Lot To Repair

Of possible interest to owners with cars with MCU failure or even cars totally bricked. I understand there are older TMC threads on this as well. According to the article, this eMMC fash card was used on Teslas S/X up to 2018.

 

[viper2ko]

So Elons response back, does that mean mcu2 cars won't be affected? I have a august 2017 S.

 

[Battpower]

So Elons response back, does that mean mcu2 cars won't be affected? I have a august 2017 S.

With regards to MCU 1 I take his comment to mean "I should care, I need to care, I can't let anyone that - so there's nothing I can do about it." From what I see, it could be (much?) less of an issue for MCU 2 depending on what's written to the eMMC chip (how much stuff) and probably the typical temperatures your car experiences, but the issue that this type of memory chip has a relatively low number of write cycles before it will start to fail (just a characteristic of the technogogy) is same for MCU1 & 2 as far as I can see.

Elon's response however you take it does not change the realities for the technology. Not just a Tesla issue, but the fact that so far from what I have read they have not significantly reduced the traffic to the eMMC chip which would be a key factor in reducing issues, along with keeping temperatures as low as poss (probably).

 

[boaterva]

You could read Elon’s comment as logging has been seriously reduced recently for whatever values of seriously and recently you want to believe.

No idea why it shouldn’t be turned off totally so future damage is minimized as soon as you load whatever version of the firmware this fix was loaded in.

At least that’s how I read it. But we’ve been down this road before (with the MCU upgrade).

 

[Tam]

So Elons response back, does that mean mcu2 cars won't be affected? I have a august 2017 S.

Yes it is affected.

Prior to 2018, MCUv1 has 8GB eMMC flash memory chip.

Since 2018, MCUv2 has been increased to 32GB eMMC flash memory chip.

The increased memory amount will delay its demise but the root cause problems cited by the article are still there.

1) Too much logging which the article thinks it's useless
2) If they don't reduce the logging, at least it should be re-directed to a volatile RAM that can be overwritten/erased with a reboot or easily replaced.

 

[whitex]

Wow, Elon is a genius. All he said was "should be much better at this point" and people will read volumes into this, everyone interpreting it any way they want to, including fanboys who will interpret it at the extreme as "both MCU1 and MCU2 no longer have this problem". Fixing logs is not sexy, so until it gets wider media attention, Elon is just brushing it off. "should be much better at this point" is along the lines of "your car gets better every update" which is the story Elon is sticking with, even though for this particular problem every update kills the EMMC flash part more and more, in different ways

1. download and store the update - writes large files to EMMC
2. apply the update (unpack it into and overwrite existing software)
3. new updates pack a lot of useless features like gaming arcade, which takes up EMMC room (so both more writes, and less room for wear leveling)

 

[Exiom]

I feel like he is responding only because this problem has finally got on the news...

Sadly I think it needs to get even more wider media coverage to get more things done... and if they admit this is an issue on their part... people who paid to get it fixed would be in a position to get their money back by suing them.

 

[whitex]

I feel like he is responding only because this problem has finally got on the news...

Sadly I think it needs to get even more wider media coverage to get more things done... and if they admit this is an issue on their part... people who paid to get it fixed would be in a position to get their money back by suing them.

This is not a liability thing, this is just poor design, that's all. If a particular brand cars rust to pieces shortly after warranty expires, owners are not entitled to any compensation, but people will know to factor that into the car depreciation, and indirectly how much the new one is worth.

 

[Battpower]

This is not a liability thing, this is just poor design, that's all. If a particular brand cars rusts to pieces shortly after warranty expires, owners are not entitled to any compensation, but people will know to factor that into the car depreciation, and indirectly how much the new one is worth.

When poor design becomes negligence, gross negligence, misrepresentation....

When goods are clearly not fit for purpose.....

When (arguably) unsolicited 'updates / patches' create new problems, exacerbate old ones and materialy reduce the value of the product, then I think there is quite a lot of potential liability.

The whole issue of regular reboots of mcu while driving, and potentially unpredictable and dangerous behavior on restart (while driving) is itself a big liability area.

 

[Exiom]

This is not a liability thing, this is just poor design, that's all. If a particular brand cars rusts to pieces shortly after warranty expires, owners are not entitled to any compensation, but people will know to factor that into the car depreciation, and indirectly how much the new one is worth.

I don't think your example is similar in comparison but if that were the case, I would have thought that is cause for recall. As a bad design, error or not is causing safety issues.

And in Tesla's case, the failure of the MCU or CID, renders the car undrivable, something that is directly caused by Tesla.

 

[Exiom]

I think we here agree or not this is a liability issue is irrelevant.

People will likely sue regardless if Tesla in some way or form suggest that they caused it, just like those currently suing for the loss of range due to an update.

I think it is unreasonable to think that the entire MCU is a "consumable".

 

[whitex]

I don't think your example is similar in comparison but if that were the case, I would have thought that is cause for recall. As a bad design, error or not is causing safety issues.

And in Tesla's case, the failure of the MCU or CID, renders the car undrivable, something that is directly caused by Tesla.

Mandatory recalls are for safety issues. Car unable to start is not a safety issue. MCU failure while driving (much less likely failure mode than the car not being able to start) might force you to pull over and get a tow truck, but no different than your ICE engine dying. All cars do break eventually, some just break sooner than others. Nobody is recalling 20 year old Fords because they can't start. The only distinction is whether it happens before or after warranty, which determines who pays for the repair.

 

[Battpower]

I think it is unreasonable to think that the entire MCU is a "consumable".

I think we could agree with that. But when a car gets this far into the 'Consumer Electronics' marketplace, 100% obsolescence in 2-3 years is the norm. Musk needs to remember he isn't selling game consoles. There is still a difference between reality and virtual reality. Real people pay real money for and make real journeys in his cars and Tesla haven't tested all (any?) of the failure scenarios that could easily occur as a result of highly unpredictable behavior of failing eMMC in MCU's.

The absolute least Tesla must do urgently is to go public with the issue and put in place verifiable mitigation measures. Who knows what a car might do if it installs corrupt update images from failing / unreliable memory?

 

[whitex]

When poor design becomes negligence, gross negligence, misrepresentation....

When goods are clearly not fit for purpose.....

When (arguably) unsolicited 'updates / patches' create new problems, exacerbate old ones and materialy reduce the value of the product, then I think there is quite a lot of potential liability.

The whole issue of regular reboots of mcu while driving, and potentially unpredictable and dangerous behavior on restart (while driving) is itself a big liability area.

The unsolicited updates is absolutely a new thing which will have to be addressed legally. It is new, so probably no precedents in the automotive industry. Excessive logging on the other hand would be just bad design.

As for restarts while driving, that would have to go to court to establish a precedent whether it is dangerous. If so, the manufacturer has to mitigate it, but not necessarily by fixing it. They could get away with disabling the driver's ability to reboot while driving, and maybe force the car into some emergency mode (screen says "PULL OVER AS SOON AS IT'S SAFE! CAR NEEDS A REBOOT" when the MCU has reboot itself). Not great customer experience, but would probably satisfy the safety angle.

 

[whitex]

I think we could agree with that. But when a car gets this far into the 'Consumer Electronics' marketplace, 100% obsolescence in 2-3 years is the norm. Musk needs to remember he isn't selling game consoles. There is still a difference between reality and virtual reality. Real people pay real money for and make real journeys in his cars

You are absolutely correct, however that is nothing the legal system will solve. It will be the marketplace which will have to resolve this. Once people start associating depreciation of a 6 year old iPhone to a 6 year old car, manufacturers may start paying attention.

 

[Battpower]

MCU failure while driving (much less likely failure mode than the car not being able to start)

With many / all owners well accustomed to rebooting MCU while driving, and with eMMC becoming progressively more unreliable / unpredictable, there is already evidence that significant system failure / unresponsiveness will be more common.

 

[Battpower]

They could get away with disabling the driver's ability to reboot while driving, and maybe force the car into some emergency mode (screen says "PULL OVER AS SOON AS IT'S SAFE! CAR NEEDS A REBOOT" when the MCU has reboot itself). Not great customer experience, but would probably satisfy the safety angle.

For UK owners aware of the challenges of our 'Smart Motorways' this paints a glorious picture of Teslas stranded in the middle of the driving lanes waiting for their cars to come back to life after a reboot.

You are dead right, not a great customer experience (or advert for the car either).

For 'The It Crowd' followers, this smacks a lot of "have you tried turning it off and back on again?"

 

[whitex]

For 'The It Crowd' followers, this smacks a lot of "have you tried turning it off and back on again?"

Sadly, all technology is going there. The complexity of designs is just too much to have product "just work". Phones recommend weekly reboots, more and more products come with hidden periodic reboot (or optional, but enabled by default, but if you disable it they fail regularly). It's the functional consolidation trend. It's presumably cheaper to have one MCU control everything rather than a simple micro-controller control HVAC, another simple microcontroller control blinkers and maybe lighting. With separated circuits, you can validate the design very well since each part does a very specific thing. They also don't all break at once, so failure modes are much gentler (and repairs are simple and cheap). But no, everyone wants to stick everything into a single chip, a single touch screen, even though that chip now needs 4, 8,16 or more powerful CPU cores. Oh, now you need to figure out the scheduling, all the software running on the same chip keeps interacting with each other creating an impossibly large number of scenarios to test. I wonder how long before someone who matters does the math and realizes that the complexity and validation costs outweigh the costs of separate hardware. The "software is cheap to replicate" dot com slogan is still firmly embedded in people's minds, but they forget that 10 $0.25 microcontrollers is still cheaper than many core SoC's, but more importantly writing, maintaining and validating ever more complex software has exponential costs. Elon is taking this functionality consolidation to the extreme, which may be a good thing for the industry - if he fails, he can serve as a great warning to others trying to do the same.

 

[whitex]

With many / all owners well accustomed to rebooting MCU while driving, and with eMMC becoming progressively more unreliable / unpredictable, there is already evidence that significant system failure / unresponsiveness will be more common.

It that turns out to be a bigger issue, Tesla will simply remove the ability to reboot while driving.