NHTSA asks Tesla to recall 158,000 [now 135,000] vehicles for eMMC failure. Voluntary Recall issued

[MP3Mike]

Same here.. feature restricted MCU2 with FM Radio, I will be fine..

It seems almost impossible for the MCU2 to be the recall solution, because even if they replaced the tuner module you still loose AM radio. And I don't think they can remove features like that for a mandatory recall. (And I doubt they would design a new tuner to add AM for MCU2 vehicles.)

It isn't like recalls have to be done instantly, or that you have to have identified a solution on day one. Look at GM and the Bolt. They have issued a recall for the battery, but they still don't have a solution to deploy to resolve the issue. (Same with Kia and their battery problems.) If software really is causing a lot of the same problems as the failing eMMC, Tesla will just have to fix the software, in addition to replacing the 8GB eMMC modules.

Then of course there is the Takata airbag recall, which started in 2016 and they are still only ~80% of the way through completing the airbag replacements on all of the recalled vehicles. So I think Tesla will be given plenty of time to come up with, and deploy, a solution.

 

[gearchruncher]

If software really is causing a lot of the same problems as the failing eMMC, Tesla will just have to fix the software, in addition to replacing the 8GB eMMC modules.

Then of course there is the Takata airbag recall, which started in 2016 and they are still only ~80% of the way through completing the airbag replacements on all of the recalled vehicles. So I think Tesla will be given plenty of time to come up with, and deploy, a solution.

Please read the NHTSA request to Tesla. Tesla has already argued that they have rolled out software that makes it better, and NHTSA found this insufficient. However, this was only for the eMMC issue. Agree that what they should really do is roll out MCU software that improves performance and stability to avoid a recall for general broken functionality. I think Tesla is hiding behind the eMMC issue as an excuse for general MCU1 issues.

As for the Takata airbags, they had to recall 60 million airbags. If they have completed 48 million in 5 years, this is 800,000 per month. Tesla has 154K to recall. It takes Takata under 6 days to do this many. NHTSA isn't going to let Tesla go years on this. 150k is nothing in automotive volumes. Also, Takata bags are only an issue if you both have an accident, and the bag itself has an issue. Tesla has said every one of these screens will fail in the next few years. They don't have years to replace them.

 

[Crutonius]

As has probably been said elsewhere the software is only part of it. The excessive writing to the flash is causing premature wear on it. Not much to be done when the hardware itself is physically damaged.

The sad thing this was completely predictable and avoidable. A competent firmware team would have recognized this possibility and worked around it.

 

[MP3Mike]

Please read the NHTSA request to Tesla.

Why do I need to read it again?

I said that they had to fix the software and replace the 8GB eMMC. (Two different issues causing the same, or similar, symptoms.)

 

[stopcrazypp]

Why do I need to read it again?

I said that they had to fix the software and replace the 8GB eMMC. (Two different issues causing the same, or similar, symptoms.)

I find it funny people still arguing for free MCU2 as the "best solution" for Tesla. While I'm sure a lot of people would be happy to get a free MCU2 (other than those who want the radio features), from what I see it's pretty much a given that the recall action would be the same as the warranty adjustment program (replacing the 8GB eMMC to 64GB eMMC).

There's a whole lot more Tesla has to deal with if they went the free MCU2 route as the recall solution:
1) Now it becomes a very strong argument that Tesla owes refunds to all owners that previously did MCU2 upgrades, regardless of the feature upgrades they got for it.
2) There are many more parts that need to be swapped (MCU, ICU and optionally radio and main screen, vs just a daughterboard swap)
3) MCU2 removes an existing feature (radio), is Tesla now on the hook to also provide the FM radio module (or even worse, have to come up with a new AM radio solution)?

So I see chances of free MCU2 as slim to none, given they already have way to just replace the 8GB eMMC to 64GB eMMC. We'll see very soon regardless in their response to NHTSA.

 

[Alysashley79]

I find it funny people still arguing for free MCU2 as the "best solution" for Tesla. While I'm sure a lot of people would be happy to get a free MCU2 (other than those who want the radio features), from what I see it's pretty much a given that the recall action would be the same as the warranty adjustment program (replacing the 8GB eMMC to 64GB eMMC).

There's a whole lot more Tesla has to deal with if they went the free MCU2 route as the recall solution:
1) Now it becomes a very strong argument that Tesla owes refunds to all owners that previously did MCU2 upgrades, regardless of the feature upgrades they got for it.
2) There are many more parts that need to be swapped (MCU, ICU and optionally radio and main screen, vs just a daughterboard swap)
3) MCU2 removes an existing feature (radio), is Tesla now on the hook to also provide the FM radio module (or even worse, have to come up with a new AM radio solution)?

So I see chances of free MCU2 as slim to none, given they already have way to just replace the 8GB eMMC to 64GB eMMC. We'll see very soon regardless in their response to NHTSA.

the only issue with the daughterboard thing is that they don’t have 158,000 of them. They can’t get 158,000 of them. (Source is someone who works at Tesla) I was told even if they could source enough the problem is that it could cost them more than their cost for mcu2.

honestly at the end of the day I don’t really care what they do as long as my mcu stops crashing. We ALL Know it’s not a “software bug or firmware bug”

I’m regards to the tekata airbag it seems like in places of the country that is both humid and hot they were replaced first as they had the most chance of having an issue apparently. When I bought my Tesla and my local service center found out it came from Dallas my airbags were overnighted abs replaced before I could even take delivery.

 

[MP3Mike]

the only issue with the daughterboard thing is that they don’t have 158,000 of them. They can’t get 158,000 of them.

For every one they replace they get one back they can refurb. They also get one back for every customer that upgrades to a MCU2. (Which may be the actual reason they lowered the price of the upgrade.)

If they only had 1,000 spares it would take less than 158 cycles to get them all replaced. The other option is that they could hire someone on-site at larger/busier service centers to replace the eMMC chip.

So, if there is a recall, it might take some time to get through them all but they have options.

 

[gearchruncher]

The other option is that they could hire someone on-site at larger/busier service centers to replace the eMMC chip.

The eMMC chip is a BGA part in the middle of a PCB. It requires serious rework equipment and experience to do. It is not a snap in chip or anything. I would not want a MCU1 that had been "repaired" by some guy with a hot air station in the back of a Tesla dealership.

 

[cousin_IT]

the only issue with the daughterboard thing is that they don’t have 158,000 of them. They can’t get 158,000 of them. (Source is someone who works at Tesla) I was told even if they could source enough the problem is that it could cost them more than their cost for mcu2.

In the end this will be a cost-driven discussion. Pro-actively announcing that they'll replace every MCU1 with an MCU2 is costly and complex. It's in their interest to slow-burn the discussion with the authorities and keep replacing parts at a low pace. If they every run out of those parts you'll see they will fix find the cheapest option. Could be an MCU2, or a revised daughterboard with a newly soldered chip only it, who knows? Let's just wait and see. It's possible some will get lucky, but probably not all

 

[cduzz]

Tesla recalled 123K cars in 2018. A much larger percentage of cars at the time.

Yes, tesla's had recalls for simpler things where they could easily get the parts (bolts,

You want a way less energy dense pack in your car? Basically nobody uses those in an EV. Why would you want a LiFePO4 pack?

In the end this will be a cost-driven discussion. Pro-actively announcing that they'll replace every MCU1 with an MCU2 is costly and complex. It's in their interest to slow-burn the discussion with the authorities and keep replacing parts at a low pace. If they every run out of those parts you'll see they will fix find the cheapest option. Could be an MCU2, or a revised daughterboard with a newly soldered chip only it, who knows? Let's just wait and see. It's possible some will get lucky, but probably not all

The MCU2 as it is right now can't talk to the old style instrument cluster or radio. So you can't swap it out without replacing the IC and the radio module (extra parts on the BOM and more time doing the work).

There aren't enough MCU1s to do the work and the refurbishing of old parts is going to get really spendy if you need to do 100,000 of them. You also can't make a large volume of MCU1s because parts can't be made or bought.

The *actual problem* is that the MCU2 doesn't have the interfaces to talk to the same systems that were on the MCU1; if it did you'd just use that.

So the *actual solution* is that the board you're putting in the "refreshed" S/X has provisions for talking to legacy interfaces. Make it mechanically compatible (or make 2 versions of the boards that have similar BOM but different layouts).

Call this "MCU2A" (still runs intel CPU of some sort, has same or similar memory and storage) -- make a 3/Y version, a "new S/X" version and an "Old S/X" version. The "old S/X" version actually has the ethernet and analog radio installed and active....

As to if tesla actually did this, well, I guess we'll see, right?

 

[viper2ko]

The MCU2 as it is right now can't talk to the old style instrument cluster or radio. So you can't swap it out without replacing the IC and the radio module (extra parts on the BOM and more time doing the work).

There aren't enough MCU1s to do the work and the refurbishing of old parts is going to get really spendy if you need to do 100,000 of them. You also can't make a large volume of MCU1s because parts can't be made or bought.

The *actual* problem is that the MCU2 doesn't have the interfaces to talk to the same systems that were on the MCU1; if it did you'd just use that.

So the *actual* solution is that the board you're putting in the "refreshed" S/X has provisions for talking to legacy interfaces. Make it mechanically compatible (or make 2 versions of the boards that have similar BOM but different layouts).

Call this "MCU2A" (still runs intel CPU of some sort, has same or similar memory and storage) -- make a 3/Y version, a "new S/X" version and an "Old S/X" version. The "old S/X" version actually has the ethernet and analog radio installed and active....

As to if tesla actually did this, well, I guess we'll see, right?

Alot of time and future time wasted to developing for another arm of hardware. At that point it would be cost and time effective to just do the mcu2 upgrade and be done with it forever

 

[MP3Mike]

There aren't enough MCU1s to do the work and the refurbishing of old parts is going to get really spendy if you need to do 100,000 of them.

So the *actual* solution is that the board you're putting in the "refreshed" S/X has provisions for talking to legacy interfaces. Make it mechanically compatible (or make 2 versions of the boards that have similar BOM but different layouts).

Call this "MCU2A" (still runs intel CPU of some sort, has same or similar memory and storage) -- make a 3/Y version, a "new S/X" version and an "Old S/X" version. The "old S/X" version actually has the ethernet and analog radio installed and active....

You think designing and manufacturing a new MCU would be cheaper? Paying someone to refurbish the daughterboards by replacing the $5 eMMC isn't going to cost that much. It is probably way less than an hour of labor.

 

[cduzz]

You think designing and manufacturing a new MCU would be cheaper? Paying someone to refurbish the daughterboards by replacing the $5 eMMC isn't going to cost that much. It is probably less than an hour of labor.

Absolutely. In the volumes we're talking about, where the boards you need to repair are spread all across the world and you have to get the work done in some modest amount of time?

You've got to pull the board, pack it up, inventory it, ship it back to a refurbishing depot where they'll have to unbox it, inventory it, diagnose it, repair all the things wrong with it, validate it, ship it back to either a central depot or whichever service center needs that part (hint, all of them).

Assuming you've got a shop that can grind through 500 per day, the simple "diagnose / repair / validate" cycle is going to require people with specialized skills, specialized tools, and an inventory of replacement parts. That's a reasonable thing to do if you're doing 5 per day, but scaling that to hundreds requires lots of people, tools, and space, and at the end of it you're not really 100% sure the result is going to be good enough to actually satisfy the NTSB. It's a bit like betting you can make an Audi Allroad with 190,000 miles work. Sure, it's just "repair work" but who's to say it isn't going to throw some other weird error 5000 miles after you've replaced the air suspension (again)? (In this case, I'm referring to "what if the flash storage isn't the only underspec part, and the touch screen controllers also fail at some rate high enough to cause the NTSB to reopen the case because they've gotten 8 complaints on your "reworked in a shop in Tijuana" parts)

The alternative is to make a new board, which involves farming the design out to a shop that specializes in that, then take the prototype that shop and make sure it does what they say, then send the design to one of hundreds of shops in china that specialize in that sort of work. You take a one-time hit to your bottom line and move on.

Scale changes everything -- a program to keep 10,000 spares in the warehouse by refurbishing / reworking the small number of cores that come back to you is radically different from "we need 150,000 and a plan to keep the feds off our back"

 

[cduzz]

Alot of time and future time wasted to developing for another arm of hardware. At that point it would be cost and time effective to just do the mcu2 upgrade and be done with it forever

The only broken part is the MCU1; replacing it with an MCU2 based system with ports to talk to the IC / radio is the fastest way forward.

In this proposal, the MCU2a or whatever *is* an intel part with only minimal provisions to talk to the legacy peripherals (analog radio / ethernet based instrument cluster).

I agree -- the arm based branch of the code should be pickled and left untouched.

You can't, in a safety recall, remove someone's AM radio -- they'll complain that they can't listen to their sports radio. Similarly, the IC is just another part that is probably just fine, shouldn't be replaced just because. The proposal is that the MCU2a has an ethernet port (talks to legacy IC) *as well as* the "second display" port that'd be used on the newer IC. That way you just reuse whatever's there. As an added bonus you get to simplify your spare parts bucket by standardizing only on the "Second display" IC; if someone with a 2013's IC starts leaking and they want it replaced, just do the MCU1 recall and replace the dead IC with the new version.

This proposed (and probably doesn't exist) MCU2A would have flags for "talk to digital or analog radio" (using 2 different ports on the board) as well as "talk to legacy or 2nd display IC" (again, 2 independent plugs on the MCU). You'd likely make 2 different versions of the MCU2A, "new car" parts that don't have any of the legacy support parts actually populated on the board and a "spare parts" version of the board which does.

Hell, you could probably put all the legacy ports on a USB daughterboard.

 

[David.85D]

The eMMC chip is a BGA part in the middle of a PCB. It requires serious rework equipment and experience to do. It is not a snap in chip or anything. I would not want a MCU1 that had been "repaired" by some guy with a hot air station in the back of a Tesla dealership.

I understand what you are saying, but the other option is probably poor working conditions, and toxic exposure somewhere in the third world. The global electronics industry is <sugar>. You know it’s a race to the bottom when industry is leaving the Philippines (avg $1.40/hr) for cheaper places.

 

[Cleo-n-Company]

I’m not even going to come close. I’m currently at 17,000 approx on this set (I have my snows on right now but come April 1st I know that I’ll probably only get a few more months before they’ll be done. I bought them Jan 2020 drove 17k until nov 1st when I put the snows on and they’re at almost 6/32.

So you officially beat me. I got about 15K from my first set. By then, between noting the wear bars and listening to the road roar - it was time to drop $1400 or so on a fresh set... sigh...

 

[gearchruncher]

I understand what you are saying, but the other option is probably poor working conditions, and toxic exposure somewhere in the third world.

As someone that designs PCB's, there are 5 places just in Seattle that do this professionally. Nobody is sending a PCB to Asia to rework a single BGA. The only two options are not Tesla doing this themselves in the back of service centers or sending it to Asia. Especially for only 150k boards, 1k at a time, 1 part.

Also, worried about Tesla's supply chain for just one component on one PCB on about 150k vehicles, when Teslas are some of the most PCB / electronics intensive cars made and they make 500k cars a year now.... Don't you worry about how your whole car was made?

 

[Cleo-n-Company]

Here's my sage-like pronouncement for the morning.

Which no one asked for.

But here goes...

It seems the dozen or so smart minds and pragmatic strategists on this single thread could completely dissect this nightmare up-and-down right-and-left.

The same group could probably take over a table in Fremont, lasso in some Tesla engineers, and come up with a viable road map within a matter of days.

This of course won't happen. But that isn't the point.

The point is Tesla does not care.

Tesla does not want to burn a single dollar or iota of brain-power to address this.

Please don't try to convince me that Tesla is doing what any other carmaker would do in the same situation.

This is what we refer to as bullshit.

It would take a trunk and a frunk to come up with all of the service bulletin and recall issues I have encountered over the years with previous vehicles.

In many of these cases - it was all proactive. I got an e-mail or a postcard in the mail asking me to bring the car in for whatever necessary fix.

None of these issues were close to the digitally induced nightmare we're now encountering.

Tesla will continue to delay, obfuscate, cry NHTSA foul, and do whatever it can to ignore this issue.

If Musk spent 1% of his energy and focus on something like this - as opposed to digging tunnels and tweeting about shorted stocks - we'd all be on our way toward a properly functioning car.

I'd recommend we all remember this sojourn when the time comes to look for another EV.

 

[stopcrazypp]

The point is Tesla does not care.

Tesla does not want to burn a single dollar or iota of brain-power to address this.

Please don't try to convince me that Tesla is doing what any other carmaker would do in the same situation.

This is what we refer to as bullshit.

I'm going to play contrarian, given we have some very relevant current events that show how manufacturers may react to calls for recall from NHTSA. For example see what Ford is saying about its recent recall (keep in mind we are talking about airbags that may explode and have killed a few people already as a result):

In its filing Thursday, Ford said its own study of its airbags found “the risks identified were so remote that they were inconsequential to safety.” The National Highway Traffic Safety Administration rejected Ford’s petition Jan. 19.

“We believe our extensive data demonstrated that a safety recall was not warranted for these driver-side airbags,” Ford said in a statement Thursday. “However, we respect NHTSA’s decision and will issue a recall.”

Bloomberg - Are you a robot?

Sidenote (GM did the same thing too last year in terms of appealing):
Bloomberg - Are you a robot?

It would take a trunk and a frunk to come up with all of the service bulletin and recall issues I have encountered over the years with previous vehicles.

In many of these cases - it was all proactive. I got an e-mail or a postcard in the mail asking me to bring the car in for whatever necessary fix.

That's just standard recall procedure required by law (if an issue was pushed to recall status), nothing to do with a specific manufacturer being proactive:

How Will I Be Notified If a Recall Is Ordered or Initiated?
Within a reasonable time after the determination of a safety defect or noncompliance, manufacturers must notify, by first-class mail, all registered owners and purchasers of the affected vehicles of the existence of the problem and give an evaluation of its risk to motor vehicle safety. The manufacturer must explain to consumers the potential safety hazards presented by the problem

https://www.nhtsa.gov/sites/nhtsa.d.../14218-mvsdefectsandrecalls_041619-v2-tag.pdf

None of these issues were close to the digitally induced nightmare we're now encountering.

Tesla will continue to delay, obfuscate, cry NHTSA foul, and do whatever it can to ignore this issue.

If Musk spent 1% of his energy and focus on something like this - as opposed to digging tunnels and tweeting about shorted stocks - we'd all be on our way toward a properly functioning car.

I'd recommend we all remember this sojourn when the time comes to look for another EV.

Well Tesla have not even officially responded to the first request by NHTSA yet (not even close to the appeal process that Ford for example went through with the airbags as above). Let's see how they respond first before drawing conclusions and bringing out the pitchforks.

 

[tkizzy]

They could simply offer as an option - free Emmc upgrade fix (keep radio) or free mcu2 upgrade ($500 change for radio)