Preventive eMMC replacement on MCU1

[Boeingpilot]

I'll add my tale of woe.....

Purchased a 2015 S90D several months ago. First inkling of issues was driving home from work (180 mile). Being a new car (to me) started to examine all the streaming audio options. Noticed after 20 minutes responsiveness of the MCU became worse and worse. Also noticed LTE not available indications. Outside air temperature on IC showed dashes. Then a 'flash', black screen and almost 3 - 4 minutes for the MCU to spontaneously reboot. Chalked that up to a random reboot. 10 days later, selecting Spotify MCU locked up, and got IC warnings of autopilot and cruise inop. Auto highbeams stopped working. Had to manually reboot to gain control. Fast forward 12 days, opened door to car and MCU reboots. Then 4 days later driving to work, set streaming and started to get extremely sluggish response, LTE with slash through it. Then the FUN starts. Car is on autosteer on the turnpike. Screen reboots and then I get alert chimes followed by aggressive automatic braking!! Next day driving home, MCU locks up and auto reboots. So... Lets see, in the course of a month 5 reboots. Tesla did examine remotely, said I was and I quote "Found internal memory running low. Cleared several file caches, freeing up a significant amount of memory for application processing."

This was after the first two reboots. The remote tech indicated that the MCU has been replaced prior to my ownership.

My observations

- This is starting to become a safety issue. The car should not be slamming on its brakes from 70 mph
- Yes the MCU was probably replaced, but my guess, with the old 8g eMMC. So a lot of good that does
- The nVidia Tegra was probably more than adequate for the older software loads. I believe that the amount of stuff the latest loads are straining the whole MCU, not just the eMMC chips
- I agree with Akikiki, Tesla is doing triage. But I also think they are denying a more fundamental problem with the MCU1.
- I know that I am, and would recommend to everyone, open a service call with Tesla every other time the MCU shows problems. I'm starting to show a pattern so when it comes push come to shove, I can show Tesla, hey, this thing's been having issues for weeks (months)

Don't get me wrong, I love the car, and I got a terrific deal on it. But, this is bush league. Tesla is no longer a startup. The Model S is a mature design now. Just like Tesla had to step up to the plate and address drive unit failures on early RWD Model Ss, they need to step up to the plate and admit they have a huge problem with the MCU1 on the S and X. While I purchased the vehicle used, I expect certain things to work reliably. While Tesla says this is 'only' the infotainment system, the reality the MCU is a major computer in the car. Problems with it can cause safety issues. The fact that it happens to control the entertainment is purely coincidental.

It's just a matter of time before a accident occurs that will be traced to the MCUs. If they think they have public relation problems with HV batteries having fires, just think what it will be like if some loss of functionality on a failing MCU causes an accident.

 

[supratachophobia]

@mymagiccarpet,

I agree with you. There is a pattern. And its not the first time. This one seems a bit more organized or explicit than we have seen earlier this year. I believe earlier this year it was no thought out, not organized, just reactionary, it was denial and forced to recognize a problem. But that's another story.

Now, let's briefly talk about what we are seeing right now. This is not denial. This also is not refusing to fix. They are fixing some. I think, its triage - worse first. Really worse first - dying first. And they don't really want to admit to an owner that his is dying, they don't want to use those words. I think, the plan (this part of the year) the plan is stall and do those first that really need it because they are near total failure. And if you (not you specifically, but you in general) are seeing problems with your car, they are resisting not refusing. They are biding their time. Hoping/waiting on more parts to be available.

I think its a supply problem. I think there's not enough MCU2 and eMMCs available to do everyone that wants it. In early summer there were dozens of S/Xs down waiting on remanufactured MCU1. This was before we saw daughter boards. Then two months later, there were daughter boards. And some people pre-emptively went in and got their daughter board replaced before there were serious problems. And simply because there was a rush - relative to just two months earlier, all the next couple of months of supply of daughter boards were used up in late July/Aug and now, they are having to ration them. Rationing what's available, but scrutinizing what can wait against what is imminently about to fail. But they don't want to talk about it.

Do you (again, not you specifically) think they are just sitting around and watching this mess get worse? No. I believe quietly there's folks working to get more MCU2 build/bought/shipped and out to SCs' And get more daughter boards bought and out to SCs. I think TonyT identified that some old daughter boards from 8 GB MCU1's are being refurbished and are making their way out to SCs and into cars. That hints that Tesla is using every trick they can to get more functioning daughter boards and MCU1s out to the SCs.

It also suggests to me that they are sensitive to the complains about the cost now of MCU1 replacement versus daughter board swap and the costs to the customer. Stop and think, if there's old daughter boards being refurbished, that means they are opening MCU1 up, removing the daughter board, using that board as a core and putting the new 64 GB eMMC on them and making them available without putting them back in the 4-5X more expensive MCU1. (Well, we know its a remanufactured MCU but instead of sending that daughter board back out to and SC where it probably cost the customer $400 or less they could wrap that refurbished board around a remanufactured MCU1 and charge (unnecessarily most time) 4-5X more for the replacement. Do we appreciate that when Tesla replaces a daughter board vice a MCU1, it takes more labor, but the labor fee to the customer is just about the same price? When they just replace the daughter board, they must stop, open the MCU, swap the board and reassemble the MCU. That's a lot more time than just taking a remanufactured MCU1 off the shelf and putting the old on in the "back to the factory box".

Okay, I have rambled on far to long. I think there's a pattern. And I think its going to change rapidly, soon, just as soon as my guess that there's a supply problem not a denial problem clears with more parts available. But then - what do I know? Just my two cents.

I think you are unequivocally correct. This is *exactly* the way Tesla would handle this based on 7 years of ownership.

 

[hybridbear]

@mymagiccarpet,

I agree with you. There is a pattern. And its not the first time. This one seems a bit more organized or explicit than we have seen earlier this year. I believe earlier this year it was no thought out, not organized, just reactionary, it was denial and forced to recognize a problem. But that's another story.

Now, let's briefly talk about what we are seeing right now. This is not denial. This also is not refusing to fix. They are fixing some. I think, its triage - worse first. Really worse first - dying first. And they don't really want to admit to an owner that his is dying, they don't want to use those words. I think, the plan (this part of the year) the plan is stall and do those first that really need it because they are near total failure. And if you (not you specifically, but you in general) are seeing problems with your car, they are resisting not refusing. They are biding their time. Hoping/waiting on more parts to be available.

I think its a supply problem. I think there's not enough MCU2 and eMMCs available to do everyone that wants it. In early summer there were dozens of S/Xs down waiting on remanufactured MCU1. This was before we saw daughter boards. Then two months later, there were daughter boards. And some people pre-emptively went in and got their daughter board replaced before there were serious problems. And simply because there was a rush - relative to just two months earlier, all the next couple of months of supply of daughter boards were used up in late July/Aug and now, they are having to ration them. Rationing what's available, but scrutinizing what can wait against what is imminently about to fail. But they don't want to talk about it.

Do you (again, not you specifically) think they are just sitting around and watching this mess get worse? No. I believe quietly there's folks working to get more MCU2 build/bought/shipped and out to SCs' And get more daughter boards bought and out to SCs. I think TonyT identified that some old daughter boards from 8 GB MCU1's are being refurbished and are making their way out to SCs and into cars. That hints that Tesla is using every trick they can to get more functioning daughter boards and MCU1s out to the SCs.

It also suggests to me that they are sensitive to the complains about the cost now of MCU1 replacement versus daughter board swap and the costs to the customer. Stop and think, if there's old daughter boards being refurbished, that means they are opening MCU1 up, removing the daughter board, using that board as a core and putting the new 64 GB eMMC on them and making them available without putting them back in the 4-5X more expensive MCU1. (Well, we know its a remanufactured MCU but instead of sending that daughter board back out to and SC where it probably cost the customer $400 or less they could wrap that refurbished board around a remanufactured MCU1 and charge (unnecessarily most time) 4-5X more for the replacement. Do we appreciate that when Tesla replaces a daughter board vice a MCU1, it takes more labor, but the labor fee to the customer is just about the same price? When they just replace the daughter board, they must stop, open the MCU, swap the board and reassemble the MCU. That's a lot more time than just taking a remanufactured MCU1 off the shelf and putting the old on in the "back to the factory box".

Okay, I have rambled on far to long. I think there's a pattern. And I think its going to change rapidly, soon, just as soon as my guess that there's a supply problem not a denial problem clears with more parts available. But then - what do I know? Just my two cents.

It all makes me wonder if at some point they’ll be forced to start giving away MCU2 because of being unable to fix MCU1. Right now Tesla says they cannot fix our MCU1 issues because it’s a kernel code issue & they don’t have anyone to fix the code.

 

[mymagiccarpet]

@mymagiccarpet,

I agree with you. There is a pattern. And its not the first time. This one seems a bit more organized or explicit than we have seen earlier this year. I believe earlier this year it was no thought out, not organized, just reactionary, it was denial and forced to recognize a problem. But that's another story.

.

I totally agree with you. Not many, if any at all, businesses have enough parts sitting around for when there is a large scale problem. So it makes sense that they are prioritizing. So when I really think about this, I think it's the company's attitude that bothers me. I would be fine if they said "we do not have enough parts so we have to prioritize those cars that will soon, so what we'll do for you is we'll mitigate the issue through SW so that you get less interruptions, but still replace it when it's needed". Seems reasonable, as long as it's not impacting safety, and that timeframe is not expressed in years. But, we don't get that. We get exactly the opposite of that.
Similarly to what happened with #batterygate, they put in so many mitigations in 70s and 85s to manage that battery without giving one ounce of truth to the customer, and with each one, they took something away from the customer.
Noone expected them to have enough batteries laying around to replace every 70 and 85's battery, nor financially would they have been able to at the time. Also, noone wins if they had gone bankrupt b/c of that. However, they could have been truthful and open from the beginning, especially considering this was happening to largely early adopters, and over time, eventually work their way to replacing the batteries. But, they, on the other hand, chose to act without notification or agreement, and continue to deceive their customers on the issue for over 1.5 years now. Hence getting hit with the lawsuit. Maybe they could have avoided it, and not piss off everyone. It's basically misstep after misstep when it comes to customer service. Each time they were forced into doing the right thing for the customer, rather than chose to do the right thing.

 

[SMAlset]

Curious if this is a symptom beyond the black screen reboots: seeing the car indicated in traffic when you log into your app's map and not seeing it moving despite the speed indicated. I occasionally will check where in the route home or running errands my husband is. Sometimes the car's red arrow will jump ahead, sometimes I have to exit that screen to the main one and then re-establish the map screen again to see his progress only to see it get "stuck" again when I know he's on the highway but moving. Is the car having trouble passing on all the driving information because of a memory issue? He has had the black screen reboots for a while now off and on and he has had them look into it. He's been trying to decide what to do and contemplating just upgrading to the MCU2 and maybe with FM radio. His car is a MS 3/2017 with the vented leather seats and it really is a great car so with little ever having been put into it in the way of repairs or expensive maintenance holding on to it at this point and upgrading has been tempting. He has an appointment later in the month.

 

[Akikiki]

Well, we know generally that Mr. Musk is not fond of the media. Please don't quote me, 'cause I can specifically quote him. But I think its pretty much understood that Tesla does not get a fair dealing in the media. Don't forget a few months ago, Tesla reassigned all their Public Relations to other jobs and disbanded that department. Does that sound like management that is happy with how they are reported in the press?

So, what would you do if your company was brow beat and beat up and lied about and abused every time they try to report or say something publicly? Yeah, after a while, you keep yo mouth shut and let them speculate even when its as bad if you formally said it.

While I don't personally like how the don't tell us what they are doing. That's the way it is. It should not shock and awe us that there's not a finite plan to explain repairs or upgrades or what we want to call re-calls that I don't think internally are referred to as re-calls. Nuff said, 'cause I am just going to start sounding redundant.

This stuff we are seeing is not permanent. It will pass. It could always be worse. Glad we are not being treated like 737 Max and parked somewhere.

 

[Akikiki]

Curious if this is a symptom beyond the black screen reboots: seeing the car indicated in traffic when you log into your app's map and not seeing it moving despite the speed indicated. I occasionally will check where in the route home or running errands my husband is. Sometimes the car's red arrow will jump ahead, sometimes I have to exit that screen to the main one and then re-establish the map screen again to see his progress only to see it get "stuck" again when I know he's on the highway but moving. Is the car having trouble passing on all the driving information because of a memory issue? He has had the black screen reboots for a while now off and on and he has had them look into it. He's been trying to decide what to do and contemplating just upgrading to the MCU2 and maybe with FM radio. His car is a MS 3/2017 with the vented leather seats and it really is a great car so with little ever having been put into it in the way of repairs or expensive maintenance holding on to it at this point and upgrading has been tempting. He has an appointment later in the month.

@SMAIset, Let's talk about losing location for a moment. Yes, I believe its related to failing eMMC. Nearly all the functions running on the MCU are services. One of if not the most execution intensive services is displaying to the center screen. The services run on the CPU, not the eMMC. The code is read from/pulled from the eMMC. When the system can read the service code from the eMMC, what's it going to do? Fail to execute/start properly. I've been told, that's what we see - er - don't see because the main screen can't read the firmware from the eMMC successfully/100% to begin executing in order to display. Some of the other services, like LTE, or WiFi, or Nav, or Charging still run when some services like LTE or WiFi, or Nav or Charging (and dozens of others) don't run. And what I have heard - read from owners experiences is sometimes between reboots the same services are not failing or saying it another way, some work and some don't.

So, your husband could be driving home and every service on his car is working proficiently - except LTE connections. We do know that streaming depends on the LTE connection, so you might ask him if streaming music worked at the time you say he's not moving on the map, but you are pretty sure he's headed home. Just an idea.

I have the same, even build month '17 that you do. I am biased now. I upgraded to MCU2 9 months ago. I LOVE IT. No mo' problems and I had some of the most famous issues that others had, and I had them starting a year before we knew this was a serious problem. I can't be more pleased with my car. Hey, I would NOT trade my '17 S, today for a brand new 2020/2021 S. I have FUSC and free premium connectivity. I don't need no stinkin' radio. AND we don't even have 1 Supercharger in our state. In two words, my advice is "Upgrade 2 MCU2 ! Oops, three words. Its worth twice what we pay for it.

I encourage our more skilled Tesla mechanics to correct me were I said something wrong about the services.

 

[SMAlset]

Thanks @Akikiki, I'll pass the info on to him. I think he is leaning strongly right now to just bite the bullet and get the upgrade to MCU2 and get some of the extra stuff that I have on my Model 3. We both have the free premium connectivity and he got free supercharging for the life of the car. He doesn't know if he wants to give up FM however and with his MS and pano roof with the center bar in the roof could even decide to add Sirius FX if they are still offering that I suppose.

Hey as far as not having a Supercharger to use your FUSC, I see from Tesla's SC map that Honolulu and Kailua are listed as getting one. Doubt they will make the 2020 opening date at this point but something to look forward to. Nice to see Maui on the list too. Wonder where on the island that will go in.

 

[Akikiki]

Hi. Then don't give up FM. Instead keep his own screen, and save $250 off the $2500. That makes swallowing the $500 for the FM radio easier. Of course he can add FM back later if he wishes, for the same price separately at $500. So, different ways of doing it.

I've read reports of at least 50 people that have upgraded to MCU2. I've seen none that said it wasn't worth it or they would not do it again.

 

[mymagiccarpet]

Well, we know generally that Mr. Musk is not fond of the media. Please don't quote me, 'cause I can specifically quote him. But I think its pretty much understood that Tesla does not get a fair dealing in the media. Don't forget a few months ago, Tesla reassigned all their Public Relations to other jobs and disbanded that department. Does that sound like management that is happy with how they are reported in the press?

So, what would you do if your company was brow beat and beat up and lied about and abused every time they try to report or say something publicly? Yeah, after a while, you keep yo mouth shut and let them speculate even when its as bad if you formally said it.

While I don't personally like how the don't tell us what they are doing. That's the way it is. It should not shock and awe us that there's not a finite plan to explain repairs or upgrades or what we want to call re-calls that I don't think internally are referred to as re-calls. Nuff said, 'cause I am just going to start sounding redundant.

This stuff we are seeing is not permanent. It will pass. It could always be worse. Glad we are not being treated like 737 Max and parked somewhere.

Am not too sure about this...
On multiple occasions they got a pass from the press, when it should have been quite the opposite. Have you seen how #batterygate has been treated? It was a no big deal by the press, noone cared when Tesla said they stole features from customers who paid for them, and do not plan on giving them back - a feature that's the most expensive piece of the car. The press just skipped it.
I understand your point around PR and how press sometimes responds, but that's not what this is about. This is about how they treat their customers. As a customer, I don't want to find out things in the press. I want them to communicate directly to me in a timely, informative, and respectful fashion. And that is not happening. They have always had an option to communicate directly to their customers and ignore the press, but they chose to deal with the press and ignore customers. The two options are not mutually exclusive. While investors can keep you in the business for a while, it is the customers who ultimately decide whether you stay in the business long term. I can't saw they have that customer orientation gene down lately.

 

[AlexP]

Anyone else being told their FPGA chip needs re-flashed instead of the daughter-board/eMMC chip replaced?

The FPGA was reflashed in my S about a month ago. I keep having reboot and response issues. The maps are not currently loading.

I dropped off the car at service today. They say it should be ready next week. We'll see what happens this time.

 

[hybridbear]

The FPGA was reflashed in my S about a month ago. I keep having reboot and response issues. The maps are not currently loading.

I dropped off the car at service today. They say it should be ready next week. We'll see what happens this time.

They did that with mine too after replacing the eMMC when things didn't improve much. Now they tell me there is no fix because it's a kernel issue & that the only solution is to pay for an upgrade to MCU2.

 

[EV-Fixme]

They did that with mine too after replacing the eMMC when things didn't improve much. Now they tell me there is no fix because it's a kernel issue & that the only solution is to pay for an upgrade to MCU2.

Try factory resetting if your settings were copied over. We have seen issues that sometimes remain when the data is moved to the new chip. We documented the issue in the eMMC thread. Consolidated eMMC Thread (MCU repair) (Black Center Screen)

 

[hybridbear]

Try factory resetting if your settings were copied over. We have seen issues that sometimes remain when the data is moved to the new chip. We documented the issue in the eMMC thread. Consolidated eMMC Thread (MCU repair) (Black Center Screen)

The Service Center has already done that twice. They did that a year ago when our issues first started becoming frequent & they did it again in November as the first step when replacing the eMMC didn't solve all the issues.

 

[EV-Fixme]

The Service Center has already done that twice. They did that a year ago when our issues first started becoming frequent & they did it again in November as the first step when replacing the eMMC didn't solve all the issues.

We have seen that it can take up to 4-5 and documented that.

 

[Akikiki]

... I want them to communicate directly to me in a timely, informative, and respectful fashion.

Yeah, agree. But what big companies do that?

... I can't saw they have that customer orientation gene down lately.

Yes, I agree. When the M3 was delivered, customer service and service centers were not prepared for the mass.

 

[yog_74]

I just received my email for my 2015 Model S. I had the MCU2 upgrade done a couple of weeks ago and one of the reasons was because I was experiencing problems. Can I get a partial refund?

 

[Akikiki]

I just received my email for my 2015 Model S. I had the MCU2 upgrade done a couple of weeks ago and one of the reasons was because I was experiencing problems. Can I get a partial refund?

Suurrrreee, why not? Wait, refund for what? And why?

 

[mymagiccarpet]

They did that with mine too after replacing the eMMC when things didn't improve much. Now they tell me there is no fix because it's a kernel issue & that the only solution is to pay for an upgrade to MCU2.

Sorry to hear you have this problem now

this is interesting, and totally irrelevant. They sold you/us the product. Which kernel is in it, and whether it can, or cannot, or it's too expensive to fix is irrelevant to a paying customer. They are responsible for the final product they shipped/delivered in its entirety.
Seems like their attempt to just get everyone onto MCU2 and profit out of that at the same time. They should just upgrade all cars to MCU2 at no, or at a very minimum cost, and that would give them props from everyone. Great customer service will create positive marketing, and impact sales. Why are they so shortsighted...there are much better ways of handling this.

 

[mymagiccarpet]

Can anyone definitively confirm what the current MCU2 upgrade is if MCU1 fails?