Think I need a new PEM Fan... Anything else to check?

[jfischer]

Huh. I just noticed that the image in post #39 has the same connector pin raised. @jfischer, did you notice this on yours?

Is it supposed to be that way?



The base of the pin on mine certainly looked like it had a shoulder on both sides just above the connector body, and none of the other pins had that. It also pushed in without a fight.

What does that pin go to?

I don’t remember the 6 pin socket being pushed in but def pin 1 of RHC34 pem fan socket was.

The 6 pin one goes to the ESS heater and would not be causing your issue. Its def the PEM fan socket and connector

here are a few more pics of the PEM CIC board.

it’s a pain in the ass to get too as you need to remove the battery side cables, then the main control board, the control board plate with crappy screws as two of mine were already stripped... then the CIC board itself. The two screws holding it to the bottom of the pem were a pain to put back in!

Some pics of the process.

 

[gregd]

I don’t remember the 6 pin socket being pushed in but def pin 1 of RHC34 pem fan socket was.

The 6 pin one goes to the ESS heater and would not be causing your issue. Its def the PEM fan socket and connector

here are a few more pics of the PEM CIC board.

it’s a pain in the ass to get too as you need to remove the battery side cables, then the main control board, the control board plate with crappy screws as two of mine were already stripped... then the CIC board itself. The two screws holding it to the bottom of the pem were a pain to put back in!

Some pics of the process.

Thanks, very helpful!

I did one more test, one that I should have done long ago... I put the 6w bulb on the motor fan connector, and measured the PEM fan side. If, as I had suspected, the fault were upstream of the connector, the load on one should have affected the other. It did not. Swapped sides, load on PEM, measured Motor, same result. So, I may in fact have a bad connector, one that failed with both negative pins (in spite of the unlikeliness) equally burned. Next step is to take a look at it.

If the pins are in fact burned, I'm going to insist that Tesla replace the CIC board and wiring harness under warranty. I warned them about the issue last fall, and they certified that all was ok. If the pins are burned, then clearly all was not ok.

Question to the group: Would it be a better risk to have the whole PEM replaced, or only the CIC board? I suspect that the "remanufacturing" process for replacement PEMs does not include replacing the IGBT insulation, so I may be trading an otherwise good PEM for one that might fail catastrophically sooner. I'm thinking just the CIC board...

Shipping the PEM to Gruber for their refurbishment involves shipping the entire car. According to Peter, it's safer to ship the car, since proper shipping crates for just the PEM are not available, and that way they can also address any other issues that might be present.

Updated wiring diagram, showing the 20 A fuses and power regulation, per discussion with Peter at Gruber this past week.

[/QUOTE]

 

[gregd]

Update and a further question:

I opened the "hatch" in the trunk, and it appears that the last time they removed the PEM for service that the wiring to the fan connector goes straight down instead of off to the passenger side of the car. So, it's accessible by reaching in through the window. If I wiggle it, fairly roughly, I can make the output under the car go from 13.4v to mv, so it's clear that there is something going on with some of the pins. I wasn't able to get the 6w lamp to light, however. Presumption is that they're burned, but I still need to get it off to see.

Question: How hard should it be to remove (and re-attach) the connector? I slide down the latch on the rear-facing side, and tug and wiggle, again fairly roughly, but no movement. Is it really that tight? (perhaps, the weather seal)? I don't want to break anything if there's a trick to getting it off smoothly. I also don't want to be in a position that I can't get it back on. Pulling down is a lot easier than pushing up, given the limited access.

 

[gregd]

Question: How hard should it be to remove (and re-attach) the connector?

Answering my own question... Slide the latch down, and then squeeze the lever underneath. It comes off pretty easily after that

So, it appears that, yes, burned contacts. PEM Positive definitely fried. Oddly, it appears that the motor side isn't so bad, so I don't understand why it's not providing the current. Perhaps they're so loose that the burned contact point is deeper inside.

Emailing this to the SC and see what they can do about it, given that they claimed the pins were perfectly good last fall. I'm claiming this is a warranty repair.

 

[jfischer]

Answering my own question... Slide the latch down, and then squeeze the lever underneath. It comes off pretty easily after that

So, it appears that, yes, burned contacts. PEM Positive definitely fried. Oddly, it appears that the motor side isn't so bad, so I don't understand why it's not providing the current. Perhaps they're so loose that the burned contact point is deeper inside.

Emailing this to the SC and see what they can do about it, given that they claimed the pins were perfectly good last fall. I'm claiming this is a warranty repair.

View attachment 564570

Yes you certainly need the connector replaced it will only get worse with time and with more arcing as the gap gets wider with the fan turning on and off.

The service LIFE of these MOLEX MLX150 connectors is only about 20 cycles (which is the times connected and unconnected) on BOTH the PEM connector and the PEM FAN connector... Kind of useless service life when for a few cents more you could have gotten the gold plated connectors which have a service life of about 10x the tin plated connectors... So every time you remote the PEM, you lessen the life if the connector and plug.

The amp rating is also low for the single motor FAN as it can approach 20 amps at 12V which is over the connector rating as well.

 

[drewski]

<snip>

Emailing this to the SC and see what they can do about it, given that they claimed the pins were perfectly good last fall. I'm claiming this is a warranty repair.

glad you got it figured out. good luck with the SC!!

 

[jason.tv]

looks like the corrosion grease started to eat away the plastic.

 

[gregd]

Right, I understand how bad the connector is for the service it's being asked to support.

The Service Center is now blaming: the blower motor (which I opted to not replace last fall), and the OVMS, which they claim "can cause false alerts". (I know, that's not possible. We've discussed this in the past.) They also say that it's not possible to do a mobile visit. I thought there was a roving Roadster repair van somewhere, no?

Additional photos, of the PEM-side of things. Looks like 3 of the 4 pins are fine. But the PEM-side Positive (upper right corner in this view, I believe) looks a bit dark... Thoughts?

 

[gregd]

looks like the corrosion grease started to eat away the plastic.

Wait, what? No, that's just it being smeared on the surface. It's pretty solid. I didn't wipe it all off before the picture.

That said, the PEM-side seemed a bit odd... At first glance it looked damp, but I think that was just the horrible angle of the light. It's a bit tight getting the trouble lamp, dental mirror, and my camera adjusted at the same time.

 

[markwj]

and the OVMS, which they claim "can cause false alerts"

For the record, OVMS has never caused false alerts. We only report on the alerts raised by the car.

OVMS used to display ALL alerts raised by the car (including debug-only ones). This was changed in October 2019 to only show non-debug alerts.

 

[gregd]

For the record, OVMS has never caused false alerts. We only report on the alerts raised by the car.

Yep. I will be making this point very clear in my letter back to the SC... My emails, btw, are CC'd to RoadsterServiceNA, on the assumption that they're listening.

Back to the problem at hand. Root cause, I think, involves more than just the PEM connector. It occurs to me that there are two other connectors that could be contributing to the issue - the ones that attach to the motor. If either of them were "loose", that would transfer load to the other side. What should have been done, for reliability sake, would have been to modify the wiring harness itself to bond both of the pairs of wires together between the PEM and Fan, as part of the "Dual fan upgrade". As it stands, we have 3 connectors that must all be perfect. There is, in fact, no redundancy or resiliency, especially when it comes to protecting the weakest (and most expensive to repair) point in the chain. The two sides are tied together at the motor, but that's too late in the chain.

Thinking back, my first reaction to disconnecting the two connectors at the motor was "Wow, that came apart easily". In fact, Gruber's decoding of the 1146 (motor fan) alerts that the car recorded last December showed an undercurrent (not zero current) fault. Is this the root cause of the car's 10 year struggle with the PEM fan connector? I bet (predict?) that one of the contacts at the motor is faulty. Probably the motor positive pin, since it's the PEM-side positive that's fried on the PEM end. If the motor positive pin at the motor wasn't making good contact, then all of the current from both sides would be flowing through the PEM positive pin, returning to their respective negative pins. This is probably why we see fewer negative pins fried at the PEM end. If a negative pin were loose at the motor end of the harness, there would be no additional load on the PEM connector pins, since (I presume) each side is separately current regulated at the negative side. The positive side is not regulated; it ties directly to APS+. Two good negative connections can push all their current into the one positive; two good positives will not sink more than one good negative's worth of current.

However, that does not answer the question of why I was not able to pull any significant current from either side of the PEM connector when cross-loading them motor positive to PEM negative. I'm guessing (concluding) multiple faults here, and yes: Using a sewing pin, I feel a nice tight friction on the two negative pins (3 & 4) at the PEM-end of the wiring harness; the two positive pins (1 & 2) are as if there's no pin even installed. They've been totally relaxed, or more likely, vaporized. So, it appears that BOTH of the connectors at the motor end have loose pins, shifting the load back and forth among the positive pins at the PEM end, eventually frying both. Is that possible? Wouldn't I see evidence of this on the motor connectors? They seem clean...

I think the next step is to remove at least the two fried pins from the wiring harness. How does one extract the pins from the connector? If I recall, there's a special tool needed? Or can I stick a simple straight pin / needle into the notch (toward the center by the "1" in the picture below), and pull the wire & connector out?

 

[gregd]

A resource, if anyone is looking for info on the connector:

Overall info page:
Molex Connector Part Number - 19418-0004

Product brochure. Pg 34 talks about "servicing" the pins.
http://www.literature.molex.com/SQLImages/kelmscott/Molex/PDF_Images/987650-2181.pdf

Complication: I see that one needs to insert an "extraction tool" (aka paper clip?) into the front end of the connector, while the latching mechanism is in a certain position. Haven't tried that yet, but I notice that the slot for the extraction tool for the PEM positive pin is melted shut... Oy.

 

[PV-EV]

If the connector is damaged and you are unable to release the pins simply cut the connector off and start fresh with a new connector and pins. These things are inexpensive. The gold ones are a good suggestion. Remember to replace the PEM side too, as it is no doubt by now compromised. I disagree with your analysis of the connector at the motor. In my experience the bad connector fails. It doesn’t cause a different connector to fail. As you know the bad connection has more resistance which causes heat, which causes the connection to have higher resistance and hence more heat and on and on until it doesn’t work.

I read an interesting engineering article about fretting of metals used in connectors due to vibration and the failure resulting from the same. I’ll try to find it and post it here. I felt at the time that it might explain the common problem this connector has in our application. Good luck on your repairs. You have all the skills needed to do this yourself.

 

[jfischer]

A resource, if anyone is looking for info on the connector:

Overall info page:
Molex Connector Part Number - 19418-0004

Product brochure. Pg 34 talks about "servicing" the pins.
http://www.literature.molex.com/SQLImages/kelmscott/Molex/PDF_Images/987650-2181.pdf

Complication: I see that one needs to insert an "extraction tool" (aka paper clip?) into the front end of the connector, while the latching mechanism is in a certain position. Haven't tried that yet, but I notice that the slot for the extraction tool for the PEM positive pin is melted shut... Oy.

A couple of side notes from my lessons learned replacing the connector and a few pics of the connectors new.


The cheap crimpers from Amazon didnt really do that great of a job crimping, even after several tries. It just wasnt a tight clean crimp.

I opted for the less expensive MOLEX crimper after tinning the wire with solder and slightly soldering the connection before the crimp. You cant put too much on as it will not allow the crimp to mold properly.

The tool to get them out is slightly smaller than a paper clip in diameter. It goes in as follows.

 

[jfischer]

Sorry forgot to attach the pics...

 

[gregd]

Thanks for the confidence, @PV-EV. My thought at this point is to replace (wire around) the current wiring harness between the PEM and fan with a new, separate cable. Tie both sets of pins together near the PEM, single pair of wires to the motor. I'm considering replacing the blower assembly too, partly just to renew it and put the whole blame-thing to bed with the Tesla SC, but the ones from Gruber come with the original unit's single power connector (vs Tesla's two), and that would mate better with the new cable. Gruber supplies a pigtail to split that single one into two, so I'd not use that. The fewer connectors, the better.

To my theory about the fan-end connectors precipitating the PEM-end failures, consider what would happen if one of the positive connections at the fan were to fail. You'd have 2 pins (negative) feeding one (positive) because things are cross connected farther down stream. That can't end well. I do need to find some specific evidence to support my theory, however, but it's the only thing that appears consistent with all the curious observations that I've seen over the years. The preference for frying only the positive pins, the repeated failures on some cars vs others, the erratic or seasonality of the errors, the odd "ease" of disconnecting the fan motor connectors (and what I'm feeling when I insert an F-type blade into them), and a few others that escape me right now. Individually, each of these can be dismissed as randomness, but collectively they all seem to point to something loose on the fan-end of the harness being the root cause of a lot of misery.

The one unknown at this point is whether I can save the PEM-side connector. 3 of the 4 pins look ok, even though one of them is associated with a fried pin (pin 2, the motor-side positive). I'm hoping that a good cleaning might salvage the 4th, though I understand that it's probably best to replace it. Just looking to not have to physically remove the PEM if I can avoid it, as I'm not very well set up here to do that. Perhaps in the mid-term, it will hold well enough to last until there's another reason to take the car apart (e.g. annual maintenance next year?), when (if?) I can get Tesla to replace the CIC board at the same time.

 

[gregd]

A couple of side notes from my lessons learned replacing the connector and a few pics of the connectors new.


The cheap crimpers from Amazon didnt really do that great of a job crimping, even after several tries. It just wasnt a tight clean crimp.

I opted for the less expensive MOLEX crimper after tinning the wire with solder and slightly soldering the connection before the crimp. You cant put too much on as it will not allow the crimp to mold properly.

The tool to get them out is slightly smaller than a paper clip in diameter. It goes in as follows.

Thanks, and for the pictures. The Molex instructions talk about locking the connector into a service "mode" prior to the extraction. I haven't tried doing that yet.

But, my current thought is to ask Gruber to make the new cable for me, assuming I order the replacement blower assembly from them, as they'd have the right tools and procedures. They seem to be genuinely interested in figuring out what's happening here, and I think my "fan-side connector theory", if I can produce physical evidence of it, will benefit many downstream.

A question I asked some time ago (different thread), but never got an answer. Is it allowed to mix gold contacts with a tin connector?

 

[jfischer]

Thanks, and for the pictures. The Molex instructions talk about locking the connector into a service "mode" prior to the extraction. I haven't tried doing that yet.

But, my current thought is to ask Gruber to make the new cable for me, assuming I order the replacement blower assembly from them, as they'd have the right tools and procedures. They seem to be genuinely interested in figuring out what's happening here, and I think my "fan-side connector theory", if I can produce physical evidence of it, will benefit many downstream.

A question I asked some time ago (different thread), but never got an answer. Is it allowed to mix gold contacts with a tin connector?

Putting into the service mode as shown here where its "popped" out about slightly.

You can use a small slotted screwdriver in both sides to pop it out. You can see the tabs just above where my fingers are on the left and right sides. work one side, then the other until it pops up.


I will note however that EVERY one I have "replaced" has cracked the white part while trying to put it into "SERVICE" mode except when its NEW. Probably UV or heat damage to the connector makes it more brittle after it has aged a bit.

As far as mixed gold and tin use on the connector, im not sure. Nothing in the specs says "dont do it"
The only difference between the two is the cycle use count of 20 tin vs 200 for gold. Amp load etc all seems to be the same.

If I was doing the CIC board and the connector again, ID opt for the gold plated connector and plug.

 

[gregd]

Thanks, very good to know. That supports my plan to simply create a new wiring harness for powering the fan.

Update on the "loose connector at the fan" theory. None of the 4 pins (2 on the motor blower side, two on the PEM) show signs of being burned, at least not that I can see. The blades look fine. I can't see into the female side, of course. Significantly, the pins on the motor blower side are definitely looser than the one for the PEM blower, and none are all that tight. So, I'm going to stick with my theory on this, that the loose connector on the motor side shifted the load to the PEM side, putting additional stress on the pin 1 of the PEM connector. But since both seem a bit on the loose side, at times, the load probably shifted back the other way, depending on ambient temperature and/or the effects of resistive heating. If so, that could account for the frying of pin 2.

So, if Tesla had replaced the blower assembly, random chance might have yielded slightly thicker blades on the connectors, but since replacing the wiring harness was not included, the $1,000 replacement likely would have been to no net effect. The issue is in the female side.

 

[gregd]

I talked to Peter at Gruber Motors this afternoon, and he's intrigued enough with my conclusion that we're going to give it a try. The "native" blower assembly has a much beefier "Deutsch" connector on it, vs the pair of F1-based ones that Tesla uses, so the plan is to get a new fan assembly, and we're going to use that better connector as-is and create a new cable harness with it on one end, and a new original PEM connector on the other, with the two pairs of circuits wired directly together in the harness. Hopefully keeping the original style (though known weak) PEM connector will prevent any (or too much) grief from Tesla when it comes time to do the next de-leafing of the car.

Fingers crossed!