Roadster PEM failure

[racevpr]

The most common problem we are seeing with Tesla Roadster Power Electronics Modules (PEM) noted in a Tech Tip we generated and reported a year ago to forums and Tesla Service Centers, is becoming rapidly more widespread.

We even spoke with the original manufacturer of the PEMs, and advised them an insulating material between the IGBT transistors and heat sinks is deteriorating, eventually shorting the transistors, and causing inverter power train failure. We believe if these cars were still under a warranty, a recall would have been issued.

To date, we have not seen a PEM unaffected by this predictable failure, and we urge an owner initiated Roadster PEM upgrade, which requires removal of the PEM from the car, shipped to service and repair shops like ours for replacement of this insulating material.

The only solution we have seen from Tesla is a PEM replacement costing just shy of $10K. A repair and upgrade from a Roadster specialty shop like ours is far less than the Tesla PEM replacement, and a more complete upgrade, as outlined next, breathes new life into the PEM and car.

We not only recommend the PEM is upgraded replacing this failing material, but since the PEM contains other aging failure prone electronic components, and is the heart of the car, we recommend a proactive replacement of other PEM wear electronics such as relays, capacitors, fuses, and switches, comparable to an Internal Combustion Engine rebuild during this repair.

Since this insulating material PEM failure can occur at any time, sitting stationary, or while driving, shorting power transistors can cause safety issues, and more serious PEM damage upstream in the driver boards. We urge owners to take proactive measures.

 

[Doug_G]

That is alarming! Thanks for the information.

 

[ecarfan]

we urge an owner initiated Roadster PEM upgrade, which requires removal of the PEM from the car, shipped to service and repair shops like ours...

Always good to hear an objective assessment from a disinterested third party.

 

[dhrivnak]

The most common problem we are seeing with Tesla Roadster Power Electronics Modules (PEM) noted in a Tech Tip we generated and reported a year ago to forums and Tesla Service Centers, is becoming rapidly more widespread.

We even spoke with the original manufacturer of the PEMs, and advised them an insulating material between the IGBT transistors and heat sinks is deteriorating, eventually shorting the transistors, and causing inverter power train failure. We believe if these cars were still under a warranty, a recall would have been issued.

To date, we have not seen a PEM unaffected by this predictable failure, and we urge an owner initiated Roadster PEM upgrade, which requires removal of the PEM from the car, shipped to service and repair shops like ours for replacement of this insulating material.

The only solution we have seen from Tesla is a PEM replacement costing just shy of $10K. A repair and upgrade from a Roadster specialty shop like ours is far less than the Tesla PEM replacement, and a more complete upgrade, as outlined next, breathes new life into the PEM and car.

We not only recommend the PEM is upgraded replacing this failing material, but since the PEM contains other aging failure prone electronic components, and is the heart of the car, we recommend a proactive replacement of other PEM wear electronics such as relays, capacitors, fuses, and switches, comparable to an Internal Combustion Engine rebuild during this repair.

Since this insulating material PEM failure can occur at any time, sitting stationary, or while driving, shorting power transistors can cause safety issues, and more serious PEM damage upstream in the driver boards. We urge owners to take proactive measures.

Is there a possiblity you would sell the parts so an onwer could make their own repair? The shipping of a 120lb PEM is not inexpensive and would add more than a week to the turn around. Thank you

 

[jeremyz]

racevpr, did you buy the Tzero from the forum member here who owned it? How much are you charging to refurbish a working PEM?

 

[racevpr]

The higher grade dielectric insulating material and IGBT Transistors we use, can be sourced.

The PEM disassembly and retrofit installation however poses some challenges and risks. For example, the current sensing boards, which have to be removed, are bolted directly into copper buss bars without steel pemnuts to protect the threads. Torquing all power connections is of course vital to reduce contact resistance, and over-torquing this connecting point easily strips the threads in the copper buss bar.

Tesla is no longer producing Roadster PEMs and a customer who had their PEM replaced by Tesla for $8500 last month, told us they admitted they are out of new PEMs and are now mandating core returns, reselling them as refurbished units, indicating they are doing PEM core repairs.

A failed DYI effort could be costly, and a shortage of PEMs is likely imminent. A broken PEM might result in a disabled car.

Through our reverse engineering efforts, we have refined the PEM repair and upgrade processes and identified the predictable failure points.

The shipping issue is relatively easy since we pay for shipping both ways in the continental US. We even pre-ship an empty foam protected box within which to insert your PEM. Service shop turnaround is typically less than a week for this service.

 

[racevpr]

racevpr, did you buy the Tzero from the forum member here who owned it? How much are you charging to refurbish a working PEM?

There are only three of the Tzero's in existence, and I am not sure if the previous owner is on TMC.

It is a remarkable car, a piece of automotive history especially in this millennium.

Preferring to honor the TMC forum guidelines, I am unable to discuss pricing.

 

[supersnoop]

Funny how this gets posted at the same time you posted a fix for sale on eBay.

 

[MileHighMotoring]

I for one am happy that you're offering to help owners out, I know in the past there was an issue of shilling on the forum but I'm willing to put that incident behind us as you're the only company out there who's trying to provide Roadster work outside of a SC.

Given that us removing and shipping the PEM is not only expensive, it's outside the capability of many of us, is this something you could fly out to a customer's town and perform off-site in our garage or something? Just a thought.

 

[Bobfitz1]

I for one am happy that you're offering to help owners out, I know in the past there was an issue of shilling on the forum but I'm willing to put that incident behind us as you're the only company out there who's trying to provide Roadster work outside of a SC.

Given that us removing and shipping the PEM is not only expensive, it's outside the capability of many of us, is this something you could fly out to a customer's town and perform off-site in our garage or something? Just a thought.

I just had a very good chat exchange with a tech from the company offering this PEM rebuild. He knew answers to all the questions I asked. They suggest finding a shop that will take out the PEM to ship, with a tech from them briefing the mechanic. There are two levels offered as described in their Ebay listing and cost of either includes shipping both ways.

 

[wiztecy]

There are two levels offered as described in their Ebay listing and cost of either includes shipping both ways.

I honestly would rather / prefer to have someone local do the PEM upgrade work. So what happens if UPS, FedEx or whoever has a bad day and crushes your PEM? Granted you'd have to insure it, and proper insurance would be the cost covering a replacement/refurbished PEM + installation. But the issue on hand is that there are no spare brand new PEMs anymore from what I've seen. Its a risk, a huge one that I personally wouldn't take. From my experience things that are fragile and heavy tend to get damaged in the shipping process. Taking the PEM out and having it serviced locally is the best solution as long as you have a competent technician.

 

[DanielFriederich]

PEM IGBTs short cut failure

It occured approx. two weeks ago to my Roadster and I talked to 2 companies which are doing a refurbish of the PEM and Tesla itself too.
I ended up opening the PEM by myself (if you are somehow skilled to use some basic tools and not afraid of switching off the 400V of the battery and measuring the connectors then an own dissasembling of the PEM out of the Roadster is doable - theres a thread here about PEM and Motor cleaning where it is described what you have to do to get the PEM out of the car. I did it already once last year to get the debris out the motor fins and cooling parts in the PEM...)

So, I took the PEM out the car and opened it up with no visible or smellable result for a burn. Turns out that Tesla used standard electronic parts to build up the PEM with 84 IGBTs (field effect transistors, 600V, 75A / 80A).
It seems like the thermal insulation and isolation of those transistors ages and by time you might get a shorted transistor to the cooling part resulting in a non funtioning PEM.

I decided to let the exchange of all IGBTs and insulation parts be done by a electronic repair shop (yamaha certified partner)... In the end the work is the same as exchanging a faulty power transistor in an hifi amp...

Somebody with a at least good basic knowledge of electronis / electronic parts and the ability to do good soldering and small mechanical work will be able to exchange faulty parts inside the PEM.

here are some fotos of the opened PEM



I will post more info on that topic after the repair has been done (right now I am collecting all the needed parts....)

 

[Kerios]

After purchasing my Roadster Sport (VIN 277) last year in March 2016, with a highish mileage of 68k and A CAC of137 it start to demonstrate symptoms indicative of dual fan failure - which I replaced myself, this cured some of the issues but within a month it failed to go and then remain in performance mode. The Roadster was under extended warrenty, and in September following a long stay with Tesla Gatwick, had a remanufactured PEM fitted - this was the second of two to be fitted whilst there, the first was from Norway and was found to be faulty, the second came from Switzerland, with markings indicated it was a EU Remanufactured unit.

From September till last weekend, the Roadster was flawless, and I had no issues other than replacing the inner wheel arch liner for the OS wheel. It went through it's annual service without issue in March and I am awaiting it to go for the 3.0 upgrade (vehicle came with battery replacement plan). It is my daily drive, rarely driven in range, charged normally and garaged accordingly - average range 160 miles.

Unfortunately, on Monday after driving down to the southwest of England, a trip of just over 180 miles - in range mode, I stopped to charge which was successful, however on then attempting to drive away, the red fault light lit up and it wouldn't move. Error message was the ominous powertrain failure restart when safe / debug showed 974 DMC FW: Pole Current Error Fault whilst charging resulted in the light moving from blue to red and back and no charge taking, restarting having no effect.

After leaving the car for 2 hours, the red fault light cleared finally and I briefly drove to another charge point - during this drive of 20 miles, temperatures were normal, regen seemed to work and performance was the usual. But, when I checked to see whether charging would work, the vehicle returned to the red fault state and wouldn't move.

Following a painful (abet free) flat bed recover (good to see tow mode, the towing eye and low loading works) of 230 miles back to Gatwick Service Centre and the subsequent excitement, I got a courtesy car, and the vehicle was left for diagnosis. The report, received later that day, indicated PEM failure again and the race is on now to try and locate a new unit from Fremont - I suspect this will take a while, weeks if not potentially months.

This is not a dig per se at Tesla, Gatwick are being as helpful as they can, but it leaves me (and others) worried that the PEM remanufacturing is not what it may sound to be. Other threads are indicating that the issue may lie with the IGBT insulation material and subsequent degradation, but one would imagine that a PEM remanufactured would have this material, connectors for the fans and other at risk parts replaced - so are Tesla doing this, or is it being out sourced?

I am fortunate that the PEM will be replaced at no cost, despite the warrenty having expired, the new unit in September came with a new 12 month warranty, but I'll be without the car for a while... I am certain there is a better solution, and one that perhaps sees the replacement of the IGBT boards with new, as in brand new, boards, which a competent technican or equivalent could fit.

Views?

 

[jaanton]

My PEM failed not quite two weeks ago. Just weeks after getting the battery upgrade. Such a bummer. Fremont service has it, but not yet said if the repair would be under the battery warranty.

If Tesla will not cover the repair, other shops seem like a good possibility. Any San Francisco area places?
Really, I need the information now.

 

[DanielFriederich]

Did you get any error codes before and while the PEM died? For example the BSM Isoliation Fault could be an indication, also the codes 905 - 907 and 1055 - 1057 (these would indicate a faulty insulation and a defective IGBT).
Did Tesla already pull the logs and say something about it? Try to enter the Service Menu in VDS and get the last recorded error messages.


[

 

[dpeilow]

@Kerios There is some talk on the German forum (if I have understood correctly) that Tesla in Switzerland has been working with a third party company to get PEMs re-manufactured. Maybe yours came from there?

@DanielFriederich is that correct info?

 

[DanielFriederich]

Yes, but to be more precise:
GS Technology (Berg Switzerland) offers a repclacement of defective IGBTs and refurbish of the insulation inside the PEM. They offer to get the cars from SC Winterthur. Not sure if SC is taking out the PEM or GS themselves. I talked to one of the engeneers at GS and got some info about their work (doing it since approx. since 2 years). They are also starting a battery project...

In the TFF forum they stated the following:
"In consultation with the Tesla service centers, in particular SC Winterthur, we will pick up defective vehicles locally."

Tesla Germany right now only sends PEMs back to the states; I am not aware of any work to be done inside the PEM over here in Europe directly by Tesla nor can I confirm an official cooperation with a third party company.


In this thread in the german tff forum where the replacement of the igbts is discussed, one driver in the north of Germany had a faulty PEM (shorted IGBT) early this year and got it repaired at his company normally dealing with (electronic) services in the wind energy market.

I am active in this TFF thread but I will post my info and ongoing efforts to do a complete exchange of the transistors with a "normal electronics guy" by myself here in this thread too since the int. forum reaches far more drivers which might be affected by this problem.

 

[DanielFriederich]

btw. there is another german guy with a 2.0 Roadster which got similar error message and behaviour after being serviced by Tesla (no 3.0! though). My latest info on it was that Tesla wanted to check with the PEM since charging worked in front of SCs eyes and after service not.... keep you guys informed!

 

[hcsharp]

Daniel thank you for the updates. Is GS Tech making any effort to add 3-phase charging to the Roadster PEM?

 

[DanielFriederich]

Daniel thank you for the updates. Is GS Tech making any effort to add 3-phase charging to the Roadster PEM?

I don't think so... I didn't talk to him about that and nothing of that cause was mentioned by them in the german forum...

Part of the design of the PEM would have to be reworked to get one phase out of three or you would need a 3 full-phase rectifying stage (I just looked up the electronic diagram and parts, seems quite easy but the voltage would be higher than just one phase...?!).. Wasnt it considered to be done already here in the forum?

I know one guy in south Germany or Switzerland who offered a 3 Phase AC to 1 Phase AC converter transformer (weighing about 75 kg...) but I never followed that story closely...