Faults: HVAC cable fault; ESS Cable Interlock fault; Vess too low during precharge, t2

[wiztecy]

So my Roadster ended up cutting out this week. Roadster has been sitting for months at 60% SOC. Made it into work fine which was 35 miles. About 8 miles into the trip back the Roadster's power shut down and I ended up with the following faults below. I did a key reset after waiting 5 mins all to have it cut out again on me. Had the Roadster towed home and issue still exists. Removed and reinserted all the cables to the Silver High Voltage controller that's under the front hood (Silver box with cooling fins). Also removed the ESS cables and reinserted them. After I did that the Roadster now complains about the left headlight and tail light being out, which they are not and working fine.

I did wash the Roadster the day before I drove it. Water did end up on top of the PEM and battery top. I ended up grabbing a towel and wiping that down. Charged afterwards fine as well as charged in the morning fully in standard mode.

Curious any ideas or clues to debug this one?

I included some snap shots of my voltage, SOC, etc as well. The odd thing is that my Vess value under the BSM screen reads "0V". Curious what others get for that value, in particular 1.5 owners. This is with the key in the off position.

 

[Gary1M]

Confirming my 1.5 Vess reads 0V as well with the key in the off position.

 

[ion_1]

saw your note to @DeedWest before you posted this, had a bad feeling something was up. Sorry to hear about this strange one with the similarities with @DeedWest. Although I may have missed it, I couldn't find a similar posting with a similar error. Also deed's is a 2.0 vs. your 1.5 so it has to be where similarities lie in the PEM and associated electronics vs the differences. Curious about @DeedWest's conclusion to the issue also. Keep us posted. Hope it works out to be something simple.

 

[Roadster]

For reference, I believe this may have been the previous post @ion_1's referring to? Regardless, the fix was a new 400V controller as mentioned here and here.

 

[thefortunes]

Strange that it did not happen immediately when you started and just a guess, but I would open up the front and put a fan on it for a day and/or let the sun dry everything out (based on the fact that you washed it the day before and I know the 400v controller as well as the "HVAC box" don't like to get wet).

When I first got mine I would get random errors after driving in the rain/fine snow. The SC would have me run the heater for an hour or so and that would dry things out enough that the errors would go away.

I eventually had the HVAC box replaced under warranty and that eliminated 99% of the errors.

Good luck!

 

[wiztecy]

Thanks everyone for the input!

I honestly do keep water away clear of that silver 400V controller in the front and don't recall getting that box wet this last time around. I would think if I had water intrusion in that controller on in the PEM that the system would not have allowed me charged that night and the following morning, not to mention taking me in 36 miles to work and 12 miles into the way back home. But now thinking about it I did hit a bump... it was a rise in the concrete highway that I usually do avoid which was 2 miles into the trip home. I'm guessing from the Roadster not being used as often and hitting that bump possibly caused a spike. But I would have thought the Roadster would have failed / faulted right after hitting the bump.

I thought I had it fixed today. Tried charging this morning, got the HVAC cable fault error and ESS precharge error but no ESS interlock error, next came the red ring of death. I then unplugged every connection to the 400V controller as well as the orange 400V controller lines that run to the PEM and reinserted them. Tried charging again and I got the amber charge ring and the Roadster began charging. Well that only lasted for 1 minute and again same errors and red ring of death again. Also received the following error in debug mode: DMC HW: Battery OverVoltage Fault". Will post the full message pic when I get a moment later. So back to the story, Iid the cable removal/reinsert trick and no luck. Also pulled the VMS plugs to do a clean reboot, nothing. I also still am getting the VDS complaining that the front left head-light and rear left tail-light are out, which they are not. That happened after unplugging the front left connector on the 400V controller the first time I tried removing/inserting the connections. Hasn't gone away, don't know if that's related to something going south on the 400v controller.

So here's something I'm wondering if a person has an answer to.... I tried removing the CF piece that's right behind the rear window, removed the 4 allen scews but can't seem to budge it loose. I don't want to force it. Anyone have a clue? There's one more HVAC connection that goes into the ESS I want to remove and re-insert however I need to remove that top CF cover in order to access it. Note I'm shutting down the APS/ESS when doing this. Never mess with this stuff live.

I disabled the APS so the battery won't drop lower, can't charge issue. I may also end up pulling the ESS fuse to eliminate any vampire / VMS drain while I'm trying to understand this.

Would love to remove the 400v controller to have it tested, and I presume only Tesla can test this. But pretty sure Tesla wants the whole care to test.

 

[ML Auto]

That cover under the rear window is only secured by those 4 screws, but gets stuck in place after years of age. But be careful prying it loose, as it is not carbon fiber, but just SMC fiberglass that cracks very easily.

 

[DeedWest]

My apologies for just now seeing this. But, yes.

The morning that my car failed, it randomly rained for about 15 minutes, and the car was outside. My SC seemed to think that by some freak accident, something got wet and it failed immediately. That was their only logical explanation apart from the part randomly failing on its own.

The 400V controller was one day shipped from Canada and was $1,566 total to replace.

From what I hear, that Roadster has driven many, many happy miles since then, error-free.

 

[wiztecy]

Looks like I'm going to pull out the 400v controller and have my hardware engineer begin doing some tests with it and see if we can identify an issue, possibly fix it if possible. Interesting tidbit and coinicidence about the wet day connection.

 

[ion_1]

Seems like water/moisture may be the common cause of failure for the 400V controller in the threads. Would be great to know where the ingress issue on the unit is located such that we can finally address it (or at least be sensitive to it). I assume Tesla retains the old unit when the replace the 400V controller so no one has their faulted unit available for analysis. Maybe you will be the one to ID the location. That would be great.

 

[wiztecy]

I haven't had the chance to remove the 400V controller. However I was able to get that fiberglass rear body filler out above the battery pack. I took pics and will post when I can get the time. There are three clips that are pressure clips, one on each side of the window and one in the middle. They bite or pinch a flat surface of the rear filler. You need to remove the water reservoir to gain clearance since the piece pulls directly towards the rear of the car. Give small tugs but don't force, work a side then the middle and pull back. It will come undone from the spring clips.

So I ended up pulling the ESS disconnect. Its easy to get too with the rear hatch open, but takes a little twisting of your hands and a feel of the disconnect pull handle. I pulled that, removed the filler piece, and then backed off the last HVAC connection on the passenger side of the battery. The connections looked a little bit tarnished, and electronic grease might be something good there in the future. So I put that connector back in, put the ESS disconnect back in, and the car in a matter of minutes rebooted. I tried turning the car on but same faults. However this time I no longer had the odd warnings of the rear and front lights being out (when they were not). I believe pulling the ESS disconnect fully reset something that the reboot of the VMS couldn't do.

I'd say be familiar and try pulling the disconnect to get familiar with it. Its good to save the ESS or in case of an emergency where you need to completely shut power down to the car.

 

[wiztecy]

Took the 400V controller out. Unbolted the top and started to do some of my own tests. There are two PCBs. One housed on the top where the non HVAC connector is connected to, top PCB has 4 large capacitors. Top PCB of the controller is the one with a heat sink to the fins of the top 400V controller case. The bottom PCB is where the 3 HVAC cables route to and has 4 small independent heat sinks. Top PCB has water-proofing / conformal coating on it to protect from moisture intrusion, however I don't see any protection on the bottom PCB. Bottom PCB has a 25 AMP fuse whereas the top PCB has a 30 AMP fuse. I tested these fuses and all were good with continuity. There are four 47 Ohm resistors on the bottom PCB (47R W21 3W Vitreous Enamelled Wirewound Resistors 47 Ohm) 5x 47R W21 3W Vitreous Enamelled Wirewound Resistors 47 Ohm | eBay

What I notice on the bottom PCB is that every hole / contact point has a nice round gold or brass ring around it, I believe so there's excellent contact on the PCB when soldered down. I noticed on one of the contact points where one of these 47 Ohm resistors is located that there's no evidence of one of these clean gold/brass rings, appears like a weird blob of solder. Possibly something failed / heated this connection to cause this artifact. All four resistors properly still read 47 Ohms from the multimeter. I don't see anything odd visually with the two PCB boards.

I think that's enough debugging I can do at the moment, may bring the entire controller into my work where my friend who's an computer engineer / electrical engineer can do some more complete testing and also give some insight and tests with that odd connection point between the PCB and resistor.

Update: Just measured 47 Ohm from where a outside connector is and across the 47Ohm resistor that has that odd solder irregularity. Most likely that connection is fine.

 

[Rotarypower]

@wiztecy please share some pictures. Thx!

 

[wiztecy]

Will do when I get the time for the pics.

So the (V_ess too low during precharge, T2) is an error code indicating that there is too much capacitance on the High Voltage Bus or possibly a leakage path that is drawing down the battery during precharge.

While I had the time I popped the PEM cover off, verified the two main fuses coming from the battery, three DCM-10 600V/10 amp fuses, and four small 10 amp fuses for the AC and various lines. All checked out to be good. Also inspected for any water damage or fried electronic parts, no signs of anything bad. I buttoned everything up, put the 400v control box in and retried things and same failures.

Next step will be finding a way to isolate and test the HVAC controller to see if that's the root issue.

 

[wiztecy]

Here's some of the pics of the opened HVAC 400V controller (top and bottom PCB), the resistor that has the odd solder pattern as well as pics inside the PEM showing the fuses.

 

[wiztecy]

So after talking to my friend, that resistor / solder point from the 400V control box lower PCB (red circle location) it appears a little bit suspicious. He said it appears it got hot there and recommends re-flowing the solder at that point. I may try that this week and see what happens. Nothing to loose and its the only odd ball thing I can see.

 

[wiztecy]

In taking out that PCB board above I found the far right HV+ line or lug (gold) is loose! Possibly vibrations took its toll on it, backing it off and then when I hit that bump in the road jostled it enough to cause a spike down the road. Still pulling the board out but that's another oddity in this mix.

That HV+ Lline is the main HVAC cable coming from the passenger side of the ESS. Now the faults are making sense. Would be sweet if this is just a simple loose connection.

 

[wiztecy]

Buttoned everything back up after tightening down the loose lug, still get the same errors. Note that if you don't put on that external ground in the HVAC 400V Control box that the Roadster will immediately fault out, plain old yellow fault on instrument panel. Nothing shows up on the VDS. Put it on properly and it goes away, so a sign of a bad ground is that yellow fault.

I'm going to pull that control box out again and see if my friend can test different components down that HV+ line. With that lug being loose and hitting that bump in my journey I believe a spike occurred causing something to fail and burn up down the line.

 

[Rotarypower]

Great work @wiztecy. Your tenacity is sure to get to the root cause. I would bite the bullet and just replace that resister, it could be breaking down with voltage, something that won't show up on a simple multimeter check. It looks quite suspicious.

 

[gregd]

Great work @wiztecy. Your tenacity is sure to get to the root cause. I would bite the bullet and just replace that resister, it could be breaking down with voltage, something that won't show up on a simple multimeter check. It looks quite suspicious.

My thought too. Is it just the camera angle, or is that end of the resistor a bit discolored?