2014 P85d High Voltage Failure

[mr_hyde]

It charged for about 16 hours yesterday and appears to finally be topped up to match its neighbors. It takes a bit of finesse because the voltage it reads while charging is higher than it reads when you disconnect the charger (obviously) and then the voltage drops a bit while it stabilizes. This is all measured in millivolts so some meters won't even show the resolution. The photo above was 4.00V at .25A but I eventually charged at 4.1V and 1.5A That was getting me a little under a mV every 10 minutes. I had 86mV to make up on the last brick and 5mV on its neighbor so 91 total to match the other 4 bricks in the module.

The next information is in very broad terms without accounting for power conversion and thermal loss, etc. but in general:

If you think about this in terms of how much 'power' the brick was getting hit with, a 50A/240V charger in your garage puts out about 12,000 watts. There are 96 bricks which means each one is absorbing about 120W using the big L2 wall charger. That will usually fill the battery in 6 hours or so but the car is also actively managing the battery temperature. My 1.5A/4V was putting out 6 watts which is tiny in comparison. Compare that to the supercharger putting out 50 or 100kW and you see charging this small amount for a long duration wasn't too stressful. Slow and steady!

 

[brainhouston]

i was charging MS with 100kwh pack the other day from 20-50% without coolant in the system at 240v 6a (~15w/brick)
no errors or complaints from the car...

 

[mr_hyde]

i was charging MS with 100kwh pack the other day from 20-50% without coolant in the system at 240v 6a (~15w/brick)
no errors or complaints from the car...

Good to know because the next trick is to plug the pack in with my extension cords and see what it looks like. I think I will still have the low iso code latched from the coolant debacle so more toolbox probably.

 

[brainhouston]

my iso f123 code wen't away w/o resets

 

[mr_hyde]

That would be nice. I had ALL the warning codes but only the one failure... w158, w142, w123, w035 AND f123

 

[brainhouston]

yeap, same for me

 

[mr_hyde]

I powered things on briefly and f123 was still there. I presume it would take time to clear on its own - it wants several samples over a period of time with no low iso before it gives up the code? The good news is my extension cords worked and the total imbalance across 96 bricks is 10mV.

 

[brainhouston]

In my case it was gone on 1st power up... but i have newer battery so maybe its diff on older ones.
I did see some youtube videos claiming u have to reset f123 error...

 

[mr_hyde]

I'm not going to celebrate too much because last time I got excited, I dumped a bunch of coolant into the pack and set myself back a couple of weeks. No coolant in the variables right now except for the bucket under the rapidmate catching the drips. Charging a couple of percent at 5amps to prove the concept then let sit overnight to further prove the concept (contactors opening and closing like they should, 12v support, etc) and then hopefully, the lid goes on tomorrow!!

 

[brainhouston]

Nice work! u got it this time

 

[mr_hyde]

I let the car take a software update so it wouldn't try tonight when I couldn't watch it. During the update, the charger stopped and an alert popped up on my phone screen. I took a gulp and waited. Communication came back after the reboot and the contactors closed again. Hopefully it behaves as expected tonight! I want to see the 12v supported the whole time and the voltage value of the bricks I was working on to shift similarly to their neighbors on SMT. I didn't mention it in the earlier post, but the manual charging got the entire module within 1mV. Hopefully the BMB can take it from there!

 

[mr_hyde]

Some updates - I cycled coolant around for an hour with a little stick pump from Amazon and managed to get no more bubbles exiting the pack. I'm certain there ARE bubbles in there because the pump is at such a low pressure and volume but it did take maybe 3/4 of a gallon of coolant so much better than dry. I had a shop towel diaper under each fitting and saw zero drips. This is the step I wished I had done last time (as well as the pressure test)...

I also let the battery run down a little running the cabin heater for a while and checked the imbalances which were still 14mV across the whole pack after dropping about 7% SOC. If I was infinitely patient and had an easy way to loop the coolant in so the battery could really work, I could spend a couple of weeks running the pack up and down heating the garage but without the coolant to condition the battery, that just isn't practical so... I decided to seal it.

The first thing to do before you seal the lid is spend an hour going through all the bags of bolts and screws you took out when you disassembled things. In my case, I knew this would be at least a month later because of the vacation schedule so I took time to label everything pretty carefully. There are different heads, shoulders, thread pitches and lengths. For the first time, I counted each fastener and each place they went and miraculously found none missing and no extras! I was stunned.

I had already spent several hours prepping both surfaces so they were free of the previous adhesive. You never want to glue on glue if you can help it. In preparation, I got four M4x .8 threaded bolts to use as guide pins for the perimeter. If anyone is trying this at home, please bring one of the screws with you to the hardware store to confirm the thread pitch. I cut off the heads so they were essentially threaded pins and put one on each corner. If you get the correct holes over these pins, the pack is basically positioned correctly. I opened the tube of goo discussed earlier and started making beads. When that was complete, I had my helper (supportive, but unenthusiastic wife) help me lift the lid onto the guide pins. That went on without drama and I started threading on fasteners.

It took a while and the 12 'through' passageway fasteners caused a momentary bit of panic when I knocked one out the bottom of the pack. I was envisioning omitting that fastener and filling the hole with a whole tube of caulk when I realized they just key and capture each other and seal with o-rings. As long as you get Teflon tape on the threads, it will seal up fine. Crisis averted.

I let it dry overnight and did a basic pressure test today. There were a few small bubbles with my soapy water but really very few considering how many fasteners there are, how many imperfections there are on the pack after removing it and the fact that I have no idea what I'm doing. Only two of the bubbles were from areas that wouldn't be under the plastic tarp which is basically a dry area unless you are one of the idiots that thinks it is 'fine' to drive a Tesla through door-high standing water. If you cruise FB long enough, someone will ask about flood damage and someone will come out of the woodwork and say the batteries are fully sealed and work fine under water... but I digress...

I pulled the offending fasteners, cleaned them again and their mating surface and re-glued. Now I am basically waiting for glue to dry. Depending on how quick the glue cures in my 50f garage, I'll pump it up again later tonight and hopefully it will hold pressure. If not, hopefully the bubbles are fewer. The good news about spraying soapy water all over your pack is how clean it is when you dry it off with paper towels!

 

[mr_hyde]

Note to self - make a generous bead around the perimeter. Too much is better than too little.

 

[Gtech]

Note to self - make a generous bead around the perimeter. Too much is better than too little.

Do I understand correctly that you pressure tested the pack itself for leaks? Did you made some special adapter for that? How about the umbrella valves?

 

[mr_hyde]

Okay, no more bubbles. Pressure testing to .19psi is a bit of a challenge. I can register about 1psi on the digital gauge with my lungs so I decided to go that route because I don't trust the regulator that low. It takes a few puffs to pressurize the pack like that, but once I can't push anymore, I can close the ball valve, count to 10 and get a nice puff back out of the tester. YMMV, but I'm pretty confident I won't get water in the pack from anything related to the sealing of the top or hump.

The thing I was dreading the most turned out to be the easiest part of the entire project. The Butyl tape used on the plastic cover has been burned into my consciousness since I did my first window regulator about a bazillion years ago. I think I still have some on the back of my hand from 1987. To get the cover off, I carfully sliced it with a razor blade so there was some left on the plastic and some left on the cover. I covered each side with the shiny side of some packaging tape and expected that to come off. Hedging my bets, I got a 30' roll of it from Amazon and I'm glad I did!

When I went to take the tape off to expose the sticky surface of the original butyl, the whole mess came right off of the lid. No problem, I still have some on the plastic sheet and a roll if I need to patch some in. The next step was to peel the tape on the plastic sheet. Guess what? Yep - it came off too! The solution was to lay the perimeter with the new tape leaving the paper sheeting in place. It is easy to turn a corner - just make small tears in the wax paper. Once that was done, I laid the plastic sheet in place and stretched it from end to end. It was a little tight so I started at the contactor and lined up the two through bolts at the back and peeled back the wax paper. I worked my way up one side and matched the holes in the middle and switched to the other side continuing to the top. Once it was in position, I did a lap and pressed it firmly into place.

If you are doing this in the summer, it probably won't be that happy about peeling - especially off the plastic. I recommend putting the whole sheet in the freezer for an hour which should make it brittle enough to pull off. I thought this would be a complete cluster and it looks like it did when I dropped the pack over 2 months ago!

The pack goes in tomorrow! I gave it one more extension cord test now that the cover is on and the car is still happy with it. The imbalance is down to 10mV so I'm excited to see how it does fully installed.

 

[brainhouston]

It took a while and the 12 'through' passageway fasteners caused a momentary bit of panic when I knocked one out the bottom of the pack. I was envisioning omitting that fastener and filling the hole with a whole tube of caulk when I realized they just key and capture each other and seal with o-rings. As long as you get Teflon tape on the threads, it will seal up fine. Crisis averted.

This is why on newer packs (like mine) top ones are left threaded, this way u won't unscrew bottom one.
I did the same, teflon tape on threads.

... unless you are one of the idiots that thinks it is 'fine' to drive a Tesla through door-high standing water. If you cruise FB long enough, someone will ask about flood damage and someone will come out of the woodwork and say the batteries are fully sealed and work fine under water... but I digress...

surprisingly i saw few 'flooded' cars on auction that were Run&Drive so some packs do survive standing water.
also, bottom breather is actually rated for some water depth with time limit... can't recall exact numbers...

 

[mr_hyde]

This is why on newer packs (like mine) top ones are left threaded, this way u won't unscrew bottom one.
I did the same, teflon tape on threads.


surprisingly i saw few 'flooded' cars on auction that were Run&Drive so some packs do survive standing water.
also, bottom breather is actually rated for some water depth with time limit... can't recall exact numbers...

My bottom sections weren't threaded into the pack - they were pressed up in with two tabs to keep them from spinning when the top was inserted. The bottom section seals with o-rings which keeps any water in the center of the barrel making it critical to have the teflon tape seal to keep that water from entering the pack. The standing water and 'Telsa will run underwater' crowd are certainly enthusiastic. Just because something can do something doesn't mean it should or that you shouldn't avoid going through deep water.

 

[mr_hyde]

This is what success looks like. The photo was taken tonight outside the restaurant where we celebrated a working Tesla. I appreciate all of the help from people like Russ, Paul, Chad, Jerry and Ray, but especially Howard who has been my thought partner on all things Tesla since we met over the summer. He lent a hand on a couple of occasions including aligning the pack for final install today. For the record, he doesn't advocate ANYONE try this at home!

Even after filling the battery with coolant, there was a lot of air in the system. Service mode does have a coolant purge sequence once you open the gateway (right turn and brake pedal) so I didn't need toolbox again - yay! The whole system took nearly a gallon after being close to what I thought would be 'mostly full'. The low coolant also threw an overheat error from the onboard charger which appears to be cooled by the system. When I had been testing things with the pack out of the car, I had just been charging at 5amps because I didn't want to put more juice through the rapidmate extensions than was necessary to do the basic testing. Today, I cranked it up to 40amps with the pack installed and it got cranky. After the coolant was burped, it took the full 40 without issue so I'm glad I didn't spend any time troubleshooting that.

What's next? I'm going to drive it and monitor things more closely including module #12 which got wet and sticky. If I have an issue in the future, I will expect it to be there but it if isn't, it will help me feel better that I did get everything cleaned up and sealed. I'll track down this thread from time to time and give updates - hopefully only with good news.

For those reading this with your own HV failure trying to decide what to do next, you certainly see the story arc. Despair and confusion (raw ignorance) at first followed by some tinkering and then some successes along with some setbacks. A few big takeaways:

1. High Voltage is deadly. I took some risks that I won't ever do again such as remove module #8 and #9 with the rest of the pack assembled. Please don't ever do that. If you need to fix something on just one of those modules, assume there are other issues in the main part of the pack you need to address too. Take days, not hours or minutes, wrapping your head around how the HV bus works and how to approach it in the least dangerous way possible. Remember that the least dangerous is to have the car towed to the Service Center.
2. Low isolation is a different animal than basic high voltage that is contained inside the pack circuit. If your battery is showing low iso, you need to take the advice in the first bullet point and crank it up 10x.
3. This isn't a weekend project. I took nearly 90 days with the 4+ week vacation in the middle. If I hadn't had the cut coolant line setback and you discount a couple of weeks for thinking and planning, it still would have taken 4+ weeks.
4. My pack was an easy one with corroded BMB boards. 029 errors with failing cells present whole other challenges and you might not want to open the pack if you don't have a path to a 14 module conversion or want to take the risk of cutting out cells or swapping modules. In short, don't drop the pack, open it, and then decide what you need to do. Plan, research and plan some more.

 

[mr_hyde]

Do I understand correctly that you pressure tested the pack itself for leaks? Did you made some special adapter for that? How about the umbrella valves?

I did make a tester for that - mostly from air compressor bits. It is nearly impossible to find a pressure regulator that will reduce a 100psi air compressor hose to .19psi and there are breathers in the pack that went above that not to mention the umbrella valves. I didn't feel like this was the most accurate test so I mostly focused on stopping any bubbles from the soapy water.

 

[Mack Dack]

Do I understand correctly that an imbalance of more than 30 millivolts will not allow the machine to start anymore?