Left the 12v on with the support of a charger for the first time since this happened and the BMS recalculated an accurate range for the SOC and the pumps chugged down a gallon of coolant into the battery which is approximately what I pushed out at the beginning of the process. Still looking for solutions to the latched 107s. If anyone in Seattle has toolbox and wants to earn a case of beer or a steak dinner or simply my gratitude, let me know!
-
Want to remove ads? Register an account and login to see fewer ads, and become a Supporting Member to remove almost all ads.
2014 P85d High Voltage Failure
- Thread starter mr_hyde
- Start date
Quite a range of emotions tonight. I ran Toolbox to clear the 107 alerts and heard the contactors close for the first time since September. I plugged in the charger and it started charging. Joy! A few minutes later, I went back into service mode to run the purge air routine on the coolant. This ran for a while but the car stopped charging which I thought was related to service mode. During the air routine, the 12v got too low and I noticed a LOT of new errors - perhaps every isolation error available. Despair!
My initial thought is that something went wrong with a coolant hose or connection and dumped a bunch of coolant. The other thought is the rapidmate didn't connect well and dumped coolant. The last thought is that I damaged something disassembling. I'll charge the 12v overnight and see if I see any drips out the umbrella valve openings (values are not in yet). At least I didn't seal the pack so it will be much easier to 'touch' safely while I try to figure this out.
My initial thought is that something went wrong with a coolant hose or connection and dumped a bunch of coolant. The other thought is the rapidmate didn't connect well and dumped coolant. The last thought is that I damaged something disassembling. I'll charge the 12v overnight and see if I see any drips out the umbrella valve openings (values are not in yet). At least I didn't seal the pack so it will be much easier to 'touch' safely while I try to figure this out.
Gtech
Active Member
I didn't do a separate isolation test before installing. I put the cover and hump on with a few screws and no glue so I could temporarily install the battery and check the BMB boards.
Update - I dropped the pack and opened the lid. There is no sign of coolant down either side nor around the rapidmate (besides the few ounces that piddle out when you drop the pack). I don't see any coolant causing the low ISO. I think the ISO errors came up after cycling the coolant but what do they say about causation and correlation? I read somewhere that isolation errors stack up and it takes a few in a row over a (short) period of time to add up to a warning on the screen. It is starting to look like the ISO wasn't related to the coolant cycling.
When I put the pack back in to test, I didn't use any of the cover-shields. Obviously, I was going to do that before sealing the pack but my impression was they were more for electrical noise isolation than actual voltage.
So the good news is I don't have a bunch of that blue goo to clean up. The bad news is the the cause of the ISO error isn't as clear. Any constructive thoughts would be appreciated.
When I put the pack back in to test, I didn't use any of the cover-shields. Obviously, I was going to do that before sealing the pack but my impression was they were more for electrical noise isolation than actual voltage.
So the good news is I don't have a bunch of that blue goo to clean up. The bad news is the the cause of the ISO error isn't as clear. Any constructive thoughts would be appreciated.
Last edited:
Gtech
Active Member
Think you have to measure what your iso is with open contactors on battery side.
Seems 3000 Kohm is a healthy value w open contactors but I think you already have this document.
If iso is less then 3000Kohm then you need to start unscrewing things untill you find the culprit
Seems 3000 Kohm is a healthy value w open contactors but I think you already have this document.
If iso is less then 3000Kohm then you need to start unscrewing things untill you find the culprit
Update as of 5pm local time... The low ISO is no longer a mystery at least. I tilted the pack to make sure there was no coolant... and there was coolant. Lots of coolant. I started to remove the modules in 'safety' order rather than the ones I thought might be leaking. #15 was certainly leaking but none inside the tray which was good. 4 other modules had coolant under the shield but none in the plastic module tray. Then I lifted up module 12 - yikes!
Plastic tray had at least a cup of coolant. I separated the trays and got it onto the bench. It has been slowly leaking coolant for a while. I must have missed it when inspecting. In my defense, there was no sign of coolant in the tray the first time I removed it and when I was inspecting, I was paying more attention to the BMB and sensor wire connections which are all fine still. Moving it around must have turned a pinhole leak into something catastrophic.
I suspect there is no fix for a coolant leak inside a module but I'm all ears. This is the first aspect of this pack that seems terminal. A 14 module conversion is about the only thing that will make this workable unless anyone has any other ideas.
Plastic tray had at least a cup of coolant. I separated the trays and got it onto the bench. It has been slowly leaking coolant for a while. I must have missed it when inspecting. In my defense, there was no sign of coolant in the tray the first time I removed it and when I was inspecting, I was paying more attention to the BMB and sensor wire connections which are all fine still. Moving it around must have turned a pinhole leak into something catastrophic.
I suspect there is no fix for a coolant leak inside a module but I'm all ears. This is the first aspect of this pack that seems terminal. A 14 module conversion is about the only thing that will make this workable unless anyone has any other ideas.
I'm getting as tired of jumping to conclusions as you probably are of reading about my misdiagnosis so, I'm going to try one more time to get this right. If you are reading (just today), my low ISO WAS caused by coolant leaking in the pack and the leak in the pack was NOT a catastrophic rupture in module 12. I put a mityvac on the coolant loop of #12 and it held 15 in. Hg for the duration of taking the photo and interrupting Howard's dinner to commiserate all of the rollercoaster dips today. There is no leak in #12 - at least not inside the module under vacuum.
My new theory (until I prove it false in a couple of hours) is that I have a crack in the coolant line going to #12 and that the coolant sprayed into module 12, collecting in the tray. Plenty also sprayed under the tray/shield too. The 2nd photo shows the bays that had coolant in them. The tilt-test I did to see the coolant lifted the driver's side (right side in the picture) so #12 leaked and probably ran downhill (lots of coolant and by the slope of the garage floor) to #13 but #15 and #2 could both be suspect or #2 could have leaked and ran down hill into #15. The connectors themselves have two o-rings and all of the suspects look to be in pretty good shape suggesting the likelihood of the plastic hose itself being cracked.
They aren't the easiest thing to wiggle off the module's nipple but I also haven't read that everyone who tries this cracks the coolant lines. This could be rookie technique for sure, but at this point, what do I do about it?
I'm assuming (this makes an ass out of me and me all day) that the rigid plastic tube is okay and it is the flexible one that has the hole. How easy are these to replace? What is the best way to pressure test them? Fun day but at least it is resting on a not-horrific note because the modules should be usable.
My new theory (until I prove it false in a couple of hours) is that I have a crack in the coolant line going to #12 and that the coolant sprayed into module 12, collecting in the tray. Plenty also sprayed under the tray/shield too. The 2nd photo shows the bays that had coolant in them. The tilt-test I did to see the coolant lifted the driver's side (right side in the picture) so #12 leaked and probably ran downhill (lots of coolant and by the slope of the garage floor) to #13 but #15 and #2 could both be suspect or #2 could have leaked and ran down hill into #15. The connectors themselves have two o-rings and all of the suspects look to be in pretty good shape suggesting the likelihood of the plastic hose itself being cracked.
They aren't the easiest thing to wiggle off the module's nipple but I also haven't read that everyone who tries this cracks the coolant lines. This could be rookie technique for sure, but at this point, what do I do about it?
I'm assuming (this makes an ass out of me and me all day) that the rigid plastic tube is okay and it is the flexible one that has the hole. How easy are these to replace? What is the best way to pressure test them? Fun day but at least it is resting on a not-horrific note because the modules should be usable.
A few updates today. The coolant leak came from the corrugated plastic (TGFS) hose for #12. It appears it was damaged when I was opening my pack using a flatbar and mallet to tap through the sealant. The flatbar made contact and cut the tube and it wasn't discovered until the coolant purge cycle ran when I was clearing the errors with toolbox. I planned to taking zero shortcuts sealing the pack just like I took zero shortcuts on the BMB boards, but treating the coolant loop as a given is definitely a shortcut I regret!
I built a pressure testing rig to test the pack (low PSI) when I sealed it so I converted it to test the coolant loop pressure. I bought a 25' coil of 5/16" aluminum fuel line on Amazon which is precisely the size of the module nipples the connectors go on. I cut those into small loops to replicate the modules that aren't in the pack yet. They pushed gently passed the two o-rings and stay nicely. One got a cap and the final one that supplies the air pressure to the system got an assortment of parts from Home Depot. Everything on this rig except the digital gauge is from there.
The pipe that got cut was obviously blowing air. On a whim, I put a little smear of permatex (Right Stuff - the name is accurate) over the cut and an hour later, it was holding 5psi. Everything else in the system is holding too. I'm not sure how #2 got wet enough to drain down to #15 but for now, I'll chalk that up to assembly error. Both modules will be tested a few more times with air and then circulate fluid before I will trust the coolant loop up in the car.
As for the original cut, it looks like my coolant loops ride higher in the pack than a few other photos I have seen. If they were 1 or 2 corrugations shorter or the barb clocked down slightly instead of slightly up, my tool wouldn't have come in contact with the pipe and this project would likely have been wrapped up by now. For people reading in the future, the lip of the pack is about .8" wide so if your tool is going in farther than that and your coolant tubes are 'up' like mine, you could damage yours too. Use caution here.
I'm NOT going to leave the cut tube in the pack with a little RTV sealing it so that is the next challenge. I have a couple of these on order from ebay because obviously, Tesla would never make a part like this available. When I get them, I'll need to figure out how they mate with the barbs and if they are even serviceable. If anyone has experience with these, please share any insights!
I built a pressure testing rig to test the pack (low PSI) when I sealed it so I converted it to test the coolant loop pressure. I bought a 25' coil of 5/16" aluminum fuel line on Amazon which is precisely the size of the module nipples the connectors go on. I cut those into small loops to replicate the modules that aren't in the pack yet. They pushed gently passed the two o-rings and stay nicely. One got a cap and the final one that supplies the air pressure to the system got an assortment of parts from Home Depot. Everything on this rig except the digital gauge is from there.
The pipe that got cut was obviously blowing air. On a whim, I put a little smear of permatex (Right Stuff - the name is accurate) over the cut and an hour later, it was holding 5psi. Everything else in the system is holding too. I'm not sure how #2 got wet enough to drain down to #15 but for now, I'll chalk that up to assembly error. Both modules will be tested a few more times with air and then circulate fluid before I will trust the coolant loop up in the car.
As for the original cut, it looks like my coolant loops ride higher in the pack than a few other photos I have seen. If they were 1 or 2 corrugations shorter or the barb clocked down slightly instead of slightly up, my tool wouldn't have come in contact with the pipe and this project would likely have been wrapped up by now. For people reading in the future, the lip of the pack is about .8" wide so if your tool is going in farther than that and your coolant tubes are 'up' like mine, you could damage yours too. Use caution here.
I'm NOT going to leave the cut tube in the pack with a little RTV sealing it so that is the next challenge. I have a couple of these on order from ebay because obviously, Tesla would never make a part like this available. When I get them, I'll need to figure out how they mate with the barbs and if they are even serviceable. If anyone has experience with these, please share any insights!
The punctured coolant hose that clips into module are marked TGFS >PA9T/PA12 TFS9< The fitting head with the yellow clip marked PA66. Other markings are manufacturing time stamp down to the minute!
PA9T, PA12, PA66 are all various types of plastic polymer
TGFS might be manufacturer ( ToyodaGosei Fluid Systems ) A tier 1-2 supplier so no catalog for us consumers. Google TGFS hoses/pipes shows a lot of different car manufacturers fuel vapor, brake booster hoses with similar TGFS markings (TFGS in capital letters and no logo)
www.toyodagosei.com
Google PA9T adhesive shows low temperature curing epoxy. Not sure this is what to use. Certainly need to handle vibration + coolant seal. Salvaged replacement pipe from ebay also had glued surface used once already so not sure if can use again with good bond. Maybe silicone hose + even pressure clamp is more predicable without bonding agent experience,
PA9T, PA12, PA66 are all various types of plastic polymer
TGFS might be manufacturer ( ToyodaGosei Fluid Systems ) A tier 1-2 supplier so no catalog for us consumers. Google TGFS hoses/pipes shows a lot of different car manufacturers fuel vapor, brake booster hoses with similar TGFS markings (TFGS in capital letters and no logo)
Functional Components | Toyoda Gosei Americas
Toyoda Gosei Americas rubber and plastic functional components that support basic vehicle functions.
www.toyodagosei.com
Google PA9T adhesive shows low temperature curing epoxy. Not sure this is what to use. Certainly need to handle vibration + coolant seal. Salvaged replacement pipe from ebay also had glued surface used once already so not sure if can use again with good bond. Maybe silicone hose + even pressure clamp is more predicable without bonding agent experience,
Last edited:
I'm also shocked at how effectively the 'quick smear' of RTV worked. I didn't even prep the surface. What if I had degreased, skuffed it, applied a full application that dried and when wrapped it with something to hold concentric pressure? Ultimately, I need to find the solution that is the most likely to hold for years, not months and I'm starting to think adding two new unions (silicone hose over barbs) is adding two variables. Curious if anyone has any experience with these?
brainhouston
Active Member
I don't see why RTV+wrap wouldn't hold it for years... maybe even shrink wrap it or zip tie over so nothing comes apart...
Tesla coolant system never gets any high pressure, n i believe the cap or tank has 5psi on it somewhere...
Tesla coolant system never gets any high pressure, n i believe the cap or tank has 5psi on it somewhere...
I want to believe that! I'm going to look at the 'do it right' options also and ultimately try to decide which will have the greatest chance of long term success. There is only one opportunity for the 'factory' seal and everything that comes later is just trying to replicate that. RTV has one failure point but replacing the hose and fitting will provide two...
Still working on the 'best' solution for the coolant line so I spent the evening working on something that will work - a rapidmate extension. This will allow me to 'install' the battery without installing it. I want to make sure all the codes and issues are cleared before I break open the caulking tubes...
I also have the parts for the HV section. That one will be easier in terms of quantity of connections but a bit harder because there will be some things that need some serious insulation. I'm not sure how much it will be needed but it will be nice to have.
I also have the parts for the HV section. That one will be easier in terms of quantity of connections but a bit harder because there will be some things that need some serious insulation. I'm not sure how much it will be needed but it will be nice to have.
The HV section will utilize the OEM HV lines that usually run to the onboard charger. The wire in the center on the female side had me puzzled for a minute because it goes directly to the BMS board with no way to integrate in into my extension harness. I finally figured out it is just contact pin that closes two contacts when the male rapidmate inserts to the battery side. The 3 bolts are B+, B- and the chassis ground.
And lastly - the coolant hose. I got the parts in today. The first image shows an ebay test piece pulled apart with a heat gun. There is a barb and an o-ring providing the seal. I dissected the other end and confirmed the end at the manifold has the same attachment. It took a lot of heat and a lot of force to pull it apart and the hose can't be reused. The 2nd image is what makes me nervous. Pulling the damaged hose off the barb could damage the pipe and/or the barb at the 45* angle. I'm pretty sure I could get it off CAREFULLY, but once I do, I would need to build some sort of tool to grab that collar and push the new hose back on.
The third image shows a successful installation of the connection piece from the first image into a new section of hose (Amazon). The hoses looks like the same part as the OEM so I'd be comfortable using it if I thought I could get it on the manifold barb. Damaging the barb or cracking the pipe would set everything back significantly so I'm still trying to decide the best path forward...
The third image shows a successful installation of the connection piece from the first image into a new section of hose (Amazon). The hoses looks like the same part as the OEM so I'd be comfortable using it if I thought I could get it on the manifold barb. Damaging the barb or cracking the pipe would set everything back significantly so I'm still trying to decide the best path forward...
Mid-week updates... I couldn't find a good way to press the new coolant tube onto the barb without making a special fork for the Doorman tool and I wasn't sure I'd be successful so I got a length of 5/16" silicone hose an slipped it over the barb with the o-ring. I've got a zip tie on the barb to hold the physical hose in place and the o-ring will take care of the sealing so the coolant part of this journey should be done. It is in the garage now holding between 13.58 and 13.60 psi depending on if I stand on one foot or two. I'll check again after this post to make sure, but I feel pretty good about this resolution.
When the packs first came out before my vacation, I checked the individual module voltages and wrote them down. They were all between 23.53v and 23.55v except for #8 at 23.50v that had the BMB failure. As noted in post #33, those bricks were all matched exactly at the resolution that meter was capable of, so I was pretty comfortable that the failing BMB played some shenanigans on the module voltage although I'm not sure how that would work... but I digress...
I went on vacation for almost 5 weeks and tried to not think about the Tesla and the 16 modules sitting on a piece of cardboard on the garage floor. When I got back from vacation, the season had changed and the garage was 20 degrees cooler. The first thing I did was check the module voltages and compared them to what they were when I left. Each one dropped .02v right down the line. I don't know if that is good or bad, but so much of what I'm playing here is 'one of these things is not like the other' and I took it as a great sign that they were all behaving the same. If there was a bad cell in the bunch using energy to generate heat, I would expect one of the modules to drop significantly more than the rest and that didn't happen.
Fast forward to a couple weeks ago when module #12 got blasted with coolant - I took a bunch of modules back out including #12, cleaned up the bottoms of the trays on the ones that got slightly wet and started drying #12 and getting it cleaned up. I've been working on the coolant leak in the meantime but today was going to be the day that was resolved so I got #12 ready to get back into the pack. I decided to check the voltage again just to be sure and unfortunately found it down .06v from the 23.52v it was 3 weeks earlier. Since I still had the pack out, I decided to measure the individual bricks and hoped to find them uniform like SMT had them during the brief time the pack was in. My good meter goes down to the mV so I was able to get more accurate numbers than when I checked #8. The first four are 3.923v, the fifth is 3.918v and the last is 3.837v. If you look at your SMT numbers, that is a huge imbalance on that last one - 86mV.
I also found a splash of coolant on the BMB. I removed it and hosed it down with contact cleaner but ultimately, I will install my spare BMB when I am done - presuming this pack survives and goes back in the car. I'm going to check the individual voltages in the morning and make sure I'm not losing more and then will try to charge up the derelict brick and see if it holds. The fun never ends!
When the packs first came out before my vacation, I checked the individual module voltages and wrote them down. They were all between 23.53v and 23.55v except for #8 at 23.50v that had the BMB failure. As noted in post #33, those bricks were all matched exactly at the resolution that meter was capable of, so I was pretty comfortable that the failing BMB played some shenanigans on the module voltage although I'm not sure how that would work... but I digress...
I went on vacation for almost 5 weeks and tried to not think about the Tesla and the 16 modules sitting on a piece of cardboard on the garage floor. When I got back from vacation, the season had changed and the garage was 20 degrees cooler. The first thing I did was check the module voltages and compared them to what they were when I left. Each one dropped .02v right down the line. I don't know if that is good or bad, but so much of what I'm playing here is 'one of these things is not like the other' and I took it as a great sign that they were all behaving the same. If there was a bad cell in the bunch using energy to generate heat, I would expect one of the modules to drop significantly more than the rest and that didn't happen.
Fast forward to a couple weeks ago when module #12 got blasted with coolant - I took a bunch of modules back out including #12, cleaned up the bottoms of the trays on the ones that got slightly wet and started drying #12 and getting it cleaned up. I've been working on the coolant leak in the meantime but today was going to be the day that was resolved so I got #12 ready to get back into the pack. I decided to check the voltage again just to be sure and unfortunately found it down .06v from the 23.52v it was 3 weeks earlier. Since I still had the pack out, I decided to measure the individual bricks and hoped to find them uniform like SMT had them during the brief time the pack was in. My good meter goes down to the mV so I was able to get more accurate numbers than when I checked #8. The first four are 3.923v, the fifth is 3.918v and the last is 3.837v. If you look at your SMT numbers, that is a huge imbalance on that last one - 86mV.
I also found a splash of coolant on the BMB. I removed it and hosed it down with contact cleaner but ultimately, I will install my spare BMB when I am done - presuming this pack survives and goes back in the car. I'm going to check the individual voltages in the morning and make sure I'm not losing more and then will try to charge up the derelict brick and see if it holds. The fun never ends!
Since information about battery packs is so impossible to find, here is a quick tutorial on how to check the individual bricks in a module that is out of the car. First, a bit on the terminology for anyone catching up...
This is true for Model S/X first generation batteries - Model 3 and Y use a different configuration and newer S/X packs use cells this size but package more into the same space. The correct terminology is Pack, Module, Brick and Cell. The Pack is the giant 1,000 pound HV 'battery' under the car. Inside the Pack are 14 or 16 Modules. Each Module is made up of 444 (in the packs of this generation) Cells which are individual 18650 'batteries' like you might see if you opened up a laptop or cordless drill battery pack. The Cells in the Module are configured in a set of 74 Cells in parallel which is called a Brick. Six of these Bricks are connected in series to make the module. A small bit of confusion comes from the scanmytesla app which most of us use to read these voltages. It calls a Brick a 'cell' which isn't accurate as described above.
Half of the cells in module are + side up and the other half are - side up. The first 74 cells are up and sitting on the same plate as the next 74 cells that are down. Three plates on the bottom connect two pairs 74p each and four plates are on the top to allow the 6 sets of 74p in series. All of this is much easier to visualize if you imaged taking six D Cells in a 3 up/3 down configuration with 7 plates to put them in series.
There are 7 sensor wires - one for each plate - connected to the BMB. The BMB uses six combinations of the seven wires to read the brick voltages. On the BMB, they are numbered 0-6. Each wire is colored but I'm not certain is Tesla was consistent. To read the individual module voltages, probe your meter on 0&1, 1&2, 2&3, etc. The lower number is always the negative side of the brick. Probing non adjacent numbers gets you voltages for groups of Bricks in series and probing from zero and 6 gets you the full Module voltage. The pins between the numbers aren't connected to anything which makes probing for the voltages easier.
Note, the image below is from one of the old BMBs I'm not using. I scraped the coating off the pins to make them easier to probe. If they are going in the pack, you want a nice coating of conformal coating to keep them from corroding.
This is true for Model S/X first generation batteries - Model 3 and Y use a different configuration and newer S/X packs use cells this size but package more into the same space. The correct terminology is Pack, Module, Brick and Cell. The Pack is the giant 1,000 pound HV 'battery' under the car. Inside the Pack are 14 or 16 Modules. Each Module is made up of 444 (in the packs of this generation) Cells which are individual 18650 'batteries' like you might see if you opened up a laptop or cordless drill battery pack. The Cells in the Module are configured in a set of 74 Cells in parallel which is called a Brick. Six of these Bricks are connected in series to make the module. A small bit of confusion comes from the scanmytesla app which most of us use to read these voltages. It calls a Brick a 'cell' which isn't accurate as described above.
Half of the cells in module are + side up and the other half are - side up. The first 74 cells are up and sitting on the same plate as the next 74 cells that are down. Three plates on the bottom connect two pairs 74p each and four plates are on the top to allow the 6 sets of 74p in series. All of this is much easier to visualize if you imaged taking six D Cells in a 3 up/3 down configuration with 7 plates to put them in series.
There are 7 sensor wires - one for each plate - connected to the BMB. The BMB uses six combinations of the seven wires to read the brick voltages. On the BMB, they are numbered 0-6. Each wire is colored but I'm not certain is Tesla was consistent. To read the individual module voltages, probe your meter on 0&1, 1&2, 2&3, etc. The lower number is always the negative side of the brick. Probing non adjacent numbers gets you voltages for groups of Bricks in series and probing from zero and 6 gets you the full Module voltage. The pins between the numbers aren't connected to anything which makes probing for the voltages easier.
Note, the image below is from one of the old BMBs I'm not using. I scraped the coating off the pins to make them easier to probe. If they are going in the pack, you want a nice coating of conformal coating to keep them from corroding.
Update today - I started to charge the low brick with 4.0V at .5A. The image shows me starting at a quarter. I got about 10mV back in about 2 hours. My bedroom happens to be right above the garage in this house, so I'm not going to let it charge overnight unattended. I'll see how much of that voltage held overnight and keep going in the morning. I didn't lose any from the previous 24 hours so the 'wet' short that drained the power from the point the low ISO showed up seems to be resolved. Tomorrow with be a grueling marathon with the charger and 1,000 q-tips cleaning 888 connections with 200 proof alcohol. I can already feel the pain in my back...
Similar threads
