Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Condensation Failure Inside Battery Pack (BMS_F107 code)

This site may earn commission on affiliate links.
Sharing some failure mode analysis with others who has been inside the battery pack (NOTE : HV can be deadly dangerous so absolutely need to be familiar with all HV protection precautions) This is @mr_hyde car with his original post here and here.

Car is 2014 MS P85D 67k miles. In Seattle always (no salted roads, rains frequently but low humidity otherwise) @mr_hyde is 2nd owner acquired recently.

Car disabled with BMS_F107 (Cell sensor voltage code) Thought it might be a loose voltage sensor wire but found the following after pack opening
  • No evidence of broken seals for water ingress anywhere (pack cover lid, front hump, fuse cover)
  • No evidence of coolant leak from any pipe fittings (no silicate residue or sticky coolant residue anywhere)
  • All umbrella valves looked fine without failed seal
  • Condensation evidence in the modules under the hump and first 3+ rows of modules from front. Rear 3+ rows looks perfectly fine.
Modules’s large and shiny electrode plate enable easy inspection of any kind of anomaly. 2 patterns were seen
  • Spore shape slightly hazy white patterns around few 18650 cells. This is likely due to corrosive electrolyte release. Perhaps this is fairly common for 18650 cells? The big shiny electrode plate captures the evidence easily. 18650 manufacturing process shows electrolyte fill ( link ) and drying ( link ) process. Not too concerned about this. Perhaps occurred early on after module + plastic cover assemebly.
  • Top plastic cover on modules have dime size indents that rest on top of electrode plates (for fitting) Many of these show condensation marks.
  • Long run sections of plastic cover touching the electrode plate shows signs of dried water flowing marks.
  • Tiny droplets of dried water marks on parts of the electrode plate not contacting the plastic cover.
  • BMB’s C26/C27 (Rev 1.5 on this car, later BMBs might have redesigned this area) are the tallest component on top of the circuit board and very close to (probably contacts) the bottom plastic tray cover. 4 BMBs shows corrosion here. Also voltage sensor wire's solder pins on module #9 (upside down mounted compared to the rest) show corrosion. There are 2 potential failure mechanisms. Conformal coating is likely sprayed on and side walls of the caps and vertically standing solder pins would have a weak / no coating layer. Also potentially conformal coating was rubbed off from contact+vibration and/or condensation formed here which eventually corroded the solder points.
IMG_1873.jpegunnamed (2).jpgIMG_1840.jpgIMG_1839.jpgIMG_1864.jpegIMG_1857.jpeg

These condensation patterns suggest the following failure mechanism
  • Since all failures are near the front of the pack and vent port is on top of the hump. It seems likely as the pack “breathes” (1000 lb of warm pack cooling will draw in external air from the vent) moist air is potentially drawn into the pack
  • Vent is on top of hump which is where some windshield water run off dumps on. Furthermore, AC drain (even improved to dump further forward of the pack) dumps nearby. Usually the hump and anything ferrous in the area (DCDC box on this 2014 MS) shows evidence of water dump + rust/corrosions in most cars that has seen salt road or rain or carwash often.
Possible Solutions
  • Place desiccant bags inside the pack prior to resealing the pack. Most available space is near where coolant enters the pack
  • Drill and install bottom serviceable desiccant cartridges like Smart Car EV ( link ) again near the coolant pack entry area.
  • My understanding is the Rav4 EV and MB B-Class (both built with variation of Tesla drive unit and battery modules) both have serviceable desiccant bags/cartridges and maybe a circulating fan?
  • Another possible solution is to remove the vent port and extend it along with an inline drying box filled with desiccant bags. However, the frunk box might be in the way for RWD cars.
NOTES
  • This is less frequent failure than water ingress through rusted fuse cover in salted road area cars. But this seems likely to occur over time depending on humidity in the region.
  • Accumulated water condensation on the electrode plate can theoretically damage the ultrasonic cell welding connections in nearby cells. But maybe BMB’s capacitors will fail first such as on this pack. But on later BMBs with different designs, not sure this will be true.
  • Will Tesla reman effort go though this level of detail to inspect all 16 BMBs? This car failed with 1 BMB (ScanMyTesla showed module 8 - note module #s reflect network ordering and IS NOT the HV serial bus order which zigzags serially from the rear. This is a potentially dangerous misunderstanding when disassembling the pack. Removing the fuse breaks the serial HV chain into 14x and 2x modules. Not split into half. Many people misunderstand this from network module # ordering) But 3 others were just behind this haven't yet completely failed. If just go by diagnostic tool directed inspection, 3 near failed BMBs would have remained in the pack.
  • Opening the pack lid non destructively is a PITA and can easily be 10-20 hours (remove carefully, clean off all sealant residues, hammer lid sealant surfaces flat etc.) @250/hr labor rate easily $3-$5k cost. Perhaps can go faster with developed tools with experience. Chevy Bolt uses a rubber gasket with bolt spacer sleeves to provide proper even pressure. This guy did it in his wheel chair ( link ) Bolt also has full service manual on disassembling inside the pack and I presume supply all parts.
 
Last edited by a moderator:
Thank you for the excellent post Howard. As mentioned, this is a clean, garaged car in the PNW with no salt and low relative humidity. The pack was perfectly dry inside with no signs of water ingress but certainly signs of *moisture* and if that is the case with this pack, it must be the case with nearly every pack of this generation and much worse in harsher climates.

The acute failures were on caps 26 and 27 of BMB 8 but three additional boards show significant corrosion and were imminent failure points - likely weeks or months down the road. I'm very glad I took the significant additional effort to 'lift the lid' and inspect the rest of the modules or I would have been revisiting this again very soon.

The internet is full of 'my battery failed' posts but there are very few details about what that really means. In my case the actual battery cells in module 8 were in perfect condition at precisely 3.91v for each brick which was the resolution/sensitivity limit for the VOM we used. I believe I speak for Howard when I say our goal is to understand and shed light on the design issues and failure points and come up with solutions to mitigate them to the extent possible.

I am curious if any vendors who have seen (significantly) broader samples than Howard and I have, can share their experiences. Is there moisture inside EVERY pack? Is it always C26 and C27 on the v1.5 boards that gets attacked first? How frequent are actual cell failures in the 18650 packs so far? When someone has a 'pack failure', what percentage are BMBs with corrosion?

A quick note on the actual repair: Dropping the pack is actually extremely easy, even in a residential garage on ramps and jack stands. Inspecting/replacing the BMBs with the modules out of the pack is easy. Opening the pack is another matter. The voltage inside the battery is deadly and I don't advise a DIYer to attempt it. If you are going to, please spend weeks, not hours or days, researching, reading and watching videos so you understand the hows and whys of the pack design. Know your error codes and if any of them show any low isolation faults, you should probably leave things alone and have an experienced shop do it. Low iso takes an already dangerous environment and turns it to 11.

From a practical standpoint, opening the lid was the absolute worst automotive task I have ever done. It took hours and it is still beat to hell. Getting it to seal back on will be even more of a chore. I would pay many hundreds of dollars for a flat new one to put back on but like most things Tesla, they won't sell me one.

12CE3E33-ECFF-4914-A5C0-BBC8D343A8B4_1_105_c.jpeg
 
There are tons of desiccant cartridges+bags available for all kinds of applications. Maybe can make a model X style fuse cover solution (under the pack, facing downwards to solve Model S upward facing cover attacked by salt + moisture + gravity) under the coolant manifold area (only large space available) and insert in bags/cartridges. Plugs with moisture indicator windows also available (Don't know if robust enough to handle all the road debris including salt)

Service interval maybe 1-2 years depending on how much drive in rain + park outside overnight + regional humidity. Smart Car's battery pack is much smaller and I'm guessing large form factor cells might be more robust (more cells = more exposed contacts) But certainly they have BMS(s) inside as well.
 
Last edited:
  • Helpful
Reactions: cleverscreenam
I love this idea but I don't love cutting into the thick and solid bottom case where it gets road spray and debris and I don't love putting a hole in the hump cover where it gets rained on. Lots of thinking to do.
 
There is room under the coolant rapidmate accessible through the hump's umbrella valve to put 2 or 3 desiccant bags and have them serviceable through the umbrella valves. The little safety screws are only annoying if you don't have the bit to remove them.
96F8ED91-CB4D-487F-A626-660360BAEAA2_1_105_c.jpeg



B6F254CC-4556-4F9B-860D-2041521B5E63_1_105_c.jpeg
 
Have u removed that orange lid??
I'm pretty sure nothing is really accessible there...
On my newer pack the lid is a bit different but there's no access to inside since the rest of the structure is bolted from inside the battery.
Pretty sure its the same for older style...
 
Have u removed that orange lid??
I'm pretty sure nothing is really accessible there...
On my newer pack the lid is a bit different but there's no access to inside since the rest of the structure is bolted from inside the battery.
Pretty sure its the same for older style...
Good call - no, I haven't looked at it yet. There is a shield that is bolted from inside but the rest just looks like sealant. I'll definitely look to understand how it works and comes out before I put bags in from the top side!
 
This 2020 article explains using ePTFE material in EV battery pack breathers to prevent moisture ingress


This appears to be Tesla's battery pack breather. PN is close but not exact. Shape seems exact. Don't know if specs are the same.


Sourcing a breather with advanced ePTFE membrane will probably be difficult in US as an individual. Most are custom and suppliers focused working with manufacturer / tier 1&2 suppliers. Likely much more option in EU.
 
Last edited:
A couple of notes on breather valve on the battery pack. Made by AGM (an AZ container venting+moisture control company founded to handle missile container storage) as noted in the above link, use a silicone rubber valve that opens and close at fairly low pressure differential to equalize the pressure between the pack interior and exterior. 3 triangular positioned holes on top are the air pathways.

There might be 2 different possibilities for moisture ingress

- Air humidity. Will condense on surfaces once drawn into the pack.
- Airway opening is mounted facing upwards (not ideal, vertical would likely be better) Its possible water drops can accumulate on top of the valve and enter whenever pack cools and valve opens to equalize pressure. Also on RWD and dual motor MS, the frunk tub might be different enough to change the windshield water runoff dump patterns above the breather.

Its not clear if one of both ingress mechanisms are occurring. FYI for those studying moisture ingress in this area.
 
Last edited:
Notes on the BMB's conformal coating inside the battery pack. Definitely looks sprayed on and uneven. There are multiple corrosion points seen
  • top side of C26/C27 (see post #3 pics) on 4 different modules.
  • module #9 (only module installed upside down from the rest) voltage sensor connector solder pin #2 and unsoldered pin just below (see post #1 pics)
This suggest there are potentially less conformal coating on sides of tall structures and board edges (probably don't matter). Shadows from spray angles might be an issue.
 
Module 9 didn't make it but the failure mode was different than the 26/27 caps on the other 4. It still had continuity but given the amount of time it takes to get to it, I took no chances and ordered a replacement. This means I had corrosion on 7,8,9,10 and 11 of my 'dry' pack. It makes me want to go back to module 6 because it sits across from 11 and somehow survived... 🤣


F844AC60-F322-4978-9C3D-5A4562E65C67_1_105_c.jpeg
 
  • Like
Reactions: spaceballs
Early M3 has TSB to replace battery pack breather with updated part due to hearing thunk over mountain passes.


Part is from NITTO and likely PTFE membrane. Z-PLUG-S diagram is 1/2 way down the page. ZH-PLUG-S (in Tesla TSB PN) listed in the same table with higher air flow rate compared to Z-PLUG-S.


Seems like M3 owners are reporting loud bangs at super chargers and this owner indicate sound is still there after breather upgrade (scroll down just before the Jan 30. 2020 update)


Don't know if these breathers are better at humidity moisture ingress than MS/X AGM silicone valve breathers or have sufficient equalization rate to prevent M3 battery pack bang noises.
 
Last edited:
Looks like M S/X battery pack has 2 breathers
  • Silicone Valve breather on top of the frontal hump in post #9
  • PTFE membrane breather on bottom in the orange plastic mount. Pics here and here (vent protective cap removed). Service manual calls this the Gore Breather. Likely manufacture is Gore and could be the high flow hydrophilic PTFE breather PN PMF200484 in product listing ( link )
  • Of course there are also the numerous umbrella valves to vent abnormally fast cell outgassing to prevent an explosive emergency event. These aren't meant to be normal operation breathers.
My understanding is the PTFE breather is continuously venting membrane. These breathers are commonly used in outdoor exposed electronics boxes to reduce condensation. The silicone valve breather on top of the hump is an actual valve that open/close around 0.2 psi to equalize pressure in case of rapid change (400ft elevation change = 0.2 psi, not sure what thermal differential = 0.2 psi)

These PTFE breathers are from Gore's outdoor electronics product line's SNAP-IN Series


These are for far smaller containers than Tesla battery pack. The battery pack isn't completely filled with batteries, electronics, bus bars and coolant tubes. The empty air volume probably far exceed the typical container volume these PTFE vents are used in. Battery container also has high aspect ratio (far wider and longer than it is tall) which might suggest needing multiple distributed breathers. No idea if this is part of the issue to the condensation found inside frontal 1/3+ of the @mr_hyde battery pack ( 5 of the frontal module BMBs have corrosion )

Condensation combined with uneven spray of conformal coating (taller objects have less coating at the base of the board (where solder joint is) from shadowing during spray) seems to cause corrosion somewhere between 5-10 year mark. Likely depending on several factors like humidity, indoor/outdoor parking, elevation change frequency etc.
 
Last edited:
  • Informative
Reactions: cleverscreenam
An interesting article by AGM (provider of the air pressure equalization breather on top of the pack hump)


It clarifies some observations and demyth a few misconceptions...
  • Basically say water will get in no matter what you do. Designer frequently underestimate the problem.
  • Liquid water proof vents (they call out Gore specifically - maker of the breather on bottom of the battery pack) as good against actual liquid water but doesn't do anything for vaporized water.
  • Gore type vents is actually counter productive when using desiccant. Desiccant will quickly saturate. I'm assuming higher external water vapor gradient will migrate towards dryer interior until desiccant becomes saturated and ineffective.
Sounds like the necessary solution is find an active drying solution, possibly seal off Gore type vents + desiccant, open pack and clean/exchange moisture corroded component every 5-10 years (depending on use)

Desiccant placement location likely need to be studied against moisture entry source(s)
 
  • Like
Reactions: cleverscreenam
Just a final update on what we know about battery pack breather / air equalization evolution.

Once again, HV pack is deadly dangerous. Do not attempt to disassemble it without expertise knowledge of the pack and HV safety. It can kill you as live HV danger points are invisible and water ingressed packs can have shorts which turns entire battery chassis with live HV with potential deadly current flow.

Original MS Pack

1. AGM 0.2 PSI silicone valve on top of the front hump on the battery pack to equalize air pressure. Valve is upward facing which is the worst orientation, water dump from windshield can land on top of valve and when the valve opens (0.2 PSI = 400 ft elevation difference), can potentially gulp the standing water in. Vertical orientation is ideal according to valve vendor. With upward facing orientation, maybe a hood is helpful to keep direct water drop from windshield run off to sit on top of the valve. In any case, valve will gulp in ambient air moisture anyway whenever needing to open.

2. PTFE membrane breather on bottom front driver side of the pack. Under the orange (also other colors) cover in post #6 PTFE is a special material ( like hard plastic ) with tiny pores to continuously flow air between interior and exterior of the pack. Usually at a much lower volume than an air eq valve like the AGM valve on the hump. PTFE membrane is good at preventing water droplets from entering (is downward facing on this pack anyways) but can not stop air moisture from entering as the pack breathes.

Pics of both here ( link )

More Recent MS Pack

There are more recent packs (from ~2021?) that doesn't have the AGM eq valve on top of the front hump. The opening for this valve is no longer present on the hump. Unfortunately can't find a picture ATM.

Maybe more recent remanufactured packs also used this same hump without the AGM air eq valve. Don't know.

Model 3 Pack

Model 3 has a PTFE breather but appears to not have a silicone air equalization valve.

How Does The Air Eq valve and Breather Work?

Without inspecting all these packs and measure each breather / valve carefully, here is a guess of how this works

Most of the older MS/X packs continuously breathes through PTFE membrane, This breathing membrane probably equalize air at a low rate but whenever the eq rate is too high, AGM valve opens and quickly eq the pack and ambient air pressure. These air pressure changes can occur thermally (charge/discharge battery) and elevation change.

M3 and latest MS pack don't have the silicone air eq valve. Therefore, M3 battery packs sometimes have a loud thunk sound while supercharging + mountain passes. Maybe MS/X without the top hump air eq valve will do this same. Don't know.

In any case, ambient moisture will enter the pack constantly. With many interior surfaces (serpentine baffles shapes are often used to condense moisture from air) inside the battery pack, moisture condensation will result. Rate probably depend on many factors. No drying agent such as desiccant bags are inside the pack.

Annual Anti-Moisture Maintenance?

Following post show drying out internal moisture failed pack with hot air through the side emergency umbrella valves holes. If the side skis and umbrella can be removed without removing battery pack (have heard conflicting answers if possible) Perhaps an annual dehumidifying maintenance through these holes would be helpful. Regarding repair moisture damaged pack without completely opening the pack, I have seen moisture disabled packs repaired temporarily without opening the pack only to fail later with BMS_F107 error as corrosion likely started on the BMB boards like the pack in original post #1

 
Last edited:
Annual Anti-Moisture Maintenance?

Following post show drying out internal moisture failed pack with hot air through the side emergency umbrella valves holes. If the side skis and umbrella can be removed without removing battery pack (have heard conflicting answers if possible) Perhaps an annual dehumidifying maintenance through these holes would be helpful.
@wk057 had been thinking about offering a service that sounds like that:

Right now, we're doing testing of some custom-made parts to be able to retrofit into these older battery packs to prevent the moisture ingress issue and better seal the pack for the long term. As part of the service we'll test for existing moisture ingress, use some custom equipment to remove any moisture from the battery pack, and give the customer an estimated idea of what we expect the long term health of their battery might be afterwards. We eventually hope to offer this as a preventative maintenance service to customers as a one-day service (scheduled in advance, of course) for < $1k, possibly by the end of the year. In my opinion, this is going to be an essential service for these vehicles once we've got it streamlined.

Hopefully that can still come to market.
 
@wk057 had been thinking about offering a service that sounds like that:



Hopefully that can still come to market.

Yes. Quite an interesting idea. Having seen the condensation moisture failure, I do wonder how much periodic service will benefit after corrosion has started. I suppose will slow it down. Don’t really know wo lots of trials and stats.