How Do I Responsibly Report a Powerwall 2 Hardware Safety Defect?

[taphil]

So I guess roast me?...

You're being so cryptic about it. Just state what it is. Whether or not you disclose the mystery now or by your chosen deadline isn't going to change the fact that there is or is not a hardware flaw that can electrocute someone tomorrow or next week, or if Tesla will need to recall or repair thousands or millions of PW that are installed. One of my PW died in under a year and took like 9 months to replace - Was this due to the flaw?

My intended goal here is to raise this issue directly with Tesla, so that I can understand how exactly they certified this product.

Do you think Telsa is going freely discuss this with you or hire you as a consultant to investigate this flaw? This is the same company with no PR dept and emails poop emojis as replies.

 

[stopcrazypp]

You're being so cryptic about it. Just state what it is. Whether or not you disclose the mystery now or by your chosen deadline isn't going to change the fact that there is or is not a hardware flaw that can electrocute someone tomorrow or next week, or if Tesla will need to recall or repair thousands or millions of PW that are installed. One of my PW died in under a year and took like 9 months to replace - Was this due to the flaw?


Do you think Telsa is going freely discuss this with you or hire you as a consultant to investigate this flaw? This is the same company with no PR dept and emails poop emojis as replies.

He already said the reason for not doing a full disclosure. He gave a deadline already and he's going to stick to it. It's coming up soon anyways, so just have some patience. He already gave hints as to what it is, so if you have sufficient knowledge in the subject, you should already know pretty much what he was getting at from reading what is posted so far.

 

[jjrandorin]

Whether or not you disclose the mystery now or by your chosen deadline isn't going to change the fact that there is or is not a hardware flaw that can electrocute someone tomorrow or next week, or if Tesla will need to recall or repair thousands or millions of PW that are installed.

There IS a difference, though. You are asking someone to break their word. That still means something to some people. It does to me anyway, and I have a lot more respect for someone who wont break their word than for someone who does.

In any case, feel free to ask again after the date specified. Of course the OP is free to share earlier if they want to, but since I already expressed this subforum is generally a bit more courteous than the rest of TMC, I would like to request that line be dropped until after the specified date.

In my (as yet unanswered) correspondence with Tesla, I mentioned I would publish my findings on JAN09, with or without their feedback.

 

[mudder]

Do you think Telsa is going freely discuss this with you or hire you as a consultant to investigate this flaw? This is the same company with no PR dept and emails poop emojis as replies.

Tesla doesn't need a consultant to fix this specific flaw. The issue is pretty obvious once you see it, and the engineering design change is trivial. The non-trivial part would be retrofitting every Powerwall unit ever sold, if in the end the certifying body deems it necessary. At this point I don't need to "freely discuss this with" anyone; it's in the certifying authority's court now... and based on how quickly both UL and Intertek replied to my complaints, I would venture a guess that it's probably in Tesla's court, too.

Given how obvious this defect is, it's probably in Tesla's best interest to hire a 3rd party to review all their high voltage designs. I'm not after that contract... I prefer designing my own products, rather than fixing someone else's problems. Also, I'm pretty much retired, although I'll keep working on passion projects until I die (e.g. my aforementioned Honda Insight lithium conversion project, which will soon morph into several NiMH Toyota hybrids). Engineering is in my blood.

...

I've worked on the other side of this situation before, back when I worked at National Instruments. We designed a CAT III mains voltage product that had a similar design flaw. It certainly wasn't a great feeling knowing that we were going to have to recall and replace thousands and thousands of shipped units, but it was the right thing to do. Our units were ultrasonically welded shut, so we had to 100% scrap them. I feel for Tesla's team members if they do in fact end up having to issue a recall. We'll see. I'm putting my money on recall, but that's out of my hands now. The wheels turn slowly, though.

 

[mudder]

There IS a difference, though. You are asking someone to break their word. That still means something to some people. It does to me anyway, and I have a lot more respect for someone who wont break their word than for someone who does.

In any case, feel free to ask again after the date specified. Of course the OP is free to share earlier if they want to, but since I already expressed this subforum is generally a bit more courteous than the rest of TMC, I would like to request that line be dropped until after the specified date.

I'm new here, but so far I suspect you're a good moderator.

 

[BGbreeder]

Consider the following hypothetical:
Suppose that you discover a serious design defect that could cause death or serious injury. You have a full time job and very little free time, but you go out of your way to cross reference the various (far from free) standards to verify the issue actually exists; fully document the issue, including an in-depth video, detailed pictures, and a full writeup; and then spend far too many hours attempting to report this issue in a way that is useful to the responsible parties.

Suppose that in doing the above, you prevent just one person from dying from electric shock. In doing so, you prevent the company that designed the product from having to pay out a $5M wrongful death lawsuit. A hardware bounty program encourages engineers to endeavor/persist/waste time going down rabbit holes they might otherwise have no obligation/desire/need to follow.

To me, looking out for the well being of one's fellow citizens would seem to be part of the essential fabric of civil society. While I might hope that there isn't an issue, if there is, I wish you the best of success.

All the best,

BG

 

[cali8484]

Doubt Tesla will do a hardware recall out of concern for the "well being of one's fellow citizens". More likely they will fight/delay it for as long as possible. Maybe a software update "recall".

 

[mudder]

Maybe a software update "recall".

If a recall is ultimately mandated, it would require physically replacing a part inside each unit.
The only way a software update could 'resolve' this hazardous hardware issue is to:
-program the Powerwall to permanently disconnect from the grid, and then;
-program the Powerwall to permanently disable the inverter.

Of course, if you did the above, then nothing on the islanded side would remain powered, thus negating the entire premise behind installing a Powerwall in the first place.

 

[jjrandorin]

I am just as curious as many of our other members as to what this actually is (and if I could even understand it when explained). As you have seen, I am fully supportive (100%) of your decision not to share until after the time limit you previously communicated to Tesla, or even not at all.

I will say though, that I understand why some of our regulars feel a bit anxious about seeing your remarks as they are, indeed, cryptic (by design).

If you are not going to share before the time specified, thats fine, but if you are not going to share until that time, I am not sure the crypticness of your statements is helping us process it much. Most of the people in this subforum (but not all), own the products, so would have some anxiety around something like this, so I ask that you keep that in mind when making these statements.

Note again that I am not trying to censure you, I approved your first post when it came to up for approval quickly, and have (fairly obviously I think) defended your right to share or not as you decide, but I also ask you to consider the position of most (but not all) of us in this subforum who are product owners and might be experiencing conflicting emotions about someone saying their 5 figure investment has a hardware problem that cant be rectified, but delaying in explaining that.

I get the delay, I do, I have said it more than once, I am just asking that you consider the other side as well.

Thanks, and hopefully this doesnt change your opinion about the site, or my actions here.

 

[getakey]

If a recall is ultimately mandated, it would require physically replacing a part inside each unit.
The only way a software update could 'resolve' this hazardous hardware issue is to:
-program the Powerwall to permanently disconnect from the grid, and then;
-program the Powerwall to permanently disable the inverter.

Of course, if you did the above, then nothing on the islanded side would remain powered, thus negating the entire premise behind installing a Powerwall in the first place.

Curious - Did you go to Harvey Mudd? Basing question on your username and your passion for engineering

 

[charlesj]

... still pretty darn courteous, quite a bit like "old internet", ...

Well, some part was almost anything goes.

 

[mudder]

I am just as curious as many of our other members as to what this actually is (and if I could even understand it when explained). As you have seen, I am fully supportive (100%) of your decision not to share until after the time limit you previously communicated to Tesla, or even not at all.

I will share my findings on JAN09.

I will say though, that I understand why some of our regulars feel a bit anxious about seeing your remarks as they are, indeed, cryptic (by design).

If you are not going to share before the time specified, thats fine, but if you are not going to share until that time, I am not sure the crypticness of your statements is helping us process it much. Most of the people in this subforum (but not all), own the products, so would have some anxiety around something like this, so I ask that you keep that in mind when making these statements.

Note again that I am not trying to censure you, I approved your first post when it came to up for approval quickly, and have (fairly obviously I think) defended your right to share or not as you decide, but I also ask you to consider the position of most (but not all) of us in this subforum who are product owners and might be experiencing conflicting emotions about someone saying their 5 figure investment has a hardware problem that cant be rectified, but delaying in explaining that.

I get the delay, I do, I have said it more than once, I am just asking that you consider the other side as well.

Couple statements to hopefully put owners' minds at ease for now:

Electrocution risk statements:
-If the Powerwall chassis is properly grounded, there is no electrocution hazard (but there is still an electrical isolation violation). With proper grounding, a fault resulting from this design defect would most likely pop the upstream breaker (if the PW firmware didn't island and shut down first).
-If the Powerwall chassis is NOT properly grounded, there is an extremely high electrocution hazard.

Thermal event risk statements:
-(single phase 230 only ((EU/etc)): If the Powerwall phases are installed according to the manufacturer's specifications, there is no thermal event hazard (but there is still an electrical isolation violation). However, if the phase installation instructions aren't exactly followed, then there is an increased chance that a thermal event could occur under very specific conditions.
-(split single phase 240 only (USA)): Even if the Powerwall is installed according to the manufacturer's specifications, there is a slight risk that a thermal event could occur under very specific conditions.
-If a thermal event occurs due to this issue, it would almost certainly remain contained within the unit. Worst case, smoke might exit the sealed enclosure, but not fire. The cells themselves are not at risk.

Summarizing the above:
-If your unit is installed properly, the worst case failure condition is smoke coming from the unit (but not fire).
-If your unit is NOT installed properly, the worst case failure condition is electrocution.

Thanks, and hopefully this doesnt change your opinion about the site, or my actions here.

My opinion hasn't changed at all. Your feedback is entirely correct.
I will share my findings on JAN09 (and also any updates I hear from either Tesla or Intertek before then).

 

[mudder]

Curious - Did you go to Harvey Mudd? Basing question on your username and your passion for engineering

No, Vanderbilt Engineering.

mudder /mŭd′ər/ (noun):
-One that performs well in muddy conditions, as a racehorse, athlete, or pickup truck.
-A horse which runs best on a muddy track; a mud-runner.
-A racehorse that exhibits a better than usual performance when the racetrack is wet or muddy.

I initially earned that nickname while racing mountain bikes.
The name carried over into my consulting career. My specialty is salvaging FUBAR engineering designs. My self-appointed official title is "Problem Solver".

 

[getakey]

No, Vanderbilt Engineering.

mudder /mŭd′ər/ (noun):
-One that performs well in muddy conditions, as a racehorse, athlete, or pickup truck.
-A horse which runs best on a muddy track; a mud-runner.
-A racehorse that exhibits a better than usual performance when the racetrack is wet or muddy.

I initially earned that nickname while racing mountain bikes.
The name carried over into my consulting career. My specialty is salvaging FUBAR engineering designs. My self-appointed official title is "Problem Solver".

Got it
You would have fit in at Mudd. I went there

 

[mudder]

Observation 1:
An aluminum tab that secures the PCBA to the enclosure is mounted coplanar to layer 1 on the PCB (see picture). The L2 mains voltage trace on layer 2 routes directly underneath a portion of this aluminum tab. The copper on layer 2 is approximately 0.38 mm below the PCB top surface (layer 1). Therefore, the spacing between chassis ground and mains voltage is only 0.38 mm. This 0.38 mm spacing is primarily FR4, plus a thin conformal coating (type unknown).

Observation 2:
The same L2 mains voltage trace (on layer 2) comes very close to the PCB edge, which is NOT conformally coated. The aluminum tab passes directly over this edge, too (coplanar to layer 1). Therefore, there is no conformal coating where the aluminum tab intersects the board edge.

Notes:
As a legal precaution, I'm only publicly reporting my factual observations. My initial goal in creating this thread was to engage Tesla's engineering team for clarifications. That is still my goal, although so far I'm disappointed in both Tesla's and Intertek's lacking communication. In particular:
-Intertek's confidentiality agreement with Tesla forbids them from disclosing any findings resulting from my complaint.
-Tesla hasn't followed up to discuss my findings. Whether or not my conclusions are correct (I think they are), Tesla's failure to respond is troubling.
-In contrast to their software bug bounty program, Tesla lacks a method to report hardware defects. To whit, Tesla Motors actually recommends reporting potential hardware defects to NHTSA, rather than directly to them. In other words, Tesla is passing the buck to a regulatory agency.

If anyone else wants to dive into the various codes and elaborate further, I recommend starting with UL 1741, which is listed on the Powerwall label. Particularly, take a look at the exceptions in sections 24.1 and also 25.4.

Overall, I'm feeling pretty apathetic about continuing to invest time and effort into this issue. I'll stick around for questions, but overall I'm just going to wash my hands and be done with this. It doesn't affect me personally and I feel like I put in enough (free) effort to clear my moral conscience. If Tesla and/or Intertek ever responds to my complaint, I'll keep going, but otherwise it's just not worth the effort. I've got too much other (paid) work to do than to keep slogging down this rabbit hole.

Stay safe out there.

 

[furuike]

I am also an electrical engineer and deal with NRTL safety compliance in product design.

Observation 1 may or may not be a problem, it depends on the insulation rating of the PCB material used as well as the conformal coating type. How do you know the PCB material is ordinary FR4? There are many high voltage PCB materials that have better voltage isolation characteristics that do not look any different, and solder mask color is no guide here. Some teflons get 500V per mil. I'm not saying they used this material, but without knowing more about the PCB construction it's insufficient evidence on its own to conclude it violates line to ground spacing.

Observation 2 does seem like a pretty straightforward problem, and it's the kind of miss you often see on PCB designs. The designers should have had safety keepouts at appropriate creepages to make DRCs if any line voltage was placed near the board edge on those layers, or else had conformal coating.

 

[furuike]

Also, to reiterate what you said above, if these are indeed defects (I haven't measured them and cannot confirm), then what it means for a powerwall owner is that there is a somewhat elevated risk of ground fault. Anyone who had a Powerwall installed by a competent company (with competent electricians) will have a proper EGC (equipment grounding conductor). This is required by the NEC. So for almost all cases, an issue like this causing a short will cause a ground fault event which will trip a circuit breaker.

 

[mudder]

Thanks for the comments, @furuike. PCB is an ordinary fiberglass, as determined by abrading the top surface to determine the weave structure. Probably a 175-class FR4, based on my experience... could be up to 600, but that would still require much more spacing than observed.

Supporting evidence: everywhere else on this PCB, Tesla has gone to GREAT lengths - and wasted tons of PCB real estate - to maintain at least 4 mm spacing between isolated modules. It seems odd that Tesla would properly enforce 4 mm clearance all over this PCB - particularly in those high voltage areas close to the aluminum bracket - except for at this one spot, which for some reason has more than an order of magnitude less spacing (i.e. 0.38 mm).

...

Do you have any thoughts on the exceptions listed in UL 1741 24.1.1? Particularly #7, which would require 0.8 mm regardless of interlayer material type? Note that #7 doesn't exactly apply to this issue, but is the least amount of required clearing I've found so far for a design like this.

Do you have any thoughts on UL 1741 25.4(b), which requires inverters to comply with Overvoltage Category IV spacing requirements (instead of Category III)?

Do you have any thoughts on UL 1741 25.4(h), which requires adherence to UL 840 spacing requirements?

...

To reiterate, I would love for Tesla to provide more information about their certification process. If they would only just respond, maybe we could close this issue out in five minutes?

 

[mudder]

Also, to reiterate what you said above, if these are indeed defects (I haven't measured them and cannot confirm), then what it means for a powerwall owner is that there is a somewhat elevated risk of ground fault. Anyone who had a Powerwall installed by a competent company (with competent electricians) will have a proper EGC (equipment grounding conductor). This is required by the NEC. So for almost all cases, an issue like this causing a short will cause a ground fault event which will trip a circuit breaker.

A secondary issue would be gradual FR4 breakdown caused by CAT IV overvoltage events. The primary reason minimum spacing is so large at CAT III and CAT IV is to prevent accumulated failure due to corona discharge following a breakdown pathway through the insulator.

If this secondary issue occurs, it could lead to a thermal event. Fortunately the PCB doesn't contain anything that could sustain a fire, so you'd likely just get internal smoke inside the sealed container, but note that as long as the corona discharge persisted, the spark ignition source would remain present (unless the upstream breaker pops). I've seen this failure in other designs before... basically the same defect as I describe here, and on a product that was ultimately recalled for exactly that reason (because it violated UL 61010-1 6.7.1.1 spacing requirements).

 

[cali8484]

If this secondary issue occurs, it could lead to a thermal event. Fortunately the PCB doesn't contain anything that could sustain a fire, so you'd likely just get internal smoke inside the sealed container, but note that as long as the corona discharge persisted, the spark ignition source would remain present (unless the upstream breaker pops).

Can the spark get over to the nearby battery cells?