Powerwall Reset During Grid Outage

[aesculus]

I had a few hour power outage yesterday where my Powerwalls immediately took over with no interruption. At that time clouds were variable and solar was intermittent. The Powerwalls were slowly being discharged and not knowing when the power would return, I began removing non critical loads from my house. The PWs started charging again when solar was present and got to 97% full. Then suddenly at about 2:10 the PWs shut down, resulting in a blackout. They restarted about 1 minute later and of course took about 5 minutes for the inverters to come back online. Some of the home load had reset itself and in one case a thermostat went wild and started reporting erroneous readings so I had to got about manually turning a few heavy loads off after the Powerwall reset. You can see all of this and a number of errors reported at the event in the attached Powerwall Dashboard images.

I am now struggling to determine what caused this event. The only thing I can think of is that I was at 97% SoC when the failure occured but the solar was very little. I also had turned off most of the load in the home. The solar may have spiked at that moment which sent a surge of power to the Powerwalls, exceeding the rate that they could accept at the current SoC and so they went into some sort of failsafe reset. Had the solar been more stable I would assume that the PWs would have just adjusted the frequency to reduce the solar generation. Other than that ????

I am not an expert to know what the error codes thrown mean and if they even support my scenario.

 

[Matt-FL]

I know when you are in an off-grid situation, once the powerwalls are full the Gateway adjusts the AC frequency to somewhere around 65hz, which tells the inverters to turn off until some battery % is siphoned off. Yours changed to 62hz for the first 20-30 minutes according to the graph in your second screenshot, which is normal when you are close to max battery already. It looks like your water heater or something similar was running at that time with 6kW usage.

Your voltage graph is strange, it looks like Grid L2 was down, but Grid L1 was possibly up (hard to see the coloration).

Definitely something with your powerwalls did not act as it should, I'd definitely put a ticket into Tesla for them to research.

 

[furuike]

You say you have 5.3kW of solar roof and 5.2kW of other PV, are these connected as strings to inverters? Do you have microinverters?

I ask because it's the inverter's responsibility to move the string-connected PV to a low power point on their I-V curve when the battery starts getting full. If your inverters didn't do this correctly (perhaps they did not respond to the increasing frequency correctly), this may have shut down your powerwall.

 

[aesculus]

You say you have 5.3kW of solar roof and 5.2kW of other PV, are these connected as strings to inverters? Do you have microinverters?

I ask because it's the inverter's responsibility to move the string-connected PV to a low power point on their I-V curve when the battery starts getting full. If your inverters didn't do this correctly (perhaps they did not respond to the increasing frequency correctly), this may have shut down your powerwall.

There are three inverters for my solar and all are monitored by CTs.

I am not sure why the frequency did not go up when the PWs were at 97% like it should have. Even though there was not much solar being produced, as soon as the clouds parted, I would get 6 kW of solar spikes as can be seen in the graph just prior to the event.

I was hoping someone would have an idea what all those errors were at the disruption time.

 

[RKCRLR]

I've had my Powerwalls reset when an excessive draw occured (i.e. heat pump tried to start). I know you said you cisconned the highg loads, just throwing that out there for information. I've also had them reset when the power was attempting to come back on but didn't do it cleanly.

Don't have any idea what the codes mean.

 

[aesculus]

I've had my Powerwalls reset when an excessive draw occured (i.e. heat pump tried to start). I know you said you cisconned the highg loads, just throwing that out there for information. I've also had them reset when the power was attempting to come back on but didn't do it cleanly.

I had to turn off heavier loads after the reset because I did not want to drain the battery, uncertain as to when the restoration would occur.

The hot tub resets itself after a power outage and for some unexplainable reason, one of my thermostats went crazy and started reporting a cooler than actual temperature so I turned it off. As soon as the power was restored it reset itself. It may be that it does not like the 62.5 kHz frequency. I never looked at that device during an grid outage before but now I will monitor it. Besides a NAS that is on a UPS, only a few electronic light switches seem to care about that frequency.

I am pretty sure there was no intermittent power restoration because the fault was actually at the bottom of my driveway and I talked to the crew when they did the restoration. The winds caused a jumper to become loose and short across a phase and that tripped a non auto resettable breaker/fuse.

 

[BGbreeder]

I had to turn off heavier loads after the reset because I did not want to drain the battery, uncertain as to when the restoration would occur.

The hot tub resets itself after a power outage and for some unexplainable reason, one of my thermostats went crazy and started reporting a cooler than actual temperature so I turned it off. As soon as the power was restored it reset itself. It may be that it does not like the 62.5 kHz frequency. I never looked at that device during an grid outage before but now I will monitor it. Besides a NAS that is on a UPS, only a few electronic light switches seem to care about that frequency.

I am pretty sure there was no intermittent power restoration because the fault was actually at the bottom of my driveway and I talked to the crew when they did the restoration. The winds caused a jumper to become loose and short across a phase and that tripped a non auto resettable breaker/fuse.

Perhaps your explanation of the cause sheds some light on the issue. Your first plot shows that L1 stayed up, (consistent with the crew's explanation) and you had some rapidly changing loads, which may have transiently exceeded the Powerwall's ability to frequency shift. IIRC, there is a limit to the rate of change of frequency that is allowed, and your "spikey" production and a nearly full Powerwall may have exceeded that.

I look forward to hearing the answer from Tesla.

All the best,

BG

 

[Matt-FL]

You mention a UPS, which may be why your system only increased frequency to 62Hz. The default is ~65Hz, however you can contact Tesla to have that reduced to keep more sensitive devices like UPS from thinking they are still offline. I did that for mine, and my frequency shift is only to 62Hz and my UPS's are moderately happy about it.

I'm also curious to hear what Tesla responds with after they review logs. Pretty sure it was some failsafe mechanism.

 

[aesculus]

UPDATE: Tesla contacted via app and email response. Waiting.

FYI: My system has been updated to the 62.5 Hz setting. I have a UPS on the NAS because that system does not like even 62 Hz. I have a few electronic light switches that also do not like that frequency and now maybe I have discovered a thermostat too, but TBD.

 

[aesculus]

Tesla returned my email request with this:

Thank you for bringing this to our attention. In review of the signals, I see the Powerwall detected the outage and the Powerwalls overloaded. Starting some loads with high inrush current during a power outage may overload the Powerwalls and cause them to stop providing backup power.

I have asked for a bit more information such as the inrush power amount. I have all new variable speed heat pumps and the hot tub was done with its cycle. The only other variable heavy loads I have (35 amp breakers) are well pumps. I would not expect those to big a big inrush and after the PWs cycled, the well pumps were not on, which they would be if they had tripped the PWs as they would only turn off after doing their work.

I am still suspecting an inrush of solar power which I stated to Tesla. I'll see if they respond.

 

[cali8484]

I have asked for a bit more information such as the inrush power amount. I have all new variable speed heat pumps and the hot tub was done with its cycle. The only other variable heavy loads I have (35 amp breakers) are well pumps. I would not expect those to big a big inrush and after the PWs cycled, the well pumps were not on, which they would be if they had tripped the PWs as they would only turn off after doing their work.

So, the hot hub is one of the backed up loads? Are these loads new after the PW's were installed? If so, the overload explanation would seem more plausible. The graphs do appear to show a pretty large power swing on the PW's right before they went down. Also, there is an alert for under voltage which would be consistent with the overload explanation. Not sure how old the PW's are but if the same loads used to work and the PW's are older then perhaps PW surge capability may have degraded?

 

[aesculus]

So, the hot hub is one of the backed up loads? Are these loads new after the PW's were installed? If so, the overload explanation would seem more plausible. The graphs do appear to show a pretty large power swing on the PW's right before they went down. Also, there is an alert for under voltage which would be consistent with the overload explanation. Not sure how old the PW's are but if the same loads used to work and the PW's are older then perhaps PW surge capability may have degraded?

The hot tub is part of the backed up loads. It always has been. But I had some super inefficient heat pumps at the time of installation (2020) and we agreed that I would have to micro manage my loads during an outage as the PWs would certainly trip if those came on, even in cooling mode.

But the HT had finished its cycle so there is no way it was the issue. Those erratic power swings are actually the solar panels. And the PWs were at 97% capacity at the time of the event. The frequency at that time was 60.2Hz which seems low to me with the PWs so fully charged. My system is actually set for 62.5 Hz on backup mode. And it seems like it should have been much higher to cause the solar inverters to back off. But after looking at the frequency graph I wonder if that is really correct or needs calibration?

NOTE: I just checked the house frequency against what the Powerwall Dashboard is reporting and its within display tolerances (0.1 Hz). I may have to force a grid outage when the weather gets more favorable to see what the conditions are of my PWs during backup mode. And maybe I should try to simulate the conditions of a 97% charged PW too if I can.

 

[aesculus]

Tesla responded to my request to determine the strength of the surge but told be that they do not have that visibility.

They also commented that if the PWs were in need of controlling the solar they would change the frequency. Of course there is no way the frequency response would have been adequate in my scenario and they probably should have shifted it before the event, but Tesla support could not/did not understand that point.

I still need to run my own experience and am waiting for the perfect day for that. Soon!

 

[cali8484]

Tesla responded to my request to determine the strength of the surge but told be that they do not have that visibility.

A bit surprising that they would not know the surge level that caused a shutdown.

 

[BGbreeder]

A bit surprising that they would not know the surge level that caused a shutdown.

I suspect that the Tesla firmware only records an out of bounds error. FWIW: if it were me, I would not design the circuit to stick around to actually try to measure the peak value as that would only invite damage to the monitoring circuit. It would be safer to get everything out of the way as fast as possible, as soon as the threshold is reached.

 

[cali8484]

I suspect that the Tesla firmware only records an out of bounds error. FWIW: if it were me, I would not design the circuit to stick around to actually try to measure the peak value as that would only invite damage to the monitoring circuit. It would be safer to get everything out of the way as fast as possible, as soon as the threshold is reached.

Sure but then they should be able to provide at least the threshold that was violated given that they actually take the time to record and report the fault.

 

[BGbreeder]

Sure but then they should be able to provide at least the threshold that was violated given that they actually take the time to record and report the fault.

Again, guessing here, that type of parameter tends to get set somewhere in firmware because the upper limit is a function of the limits on the control hardware (Tesla) and (probably) design goals for minimizing the voltage to which the customer's equipment is subjected to (NEMA/CBMEA/ANSI). In the latter case, the generally accepted number is 126-127V, per ANSI.

Voltage Tolerance Standard – ANSI C84.1 – Voltage Disturbance

https://durastudio.com/sites/default/files/ansi_c84-1-20xx_2016_revision_draft_2019-02-15.pdf

NEMA, the National Electrical Manufacturers Association recommends devices tolerate up to 10% over voltage, starting from 115V, so 126.5V/253V, nearly the same upper voltage.

But what really matters is how long the voltage is out of specification and how far out it is, as that gets to applied power/damage. To that goal CBEMA, ITIC, and EEPRI suggest this which I file under the "do not exceed" levels;

Notice that the lower scale is logarithmic, and a single cycle is roughly mid graph, making everything on the left very short transient voltage levels. My personal bet is that Tesla's numbers, plural, are within the upper blue line.

All the best,

BG

 

[cali8484]

Again, guessing here, that type of parameter tends to get set somewhere in firmware because the upper limit is a function of the limits on the control hardware (Tesla) and (probably) design goals for minimizing the voltage to which the customer's equipment is subjected to (NEMA/CBMEA/ANSI). In the latter case, the generally accepted number is 126-127V, per ANSI.

Voltage Tolerance Standard – ANSI C84.1 – Voltage Disturbance

https://durastudio.com/sites/default/files/ansi_c84-1-20xx_2016_revision_draft_2019-02-15.pdf

NEMA, the National Electrical Manufacturers Association recommends devices tolerate up to 10% over voltage, starting from 115V, so 126.5V/253V, nearly the same upper voltage.

But what really matters is how long the voltage is out of specification and how far out it is, as that gets to applied power/damage. To that goal CBEMA, ITIC, and EEPRI suggest this which I file under the "do not exceed" levels;

Notice that the lower scale is logarithmic, and a single cycle is roughly mid graph, making everything on the left very short transient voltage levels. My personal bet is that Tesla's numbers, plural, are within the upper blue line.

All the best,

BG

It would be very unusual if Tesla is triggering overload fault based on voltage. For failsafe overload/short-circuit protection it's typically based on current. However, Tesla could be using a similar type of inverse definite minimum time (IDMT) curve for current. For any overload fault, Tesla would certainly know the point on the curve that triggered the fault based on measured current levels.

 

[CrazyRabbit]

What inverter profiles are you running on your none Tesla inverters?
I’m 99% sure your that is your issue.

I run “IEEE 1547 FW 61 to 62 Hz” on my enphase inverters.

 

[aesculus]

What inverter profiles are you running on your none Tesla inverters?
I’m 99% sure your that is your issue.

I run “IEEE 1547 FW 61 to 62 Hz” on my enphase inverters.

If this was directed at me I don't understand the question or how to check. Both of my non Tesla inverters are Fronius which respond nicely to minute adjustments to frequency when near full in the past. But that is with stable solar output and not where clouds are broken and the solar it quite dynamic.