So I just had a really weird powerwall failure

[BrettS]

This was a really weird experience and I’m still not quite sure what to make of it. About an hour ago I was in the middle of a pretty bad thunderstorm. I was operating off grid at the time with my service disconnect off. Suddenly there was a *loud* crack of thunder. I even wonder if it’s possible that lightning hit my house, but whatever it was it definitely affected some things in my house. My fish tank automation system went offline and my UPS was very unhappy. The UPS stopped powering it’s devices and was just continuously screaming. Other than that my house still had power and was operating normally.

Since the UPS was making a horrible noise I went to check that out first. After trying a few things the only way I could finally get it to come back was to disconnect the power to the UPS and then disconnect it’s battery. The loud beeping finally stopped. I waited a little while, then reconnected the battery, then reconnected the UPS to power and at that point it was happy again and the devices plugged into the UPS came back online.

Then I went over to the fish tank to try to reset the controller there, but about that time the entire house lost power. Which seemed very strange because, like I said, I was already operating on the powerwalls, and the last time I had checked, maybe 20 or 30 minutes earlier the powerwalls were at like 94%

I pulled up the tesla app to see if it could show me what was going on, but all it showed was that the grid was off and the powerwalls were at 0%, which I knew was impossible. There’s no way 4 powerwalls went from 94% to 0% in 30 minutes. I went out to the garage and powercycled the powerwalls to see if that would reset them, but it didn’t make a difference. I also went out and hit the reset button on the energy gateway and that didn’t bring things back online either.

I turned the service disconnect back on to reconnect me to the grid, and luckily that powered my house again, but interestingly the tesla app was still showing the grid disconnected and the powerwalls at 0%.

The really weird thing was that the powerwalls went offline like 10 minutes after whatever loud thunder event caused the issues with the UPS and the fish tank. I could kind of understand if they went offline right when it happened, but it was a good 10 minutes after that the house lost power.

I wound up calling Tesla and he was able to connect to my system and he could see in the logs where the powerwalls dropped offline, but couldn’t really come up with any reason as to why they would do that. He also had me power cycle one of the powerwalls and at that point the app seemed to come back online and it showed that my house was being fed by the grid and the powerwalls were at 91% (which was a whole lot more believable than 0%).

He spent some time checking the logs, but didn’t really have any explanation for me. I went ahead and threw the service disconnect again and the powerwalls took over again immediately.

Definitely an odd experience.

 

[charlesj]

WOW, what an experience. Overvoltage shutting things down? But 10 min after the fact is strange indeed.
What kind of notification did you get about the grid being down on your app? Am curious as I have yet to see a good warning, especially a badge and tone on the phone.

 

[jboy210]

I wonder if the powerwall itself saw a surge and took itself offline but the gateway assumed it was online. Then once you manually cycled the powerwall the gateway figure out what was going one.

 

[BrettS]

WOW, what an experience. Overvoltage shutting things down? But 10 min after the fact is strange indeed.
What kind of notification did you get about the grid being down on your app? Am curious as I have yet to see a good warning, especially a badge and tone on the phone.

Yeah, I really don’t know. The technician seemed to think that it could have shut down due to an overload, but frankly I had nothing out of the ordinary running in my house at the time and even if the AC compressor was on the house wouldn’t have been drawing more than 3.5 or 4kW and there’s no way that could have overloaded 4 powerwalls.

Although now that I’m thinking about it, I wonder if it’s possible that two or three of the powerwalls might have gone offline at the time of the event and then maybe the AC compressor started up and that could have been enough to overload one or two powerwalls. This is just speculation, of course. I don’t even know if it’s possible for some of the powerwalls to go offline, but not all of them or even if or how I would be able to identify that they were in that state.

As far as notification on the app, I haven’t ever gotten a grid outage notification. I know there was another thread about this, but since I still don’t have my PTO I haven’t worried about it too much. If I’m still having issues with notifications after I get my PTO I’ll call support and see if they can figure out what’s going on.

 

[BrettS]

I wonder if the powerwall itself saw a surge and took itself offline but the gateway assumed it was online. Then once you manually cycled the powerwall the gateway figure out what was going one.

Yeah, I don’t know. I didn’t start trying to cycle the power on the powerwalls or reset the gateway until after everything went offline and the while house lost power. And that didn’t happen for about 10 minutes after the initial incident. But honestly at this point I still don’t know why it went offline or even why it later came back online since my initial power cycling didn’t seem to make a difference.

 

[jboy210]

Yeah, I don’t know. I didn’t start trying to cycle the power on the powerwalls or reset the gateway until after everything went offline and the while house lost power. And that didn’t happen for about 10 minutes after the initial incident. But honestly at this point I still don’t know why it went offline or even why it later came back online since my initial power cycling didn’t seem to make a difference.

There is that 300 seconds (5 minutes) event cycle that seems to be baked into these systems. I wonder if something like this had an impact on things taking a while to reset and coming back up. But, it sure is a strange one. Glad we only have one thunderstorm every 5 or 6 years. I remember living in Virginia and having them every other day in the summer.

 

[power.saver]

This was a really weird experience and I’m still not quite sure what to make of it. About an hour ago I was in the middle of a pretty bad thunderstorm. I was operating off grid at the time with my service disconnect off. Suddenly there was a *loud* crack of thunder. I even wonder if it’s possible that lightning hit my house, but whatever it was it definitely affected some things in my house. My fish tank automation system went offline and my UPS was very unhappy. The UPS stopped powering it’s devices and was just continuously screaming. Other than that my house still had power and was operating normally.

Since the UPS was making a horrible noise I went to check that out first. After trying a few things the only way I could finally get it to come back was to disconnect the power to the UPS and then disconnect it’s battery. The loud beeping finally stopped. I waited a little while, then reconnected the battery, then reconnected the UPS to power and at that point it was happy again and the devices plugged into the UPS came back online.

Then I went over to the fish tank to try to reset the controller there, but about that time the entire house lost power. Which seemed very strange because, like I said, I was already operating on the powerwalls, and the last time I had checked, maybe 20 or 30 minutes earlier the powerwalls were at like 94%

I pulled up the tesla app to see if it could show me what was going on, but all it showed was that the grid was off and the powerwalls were at 0%, which I knew was impossible. There’s no way 4 powerwalls went from 94% to 0% in 30 minutes. I went out to the garage and powercycled the powerwalls to see if that would reset them, but it didn’t make a difference. I also went out and hit the reset button on the energy gateway and that didn’t bring things back online either.

I turned the service disconnect back on to reconnect me to the grid, and luckily that powered my house again, but interestingly the tesla app was still showing the grid disconnected and the powerwalls at 0%.

The really weird thing was that the powerwalls went offline like 10 minutes after whatever loud thunder event caused the issues with the UPS and the fish tank. I could kind of understand if they went offline right when it happened, but it was a good 10 minutes after that the house lost power.

I wound up calling Tesla and he was able to connect to my system and he could see in the logs where the powerwalls dropped offline, but couldn’t really come up with any reason as to why they would do that. He also had me power cycle one of the powerwalls and at that point the app seemed to come back online and it showed that my house was being fed by the grid and the powerwalls were at 91% (which was a whole lot more believable than 0%).

He spent some time checking the logs, but didn’t really have any explanation for me. I went ahead and threw the service disconnect again and the powerwalls took over again immediately.

Definitely an odd experience.

Did you notice if your inverters were working between the time of the thunder and when you lost all power?

 

[BrettS]

Did you notice if your inverters were working between the time of the thunder and when you lost all power?

The inverters had been off since the storm started simply because there wasn’t enough sun due to the clouds and heavy rain.

 

[gpez]

@BrettS please do update us when (if?) you figure out what happened.

There is a way you can query the Powerwall for the latest "grid faults" though I'm not sure that's exactly what you're looking for.

vloschiavo/powerwall2
GET /api/system_status/grid_faults

 

[BrettS]

@BrettS please do update us when (if?) you figure out what happened.

There is a way you can query the Powerwall for the latest "grid faults" though I'm not sure that's exactly what you're looking for.

vloschiavo/powerwall2
GET /api/system_status/grid_faults

Honestly I don’t think I will figure it out. Unless it happens again (which I certainly hope it doesn’t). I’ve thought about it a little more and my best guess is what I mentioned above. The event (whatever it was) took two or three of my powerwalls offline, but since at least one remained online and powering the house I didn’t know there was a problem. Until my thermostat called for AC and the compressor starting up overloaded the one or two powerwalls that were still online and caused them to shut down.

That’s the only thing I can think of that explains why the system went offline 10 minutes after the event. But I’m not sure there’re really any to know for sure if that’s what happened or not.

If that api call just comes up with grid faults then I’m not sure how helpful it will be since I was already off grid, both at the time of the event and at the time that the whole system went offline.

 

[gpez]

Honestly I don’t think I will figure it out. Unless it happens again (which I certainly hope it doesn’t). I’ve thought about it a little more and my best guess is what I mentioned above. The event (whatever it was) took two or three of my powerwalls offline, but since at least one remained online and powering the house I didn’t know there was a problem. Until my thermostat called for AC and the compressor starting up overloaded the one or two powerwalls that were still online and caused them to shut down.

That’s the only thing I can think of that explains why the system went offline 10 minutes after the event. But I’m not sure there’re really any to know for sure if that’s what happened or not.

If that api call just comes up with grid faults then I’m not sure how helpful it will be since I was already off grid, both at the time of the event and at the time that the whole system went offline.

If you haven't already maybe escalate to T2? Obviously that shouldn't happen and you wouldn't want to be left high and dry during an actual grid outage. Ever find out if your house was actually struck?

The grid_faults API isn't terribly well documented so who knows what its actually doing, may be worth checking out just for fun.

 

[gpez]

Just noticed this API documented, that may be interesting too.

GET /api/powerwalls Use this URL to determine how many power walls you have, their serial numbers, and if they are in sync (assuming more than one powerwall).

request: curl --cacert cacert.pem https://powerwall/api/powerwalls

 

[BrettS]

So I just called support back to see if they could see anything in the logs to confirm or deny my theory or see anything that the other person might have missed.

Unfortunately it doesn’t really look like there’s much there. She said that she could see the system go offline and lose communication when the event happened, but apparently whatever happened affected the system as well as she said there was really nothing in the log from when it went offline until it was suddenly back online.

She did say that she thought that my theory was feasible, but there was no evidence to confirm or deny it.

In the end I think I’m reasonably happy even in not knowing. Obviously something major happened as evidenced by my UPS. The loads it backed up were offf and it was just continuously beeping. I had to unplug the UPS and disconnect it’s battery to get it to reset. The Powerwalls did shut off, which is less than ideal, but they also came back online fairly easily without any damage or lasting effects. I’ve never experienced anything like that event before and I certainly hope I never do again, but in any case it’s certainly not likely to be a common problem.

I spent some time walking around the house to see if I could find any evidence of a lightning strike, but I didn’t see anything. I didn’t get out a ladder and go up to the roof or anything, so it’s possible that I missed something, but it didn’t appear to be a direct strike in any case.

 

[CrazyRabbit]

you might check the data cable from the GW to the PW's, perhaps they missed the ground wire on one of the PW's.
trip the PW breakers and shut off the powerwalls before sticking your fingers behind cover plates, would want to see you crash!

 

[bob_p]

Would recommend simulating a power outage and running a more controlled off-grid test to verify everything works through the transition from grid to solar/PowerWall power.

If UPS boxes are having problems running when there isn't any grid power, the Gateway may be setting the powerline frequency too high (which is used to shutoff solar power), which can be lowered by Tesla's technical support.

Even after doing that, one of our UPS boxes refuses to work when the grid is offline - and I've taken that UPS out of service.

If the PowerWalls are shutting off when switching to off-grid power, it's possible there is something that is pulling more power than the PowerWalls can provide. With 4 PowerWalls, they should be able to sustain 20KW. But if you have a high load on the system prior to losing power (EV charging, pool pumps, electric ovens, air conditioners) and an air conditioner compressor tries to turn on a surge could be more than the PowerWalls can supply - and might cause problems.

We have a 4 PowerWalls and 3 HVAC systems. While operating without grid power, we had problems when the 2nd or 3rd air conditioner tried to start up. This didn't cause the PowerWalls to go offline - only that the compressors couldn't get enough power to startup - so they refused to run. After we installed hard starts on all 3 compressors, we haven't had any problems.

The issue the OP encountered should be researched further to determine if it can be reproduced - and then a solution found, otherwise the OP is at risk of not having backup power when the grid goes out.

 

[BrettS]

Would recommend simulating a power outage and running a more controlled off-grid test to verify everything works through the transition from grid to solar/PowerWall power.

Haha, I’m quite confident that I can operate off grid I’ve been doing an ‘extended off grid test’ essentially since my system was installed in late June. Right now the app reports that I have spent more than 1000 hours off grid. It hasn’t been one solid block, though, I have been going back on grid for short times for various reasons, such as car charging. So far my longest continuous stint off grid was 228 hours and 33 minutes. I had been off grid for several days before the issue happened on Sunday. I did go back on grid after the problem happened as part of my attempt to bring everything back online, but as pretty much as soon as I had power and things were reporting properly in the app I threw the service disconnect again and verified that the powerwalls took over with no problem, and I have been off grid again since then.

I still don’t know what it was that took my system offline, but I’m sure that it was related to that huge thunderclap and not just a problem with my system being off grid.

If UPS boxes are having problems running when there isn't any grid power, the Gateway may be setting the powerline frequency too high (which is used to shutoff solar power), which can be lowered by Tesla's technical support.

Tesla has already lowered my frequency to 62.5Hz and my UPS has no problem with that. As I said above, this was more than the UPS being unhappy with the power line quality... the UPS completely locked up and stopped providing power to the devices downstream. The only way I could get it to reset was to unplug the UPS and disconnect it’s battery.

If the PowerWalls are shutting off when switching to off-grid power, it's possible there is something that is pulling more power than the PowerWalls can provide. With 4 PowerWalls, they should be able to sustain 20KW. But if you have a high load on the system prior to losing power (EV charging, pool pumps, electric ovens, air conditioners) and an air conditioner compressor tries to turn on a surge could be more than the PowerWalls can supply - and might cause problems.

Again, the powerwalls don’t shut off when I switch off grid. It was just this one event. During my ‘extended off grid test’ I’ve regularly used my electric oven, electric drier, as well as charged my car, and of course my AC compressor has been cycling on and off the entire time and I’ve never overloaded the powerwalls. At the time of this event there were no high demand appliances on, so the power draw would have been pretty minimal. Typically in the middle of the day my house draws about 1.6kW when the AC isn’t running and about 3.4kW when it is running, so I wouldn’t have been anywhere close to overloading 4 powerwalls.

I do appreciate your thoughts and if this was the first time that I had gone off grid and had a failure it certainly would warrant further investigation and testing, but at this point I’m confident that my system can run off grid for extended periods of time and that this was a one time issue related to that thunder/lightning event.

 

[DermMD]

It is quite possible the Powerwall is fine. There's an old thread on the Tesla Power discussion board discussing how some UPS systems will not work with Powerwall. One of the contributors discusses the science. Something to do with the sine wave frequency the UPS operates on being different from the frequency of the Powerwall. Hence creating incongruence. He also suggests a solution.

OOPS, I see you responded to this idea already

 

[LadyLion]

I would like to ask what happens when one's internet goes down completely due to your internet provider failure.
Last fall during the PSPS, public safety power shutdown, during nearby California wildfires, Comcast shut down internet due to their lack of generator emergency power to keep it operating. We also lost cable and phone with Comcast and ATT cell service.
If we lose internet and/ or cell service this year during PSPS, will the app will be unavailable I assume?.
And what about the rest of the system of panels and powerwalls?
Anyone have any experience with these situations?

Edit: Could a thunderstorm have caused an internet glitch/short outage?

 

[cridinger82]

The system will still operate under the settings it was last setup for... now I do believe the app itself is cloud based so you would in fact lose access if you lost cellular. Wifi and internet is is not required. Cellular is though as a final backup.

 

[gpez]

The system will still operate under the settings it was last setup for... now I do believe the app itself is cloud based so you would in fact lose access if you lost cellular. Wifi and internet is is not required. Cellular is though as a final backup.

Even with no internet connection you can still log in to and see the status of the Powerwall on your local (wifi or ethernet) network.