My system is down due to gateway communication issue

[kairojya]

I’ve been using the system for the past 3 months while waiting for PTO. I received PTO today, 115~ days since passing my install inspection, and my system is broken. The irony . Tesla support submitted a ticket and told me to give them 3-5 days to remotely diagnose the system. They’ll either email me that they fixed it, or someone will eventually contact me regarding scheduling an onsite visit to further diagnose.

My Powerwall+, Powerwall 2, 3x340w panels, and Backup Gateway 2 were installed this June. As of midday yesterday the Powerwall+’s gateway left my wifi network, and 2 hours after that I no longer have any data from the meters showing up in the Tesla app. The lights on the Powerwalls are flashing green (not pulsating). There is a musical tone playing every so often from something in the Powerwall+’s inverter/gateway. The LED light on the inverter/gateway (bottom side in this orientation) is blinking once every 4 seconds. According to Tesla documentation this indicates that the Powerwalls cannot talk to the gateway (my BG2?). I have tried following Tesla’s troubleshooting guide to restart the system but after multiple attempts, including with Tesla Energy Support (Powerwall) on the phone, it still isn’t working. I can’t see the TEG wifi network being broadcasted either. I’ve tried connecting the Powerwall+’s inverter/gateway into my network via ethernet and it does not connect. I’ve tried pressing the reset button in my Backup Gateway 2. I’ve tried turning off everything: Powerwall toggle switches, AC disconnect blade, breakers for both Powerwalls within my BG2, and my main panel main breaker. The support agent I talked to thinks that the system can’t talk to one of the CTs but I think the only “remote” CT I have is the Neurio that is within my Powerwall+’s inverter / gateway.

Any suggestions?

Additional Info:
I created my own data logger which was monitoring the system via my Powerwall+‘s gateway local API. It took Tesla over 3 months to file for PTO with my utility company. The utility company came out last Thursday to inspect everything and sign off for PTO (I already had a smart meter). My system was never put into “Self Consumption Only“ mode by Tesla, but my advisor told me to turn the system on a few weeks ago so that Tesla could remotely configure my system in that mode until I get PTO. The system worked fine for 3 days since my utility company inspection. All they did was open up the meter panel (painted blue, to the left of my BG2 in the attachment) to make sure Tesla didn’t tie into the system up in the meter panel area. I don’t know if I have a CT anywhere in there. I have not removed its deadfront.

 

[kairojya]

Update: system is fixed.

A full power cycle of the home was needed, including opening my service disconnect within my meter panel (the portion beneath the meter in my photos in my post above). Initially I only power cycled the Tesla equipment: toggle switches on the Powerwalls, AC disconnect (blade switch), and breakers within my Backup Gateway 2 (1 for each Powerwall). When I called Tesla Support they told me to open the main panel breaker, which I assumed meant the home load panel. So I powered off all the Tesla equipment and opened my main panel (home loads) main breaker at the bottom in my photo above. I tried the same thing this morning and after 5 minutes I could still see a blinking light through the reset hole of the Backup Gateway 2’s deadfront. Then I opened my service disconnect breaker and the light within the BG2 turned off! The lights on the front of my Powerwalls were completely off too but I think that occurs in an earlier step.

Once I started closing breakers the system turned on normally. I could see the TEG SSID again, and it connected to my home’s network, and I see data again in the Tesla app.

 

[dandrewk]

Update: system is fixed.

A full power cycle of the home was needed, including opening my service disconnect within my meter panel (the portion beneath the meter in my photos in my post above). Initially I only power cycled the Tesla equipment: toggle switches on the Powerwalls, AC disconnect (blade switch), and breakers within my Backup Gateway 2 (1 for each Powerwall). When I called Tesla Support they told me to open the main panel breaker, which I assumed meant the home load panel. So I powered off all the Tesla equipment and opened my main panel (home loads) main breaker at the bottom in my photo above. I tried the same thing this morning and after 5 minutes I could still see a blinking light through the reset hole of the Backup Gateway 2’s deadfront. Then I opened my service disconnect breaker and the light within the BG2 turned off! The lights on the front of my Powerwalls were completely off too but I think that occurs in an earlier step.

Once I started closing breakers the system turned on normally. I could see the TEG SSID again, and it connected to my home’s network, and I see data again in the Tesla app.

I'm a bit unclear: When you said you turned everything off, the gateway's reset light was still blinking after five minutes? Did you take any other physical steps, or was it just a question of time before the PW/gateway to run out of juice?

 

[Fruitcake]

Update: system is fixed.

A full power cycle of the home was needed, including opening my service disconnect within my meter panel (the portion beneath the meter in my photos in my post above). Initially I only power cycled the Tesla equipment: toggle switches on the Powerwalls, AC disconnect (blade switch), and breakers within my Backup Gateway 2 (1 for each Powerwall). When I called Tesla Support they told me to open the main panel breaker, which I assumed meant the home load panel. So I powered off all the Tesla equipment and opened my main panel (home loads) main breaker at the bottom in my photo above. I tried the same thing this morning and after 5 minutes I could still see a blinking light through the reset hole of the Backup Gateway 2’s deadfront. Then I opened my service disconnect breaker and the light within the BG2 turned off! The lights on the front of my Powerwalls were completely off too but I think that occurs in an earlier step.

Once I started closing breakers the system turned on normally. I could see the TEG SSID again, and it connected to my home’s network, and I see data again in the Tesla app.

>Then I opened my service disconnect breaker and the light within the BG2 turned off!

Just to be clear, by "service disconnect breaker" do you mean the breaker in the Gateway?

Fruitcake

 

[kairojya]

I'm a bit unclear: When you said you turned everything off, the gateway's reset light was still blinking after five minutes? Did you take any other physical steps, or was it just a question of time before the PW/gateway to run out of juice?

The light was still blinking in my gateway (BG2) when I turned everything off except for my utility service breaker. When I was on the phone with Tesla support they did not tell me to specific disconnect my home from the grid (via my utility service breaker/disconnect). They only told me to turn off the things I already did and to flip/open/turn-off the main breaker in my home’s main load panel. The agent had me wait 5 minutes and then I powered everything back on.

The next day I repeated the same procedure but this time after 5 minutes I looked into the gateway’s reset hole and saw the light in there was still blinking. At this point I assumed that the gateway was still powered on by either my grid connection or the Powerwalls. Since I didn’t have any other buttons/switches to use to further turn off the Powerwalls (lol), I tried flipping my utility service disconnect which isolated my home from the grid. That did the trick for me, the light in the gateway went out. I waited 5-6 minutes (less than 10) and powered things back on in the reverse order and my system started working again.

>Then I opened my service disconnect breaker and the light within the BG2 turned off!

Just to be clear, by "service disconnect breaker" do you mean the breaker in the Gateway?

Fruitcake

No, I am referring to a breaker in my meter panel which disconnects my home from the grid. I should have called it “utility service disconnect” maybe?

I did also flip/open/turned-off the breakers within my gateway. I should have included a pic of my gateway with the cover open but deadfront on so that the labels were visible. I should have also included a picture of my meter panel cover opened to show the utility service breaker/disconnect.

 

[dandrewk]

The light was still blinking in my gateway (BG2) when I turned everything off except for my utility service breaker. When I was on the phone with Tesla support they did not tell me to specific disconnect my home from the grid (via my utility service breaker/disconnect). They only told me to turn off the things I already did and to flip/open/turn-off the main breaker in my home’s main load panel. The agent had me wait 5 minutes and then I powered everything back on.

The next day I repeated the same procedure but this time after 5 minutes I looked into the gateway’s reset hole and saw the light in there was still blinking. At this point I assumed that the gateway was still powered on by either my grid connection or the Powerwalls. Since I didn’t have any other buttons/switches to use to further turn off the Powerwalls (lol), I tried flipping my utility service disconnect which isolated my home from the grid. That did the trick for me, the light in the gateway went out. I waited 5-6 minutes (less than 10) and powered things back on in the reverse order and my system started working again.

I mentioned this in the other thread here. The capacitance of the system will keep the gateway and PW powered for several minutes after you cut off all power to they system. For me, it took about 9 minutes for the PW and gateway lights to turn off.

 

[kairojya]

I mentioned this in the other thread here. The capacitance of the system will keep the gateway and PW powered for several minutes after you cut off all power to they system. For me, it took about 9 minutes for the PW and gateway lights to turn off.

Thanks. It may have been a coincidence that my gateway finally powered off after I disconnected my home from the grid. Perhaps I didn’t wait long enough during my prior ”full” system power cycle previously. If the system acts up again I’ll do the steps more slowly and see what happens. I would have thought the Tesla support agent would have told me to wait until I see the power light in the gateway turn off, rather than “lets wait 5 minutes and power everything back up”.

 

[slcasner]

The next day I repeated the same procedure but this time after 5 minutes I looked into the gateway’s reset hole and saw the light in there was still blinking. At this point I assumed that the gateway was still powered on by either my grid connection or the Powerwalls. Since I didn’t have any other buttons/switches to use to further turn off the Powerwalls (lol), I tried flipping my utility service disconnect which isolated my home from the grid. That did the trick for me, the light in the gateway went out. I waited 5-6 minutes (less than 10) and powered things back on in the reverse order and my system started working again.

Yes, this makes sense. There is a direct connection from the meter panel to the gateway and then from the gateway to the main load panel. So flipping the breaker in the main load panel does not help to isolate the gateway from the grid. The gateway is interposed between the meter and the main load panel so that the relay in the gateway can open to allow your main load panel to be energized when the grid is down without sending power back to the grid. But even when the relay is open, the gateway is still maintaining a connection to the grid to sense when the grid comes back up. It also has to be powering itself from that connection otherwise it would not be able to close the relay again and let power come to your main load panel if the Powerwalls were off.

 

[Fruitcake]

Yes, this makes sense. There is a direct connection from the meter panel to the gateway and then from the gateway to the main load panel. So flipping the breaker in the main load panel does not help to isolate the gateway from the grid. The gateway is interposed between the meter and the main load panel so that the relay in the gateway can open to allow your main load panel to be energized when the grid is down without sending power back to the grid. But even when the relay is open, the gateway is still maintaining a connection to the grid to sense when the grid comes back up. It also has to be powering itself from that connection otherwise it would not be able to close the relay again and let power come to your main load panel if the Powerwalls were off.

This makes sense. I even wondered this as Tesla's troglodyte first-level support had me go through the same procedure - turn off the PWs, the PV, the main breaker to the house - but not touch the Gateway or anything northbound of that. I will remember this the next time their vacuous support representatives ask me to go through their erroneously-cheerful-sounding "whole home shutdown."

I looked at my utility meter (which is now digital, apparently of the variety required for home PV systems) and I don't see any obvious switch to flip or lever to pull. Any thoughts?

Fruitcake

 

[slcasner]

I looked at my utility meter (which is now digital, apparently of the variety required for home PV systems) and I don't see any obvious switch to flip or lever to pull. Any thoughts?

The meter itself does not include any means to disconnect power (other than pulling the meter out of its socket, which requires breaking the utility's seal). The service disconnect breaker that @kairojya referred to is a breaker in the same box with the meter. @Fruitcake, if your meter is on a small box all by itself that does not contain a breaker then your Gateway would have a main breaker installed to act as the main service disconnect. Opening that breaker should remove power from the control circuit in the Gateway as well.

Now, for any home with the Tesla Backup Switch installed and controlled by the Powerwall+ instead of using a Gateway, then the relay for grid isolation is in the Backup Switch installed as an adapter ring between the meter and its socket. If somehow the control circuit in the Backup Switch needs a power cycle then you have a problem because there is no way to interrupt power upstream from the Backup Switch.

 

[kairojya]

Thanks for the insight into that. Not being able to restart the backup switch makes me somewhat glad I have a gateway. Though it would be easier if I had my utility service disconnect breaker within the gateway rather than fiddling with my meter panel. I will update this thread with a picture of my meter panel with its cover open, as well as my gateway with its cover open but deadfront in place. My gateway photo above has the dead front removed so it isn’t immediately obvious where the service disconnect breaker would be. The short (5~ min) gateway install overview video on Tesla’s website explains that part though.

So the gateway (Backup Gateway 2 in my case) is powered by either the grid or Powerwalls as it effectively sits between those 2 power sources? I’ll add that with my system configuration, the gateway (BG2) does not broadcast its SSID as the Powerwall+’s gateway/inverter acts as the main system controller. Only its SSID is broadcasted during normal operation. My install auditor and multiple Tesla support agents have confirmed this. It had me very confused after my install in June as I was trying to figure out how to connect to the gateway (BG2) per Tesla’s documentation online.

 

[jboy210]

I once had a similar glitch once. I followed the procedure the installers showed me and went backward from the inverters to the gateway box shutting things off. Waited a few minutes and went forward from the PG&E box (which I left on) to the solar.

 

[slcasner]

Thanks for the insight into that. Not being able to restart the backup switch makes me somewhat glad I have a gateway.

What I said might be wrong. I forgot about the white pushbutton switch on the bottom of the Backup Switch near the connector. That pushbutton might remove power from the circuit in the Backup Switch.

So the gateway (Backup Gateway 2 in my case) is powered by either the grid or Powerwalls as it effectively sits between those 2 power sources?

Right. If you think about it, that has to be the case. If the grid is down, the gateway needs to be powered by the batteries so it can keep operating. And since we don't want the house to be without power if the batteries have failed or been turned off but the grid is still up, the gateway needs to be powered by the grid so it can keep the relay closed to allow the grid power to reach the house.

I’ll add that with my system configuration, the gateway (BG2) does not broadcast its SSID as the Powerwall+’s gateway/inverter acts as the main system controller.

In an installation like yours, the controller in the Powerwall+ is responsible for telling the gateway to open or close the relay. I don't know how power is connected among the battery, inverter and controller parts of the Powerwall+, so if the grid is down and the battery switch on the Powerwall+ is turned off (also for any other Powerwalls) then the house would be without power. If the grid then comes back up, the gateway could be powered from the grid, but what about the controller in the Powerwall+? Would the gateway be able to close the relay on its own? I don't know, that is an interesting question. I saw a comment in one of these threads recently that Tesla may be changing the firmware to make the gateway be the main system controller in this configuration, but I haven't seen anything definite.

My own Solar Roof installation is supposed to be done over the next two weeks and I will also have a Powerwall+ and Gateway 2, so I may have an opportunity to learn more.

 

[Fruitcake]

The meter itself does not include any means to disconnect power (other than pulling the meter out of its socket, which requires breaking the utility's seal). The service disconnect breaker that @kairojya referred to is a breaker in the same box with the meter. @Fruitcake, if your meter is on a small box all by itself that does not contain a breaker then your Gateway would have a main breaker installed to act as the main service disconnect. Opening that breaker should remove power from the control circuit in the Gateway as well.

Now, for any home with the Tesla Backup Switch installed and controlled by the Powerwall+ instead of using a Gateway, then the relay for grid isolation is in the Backup Switch installed as an adapter ring between the meter and its socket. If somehow the control circuit in the Backup Switch needs a power cycle then you have a problem because there is no way to interrupt power upstream from the Backup Switch.

Yes, my Tesla Gateway has a breaker. In fact, I recall the installers opening it several times to test that the PWs would backup the loads in my home - including a soft-start AC. I am guessing I should open that breaker the next time I need to perform a "whole home shutdown"

Fruity

 

[kairojya]

What I said might be wrong. I forgot about the white pushbutton switch on the bottom of the Backup Switch near the connector. That pushbutton might remove power from the circuit in the Backup Switch.

Right. If you think about it, that has to be the case. If the grid is down, the gateway needs to be powered by the batteries so it can keep operating. And since we don't want the house to be without power if the batteries have failed or been turned off but the grid is still up, the gateway needs to be powered by the grid so it can keep the relay closed to allow the grid power to reach the house.

In an installation like yours, the controller in the Powerwall+ is responsible for telling the gateway to open or close the relay. I don't know how power is connected among the battery, inverter and controller parts of the Powerwall+, so if the grid is down and the battery switch on the Powerwall+ is turned off (also for any other Powerwalls) then the house would be without power. If the grid then comes back up, the gateway could be powered from the grid, but what about the controller in the Powerwall+? Would the gateway be able to close the relay on its own? I don't know, that is an interesting question. I saw a comment in one of these threads recently that Tesla may be changing the firmware to make the gateway be the main system controller in this configuration, but I haven't seen anything definite.

My own Solar Roof installation is supposed to be done over the next two weeks and I will also have a Powerwall+ and Gateway 2, so I may have an opportunity to learn more.

That all makes sense. But now I’m worried about the grid being available but my home having no power due to either the gateway and/or Powerwall+ controller failing! I don’t know enough about these systems to know if they could be designed to fail in a state that would allow the home to still be powered by the grid. It would be terrible waiting on parts and on-site service from Tesla with no power to the home for (weeks, months?) while the grid was available the whole time.

In the failure case that I started this thread under, all of my Tesla equipment was inoperable but I still had power to the home at least with the grid being available.

 

[slcasner]

That all makes sense. But now I’m worried about the grid being available but my home having no power due to either the gateway and/or Powerwall+ controller failing! I don’t know enough about these systems to know if they could be designed to fail in a state that would allow the home to still be powered by the grid. It would be terrible waiting on parts and on-site service from Tesla with no power to the home for (weeks, months?) while the grid was available the whole time.

That's a valid point. In my old PV system that's just been removed before installation of the Solar Roof there was a pair of circuit breakers ganged together with an arm that turned one off if you turned the other one on. These allowed bypassing the inverter so the backed-up circuits could be powered if the inverter failed.

In the failure case that I started this thread under, all of my Tesla equipment was inoperable but I still had power to the home at least with the grid being available.

From your description it sounds like the Gateway was still functioning, it just was not getting correct inputs.

 

[Fruitcake]

That all makes sense. But now I’m worried about the grid being available but my home having no power due to either the gateway and/or Powerwall+ controller failing! I don’t know enough about these systems to know if they could be designed to fail in a state that would allow the home to still be powered by the grid. It would be terrible waiting on parts and on-site service from Tesla with no power to the home for (weeks, months?) while the grid was available the whole time.

In the failure case that I started this thread under, all of my Tesla equipment was inoperable but I still had power to the home at least with the grid being available.

> It would be terrible waiting on parts and on-site service from Tesla with no power to the home for (weeks, months?) while the grid was available the whole time

I raised this exact hypothetical scenario with Tesla's "support" personnel while being made to wait a month for onsite repairs. (I wasn't without power, hence the term "hypothetical.") I posed this scenario to at least 3 of 4 people. Each of them stammered incoherently and non-specifically, something to the effect of "we wouldn't let you wait for that long" without offering any concrete time horizons or plans of action.

Worse come to worst, in a situation like that (e.g. hard down with no power and no Tesla support in sight) you could probably get your local utility to bypass all of the Tesla equipment, basically returning your home to the same 100% grid-dependent state you spent all that money to avoid in the first place.

Tesla seems to be occupying some sort of quasi-utility status, in that they have the scope and obligations of a utility company, but without the regulatory apparatus to force them to meet customary & reasonable SLAs for their customers.

I'm (barely) tolerating this current issue for now. The next time there's a failure and Tesla makes me wait a month or more for repair, I'm going to perform a rip/replace of their hardware with that of a local solar provider and bill Tesla for the cost.

Fruitcake

 

[yblaser]

> It would be terrible waiting on parts and on-site service from Tesla with no power to the home for (weeks, months?) while the grid was available the whole time

I raised this exact hypothetical scenario with Tesla's "support" personnel while being made to wait a month for onsite repairs. (I wasn't without power, hence the term "hypothetical.") I posed this scenario to at least 3 of 4 people. Each of them stammered incoherently and non-specifically, something to the effect of "we wouldn't let you wait for that long" without offering any concrete time horizons or plans of action.

Worse come to worst, in a situation like that (e.g. hard down with no power and no Tesla support in sight) you could probably get your local utility to bypass all of the Tesla equipment, basically returning your home to the same 100% grid-dependent state you spent all that money to avoid in the first place.

Tesla seems to be occupying some sort of quasi-utility status, in that they have the scope and obligations of a utility company, but without the regulatory apparatus to force them to meet customary & reasonable SLAs for their customers.

I'm (barely) tolerating this current issue for now. The next time there's a failure and Tesla makes me wait a month or more for repair, I'm going to perform a rip/replace of their hardware with that of a local solar provider and bill Tesla for the cost.

Fruitcake

I don't know about your utility but here they aren't going to touch anything on the house side of the meter.

I would hope Tesla would address something like in this hypothetical situation immediately. Someone posted recently that they had a roof leak due to the solar install and Tesla came out next day. I would assume that the liability or perceived liability would spur an accelerated response. Support is not going to know how they would respond to a situation that they have probably never experienced.

 

[Fruitcake]

I don't know about your utility but here they aren't going to touch anything on the house side of the meter.

I would hope Tesla would address something like in this hypothetical situation immediately. Someone posted recently that they had a roof leak due to the solar install and Tesla came out next day. I would assume that the liability or perceived liability would spur an accelerated response. Support is not going to know how they would respond to a situation that they have probably never experienced.

>I don't know about your utility but here they aren't going to touch anything on the house side of the meter.

Perhaps a local electrician, then.

Regardless (and to use your roof leak analogy) it seems that you have to suffer some form of tangible damage before Tesla responds in a prompt manner. Because once you suffer measurable, financial damages, you can take Tesla to court.

Fruicake

 

[Mark-R-1]

That all makes sense. But now I’m worried about the grid being available but my home having no power due to either the gateway and/or Powerwall+ controller failing! I don’t know enough about these systems to know if they could be designed to fail in a state that would allow the home to still be powered by the grid. It would be terrible waiting on parts and on-site service from Tesla with no power to the home for (weeks, months?) while the grid was available the whole time.

In the failure case that I started this thread under, all of my Tesla equipment was inoperable but I still had power to the home at least with the grid being available.

There is a way even if the gateway "trips", there is a switch under the "do not touch sticker" that can trip (i found out the hard way!) and it was half way between grid and off grid, Tesla support instructed me to set it back to grid and power from the grid came back. Our gateway is still dead but at least we could get the power back on!