Daily reset/shutdown of my Powerwall+ and/or Inverter after coolant top-off?

[Bitslizer]

Yes that is strange. My came back at close to 100%. Though today is cloudy, so that is also causing an effect

monitor installer mode system page when you push the button. share your screenshot to see if it matches mine. flip the breaker too?

A bit disappointed we aren't able to nail this down yet heh....

any chance you can open up your gateway and confirm the button wiring? is it plugged into the gateway or leading back to the PW+ inverter?

 

[Bitslizer]

What I find interesting about your photos is that your system cones back at full power. Mine comes back every day at about 1 kw lower output than before the shutdown. View attachment 694976View attachment 694977

I suspect 1 of your string aren't starting backup. Get a screenshot of the installer mode system page with the string status before and after the dip

 

[bryan995]

monitor installer mode system page when you push the button. share your screenshot to see if it matches mine. flip the breaker too?

A bit disappointed we aren't able to nail this down yet heh....

any chance you can open up your gateway and confirm the button wiring? is it plugged into the gateway or leading back to the PW+ inverter?

The button does nothing no change in the installer mode screen. Not change to production. No clicks.

The estop cable comes in through the wall and connects to the top right via the AUX plug. It does not directly continue onto the inverter or PWs... eek !

 

[Bitslizer]

The button does nothing no change in the installer mode screen. Not change to production. No clicks.

The estop cable comes in through the wall and connects to the top right via the AUX plug. It does not directly continue onto the inverter or PWs... eek !

they may have wired it the old way which "may" not be correct since the roll out of the PW+ I can take a picture of where the cable plug in on my pw+ inverter.

The TEG used to be the BRAIN of the entire Tesla system, but that's now taken over by the PW+Inverter. Button is telling the TEG to issue the rapid shutdown command which I suspect is not getting communicated back to the Inverter

So i guess that MAY explain why you are STILL dropping out as your pushing the button didn't cause a rapid shutdown.

You can TRY if the system DISABLED/ENABLED or the Power Flow STOP SYSTEM may also reset the 24 hours timer. I was saying try the E-stop button as that's what I saw the installer did while fixing my wiring that as a side effect seem to have caused my dip and set the dip time. the other clue was the 24 hours + 10 mins.

 

[kairojya]

I used the e-stop switch button this evening. Will see how solar production goes tomorrow.

Is there a way to tell that the inverter is off/rebooting via the local API? I checked the /api/status endpoint before I used the e-stop today and it reports that my system has been up since the last update 21.24.1. I figured if the inverter is rebooting daily, these values should be different right? Maybe this is the entire system and our problem is just with the inverter.

"start_time":"2021-08-04 02:19:10 +0800"
"up_time_seconds":"162h17m35.907119541s"

I checked the endpoint for 10 minutes after I pressed the e-stop switch, before turning it to reset it. The start_time did not change, and the up_time_seconds kept increasing. I saw the same thing after turning the e-stop to reset it (i.e. turn the system back on).


Edit: somehow I missed that there were 2 new pages of back and forth on this topic. My e-stop is mounted right beside my PW+ and I can easily see where the wires enter the inverter and where they connect. Let me know if you’d like pictures.

 

[Bitslizer]

I used the e-stop switch button this evening. Will see how solar production goes tomorrow.

Is there a way to tell that the inverter is off/rebooting via the local API? I checked the /api/status endpoint before I used the e-stop today and it reports that my system has been up since the last update 21.24.1. I figured if the inverter is rebooting daily, these values should be different right? Maybe this is the entire system and our problem is just with the inverter.

"start_time":"2021-08-04 02:19:10 +0800"
"up_time_seconds":"162h17m35.907119541s"

I checked the endpoint for 10 minutes after I pressed the e-stop switch, before turning it to reset it. The start_time did not change, and the up_time_seconds kept increasing. I saw the same thing after turning the e-stop to reset it (i.e. turn the system back on).


Edit: somehow I missed that there were 2 new pages of back and forth on this topic. My e-stop is mounted right beside my PW+ and I can easily see where the wires enter the inverter and where they connect. Let me know if you’d like pictures.

That seem to be the power on time (mine matches when the installer initially turn on the inverters). I suspect that number won't change unless the inverter totally lost AC power

What about api/system_status
The date time for Last toggle? Is that closed to when you hit the button or off by about 1 hour?

 

[darryllee]

What about api/system_status
The date time for Last toggle? Is that closed to when you hit the button or off by about 1 hour?

Oh wow! I never looked at api/system_status !

So yes, I see this:

  "last_toggle_timestamp": "2021-08-10T12:29:08.318181587-07:00",

That matches up with my outage this afternoon. But it's around the same time when I logged into the system in Installer mode to try and see if there were any alerts.

So I just repeated the process, and yup, it's literally when I toggled the switch. So... not helpful in telling us what's going on with the INVERTER.

Ok, time to try the RESET...

And...nothing. I heard a buzz and clunk coming from the Powerwall+ that sounded very similar to the sound when I toggle the switch. But no change in last_toggle_timestamp, no change in power flowing from the Powerwall+ to my house...

Oh, before that I also clicked on [STOP SYSTEM] in the web UI and also heard the buzz (shutdown?). These API endpoints all went dead:

• system_status
• system_status/soe
• meters/aggregates

But good old status, which doesn't require authentication at all, continued chugging away, and indeed, the start_time didn't change, and uptime continued to increment. Also system/update/status which monitors software updates kept recording info. And sitemaster did show this:

{"status":"StatusDown","running":false,"connected_to_tesla":true,"power_supply_mode":false,"can_reboot":""}

And the devices/vitals continued to provide its logging.

So then, I still don't know how to completely restart/shut down the system to the point where status will show a new start_time and the uptime resets, if that's even possible.

 

[darryllee]

Is there a way to tell that the inverter is off/rebooting via the local API? I checked the /api/status endpoint before I used the e-stop today and it reports that my system has been up since the last update 21.24.1. I figured if the inverter is rebooting daily, these values should be different right? Maybe this is the entire system and our problem is just with the inverter.

"start_time":"2021-08-04 02:19:10 +0800"
"up_time_seconds":"162h17m35.907119541s"

I checked the endpoint for 10 minutes after I pressed the e-stop switch, before turning it to reset it. The start_time did not change, and the up_time_seconds kept increasing. I saw the same thing after turning the e-stop to reset it (i.e. turn the system back on).

Huh. My I just noticed my start_time is very close to yours. I guess we were part of the same rollout group:

"start_time":"2021-08-04 02:42:10 +0800"

 

[kairojya]

I’ve noticed my start_time and up_time_seconds reset each time my system has a firmware update.

Like @darryllee sees, my last_toggle_timestamp did not change when I used the e-stop/RSD switch. It shows the same value as my start_time (from /status/):

"last_toggle_timestamp":"2021-08-04T02:27:38.712411346-04:00"

I’ll check for my solar production dropout around 10:10 AM EST today and report back. The sitemaster endpoint didn’t change either when I tried the e-stop/RSD yesterday. I’ve not used the e-stop/RSD before.

I typically just turn the power switch of both of my Powerwalls off as a way to turn the system off without having to go outside (my setup is in my garage). I’ve seen that this action results in the same behavior as using my A/C disconnect blade thing outside (though I understand this is a software power off where the AC disconnect is a hardware one). I still don’t fully understand what actually turns off when I try various methods, as the local API is still reachable in all scenarios that I’ve tried.

 

[Bitslizer]

I think there's a power line between the pw and the inverter keeping it alive even with the various breaker and disconnect being thrown

 

[kairojya]

Oh wow! I never looked at api/system_status !

Off topic: from what I’ve read on here, system_status is thought to be good for monitoring degradation over time. Look for ‘nominal_full_pack_energy’. You’ll find it at the system level, and then listed for each Powerwall. At least this is what I have been tracking. I’ve noticed the value jump (350W~ per each of my 2 Powerwalls) after what I’ll call a full system restart like what happens around the time I’ve seen my firmware version change. I assume there’s some kind of recalculation that occurs upon system start which differs from the ongoing calculation while the system is up.

 

[Bitslizer]

So what started as a beautiful sunny morning turn into a cloudy and now thunderstorms, given that I was expecting the dip to occur around 9:30...9:45ish this really doesn't help. I'm monitoring the installer system screen and didn't notice it going into "wait for solar" state or the 6 steps self-test so i guess that's a positive. My panels maybe generating 50w total right now?

Side note - Around 9am (before the expected daily dip) the installer system dark gray top bar did flashed red for less than a second, but I didn't see alert or error message and it was only a brief flash less than a second so no clue what it was.

 

[kairojya]

I didn’t have a drop to 0 today! Even though I couldn’t tell if my e-stop use last night had any effect. I was polling the sitemaster endpoint around the time I expected it to reset based on yesterday’s reset. I didn’t see any state change. I’ll poll it this evening around the time I used the e-stop yesterday evening.

I have not been monitoring the installer page.

 

[Bitslizer]

I didn’t have a drop to 0 today! Even though I couldn’t tell if my e-stop use last night had any effect. I was polling the sitemaster endpoint around the time I expected it to reset based on yesterday’s reset. I didn’t see any state change. I’ll poll it this evening around the time I used the e-stop yesterday evening.

I have not been monitoring the installer page.

great! if you confirm a state change this evening, that will help to confirm my theory its a 24 hours timer of some sort... last self test maybe?

 

[Bitslizer]

at 10:22 i caught my screen going into wait for solar, but production been stuck near 0 for a while due to the thunderstorm, and this is way later than I expected the daily dip to occur, so who knows...... the string that I have problem not starting up in the morning is stuck at 370v while the other 2 string is less than 8v.....

edit --

I did have a dip around 10:15 yesterday, that "may" have been associated with me doing a disabled/enabled to get the problematic string producing again, i vaguely remember this but not 100% sure

10:30, its doing the 6 step self test

10:33 its done... being so cloudy/rainy, my problem string is now "Not connected" again, this mimic my cloudy morning failed to wake up issue.

edit 2 --

at 10:50 I will do a disabled/enabled to try to wake up the not connected string... documenting it here so I don't have to "vaguely remember" lol

edit 3 --

pretty much on the dot 10 min (11am) it went into self-test mode, finish around 11:03, didn't wake up my string but that's a different story

 

[Bitslizer]

at 10:22 i caught my screen going into wait for solar, but production been stuck near 0 for a while due to the thunderstorm, and this is way later than I expected the daily dip to occur, so who knows...... the string that I have problem not starting up in the morning is stuck at 370v while the other 2 string is less than 8v.....

edit --

I did have a dip around 10:15 yesterday, that "may" have been associated with me doing a disabled/enabled to get the problematic string producing again, i vaguely remember this but not 100% sure

10:30, its doing the 6 step self test

10:33 its done... being so cloudy/rainy, my problem string is now "Not connected" again, this mimic my cloudy morning failed to wake up issue.

edit 2 --

at 10:50 I will do a disabled/enabled to try to wake up the not connected string... documenting it here so I don't have to "vaguely remember" lol

edit 3 --

pretty much on the dot 10 min (11am) it went into self-test mode, finish around 11:03, didn't wake up my string but that's a different story

disabled/enabled again and up and running at 11:47am with that problem string woked up now that its brighter out although still cloudy, 360v/3.1a ish 11x340w string

 

[darryllee]

So I finally found something of interest: api/vitals data shows this change from 12:25PM PDT to 12:30PM PDT:

PVAC_State: PVAC_Active
...
PVAC_InvState: INV_Grid_Connected
PVAC_PvState_A: PV_Active
PVAC_PvState_B: PV_Active
PVAC_PvState_C: PV_Active
PVAC_PvState_D: RPV_Active_Parallel

..
PVS_State: PVS_Active
PVS_SelfTestState: PVS_SelfTestOff

To:

PVAC_State: PVAC_Standby
...
PVAC_InvState: INV_Disconnected
PVAC_PvState_A: PV_Disabled
PVAC_PvState_B: PV_Disabled
PVAC_PvState_C: PV_Disabled
PVAC_PvState_D: PV_Disabled
...
PVS_State: PVS_SelfTestIsolation
PVS_SelfTestState: PVS_SelfTesting

And this line is only present during the downtime:
PVAC_a014_PVS_disabled_relay

So ayup. As everyone suspects, it's running a test. The question is, how do we make it stop, or schedule it for not during the middle of the day during production!?

 

[Bitslizer]

So I finally found something of interest: api/vitals data shows this change from 12:25PM PDT to 12:30PM PDT:

PVAC_State: PVAC_Active
...
PVAC_InvState: INV_Grid_Connected
PVAC_PvState_A: PV_Active
PVAC_PvState_B: PV_Active
PVAC_PvState_C: PV_Active
PVAC_PvState_D: RPV_Active_Parallel

..
PVS_State: PVS_Active
PVS_SelfTestState: PVS_SelfTestOff

To:

PVAC_State: PVAC_Standby
...
PVAC_InvState: INV_Disconnected
PVAC_PvState_A: PV_Disabled
PVAC_PvState_B: PV_Disabled
PVAC_PvState_C: PV_Disabled
PVAC_PvState_D: PV_Disabled
...
PVS_State: PVS_SelfTestIsolation
PVS_SelfTestState: PVS_SelfTesting

And this line is only present during the downtime:
PVAC_a014_PVS_disabled_relay

So ayup. As everyone suspects, it's running a test. The question is, how do we make it stop, or schedule it for not during the middle of the day during production!?

So you pressed the web page stop button yesterday and it still doing the self test today?

1) when did you stop system?
2) when was yesterday's daily self test?

 

[abek]

So I finally found something of interest: api/vitals data shows this change from 12:25PM PDT to 12:30PM PDT:

PVAC_State: PVAC_Active
...
PVAC_InvState: INV_Grid_Connected
PVAC_PvState_A: PV_Active
PVAC_PvState_B: PV_Active
PVAC_PvState_C: PV_Active
PVAC_PvState_D: RPV_Active_Parallel

..
PVS_State: PVS_Active
PVS_SelfTestState: PVS_SelfTestOff

To:

PVAC_State: PVAC_Standby
...
PVAC_InvState: INV_Disconnected
PVAC_PvState_A: PV_Disabled
PVAC_PvState_B: PV_Disabled
PVAC_PvState_C: PV_Disabled
PVAC_PvState_D: PV_Disabled
...
PVS_State: PVS_SelfTestIsolation
PVS_SelfTestState: PVS_SelfTesting

And this line is only present during the downtime:
PVAC_a014_PVS_disabled_relay

So ayup. As everyone suspects, it's running a test. The question is, how do we make it stop, or schedule it for not during the middle of the day during production!?

Interesting, indeed. And a good question. A daily self-test would likely be a simple configuration setting somewhere in the system. But where to find it, and in which system - Tesla’s servers or or in the installed equipment?

 

[Bitslizer]

So I finally found something of interest: api/vitals data shows this change from 12:25PM PDT to 12:30PM PDT:

PVAC_State: PVAC_Active
...
PVAC_InvState: INV_Grid_Connected
PVAC_PvState_A: PV_Active
PVAC_PvState_B: PV_Active
PVAC_PvState_C: PV_Active
PVAC_PvState_D: RPV_Active_Parallel

..
PVS_State: PVS_Active
PVS_SelfTestState: PVS_SelfTestOff

To:

PVAC_State: PVAC_Standby
...
PVAC_InvState: INV_Disconnected
PVAC_PvState_A: PV_Disabled
PVAC_PvState_B: PV_Disabled
PVAC_PvState_C: PV_Disabled
PVAC_PvState_D: PV_Disabled
...
PVS_State: PVS_SelfTestIsolation
PVS_SelfTestState: PVS_SelfTesting

And this line is only present during the downtime:
PVAC_a014_PVS_disabled_relay

So ayup. As everyone suspects, it's running a test. The question is, how do we make it stop, or schedule it for not during the middle of the day during production!?

Big red button. Do you see this in your pw+ inverter? When jumped it's inverter on/normal operation. If you open the jumper, it should trigger a e-stop like pushing the big red button. My button is wired to this. Picture was before they wire the button