Weird Gateway behavior after recent firmware rollback

[RunDeep]

I have a three PW/11.1 kW solar install. I noticed on Saturday that I had about six backup events, Sunday I had 24 and today before 9am I had over 100, all very short (<1 min). The PWs were also not discharging like they should, instead power was coming from the grid with the PWs well above the 10% self-powered mode limit (see my other post). I called Tesla support and they escalated it to Tier 2; I suggested resetting the TEG and they said it was worth a try. Also, the firmware on the TEG was downgraded late Friday/early Saturday. When I went to reset the TEG, I found that the grid contactor was cycling frequently, about every 20 seconds. I decided that was a bad thing and opened the breaker to the TEG in the electrical panel instead. At that point the contactor stopped cycling as expected.

I called the installer (Cobalt in Mountain View) and they sent a tech out to take a look. We tried resetting the TEG but the main contactor went right back to cycling. Nothing leapt out at him so he called Tesla. They consulted with tier 2 and suggested contacting PG&E since they saw a lot of errors related to the grid. They also saw some neutral connection errors but all the neutral connections in the Gateway looked solid. I actually called Tesla twice, the first guy was reading the errors he was seeing out loud and there were a bunch of weird sounding ones; gonna follow up if I get the call back I was promised...

I put a Fluke meter with a Max/Min function on the grid side of a breaker in the panel and watched the bus freq for a few hours after isolating the house from the grid. I saw a max of 63.993 Hz and min of 59.961 Hz; this was isolated from the PWs so it wasn't the TEG shutting off the solar. I monitored the grid voltage in the afternoon and it was pretty normal looking. PG&E sent a guy out a few hours after I called, he put a dummy load on the input lines and the voltages didn't sag significantly so he said it looked good from PG&E's side. He didn't think the freq deviation was an issue and he didn't see any issues with the neutrals. This is in an older residential area and a neighbor about 10 houses away with PWs isn't seeing any events.

Some open questions I have/looking for opinions:

• Is this related to the firmware downgrade since the system had been running great before the downgrade? Also, during the Cobalt tech's call, Tesla said there was something going on related to the firmware downgrade and they had scheduled a firmware update for me. api/system/update/status is now showing state: "/update_staged" which is a change from earlier today.
• Why is the contactor cycling so frequently, I thought it was supposed to monitor the grid for five minutes before connecting back to the grid?
• Why were the PWs not discharging in self-consumption mode, somehow related to the contactor cycling? They are discharging normally with the grid disconnected.
• I have 11.1 kW of SunPower panels with micro-inverters. I thought when the PW was nearly full, the TEG would raise the bus frequency to 6x Hz (I requested they lower it previously and it appears to be around 61.5 Hz though the ticket is still open) and all the optimizers would shutdown. I normally peak in the high 8/low 9 kWs and I spent a good part of today with ~3 kW showing on the app and TEG web page and the PWs at 98%. Shouldn't it be max or nothing?

 

[bemyax]

On your last bullet point, but not the focus of this thread. I have APSystems microinverters. The spec sheet the installers gave me says they are supposed to kick off, dead, at 62.5 Hz. However, they apparently have a newer firmware. When I went off grid at 90%, but still charging on a sunny day, the Powerwall gradually increased the frequency. The microinverters were gradually reducing their output. By the time I got to 62.5 Hz, they were off. On a different day I waited until the PW was full to try it. The solar flow did instantly stop. I don't know what frequency was created because I was distracted by the fact that the high frequency crashed the HVAC computer. Anyway, carry on with the original topic.

 

[BGbreeder]

@RunDeep any update on your issue?

All the best,

BG

 

[Vines]

I didn't see this earlier, but hopefully it's all been settled now?

Let us know so others may learn from your experience. It is pretty strange that the PGE grid was not frequency-stable, as it sounded from your post there.

The new UL1741 Rule 21 SA allows "ride through" so even a brief but total shutdown of the grid will allow a PV system to restart in less than 5 seconds.

 

[RunDeep]

Sorry, I replied in another thread: Gateway waiting time to reconnect to grid Turns out that the gross power was fine, there was arcing at one of the meter contacts/blades to the associated socket jaw in the panel which was causing high frequency noise (my guess) that threw off the PW. The linked post has more details. As an aside, SquareD doesn't sell replacement jaws, you have to go up to the whole meter base for about $700. Definitely filing a claim with PG&E.