Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

PW2 Failure: Observations and Timeline

This site may earn commission on affiliate links.
I'm making this post to help other PW2 customers who might experience a similar failure with their PW2. The content below are all statements of fact. No ranting or complaining has been included. The one personal comment I'd like to make is this: I was well aware of the customer service reputation and resolution times when I bought my system in 2017. I just never thought it would happen to me....on to the story.

System Description: 8.7kW DC, Delta Soliva 7.6 inverter, Tesla Backup Gateway gen 1, single PW2, partial home backup. This was a Solar City/Tesla Energy installation.

Runtime: Continuous operation with no problems from January 2018 to August 21, 2020. Operated in solar self-consumption mode. Winter reserve at 33% SOC, summer reserve 70% SOC. Lifetime PW throughput: ~6.6 MWh

Failure: On Aug 21, 2020 at 4:45 PM, an abrupt PW failure occurred. There was lightning in the area at the time. I can only observe that we did not have any blips, flicker or brownout in the home during the storm. The solar continued to operate normally and the gateway was online and all network connections were good. The failure of the PW manifested itself as follows: At the time of failure, all energy flow to/from PW stopped. The displayed SOC went from 80% to 0% (instantly). The PW2 LED began flashing green indicating loss of communication from PW to Gateway. See attached images.

Attached images: 1) View from app of PW2 state post-failure. 2) View of power flows from app post-failure. 3) Energy flow and SOC during day of failure.

Service Timeline:
8/21/20: Attempt reset procedure from PW2 manual. No effect.
8/22/20: I called Tesla Energy support. They offer a slightly different reset procedure. Try that. No effect
8/26/20: I was expecting to hear from Tesla but they were not contacting me. I called back in again to find that my case had been escalated to L2 support on 8/23. They said the turnaround time is five days. Ok; I need to wait a few more days.
8/31/20: I was expecting to hear from L2 but they were not contacting me. I called Tesla Energy and found out my case had been escalated to a service tech visit and my case was waiting in a scheduling queue. They patched me through to scheduling and I set up an appointment for 9/3/20.
9/3/20: Service tech visit. After a few hours, the problem was diagnosed as a failed synchronizer. This will take about a week to show up in the local warehouse, then the replacement can be scheduled.
9/11/20: Check back on status of synchronizer. Still not arrived in the warehouse.

I will update the thread once the synchronizer replacement has been made. It remains possible that this is not going to fix the problem. If anyone could describe the synchronizer unit and its function I'd be interested to hear about it. My web searches on the subject have come up empty.
 

Attachments

  • power flow.png
    power flow.png
    477.3 KB · Views: 89
  • powerwall.png
    powerwall.png
    469.7 KB · Views: 60
  • trends.png
    trends.png
    82 KB · Views: 61
Definitely some similarities there. My event seemed to happen during a storm with lightning as well and also caused the powerwalls to report a 0% SOC when in reality it was in the 90’s. Of course in my case the powerwalls were able to be reset and came back online after the event. I have not experienced anything similar happening since then either.

I agree with your assessment of what a “synchronizer” could be. I’ve never heard that term before, but any grid tied inverter would need to be synchronized with the grid.
 
<< UPDATE >> The repair has been completed. I will post the full timeline from failure (Aug. 21) to re-commissioning (Oct. 20) below. I learned a few additional details when the Tesla electricians made the repair. They had originally told me the failed component - the synchronizer - was in the PW2, but I think they were just being a bit loose with their description. When they actually made the repair, they did not even touch the PW2. The synchronizer is in the gateway. In fact, I feel like what they replaced was the gateway itself. It is a black plastic brick that sits right in the middle of the gateway cabinet. It looks to be a controller, synchronizer, switch, etc. Once you take this thing out there is not much left to look at in the gateway other than physical connections and wiring. When the gateway was re-commissioned, it came to life with two year old software (only allowing the choice between Backup Only mode and Self-Powered mode). Over the course of two days, it eventually caught up to the most recent software rev, including time of use, Storm Watch and Car Charging options.

Note 1: When the new gateway was commissioned it came to life with a different serial number. As a result, I have two PW2s showing up in my app now. The old "dead" one that has all my historical data prior to the repair, and the new one that is showing live power flows and contains historical data since the repair. This is annoying and I'm going to call in to see if they can overlay the data sets for me and delete the old system.

Note 2: My MS was plugged in overnight after the gateway replacement. It charged off-schedule starting at midnight. I believe this is a software bug related to a revision mismatch between the MS and the new gateway. Even after the gateway OTA updates were complete it still seems to be charing off-schedule starting at midnight. I think a reboot of my MS MCU is in order.

Note 3: The diagnostic visit in September required about four hours of on-site troubleshooting. The swap out and re-commissioning of the gateway took less than two hours. There were no power outages required for either visit.

Note 4: The greater Houston area has enough Tesla energy work to keep the electrician really busy. When he showed up to replace the gateway there was a trainee working with him. This is good news.

Note 5: The electricians upgraded some wiring to/from the gateway to a higher gauge. They said they were doing this on all the gateways they worked on.

Final Service Timeline:
8/21/20: Attempt reset procedure from PW2 manual. No effect.
8/22/20: I called Tesla Energy support. They offer a slightly different reset procedure. Try that. No effect
8/26/20: I was expecting to hear from Tesla but they were not contacting me. I called back in again to find that my case had been escalated to L2 support on 8/23. They said the turnaround time is five days. Ok; I need to wait a few more days.
8/31/20: I was expecting to hear from L2 but they were not contacting me. I called Tesla Energy and found out my case had been escalated to a service tech visit and my case was waiting in a scheduling queue. They patched me through to scheduling and I set up an appointment for 9/3/20.
9/3/20: Service tech visit. After a few hours, the problem was diagnosed as a failed synchronizer. This will take about a week to show up in the local warehouse, then the replacement can be scheduled.
9/11/20: Check back on status of synchronizer. Still not arrived in the warehouse.
9/29/20: Check back again. Find out that the person who was supposed to order the new component had left Tesla and nobody picked up his list of outstanding work items. Order submitted.
10/6/20: Check back again. Find out the synchronizer has arrived at the Houston warehouse.
10/7/20: Appointment text arrives from Tesla Solar. Date of replacement set for 10/20/20.
10/20/20: Two hour service visit during scheduled window. New gateway commissioned. A couple of hours after the visit, a new gateway shows up in my Tesla App.
10/23/20: OTA software updates on the new gateway seem to be complete now. All functionality restored in Tesla App.

I hope this information proves helpful to other users out there. When I encountered this problem I really struggled to find much published around this particular type of failure. - Sepps
 
I’m glad they were finally able to get things sorted for you. It’s unfortunate that it took so much time though.

Note 1: When the new gateway was commissioned it came to life with a different serial number. As a result, I have two PW2s showing up in my app now. The old "dead" one that has all my historical data prior to the repair, and the new one that is showing live power flows and contains historical data since the repair. This is annoying and I'm going to call in to see if they can overlay the data sets for me and delete the old system.

Yeah, that definitely sounds like they replaced the whole gateway. Or at least the innards of it. Having two systems with two sets of data certainly sounds annoying though. Hopefully they have a way to merge them for you. Let us know what the outcome of that is.

Note 2: My MS was plugged in overnight after the gateway replacement. It charged off-schedule starting at midnight. I believe this is a software bug related to a revision mismatch between the MS and the new gateway. Even after the gateway OTA updates were complete it still seems to be charing off-schedule starting at midnight. I think a reboot of my MS MCU is in order.

I have a hard time imagining how this could be related. The MS and the gateway really have nothing to do with each other. The charging options in the new gateway firmware only apply during an outage and wouldn’t affect the MS charging schedule at all. That just determines whether or not the car can start (or continue) charging when the power is out.

The timing is certainly interesting, but I still think it was probably unrelated.