TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker and becoming a Supporting Member. For more info: Support TMC

New Wall Connector load sharing protocol

Discussion in 'Model S: Battery & Charging' started by ctr, Jul 1, 2016.

  1. brkaus

    brkaus Well-Known Member

    Joined:
    Jul 8, 2014
    Messages:
    6,582
    Location:
    Austin, TX
    So I'm picturing the usage model being able to add a programmable "master" to control the existing HPWC(s) as slaves.

    What granularity of control do you have? Can you tell the slave that no power is available, or is there a minimum? I'm considering doing this for learning, fun, and just to have some practical motivation, to make sure my time of use is enforced.
     
  2. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    BTW, IN and OUT pins are not on a separate bus as I theorized. A voltmeter shows no resistance between D+ IN and D+ OUT (or D- IN and D- OUT), but there is resistance between D+ and D-. Also, I've switched between IN and OUT pins while the system is on (using external barrel connectors) and they're both sending the same data.

    This means that slave HPWCs can't establish who is their immediate parent or child is in the chain by using the pins. Instead, all messages sent by any HPWC are received by every other HPWC in the chain, so they must negotiate for a unique ID and obey some sort of rules to avoid talking over each other. Each device seems pre-programmed with a unique id but I've seen evidence they will pick a new random id if they sense another device on the chain with their id. IDs are 2 bytes long which means 64k values and thus two devices happening to carry the same ID is easily possible, if unlikely.

    The other reason this is important is if ctr connected a logger to both IN and OUT pins expecting that the IN pins would only be used by the HPWC for receiving messages and OUT pins for sending (a logical assumption), that might explain why he logged messages twice as long as I've been seeing. Basically, IN and OUT transmit the same data and I'm not sure what effect that would have on his logger.

    I see a lot of evidence ctr's messages are corrupted. For example, one of the "I'm here" messages Master HPWCs send when first turned on that ctr reported:
    e1fc 7801 00b3 0000 0000 0000 0d00 fec0 fcc0 01e1 b378 0000 0000 0000 0000 c00d c0fe

    The second part of the above message, with spaces between each byte looks like this:

    c0 fc c0 01 e1 b3 78 00 00 00 00 00 00 00 00 c0 0d c0 fe

    That looks similar to US message posted by IanAber:

    c0 fc e1 03 8c 49 00 00 00 00 00 00 00 00 b9 c0 fe

    except there's an extra c0 01 in ctr's message.

    Applying checksum logic to ctr's message: c0+01+e1+b3+78+c0=38d Dropping all but the last 8 bits leaves 8d which doesn't match the 0d checksum byte at the end of the message. If we get rid of the c0 01 part: e1+b3+78+c0=2cc which still doesn't match. From my research, ctr's message is still invalid even if you delete c0 01 because a valid message should have 00 in byte 14 and a non-zero in byte 6. I suspect the message has data from IN and OUT pins interleaved in some way.

    If ctr's logs are corrupted, that's good news in that EU units probably use the same 17 byte protocol as US units.
     
  3. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    #83 WinterDragoness, Nov 8, 2017
    Last edited: Nov 8, 2017
    I've got Perl on a Raspberry Pi handling communication with the HPWC. I can fake being a master or a slave without setting off any red error lights. Fake master seems to work correctly but fake slave still has the real master sending me that fd e0 message I mentioned in the previous post every second or two as if it doesn't like my response.

    I thought maybe that was just normal behavior so I tried plugging the car in to see if fake slave got any status updates on the power remaining while the car charged. I got nothing. The fd e0 messages received every second continued to contain the same string of 00 00 00 00 00 00 00 and no new types of message appeared.

    Assuming I'm faking my slave properly, my next theory is that slaves change their response to fd e0 when they're actively charging a car and only at that point will master tell them how much power they can use. If it works that way, maybe master communicating every second makes sense, but... it doesn't seem right. We'll see tomorrow.

    Also, one of my $30 RS485 cables was producing a single bad byte compared to the other cable which makes me question both of them. Adding termination resistor or moving closer to HPWC did not fix it. Could be I'm still not seeing quite what the HPWC is actually sending, or it's not seeing quite what I'm sending. Will be a few days before MITE46's suggested cheap adapter arrives for comparison.

    I also wondered if 9600 baud could be wrong so I checked the signal with my oscilloscope and it is indeed 9600 baud. There's still an outside chance the 8N1 setting could be wrong, though it seems unlikely I would have gotten this far if it were.
     
  4. MITE46

    MITE46 Member

    Joined:
    Aug 26, 2014
    Messages:
    185
    Location:
    Los Altos / San Marino
    Should have gone with the $5 USB adapter...=)

    Why do you have a 40 in the command below?
    Switching HPWC rotary switch to 8 (40A master mode), I fake a message from a slave with 0A 51 id:
    C0 FD E2 0A 51 45 1F 40 00 00 00 00 00 00 E1 C0 FE
     
  5. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    #85 WinterDragoness, Nov 9, 2017
    Last edited: Nov 9, 2017
    Because when I put my HPWC in slave mode and just watch, the only message it sends is:
    c0 fd e2 03 2e 68 1f 40 00 00 00 00 00 00 da c0 fe

    where 03 2e seems to be a hard coded HPWC ID, 68 is a seemingly random value that only changes when HPWC is power cycled or reset, and 1f 40 always remains the same.

    I once sent this command by mistake to a Master HPWC:
    C0 FD E2 0A 51 03 2E 69 1F 40 00 00 00 00 68 C0 FE

    where 2E 69 appeared where 1F 40 normally would. It immediately started blinking red led 6 times with top green led on and had to be reset to stop doing that. Manual says that means "The networked Wall Connectors have different maximum current capabilities". So my theory is 1F 40 refers to the max rating of the charger. 0x1F40 = 8000 in base 10 which could mean 80.00 Amps which is the max rating of my HPWC.
     
    • Helpful x 1
  6. Ross

    Ross Member

    Joined:
    Jun 4, 2016
    Messages:
    22
    Location:
    ATL
    @CDragon

    Would you be willing to share the PI code you're using?

    I have a 4 unit load share at my business, I could set something to listen to the units as people plug/unplug.
     
  7. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    #87 WinterDragoness, Nov 9, 2017
    Last edited: Nov 9, 2017
    Sure! Half the reason I'm doing this is to spread the ability to control HPWCs to track solar output or other peak green energy sources like night wind. Doing so saves some energy and avoids fossil sources.

    However, until my Perl code is successfully commanding an HPWC to do something useful, I don't think it's worth sharing. If you'd like to get me some logs, all you need is the hardware (the Digikey cable or the cheaper JBTek adapter which, coincidentally just arrived from Las Vegas despite only ordering it yesterday morning!) you can run logging with standard unix commands in Raspberry Pi:

    stty -F /dev/ttyUSB0 raw -echo 9600 cs8 -cstopb -parenb
    od -t x1 < /dev/ttyUSB0

    If you have more than just the RS485 adapter plugged in to Pi's USB, RS485 won't necessarily appear at ttyUSB0 (and may change each time Pi boots). You can find it with this command:

    dmesg |grep ttyUSB
    which will output something like this:

    [ 9.172050] usb 1-1.2: ch341-uart converter now attached to ttyUSB0
    [ 9.403061] usb 1-1.3: FTDI USB Serial Device converter now attached to ttyUSB1

    The ch341 adapter is the cheap one and FTDI USB is the expensive one. The expensive one has extra wires for termination, external power, and has Tx and Rx LEDs that light on activity. I think it also has some advanced features you can program but probably nothing useful for this task. Oh, and there are a fair number of reviews complaining the cheap one failed within months which is part of why I avoided it, but then I received an expensive one that was bad so who knows. Also, the cheap one is easier to wire for this project. I had to securely insulate the 4 extra wires on the expensive one so they can't touch anything accidentally and read the manual to find out which were D+ (it's orange) and D- (it's yellow).

    Both adapters were picked up by the latest Raspberry Pi kernel and work fine with the logging command. Both are showing me the same output from the HPWC so it looks likely my second FTDI cable is good. I've read some older kernels may occasionally freeze them or not recognize them. If you have problems, you can update your pi to the latest kernel with these commands, though beware, if you use the pi for other things, sometimes updates break random things:

    sudo apt-get install rpi-update
    sudo rpi-update​

    Reboot.

    Beware that plugging or unplugging either of these rs485 adapters seems to crash the pi. They're both running together off a 1A power supply to the pi using an ethernet cable (no kb, mouse, or monitor). I've had so many problems with weird glitches from unstable power on Pis, I recommend a 2A power supply if you're ordering new parts anyway. I suspect 1A with USB wifi and RS485 won't be stable.

    P.S.
    I've discovered that a slave reports 04 00 00 00 19 00 00 instead of 00 00 00 00 19 00 00 when slave is plugged in to a car. Sending the "plugged in" code to master causes it to send 05 0f a0 00 00 00 00 instead of 00 00 00 00 00 00 00 in those once-per-second handshakes. At that point, plugging in master changes the data to 05 07 d0 01 00 00 00 and the HPWC refuses to supply power to the car. I assume it's waiting for a status change from the slave but I'll need to send these new data blocks to the slave to see how it should respond. This would be so much easier with two HPWCs...

    Main point is it looks like my unlikely theory was correct: Master doesn't tell the slave anything useful unless the slave reports that it's plugged in to a car.
     
    • Informative x 1
  8. Ross

    Ross Member

    Joined:
    Jun 4, 2016
    Messages:
    22
    Location:
    ATL
    Should have been more clear. I am offering to log data send/receive from the wall units, as well as run through scenarios here on site. We have multiple vehicles here and we're watching this thread with interest.

    If you have anything specific, like "Plug in a car to Master, then Plug in a car to slave and report back the logs gathered" I'd be willing to do that. Going even further, I could hook up an RPI to each one and have them dump traffic they receive into a webpage somewhere.
     
  9. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    Hi Ross,

    I generally understood that's what you were offering. Thanks! Each chain of up to 4 HPWCs uses a single data bus so you just need one pi and one RS485 adapter (I recommend at least 2 rs485s just to double check you didn't get a bad one like I did). You can connect that pi logger anywhere in the chain (may as well wire it to Master's IN pins) and watch what all 4 are saying to each other.

    For now, I'd say:
    1. Unplug all cars
    2. Start logging
    3. Plug in all cars
    4. Wait 2 minutes, then write down how many amps each are charging at.
    5. Unplug all cars
    6. Send me log and how many amps for each car
    I may have other suggested tests once I figure out the protocol better but this log would show how more than two HPWCs interact while cars are charging and not charging.
     
  10. MITE46

    MITE46 Member

    Joined:
    Aug 26, 2014
    Messages:
    185
    Location:
    Los Altos / San Marino
    #90 MITE46, Nov 9, 2017
    Last edited: Nov 9, 2017
    1f40 is pilot but usually not in that location. The slave command you are sending seems to be similar to what is received after a reset is sent.

    Make sure to maintain the heartbeat, the whole time.

    We also see messages once a second during a charge session. Makes sense as if another car plugs in, it has to adapt very quickly, especially if sharing a circuit.
     
  11. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    #91 WinterDragoness, Nov 9, 2017
    Last edited: Nov 10, 2017
    SUCCESS!!!!!1!!11!!!1

    I've tricked my TWC in slave mode into taking charge limiting instructions from a Raspberry Pi. The FOOL! Now I'm off to apply the technique to some hardware which I'll sell for a monthly subscription fee and never tell anyone how I did it! BWAHAHAHAHAHA!!!11!!!1!!

    Actually, I'll just clean up the Perl code and release it soon. There are still some details I want to figure out first.

    To answer brkaus's earlier question, the protocol lets you control the power limit in 0.01A increments from 0.01A to 80.00A. Niiiiice. The protocol does not seem to allow you to set a 0A limit, but setting 0.01A shows the charger draws no more than its 0.25A idle power. In fact, the car's UI and power use logs show the car doesn't even try to charge till you set the limit to at least 1A. I made a script that set 0.01A limit for 60 seconds, then added 0.5A every 10 seconds and took a video I can post if anyone's interested. The car only displays the charge rate to the nearest amp but logging the protocol shows the actual power draw increases to near the limits I was setting (1.01A, 1.51A, 2.01A, 2.51A, etc).

    The next hurdle is finding an outdoor enclosure with an LCD touchscreen or an LCD and buttons or maybe just buttons and a couple status lights. Basically, I want to house the pi and allow changing from track solar panel power charging mode to charge from wall power mode.

    And since MITE46 keeps posting cool pics of his lab, check out my professional lab mirror that I scientifically adjusted to show me the status lights on the TWC through our security camera:

    Lab mirror.jpg

    This delicate instrument allows me to fine tune settings from the comfort of my lab chair while watching for the dreaded blinking red LED that marks failure of the experiment.
     
    • Like x 3
    • Funny x 3
    • Helpful x 2
    • Love x 1
  12. M_VdM

    M_VdM Member

    Joined:
    Mar 31, 2016
    Messages:
    40
    Location:
    Belgium
    @CDragon: Great Job ! :cool:
    R-E-S-P-E-C-T

    Thanks for sharing your results ... I' can hardly wait to get a look.

    That should keep me busy for a while, till my Model 3 comes along ... end of next year in Europe :rolleyes:
     
  13. spottyq

    spottyq Member

    Joined:
    May 7, 2015
    Messages:
    278
    Location:
    Belgium
    Great work @CDragon ! (And thanks for the funny description ! ;-) )

    I guess that will interest some owners with solar and some technical knowledge…
     
  14. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    The c0 fd e2 03 2e 68 1f 40 00 00 00 00 00 00 da c0 fe (what I'm now calling "link ready") that's sent by a slave TWC is not just sent at boot, it's sent every 10 seconds until it gets what you call a "heartbeat" message from master TWC. This is part of why I thought it wrong to see the heartbeat every second - I assumed all messages would be at 10s intervals or maybe just sent as needed. Once master starts sending heartbeat, "link ready" is never sent again unless master stops sending heartbeat.

    Also, if master sees a link ready message, it immediately starts transmitting heartbeat directed to the TWC ID contained in the link ready message. If slave sees one of master's boot messages, I believe it sends link ready immediately instead of waiting up to 10s. From what I can tell, a master won't send heartbeat to a particular slave's TWC ID until it sees a link ready message from the slave.

    Since you presumably started with two TWCs and watched how they talked, you probably never bothered learning how to emulate a slave TWC with the real one set to master. Not that that knowledge is really useful other than in protocol reversing, but it's kind of interesting.
     
  15. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    #95 WinterDragoness, Nov 11, 2017
    Last edited: Nov 11, 2017
    Hey TheNoOne, I wanted to officially thank you for your post on the checksum byte awhile back. That one bit of information was probably the thing that saved me the most time in figuring out the protocol.

    I also just realized that you were posting that info based on your own EU wall charger, and that your messages were 17 bytes long, so I think that's proof that EU chargers use the same protocol as US chargers. Yay!

    One odd thing is you say your master-mode charger sent a 16-byte idle message every 4 hours:
    c0 fc 1d 00 00 00 00 00 00 00 00 00 00 00 1d c0

    I suspect the message actually contains an extra fe byte at the end but that your logger wasn't catching it. I've found that the last byte in a message can be corrupted if you don't include a 120ohm termination resistor in parallel with the D+ D- pins (like this). Although I've only seen the final fe byte get corrupted into two bytes (02 00) whereas I think you're saying there was no byte after the last c0? If that's not corruption, it's very odd they would change the protocol in this one case.

    I got slowed down when I discovered that I needed to handle messages over 17 bytes long. Why?
    The protocol uses c0 to mark the next byte as having special meaning:
    • c0 fb, c0 fc, and c0 fd mark the start of various messages.
    • c0 fe marks the end of a message.
    Therefore, c0 can not appear within a message. They could have just used c0 c0 to mean an actual c0 byte (doubling a special character is commonly used to escape it), but instead some insane person decided that an actual c0 byte would be represented by... db dc! Because why not introduce another special value that needs escaping?

    Ok, so what about an actual db byte in a message? Why, it's represented by db dd of course! Certainly wouldn't want to escape db using db db (by doubling it) because that's just too obvious. *facepalm*

    Maybe they had some good reason for this crazy method of escaping special values but it makes no sense to me.
    I've confirmed fb, fc, fd, and fe are not special values and I hope there aren't others lurking.

    Anyway, my code should be released this weekend assuming I don't find any other weird things to deal with.
     
    • Like x 1
    • Love x 1
  16. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    #96 WinterDragoness, Nov 11, 2017
    Last edited: Nov 11, 2017
    The day has arrived..

    The veil will be lifted...

    The secrets of the RS485 TWC (aka HPWC II) load-sharing protocol can now be found at TWCManager.

    About 30 mins before the sun got too low, I got it scaling the car charger power to solar panel output. Here's a graph from my TED monitor:

    First time tracking solar output to car charger3.jpg
    Each dot is a minute of averaged power use and solar is negative because it's generating power instead of using power. I started car charging at the red line. The car is not taking quite as much power as the panels are producing because the TWC or the car are being overly conservative, drawing only 2.7A when told they can draw 3.51A (for example). The car itself stops charging when the limit drops below 1A but TED stopped reporting power use somewhere around 1.3A. That's just a limitation of the TED "Spyder" sensors.

    I did find one issue that may have no great solution:

    If TWCManager is stopped or crashes for more than maybe 30 seconds (haven't timed it exactly), the slave wall charger starts blinking red 4 times and stops charging the car. The car will soon say "Charge cable fault" on the smaller display.

    When TWCManager is re-started, the TWC light turns green and the car stops showing any error, yet the car will refuse to charge until you re-plug the charging cable. In this condition, the TWC reports protocol state 08 when TWCManager reconnects, then reports state 03 after the manager gives it a new charger power limit. State 03 seems to mean the car refuses to charge (ie it enters state 03 if you hit "Stop charging" on the car's UI). Yet if you press "Start charging" on the UI, nothing happens. No error, no feedback, and the button still shows "Start charging". Car shows 0/10A available for charging (or whatever the max charge set by TWCManager is) but won't charge until you unplug and replug.

    I plan to try some things like just waiting a long time, but I'm not sure there's a solution other than to have TWCManager email or text a warning telling you to re-plug. Of course depending how it's crashed, that may not be possible either. Usually what crashes in a raspberry pi are the USB devices rather than the CPU, though since wifi is USB it may crash and not be able to email. I think they may have worked those problems out by now but I'm still using older USB-crashy firmware on original a Pi 1B.

    Other than problems with USB wifi locking up, I've run home automation on a Pi with an ethernet connection for a few years and I don't remember it ever locking up. It runs a USB Insteon controller that has also been trouble free (other than I had to replace its crappy capacitors once which is a common problem).

    Wow, I just noticed they've come out with Pi Zero W with built in wifi and a small enough form factor it has some hope of fitting in the TWC in an insulated case. And only $10 frickin dollars? Holy cow... That's cheaper than an FTDI RS485 cable!

    The problem is powering the pi. I don't know if it's safe to steal 1 to 2A of power from a 5V power supply in the TWC, or if it even supplies 5V power. If not, the Pi plus a "wall wart" power supply could be housed in the "spacer" adapter that comes with the TWC and fits behind it. 120V power for the wall wart could even be taken from the TWC's main power. That probably breaks electrical codes, though possibly not as long as all wiring remains in the fire-resistant enclosure.

    Anyway, as long as TWCManager doesn't stop running, I've left it telling the charger there's a 0.01A limit for up to 2 hours so far and when I change it to 10A the car starts charging within ~30s even if its screens have turned off. I'm doing a test now leaving it running all night to make sure it starts charging in the morning once sun hits the solar panels.
     
    • Informative x 6
    • Love x 2
    • Helpful x 1
  17. MITE46

    MITE46 Member

    Joined:
    Aug 26, 2014
    Messages:
    185
    Location:
    Los Altos / San Marino
    Yay! I am glad you got it working. Did you see the voltage and cumulative energy counters?
     
    • Helpful x 1
  18. WinterDragoness

    Joined:
    Nov 14, 2015
    Messages:
    210
    Location:
    Southern California
    #98 WinterDragoness, Nov 11, 2017
    Last edited: Nov 11, 2017
    Erm... no? Heartbeat data sent from the slave contains only 7 bytes which look like this while charging:

    01 02 b3 02 3f 00 00

    01 is a state code (apparently charging, in this case), 02 b3 is max amps (6.91A), 02 3f is actual amps being used (5.75A) and the last two bytes are always 00 even after charging for 30 mins. I've never seen them contain anything other than 00 under any circumstance.

    Are the last bytes supposed to contain voltage or cumulative charge? If so, mine aren't working. =(

    I also didn't get any other message types as long as the heartbeat was going. It should display "***UNKNOWN MESSAGE TYPE" if anything not seen before comes in.

    My wall charger was ordered Aug 23, 2016 while the new charger version was first released in mid April 2016 so I wouldn't think I got a super early version, but maybe they added to the protocol later?
     
  19. MITE46

    MITE46 Member

    Joined:
    Aug 26, 2014
    Messages:
    185
    Location:
    Los Altos / San Marino
    Hmm, maybe. Easiest way to tell is to get another HPWC and see if those messages come across...we see them every 10 seconds or so.
     
  20. K-MTG

    K-MTG Sunshade Captain of TMC

    Joined:
    Oct 24, 2015
    Messages:
    4,815
    Location:
    Irvine, CA
    It would be great if someone can make a video showcasing this...projects like these make me wanna go back to CS...I guess its never too late to learn something new! But a video with step by step instructions would be very very helpful
     

Share This Page

  • About Us

    Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.
  • Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


    SUPPORT TMC