TMC is an independent, primarily volunteer organization that relies on ad revenue to cover its operating costs. Please consider whitelisting TMC on your ad blocker or making a Paypal contribution here: paypal.me/SupportTMC

Strange network behavior... car incessantly attacking my nameserver

Discussion in 'Model S: User Interface' started by FlasherZ, Feb 18, 2016.

  1. FlasherZ

    FlasherZ Sig Model S + Sig Model X + Model 3 Resv

    Joined:
    Jun 21, 2012
    Messages:
    7,019
    For all of you other network geeks, I'd like to know if you've seen this odd behavior. Tonight, I noticed that the car no longer had connectivity through Tesla's VPN tunnel while on Wi-Fi. It had Slacker, the web browser worked, etc., but the remote app couldn't "see" the car and all my API calls returned "car unavailable".

    I logged into the management portal for my AP's (Meraki), did a packet capture, and saw the following:

    Code:
    16:56:06.152567 IP y.y.y.y.53 > x.x.x.x.15413: 56407| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.38, A 209.10.208.44, A 209.10.208.49, A 209.11.133.30, A 209.11.133.29, A 209.10.208.51, A 209.11.133.23, A 209.11.133.40, A 205.234.27.254, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.27, A 209.10.208.43, A 209.11.133.39, A 209.10.208.48, A 209.10.208.42, A 209.10.208.47, A 205.234.27.198, A 205.234.27.234, A 209.10.208.50, A 209.10.208.45, A 209.11.133.24, A 209.11.133.28, A 209.10.208.46 (488)
    16:56:06.068971 IP y.y.y.y.53 > x.x.x.x.15413: 56407| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.38, A 209.10.208.44, A 209.10.208.49, A 209.11.133.30, A 209.11.133.29, A 209.10.208.51, A 209.11.133.23, A 209.11.133.40, A 205.234.27.254, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.27, A 209.10.208.43, A 209.11.133.39, A 209.10.208.48, A 209.10.208.42, A 209.10.208.47, A 205.234.27.198, A 205.234.27.234, A 209.10.208.50, A 209.10.208.45, A 209.11.133.24, A 209.11.133.28, A 209.10.208.46 (488)
    16:56:11.085548 IP y.y.y.y.53 > x.x.x.x.47240: 31174| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.30, A 209.10.208.51, A 209.10.208.43, A 209.10.208.47, A 209.11.133.38, A 205.234.27.234, A 209.11.133.22, A 209.11.133.28, A 209.11.133.24, A 209.11.133.27, A 205.234.27.254, A 209.11.133.23, A 209.10.208.50, A 209.10.208.48, A 209.11.133.40, A 209.11.133.39, A 205.234.27.198, A 209.10.208.44, A 209.10.208.45, A 209.11.133.12, A 209.10.208.46, A 209.11.133.29, A 209.10.208.42, A 209.10.208.75, A 209.10.208.49 (488)
    16:56:11.100927 IP y.y.y.y.53 > x.x.x.x.57265: 50032| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.10.208.75, A 209.11.133.29, A 209.10.208.45, A 209.10.208.49, A 209.11.133.23, A 205.234.27.254, A 209.10.208.51, A 209.10.208.43, A 209.11.133.38, A 209.10.208.47, A 209.11.133.28, A 209.11.133.40, A 209.10.208.48, A 205.234.27.198, A 209.11.133.12, A 209.11.133.27, A 209.11.133.24, A 209.11.133.22, A 209.10.208.42, A 209.10.208.44, A 209.11.133.30, A 209.11.133.39, A 205.234.27.234, A 209.10.208.50, A 209.10.208.46 (488)
    16:56:16.114600 IP y.y.y.y.53 > x.x.x.x.32164: 47423| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.22, A 209.11.133.30, A 209.10.208.43, A 209.10.208.50, A 209.11.133.23, A 209.11.133.24, A 209.11.133.12, A 205.234.27.234, A 209.10.208.75, A 205.234.27.254, A 209.11.133.40, A 209.10.208.51, A 209.10.208.42, A 209.10.208.46, A 209.11.133.38, A 205.234.27.198, A 209.11.133.27, A 209.10.208.48, A 209.10.208.44, A 209.10.208.49, A 209.10.208.45, A 209.11.133.39, A 209.10.208.47, A 209.11.133.28, A 209.11.133.29 (488)
    16:56:16.125739 IP y.y.y.y.53 > x.x.x.x.34201: 43294| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.29, A 209.10.208.45, A 205.234.27.234, A 209.10.208.44, A 209.11.133.24, A 209.10.208.43, A 205.234.27.254, A 209.10.208.50, A 209.10.208.47, A 209.11.133.22, A 209.10.208.42, A 205.234.27.198, A 209.11.133.30, A 209.11.133.39, A 209.10.208.49, A 209.10.208.51, A 209.10.208.46, A 209.11.133.40, A 209.10.208.75, A 209.11.133.38, A 209.11.133.28, A 209.11.133.12, A 209.10.208.48, A 209.11.133.23, A 209.11.133.27 (488)
    
    Every 5 seconds, the car was querying my nameserver (BIND9). This continued for well over 15 minutes. There were no errors on the nameserver, just query after query after query. I didn't see a problem with the query or response, it just seemed that the car was ignoring them and retrying again.

    I fixed it by turning off Wi-Fi in the car, stopping my nameserver, and turning Wi-Fi back on. The car then tries to send nameserver requests for a while, gets frustrated, and gives up. I then restarted the nameserver and then the car did a query and reconnected its tunnel.

    (One other behavior - it appears the car cannot use a secondary nameserver. When I took my primary nameserver down, it just kept sending queries despite getting back port unreachable messages. It never tried to contact the backup nameserver.)

    Has anyone else seen this behavior?
     
  2. AWDtsla

    AWDtsla Active Member

    Joined:
    Mar 3, 2013
    Messages:
    2,134
    Location:
    NE
    Every 5 seconds sounds pretty slow. Answer should be cached.
     
  3. FlasherZ

    FlasherZ Sig Model S + Sig Model X + Model 3 Resv

    Joined:
    Jun 21, 2012
    Messages:
    7,019
    I'm unsure that the car was ever getting the response in the first place. It kept asking, and my nameserver kept responding, but the car just continued asking. So if it never heard a response in the first place, it couldn't cache it.

    My nameserver was serving a cached response, from what I could see.
     
  4. dkonigs

    dkonigs Member

    Joined:
    Apr 11, 2015
    Messages:
    74
    Location:
    San Jose, CA
    When I first got my car, I couldn't get the mobile app to connect to it when the car was on my home WiFi. Upon packet sniffing, I think I saw similar behavior (but don't recall completely, since its been a while). I then configured my DHCP server to give the car Google's DNS server IPs instead, and suddenly everything started working fine.
     
  5. scottm

    scottm Active Member

    Joined:
    Jun 13, 2014
    Messages:
    1,262
    Location:
    Canada
    Just an honest question: why does anybody bother with WiFi? just turn it off an leave it off...
     
  6. jerry33

    jerry33 S85 - VIN:P05130 - 3/2/13

    Joined:
    Mar 8, 2012
    Messages:
    12,752
    Location:
    Texas
    Originally it was stated that Wi-Fi would get updates faster and maps would only be updated through Wi-Fi. I don't believe that's still the case (although my Wi-Fi connection is very weak and I haven't received a map upgraded in the three years I've had my Model S). If you don't have LTE, Wi-Fi through a phone's hotspot can be faster.
     
  7. FlasherZ

    FlasherZ Sig Model S + Sig Model X + Model 3 Resv

    Joined:
    Jun 21, 2012
    Messages:
    7,019
    Not to mention that Tesla originally told owners they would have to start paying for data connectivity to the cars... but that's a difficult one because of all the other data they get.
     
  8. thecloud

    thecloud As rhythm raced inside, the ship came alive

    Joined:
    Nov 24, 2014
    Messages:
    565
    Location:
    Sunnyvale, CA
    Yes, I see the same behavior on a daily basis. I run a caching DNS server on my internal network at 192.168.11.11, and the DHCP server is configured to specify that server for DNS. The car keeps waking up periodically, asks to resolve 'vpn.vn.teslamotors.com', and the server keeps sending the same valid response from its cache. Eventually the car stops asking, and then it starts up again later.

    Code:
    21:09:33.544624 IP 192.168.11.102.38947 > 192.168.11.11.53:  38171+ A? vpn.vn.teslamotors.com. (40)
    21:09:33.545559 IP 192.168.11.11.53 > 192.168.11.102.38947:  38171| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24, A 209.11.133.27, A 209.11.133.28, A 209.11.133.29 (506)
    21:09:38.558548 IP 192.168.11.102.16349 > 192.168.11.11.53:  20048+ A? vpn.vn.teslamotors.com. (40)
    21:09:38.559858 IP 192.168.11.11.53 > 192.168.11.102.16349:  20048| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24, A 209.11.133.27, A 209.11.133.28 (506)
    21:09:38.578473 IP 192.168.11.102.12116 > 192.168.11.11.53:  64125+ A? vpn.vn.teslamotors.com. (40)
    21:09:38.579472 IP 192.168.11.11.53 > 192.168.11.102.12116:  64125| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.28, A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24, A 209.11.133.27 (506)
    21:09:43.589701 IP 192.168.11.102.21944 > 192.168.11.11.53:  25485+ A? vpn.vn.teslamotors.com. (40)
    21:09:43.591042 IP 192.168.11.11.53 > 192.168.11.102.21944:  25485| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.27, A 209.11.133.28, A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24 (506)
    21:09:43.600824 IP 192.168.11.102.64403 > 192.168.11.11.53:  10655+ A? vpn.vn.teslamotors.com. (40)
    21:09:43.601748 IP 192.168.11.11.53 > 192.168.11.102.64403:  10655| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.24, A 209.11.133.27, A 209.11.133.28, A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23 (506)
    
    I've reported the issue to Tesla but never heard anything back.
     
  9. FlasherZ

    FlasherZ Sig Model S + Sig Model X + Model 3 Resv

    Joined:
    Jun 21, 2012
    Messages:
    7,019
    I'm wondering (complete WAG here) if they're running into a 512-byte buffer problem with the DNS query - I saw it as 488 on the payload, you see it as 506. I remember this happening years ago with some other services that had extra-large query responses.
     
  10. green1

    green1 Active Member

    Joined:
    Mar 25, 2014
    Messages:
    4,105
    Location:
    Calgary, Alberta, Canada
    Could you test that by setting up your name server to reply with a smaller payload? Say only half the IPs, and see if it solves it?
     
  11. Tree95

    Tree95 Member

    Joined:
    Oct 24, 2014
    Messages:
    192
    Location:
    Usa
    I live in a rural area with no cellular data. WiFi allows Tesla to push updates, and allows me to see the charge state with the app without walking out to the garage.
     
  12. thecloud

    thecloud As rhythm raced inside, the ship came alive

    Joined:
    Nov 24, 2014
    Messages:
    565
    Location:
    Sunnyvale, CA
    I could definitely test that theory by setting my BIND server up as authoritative for the vn.teslamotors.com domain. I'll play around with it tomorrow.
     
  13. strider

    strider Active Member

    Joined:
    Oct 20, 2010
    Messages:
    2,918
    Location:
    NE Oklahoma
    #13 strider, Feb 19, 2016
    Last edited: Feb 19, 2016
    I've never been able to contact our car over WiFi (though maps, radio, browser work). I thought it had something to do w/ my WiFi (Aruba) not NATing the IPSec properly but haven't taken the time to troubleshoot. I also run a local name server (OS X Server - I got tired of manually patching FreeBSD) so maybe it's related. I'll try bouncing named.

    -Update-
    That fixed it. Bizarre. I didn't even touch the car. I just did a kill -HUP of named and then fired up the app. It gave the the "Waking car" message and then connected (previously is would hang on "Waking car"). Definitely something strange going on.
     
  14. wk057

    wk057 Senior Tinkerer

    Joined:
    Feb 23, 2014
    Messages:
    4,714
    Location:
    Hickory, NC, USA
    The way wifi works on the car is weird. Wifi is handled by an independent MCU board that runs linux. It then NATs wifi traffic for the CID along with running dnsmasq. The CID runs its own dnsmasq as well to split queries for Tesla's VPN hosts off to their internal nameserver. All of this is coordinated by a process on the CID....... and I've seen it get confused quite a few times.

    Most likely what was happening was the car was sending the DNS request out via wifi and expecting a response via cellular or Tesla's tunnel... thus the constant retries.

    If your home wifi IP mask is in the same range as anything internal to the car is screws things up too. Works half-assed, breaks sometimes, sometimes acts randomly.
     
  15. Cosmacelf

    Cosmacelf Active Member

    Joined:
    Mar 6, 2013
    Messages:
    3,399
    Location:
    San Diego
    Wk057, could you explain that last paragraph in more detail. What ranges does the car use?
     
  16. FlasherZ

    FlasherZ Sig Model S + Sig Model X + Model 3 Resv

    Joined:
    Jun 21, 2012
    Messages:
    7,019
    #16 FlasherZ, Feb 20, 2016
    Last edited: Feb 20, 2016
    It's not in this case. I have my own BGP-advertised public IP address blocks and the car gets a public IP from them.

    I don't think it's the car sending/expecting a response via different paths - if I bounce the DNS server (as strider did above), then the car goes into happy mode again once it gets a smaller query.

    The query on the wire and the query response all look good, so I don't think it's anything having to do with BIND.

    I did confirm, though, that the message size is > 512 bytes (525 bytes from one of my nameservers, 555 from the other), so that seems like the best target to look at right now. I dropped a note to the engineering team so they can look at it.

    I also configured BIND to use a maximum UDP size of 512 bytes by setting "max-udp-size 512" in the options configuration. When a query is over 512 bytes, then BIND will retry by advertising a smaller EDNS size to authoritative servers. This should make it work temporarily, until Tesla fixes its ability to receive queries > 512 bytes.

    Before:
    ;; Query time: 0 msec
    ;; WHEN: Sat Feb 20 08:35:32 CST 2016
    ;; MSG SIZE rcvd: 555

    After:
    ;; Query time: 0 msec
    ;; WHEN: Sat Feb 20 08:36:45 CST 2016
    ;; MSG SIZE rcvd: 471
     
  17. thecloud

    thecloud As rhythm raced inside, the ship came alive

    Joined:
    Nov 24, 2014
    Messages:
    565
    Location:
    Sunnyvale, CA
    Well, it looks like FlasherZ's WAG was right. After I set up BIND to return a single IP address when it asked to resolve the hostname, I got exactly one query from the car, and the app subsequently connected to the car right away.

    Code:
    11:32:04.363872 IP 192.168.11.102.21406 > 192.168.11.11.53:  4052+ A? vpn.vn.teslamotors.com. (40)
    11:32:05.639689 IP 192.168.11.11.53 > 192.168.11.102.21406:  4052* 3/1/1 CNAME usvpn.vn.teslamotors.com., CNAME vpn-209-10-208-42.vn.teslamotors.com., A 209.10.208.42 (157)
    
    Note this response is 157 bytes, compared to 506 bytes just before I switched the configuration:

    Code:
    11:31:54.344006 IP 192.168.11.102.30306 > 192.168.11.11.53:  21672+ A? vpn.vn.teslamotors.com. (40)
    11:31:54.344931 IP 192.168.11.11.53 > 192.168.11.102.30306:  21672| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24, A 209.11.133.27, A 209.11.133.28, A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47 (506)
    
     
  18. FlasherZ

    FlasherZ Sig Model S + Sig Model X + Model 3 Resv

    Joined:
    Jun 21, 2012
    Messages:
    7,019
    Thanks for the confirmation. You can use the global option "max-udp-size 512;" to make the server hand better query responses to the car until they fix it.
     
  19. thecloud

    thecloud As rhythm raced inside, the ship came alive

    Joined:
    Nov 24, 2014
    Messages:
    565
    Location:
    Sunnyvale, CA
    #19 thecloud, Feb 20, 2016
    Last edited: Feb 20, 2016
    Yup, that's definitely a better approach. Thanks for the explanation and workaround!

    Here's a 'dig' command I found on the net which will tell you whether a given DNS server has a udp packet size limit. This is the output after setting the global configuration limit to 512; previously, it reported 4096.

    Code:
    $ dig @192.168.11.11 +noall +comments +bufsize=1 query
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 36410
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    
     
  20. president_ltd

    president_ltd Member

    Joined:
    Feb 8, 2013
    Messages:
    63
    Location:
    australia
    my guess is they did a relatively simple fix for CVE-2015-7547 to stop people using that as an attack vector and got it slightly wrong.
    my guess is they just did an iptables rule to block dns responses >512 bytes without necessarily realizing their own servers do this because of the CNAME.
     

Share This Page