Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Strange network behavior... car incessantly attacking my nameserver

This site may earn commission on affiliate links.

FlasherZ

Sig Model S + Sig Model X + Model 3 Resv
Jun 21, 2012
7,030
1,032
For all of you other network geeks, I'd like to know if you've seen this odd behavior. Tonight, I noticed that the car no longer had connectivity through Tesla's VPN tunnel while on Wi-Fi. It had Slacker, the web browser worked, etc., but the remote app couldn't "see" the car and all my API calls returned "car unavailable".

I logged into the management portal for my AP's (Meraki), did a packet capture, and saw the following:

Code:
16:56:06.152567 IP y.y.y.y.53 > x.x.x.x.15413: 56407| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.38, A 209.10.208.44, A 209.10.208.49, A 209.11.133.30, A 209.11.133.29, A 209.10.208.51, A 209.11.133.23, A 209.11.133.40, A 205.234.27.254, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.27, A 209.10.208.43, A 209.11.133.39, A 209.10.208.48, A 209.10.208.42, A 209.10.208.47, A 205.234.27.198, A 205.234.27.234, A 209.10.208.50, A 209.10.208.45, A 209.11.133.24, A 209.11.133.28, A 209.10.208.46 (488)
16:56:06.068971 IP y.y.y.y.53 > x.x.x.x.15413: 56407| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.38, A 209.10.208.44, A 209.10.208.49, A 209.11.133.30, A 209.11.133.29, A 209.10.208.51, A 209.11.133.23, A 209.11.133.40, A 205.234.27.254, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.27, A 209.10.208.43, A 209.11.133.39, A 209.10.208.48, A 209.10.208.42, A 209.10.208.47, A 205.234.27.198, A 205.234.27.234, A 209.10.208.50, A 209.10.208.45, A 209.11.133.24, A 209.11.133.28, A 209.10.208.46 (488)
16:56:11.085548 IP y.y.y.y.53 > x.x.x.x.47240: 31174| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.30, A 209.10.208.51, A 209.10.208.43, A 209.10.208.47, A 209.11.133.38, A 205.234.27.234, A 209.11.133.22, A 209.11.133.28, A 209.11.133.24, A 209.11.133.27, A 205.234.27.254, A 209.11.133.23, A 209.10.208.50, A 209.10.208.48, A 209.11.133.40, A 209.11.133.39, A 205.234.27.198, A 209.10.208.44, A 209.10.208.45, A 209.11.133.12, A 209.10.208.46, A 209.11.133.29, A 209.10.208.42, A 209.10.208.75, A 209.10.208.49 (488)
16:56:11.100927 IP y.y.y.y.53 > x.x.x.x.57265: 50032| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.10.208.75, A 209.11.133.29, A 209.10.208.45, A 209.10.208.49, A 209.11.133.23, A 205.234.27.254, A 209.10.208.51, A 209.10.208.43, A 209.11.133.38, A 209.10.208.47, A 209.11.133.28, A 209.11.133.40, A 209.10.208.48, A 205.234.27.198, A 209.11.133.12, A 209.11.133.27, A 209.11.133.24, A 209.11.133.22, A 209.10.208.42, A 209.10.208.44, A 209.11.133.30, A 209.11.133.39, A 205.234.27.234, A 209.10.208.50, A 209.10.208.46 (488)
16:56:16.114600 IP y.y.y.y.53 > x.x.x.x.32164: 47423| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.22, A 209.11.133.30, A 209.10.208.43, A 209.10.208.50, A 209.11.133.23, A 209.11.133.24, A 209.11.133.12, A 205.234.27.234, A 209.10.208.75, A 205.234.27.254, A 209.11.133.40, A 209.10.208.51, A 209.10.208.42, A 209.10.208.46, A 209.11.133.38, A 205.234.27.198, A 209.11.133.27, A 209.10.208.48, A 209.10.208.44, A 209.10.208.49, A 209.10.208.45, A 209.11.133.39, A 209.10.208.47, A 209.11.133.28, A 209.11.133.29 (488)
16:56:16.125739 IP y.y.y.y.53 > x.x.x.x.34201: 43294| 26/0/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.29, A 209.10.208.45, A 205.234.27.234, A 209.10.208.44, A 209.11.133.24, A 209.10.208.43, A 205.234.27.254, A 209.10.208.50, A 209.10.208.47, A 209.11.133.22, A 209.10.208.42, A 205.234.27.198, A 209.11.133.30, A 209.11.133.39, A 209.10.208.49, A 209.10.208.51, A 209.10.208.46, A 209.11.133.40, A 209.10.208.75, A 209.11.133.38, A 209.11.133.28, A 209.11.133.12, A 209.10.208.48, A 209.11.133.23, A 209.11.133.27 (488)

Every 5 seconds, the car was querying my nameserver (BIND9). This continued for well over 15 minutes. There were no errors on the nameserver, just query after query after query. I didn't see a problem with the query or response, it just seemed that the car was ignoring them and retrying again.

I fixed it by turning off Wi-Fi in the car, stopping my nameserver, and turning Wi-Fi back on. The car then tries to send nameserver requests for a while, gets frustrated, and gives up. I then restarted the nameserver and then the car did a query and reconnected its tunnel.

(One other behavior - it appears the car cannot use a secondary nameserver. When I took my primary nameserver down, it just kept sending queries despite getting back port unreachable messages. It never tried to contact the backup nameserver.)

Has anyone else seen this behavior?
 
Every 5 seconds sounds pretty slow. Answer should be cached.

I'm unsure that the car was ever getting the response in the first place. It kept asking, and my nameserver kept responding, but the car just continued asking. So if it never heard a response in the first place, it couldn't cache it.

My nameserver was serving a cached response, from what I could see.
 
When I first got my car, I couldn't get the mobile app to connect to it when the car was on my home WiFi. Upon packet sniffing, I think I saw similar behavior (but don't recall completely, since its been a while). I then configured my DHCP server to give the car Google's DNS server IPs instead, and suddenly everything started working fine.
 
Just an honest question: why does anybody bother with WiFi? just turn it off an leave it off...
Originally it was stated that Wi-Fi would get updates faster and maps would only be updated through Wi-Fi. I don't believe that's still the case (although my Wi-Fi connection is very weak and I haven't received a map upgraded in the three years I've had my Model S). If you don't have LTE, Wi-Fi through a phone's hotspot can be faster.
 
Originally it was stated that Wi-Fi would get updates faster and maps would only be updated through Wi-Fi. I don't believe that's still the case (although my Wi-Fi connection is very weak and I haven't received a map upgraded in the three years I've had my Model S). If you don't have LTE, Wi-Fi through a phone's hotspot can be faster.

Not to mention that Tesla originally told owners they would have to start paying for data connectivity to the cars... but that's a difficult one because of all the other data they get.
 
Yes, I see the same behavior on a daily basis. I run a caching DNS server on my internal network at 192.168.11.11, and the DHCP server is configured to specify that server for DNS. The car keeps waking up periodically, asks to resolve 'vpn.vn.teslamotors.com', and the server keeps sending the same valid response from its cache. Eventually the car stops asking, and then it starts up again later.

Code:
21:09:33.544624 IP 192.168.11.102.38947 > 192.168.11.11.53:  38171+ A? vpn.vn.teslamotors.com. (40)
21:09:33.545559 IP 192.168.11.11.53 > 192.168.11.102.38947:  38171| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24, A 209.11.133.27, A 209.11.133.28, A 209.11.133.29 (506)
21:09:38.558548 IP 192.168.11.102.16349 > 192.168.11.11.53:  20048+ A? vpn.vn.teslamotors.com. (40)
21:09:38.559858 IP 192.168.11.11.53 > 192.168.11.102.16349:  20048| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24, A 209.11.133.27, A 209.11.133.28 (506)
21:09:38.578473 IP 192.168.11.102.12116 > 192.168.11.11.53:  64125+ A? vpn.vn.teslamotors.com. (40)
21:09:38.579472 IP 192.168.11.11.53 > 192.168.11.102.12116:  64125| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.28, A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24, A 209.11.133.27 (506)
21:09:43.589701 IP 192.168.11.102.21944 > 192.168.11.11.53:  25485+ A? vpn.vn.teslamotors.com. (40)
21:09:43.591042 IP 192.168.11.11.53 > 192.168.11.102.21944:  25485| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.27, A 209.11.133.28, A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24 (506)
21:09:43.600824 IP 192.168.11.102.64403 > 192.168.11.11.53:  10655+ A? vpn.vn.teslamotors.com. (40)
21:09:43.601748 IP 192.168.11.11.53 > 192.168.11.102.64403:  10655| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.11.133.24, A 209.11.133.27, A 209.11.133.28, A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47, A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23 (506)

I've reported the issue to Tesla but never heard anything back.
 
I'm wondering (complete WAG here) if they're running into a 512-byte buffer problem with the DNS query - I saw it as 488 on the payload, you see it as 506. I remember this happening years ago with some other services that had extra-large query responses.
 
I'm wondering (complete WAG here) if they're running into a 512-byte buffer problem with the DNS query - I saw it as 488 on the payload, you see it as 506. I remember this happening years ago with some other services that had extra-large query responses.
Could you test that by setting up your name server to reply with a smaller payload? Say only half the IPs, and see if it solves it?
 
I've never been able to contact our car over WiFi (though maps, radio, browser work). I thought it had something to do w/ my WiFi (Aruba) not NATing the IPSec properly but haven't taken the time to troubleshoot. I also run a local name server (OS X Server - I got tired of manually patching FreeBSD) so maybe it's related. I'll try bouncing named.

-Update-
That fixed it. Bizarre. I didn't even touch the car. I just did a kill -HUP of named and then fired up the app. It gave the the "Waking car" message and then connected (previously is would hang on "Waking car"). Definitely something strange going on.
 
Last edited:
The way wifi works on the car is weird. Wifi is handled by an independent MCU board that runs linux. It then NATs wifi traffic for the CID along with running dnsmasq. The CID runs its own dnsmasq as well to split queries for Tesla's VPN hosts off to their internal nameserver. All of this is coordinated by a process on the CID....... and I've seen it get confused quite a few times.

Most likely what was happening was the car was sending the DNS request out via wifi and expecting a response via cellular or Tesla's tunnel... thus the constant retries.

If your home wifi IP mask is in the same range as anything internal to the car is screws things up too. Works half-assed, breaks sometimes, sometimes acts randomly.
 
The way wifi works on the car is weird. Wifi is handled by an independent MCU board that runs linux. It then NATs wifi traffic for the CID along with running dnsmasq. The CID runs its own dnsmasq as well to split queries for Tesla's VPN hosts off to their internal nameserver. All of this is coordinated by a process on the CID....... and I've seen it get confused quite a few times.

Most likely what was happening was the car was sending the DNS request out via wifi and expecting a response via cellular or Tesla's tunnel... thus the constant retries.

If your home wifi IP mask is in the same range as anything internal to the car is screws things up too. Works half-assed, breaks sometimes, sometimes acts randomly.

It's not in this case. I have my own BGP-advertised public IP address blocks and the car gets a public IP from them.

I don't think it's the car sending/expecting a response via different paths - if I bounce the DNS server (as strider did above), then the car goes into happy mode again once it gets a smaller query.

The query on the wire and the query response all look good, so I don't think it's anything having to do with BIND.

I did confirm, though, that the message size is > 512 bytes (525 bytes from one of my nameservers, 555 from the other), so that seems like the best target to look at right now. I dropped a note to the engineering team so they can look at it.

I also configured BIND to use a maximum UDP size of 512 bytes by setting "max-udp-size 512" in the options configuration. When a query is over 512 bytes, then BIND will retry by advertising a smaller EDNS size to authoritative servers. This should make it work temporarily, until Tesla fixes its ability to receive queries > 512 bytes.

Before:
;; Query time: 0 msec
;; WHEN: Sat Feb 20 08:35:32 CST 2016
;; MSG SIZE rcvd: 555

After:
;; Query time: 0 msec
;; WHEN: Sat Feb 20 08:36:45 CST 2016
;; MSG SIZE rcvd: 471
 
Last edited:
Well, it looks like FlasherZ's WAG was right. After I set up BIND to return a single IP address when it asked to resolve the hostname, I got exactly one query from the car, and the app subsequently connected to the car right away.

Code:
11:32:04.363872 IP 192.168.11.102.21406 > 192.168.11.11.53:  4052+ A? vpn.vn.teslamotors.com. (40)
11:32:05.639689 IP 192.168.11.11.53 > 192.168.11.102.21406:  4052* 3/1/1 CNAME usvpn.vn.teslamotors.com., CNAME vpn-209-10-208-42.vn.teslamotors.com., A 209.10.208.42 (157)

Note this response is 157 bytes, compared to 506 bytes just before I switched the configuration:

Code:
11:31:54.344006 IP 192.168.11.102.30306 > 192.168.11.11.53:  21672+ A? vpn.vn.teslamotors.com. (40)
11:31:54.344931 IP 192.168.11.11.53 > 192.168.11.102.30306:  21672| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24, A 209.11.133.27, A 209.11.133.28, A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47 (506)
 
Well, it looks like FlasherZ's WAG was right. After I set up BIND to return a single IP address when it asked to resolve the hostname, I got exactly one query from the car, and the app subsequently connected to the car right away.

Code:
11:32:04.363872 IP 192.168.11.102.21406 > 192.168.11.11.53:  4052+ A? vpn.vn.teslamotors.com. (40)
11:32:05.639689 IP 192.168.11.11.53 > 192.168.11.102.21406:  4052* 3/1/1 CNAME usvpn.vn.teslamotors.com., CNAME vpn-209-10-208-42.vn.teslamotors.com., A 209.10.208.42 (157)

Note this response is 157 bytes, compared to 506 bytes just before I switched the configuration:

Code:
11:31:54.344006 IP 192.168.11.102.30306 > 192.168.11.11.53:  21672+ A? vpn.vn.teslamotors.com. (40)
11:31:54.344931 IP 192.168.11.11.53 > 192.168.11.102.30306:  21672| 26/2/0 CNAME usvpn.vn.teslamotors.com., A 209.10.208.48, A 209.10.208.49, A 209.10.208.50, A 209.10.208.51, A 209.10.208.75, A 209.11.133.12, A 209.11.133.22, A 209.11.133.23, A 209.11.133.24, A 209.11.133.27, A 209.11.133.28, A 209.11.133.29, A 209.11.133.30, A 209.11.133.38, A 209.11.133.39, A 209.11.133.40, A 205.234.27.198, A 205.234.27.234, A 205.234.27.254, A 209.10.208.42, A 209.10.208.43, A 209.10.208.44, A 209.10.208.45, A 209.10.208.46, A 209.10.208.47 (506)

Thanks for the confirmation. You can use the global option "max-udp-size 512;" to make the server hand better query responses to the car until they fix it.
 
Thanks for the confirmation. You can use the global option "max-udp-size 512;" to make the server hand better query responses to the car until they fix it.
Yup, that's definitely a better approach. Thanks for the explanation and workaround!

Here's a 'dig' command I found on the net which will tell you whether a given DNS server has a udp packet size limit. This is the output after setting the global configuration limit to 512; previously, it reported 4096.

Code:
$ dig @192.168.11.11 +noall +comments +bufsize=1 query
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 36410
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
 
Last edited:
I did confirm, though, that the message size is > 512 bytes (525 bytes from one of my nameservers, 555 from the other), so that seems like the best target to look at right now. I dropped a note to the engineering team so they can look at it.

my guess is they did a relatively simple fix for CVE-2015-7547 to stop people using that as an attack vector and got it slightly wrong.
my guess is they just did an iptables rule to block dns responses >512 bytes without necessarily realizing their own servers do this because of the CNAME.