Strange network behavior... car incessantly attacking my nameserver

wk057 · Feb 20, 2016

president_ltd said:
my guess is they did a relatively simple fix for CVE-2015-7547 to stop people using that as an attack vector and got it slightly wrong.
my guess is they just did an iptables rule to block dns responses >512 bytes without necessarily realizing their own servers do this because of the CNAME.

They did not.

thecloud · Feb 20, 2016

president_ltd said:
my guess is they did a relatively simple fix for CVE-2015-7547 to stop people using that as an attack vector and got it slightly wrong.
my guess is they just did an iptables rule to block dns responses >512 bytes without necessarily realizing their own servers do this because of the CNAME.

Unlikely. This behavior has been the same for over a year (I previously reported it to Tesla on Thu, 5 Feb 2015), and probably has always been this way. All this time, it had not occurred to me there was an issue with the response, whose contents appeared valid and which my server was effectively just passing along from the upstream DNS. Now I know better.

And speaking of that CVE, the potential overflow occurs only when a response length is greater than 2048, but since the Tesla misbehavior happens with lengths >512, and since old routers might drop larger UDP packets anyway, 512 seems to be the right place to set the limit for now.

strider · Feb 20, 2016

Thanks everyone! I'll add that option to my named.conf. Now to figure out which of the 23 "include" files OS X Server uses to add it to.

jgallops · Feb 21, 2016

Now I know why I have been having the connectivity issues on my network. I also reported it to Tesla a number of months back and I got the standard message about giving the problem to the local SC.

Rather than jump through hoops playing with my DNS, I just banned the car's MAC address from my WiFi so the car can't connect. Let Tesla pay for the data until they fix their problem. Now my iPhone app works all the time instead of almost never when on my network.

Many thanks for the research!

Cosmacelf · Feb 21, 2016

Out of curiosity, why does querying an external name server work ok? Do they by default only have 512 byte UDP packets?

FlasherZ · Feb 21, 2016

Cosmacelf said:
Out of curiosity, why does querying an external name server work ok? Do they by default only have 512 byte UDP packets?

It appears that Google's external servers are configured for 512 byte UDP maximum. I tested using various open recursive resolvers out there and found most of them are giving big replies.

thecloud · Feb 21, 2016

Cosmacelf said:
Out of curiosity, why does querying an external name server work ok? Do they by default only have 512 byte UDP packets?

By default, current versions of the BIND name server allow a maximum size of 4096 bytes for UDP DNS responses, unless configured otherwise. (That's done by setting the max-udp-size option in named.conf, per the manual: "The usual reason for setting max-udp-size to a non-default value is to get UDP answers to pass through broken firewalls that block fragmented packets and/or block UDP packets that are greater than 512 bytes.")

So, the answer to your question depends on how the particular external name server is configured. Google's public name servers are configured with a max udp value of 512, so won't have the problem. Comcast's DNS, on the other hand, has a max udp value of 4000. (Not 4096... they've deliberately set it at 4000 for some reason.)

bollar · Feb 21, 2016

strider said:
Thanks everyone! I'll add that option to my named.conf. Now to figure out which of the 23 "include" files OS X Server uses to add it to.

When you sort that out, please share with the rest of us!

bmah · Feb 21, 2016

strider said:
Thanks everyone! I'll add that option to my named.conf. Now to figure out which of the 23 "include" files OS X Server uses to add it to.

bollar said:
When you sort that out, please share with the rest of us!

Or maybe you could just go back to FreeBSD. :smile:

/me ducks

(Seriously, way cool to be on a car forum and see a discussion of UDP packet sizes used by DNS servers.)

strider · Feb 22, 2016

bmah said:
Or maybe you could just go back to FreeBSD. :smile:

/me ducks

(Seriously, way cool to be on a car forum and see a discussion of UDP packet sizes used by DNS servers.)

Ha! Yeah, I've always enjoyed the level of nerdiness here. Way to work in IRC there too

But at the expense of turning in my nerd card I need help

Here's the first part of my named.conf (I found it in /Library/Server/named)

Code:

include "/Library/Server/named/rndc.key";
options {
        directory "/Library/Server/named";
        listen-on-v6 {
                "any";
        };
        allow-recursion {
                com.apple.ServerAdmin.DNS.public;
        };
        max-udp-size 512;
        allow-transfer {
                none;
        };
        forwarders {
                8.34.34.34;
                8.35.35.35;
        };
};

You can see I added the max-udp-size to the options section. I did a kill -HUP (and even used the GUI to stop/start the service) But when I run the dig command I get:
dig @127.0.0.1 +noall +comments +bufsize=1 query
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 23879
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096

It's like it's ignoring that option. Just need a sanity check that I'm putting it in the right place (I've been a Director for the last 5 years so am getting more familiar with PowerPoint and Excel than I am vi any more

) before I go trying to figure out how else they've ba$terdized BIND.

FlasherZ · Feb 22, 2016

Well, there are two options... There's the edns size, and max-udp-size. Use dig to ask for vpn.vn.teslamotors.com and check the response size at the bottom.

strider · Feb 22, 2016

It gave me:
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Feb 22 16:31:01 2016
;; MSG SIZE rcvd: 460

So maybe it's working. I'm able to talk to the car over WiFi (but of course I also just HUP'd named to make the change). I also set edns-udp-size 512 as well and I get the same result. I'll keep my eye on it. Strange that thecloud's dig command still shows the 4096 number.

FlasherZ · Feb 22, 2016

It is working.

I have only max-udp-size 512 set, my EDNS setting is still default at 4k. This is because a smart EDNS resolver won't have a problem, only Tesla's basic DNS queries have issues...

You have to use dig with +noedns to make it do a standard query, in which case you can see it get truncated at 512:

Code:

; <<>> DiG 9.9.5-3ubuntu0.7-Ubuntu <<>> +noedns vpn.vn.teslamotors.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45197
;; flags: qr rd ra; QUERY: 1, ANSWER: 26, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;vpn.vn.teslamotors.com.                IN      A

;; ANSWER SECTION:
vpn.vn.teslamotors.com. 0       IN      CNAME   usvpn.vn.teslamotors.com.
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.40
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.23
usvpn.vn.teslamotors.com. 0     IN      A       205.234.27.234
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.46
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.75
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.48
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.28
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.43
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.22
usvpn.vn.teslamotors.com. 0     IN      A       205.234.27.198
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.39
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.30
usvpn.vn.teslamotors.com. 0     IN      A       205.234.27.254
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.45
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.44
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.50
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.49
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.24
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.42
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.38
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.51
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.29
usvpn.vn.teslamotors.com. 0     IN      A       209.10.208.47
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.12
usvpn.vn.teslamotors.com. 0     IN      A       209.11.133.27

;; AUTHORITY SECTION:
teslamotors.com.        129847  IN      NS      udns1.ultradns.net.
teslamotors.com.        129847  IN      NS      udns2.ultradns.net.

;; Query time: 0 msec
;; SERVER: 
;; WHEN: Mon Feb 22 20:26:23 CST 2016
;; MSG SIZE  rcvd: 512

bollar · Feb 22, 2016

You guys are smarter than I am...

Code:

[FONT=Menlo]include "/Library/Server/named/rndc.key";[/FONT]
[FONT=Menlo]options {[/FONT]
[FONT=Menlo]        directory "/Library/Server/named";[/FONT]
[FONT=Menlo]        allow-recursion {[/FONT]
[FONT=Menlo]                com.apple.ServerAdmin.DNS.public;[/FONT]
[FONT=Menlo]        };[/FONT]
[FONT=Menlo]        max-udp-size 512;[/FONT]
[FONT=Menlo]        allow-transfer {[/FONT]
[FONT=Menlo]                none;[/FONT]
[FONT=Menlo]        };[/FONT]
[FONT=Menlo]        forwarders {[/FONT]
[FONT=Menlo]                208.67.222.222;[/FONT]
[FONT=Menlo]                208.67.220.220;[/FONT]
[FONT=Menlo]        };[/FONT]
[FONT=Menlo]};

[/FONT]
[FONT=Menlo]server:named mediaserver$ dig @127.0.0.1 +noedns vpn.vn.teslamotors.com

[/FONT]
[FONT=Menlo];; Query time: 1 msec[/FONT]
[FONT=Menlo];; SERVER: 127.0.0.1#53(127.0.0.1)[/FONT]
[FONT=Menlo];; WHEN: Mon Feb 22 22:14:25 2016[/FONT]
[FONT=Menlo];; MSG SIZE  rcvd: 671

[/FONT]

FlasherZ · Feb 22, 2016

Unfortunately, I can't help you. All of my stuff is Linux/FreeBSD. The command is correct. Any syslog evidence on named startup?

thecloud · Feb 23, 2016

First off, I missed spotting that 'query' should not have been a literal argument to the 'dig' command I posted earlier. It should be replaced with an actual query string (in this case, 'vpn.vn.teslamotors.com').

A non-zero bufsize argument to 'dig' causes an EDNS query to be sent from the client; I've changed it to 2048 below so it's clear that the client is advertising it has a 2048-byte buffer to receive the response. If your server's max-udp-size option is set to 512, then the response from the server should not be larger than 512, regardless of what the client advertises. (And if you pass +bufsize=0, that is identical to +noedns, in which case I'd also expect the reply to be limited to 512 since it's a non-EDNS query.)

EDNS query:

Code:

$ dig @127.0.0.1  +comment +bufsize=2048 vpn.vn.teslamotors.com
...
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
...

Standard query (non-EDNS):

Code:

$ dig @127.0.0.1  +comment +bufsize=0 vpn.vn.teslamotors.com

bollar, what values do you see below OPT PSEUDOSECTION with the first version of the dig command above?

bollar · Feb 23, 2016

thecloud said:
bollar, what values do you see below OPT PSEUDOSECTION with the first version of the dig command above?

Thanks for the help! I see that the message size is less than 512 this morning, but can't connect to the car once it's connected to WiFi.

Code:

[FONT=Menlo]$ dig @127.0.0.1  +comment +bufsize=2048 vpn.vn.teslamotors.com[/FONT][FONT=Menlo]
[/FONT]
[FONT=Menlo]; <<>> DiG 9.8.3-P1 <<>> @127.0.0.1 +comment +bufsize=2048 vpn.vn.teslamotors.com[/FONT]
[FONT=Menlo]; (1 server found)[/FONT]
[FONT=Menlo];; global options: +cmd[/FONT]
[FONT=Menlo];; Got answer:[/FONT]
[FONT=Menlo];; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21439[/FONT]
[FONT=Menlo];; flags: qr rd ra; QUERY: 1, ANSWER: 26, AUTHORITY: 0, ADDITIONAL: 1[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo];; OPT PSEUDOSECTION:[/FONT]
[FONT=Menlo]; EDNS: version: 0, flags:; udp: 4096[/FONT]
[FONT=Menlo];; QUESTION SECTION:[/FONT]
[FONT=Menlo];vpn.vn.teslamotors.com.        IN    A[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo];; ANSWER SECTION:[/FONT]
[FONT=Menlo]vpn.vn.teslamotors.com.    300    IN    CNAME    usvpn.vn.teslamotors.com.[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.28[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.46[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.43[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.45[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.38[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.44[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.75[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.40[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.47[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.12[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.39[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.22[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.50[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.51[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    205.234.27.198[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.42[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.30[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.24[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.29[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.49[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.10.208.48[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    205.234.27.234[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.23[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    205.234.27.254[/FONT]
[FONT=Menlo]usvpn.vn.teslamotors.com. 300    IN    A    209.11.133.27[/FONT]
[FONT=Menlo]
[/FONT]
[FONT=Menlo];; Query time: 15 msec[/FONT]
[FONT=Menlo];; SERVER: 127.0.0.1#53(127.0.0.1)[/FONT]
[FONT=Menlo];; WHEN: Tue Feb 23 06:52:28 2016[/FONT]
[FONT=Menlo];; MSG SIZE  rcvd: 471[/FONT]

And here's the entire named.conf, just in case it's helpful:

Code:

include "/Library/Server/named/rndc.key";options {
	directory "/Library/Server/named";
	allow-recursion {
		com.apple.ServerAdmin.DNS.public;
	};
	max-udp-size 512;
	allow-transfer {
		none;
	};
	forwarders {
		208.67.222.222;
		208.67.220.220;
	};
};
controls {
	inet 127.0.0.1 port 54 allow {
		"any";
	} keys {
		"rndc-key";
	};
};
acl "com.apple.ServerAdmin.DNS.public" {
	localnets;
};
logging {
	channel "_default_log" {
		file "/Library/Logs/named.log";
		severity info;
		print-time yes;
	};
	category "default" {
		"_default_log";
	};
};
view "com.apple.ServerAdmin.DNS.public" {
	zone "1.1.10.in-addr.arpa" IN {
		type master;
		file "db.1.1.10.in-addr.arpa";
		allow-transfer {
			none;
		};
		allow-update {
			none;
		};
	};
	zone "bollar.com" IN {
		type master;
		file "db.bollar.com";
		allow-transfer {
			com.apple.ServerAdmin.DNS.public;
		};
		allow-update {
			none;
		};
	};
	zone "0.0.127.in-addr.arpa" IN {
		type master;
		file "named.local";
		allow-update {
			none;
		};
	};
	zone "localhost" IN {
		type master;
		file "localhost.zone";
		allow-update {
			none;
		};
	};
	zone "0.0.10.in-addr.arpa" IN {
		type master;
		file "db.0.0.10.in-addr.arpa";
		allow-transfer {
			none;
		};
		allow-update {
			none;
		};
	};
	zone "141.119.216.in-addr.arpa" IN {
		type master;
		file "db.141.119.216.in-addr.arpa";
		allow-transfer {
			none;
		};
		allow-update {
			none;
		};
	};
	zone "." IN {
		type hint;
		file "named.ca";
	};
};

mknox · Feb 23, 2016

Pardon my complete ignorance here, but could any of this have an adverse effect on my home network? My family has been complaining about sporadic internet (not WiFi) drop outs and there seems to be some correlation to when my car is at home and connected.

green1 · Feb 23, 2016

mknox said:
Pardon my complete ignorance here, but could any of this have an adverse effect on my home network? My family has been complaining about sporadic internet (not WiFi) drop outs and there seems to be some correlation to when my car is at home and connected.

The reports are of it "attacking" the name server every 5 seconds or so, in network terms that's fairly slow, so I don't see it negatively impacting performance of the rest of the network.

strider · Feb 23, 2016

Bah. Mine is back to MSG SIZE rcvd: 671 So that command is not being honored in OS X server for some reason. So bollar, I'm not smarter than you, for some reason right after restarting named the messages are shorter and then they get longer. Going to have to keep digging on why that option is being ignored.

Strange network behavior... car incessantly attacking my nameserver

Former Tesla Tinkerer

As rhythm raced inside, the ship came alive

Active Member

Member

Well-Known Member

Sig Model S + Sig Model X + Model 3 Resv

As rhythm raced inside, the ship came alive

Disgruntled Member

Moderator, Supercharger Hunter

Active Member

Sig Model S + Sig Model X + Model 3 Resv

Active Member

Sig Model S + Sig Model X + Model 3 Resv

Disgruntled Member

Sig Model S + Sig Model X + Model 3 Resv

As rhythm raced inside, the ship came alive

Disgruntled Member

Well-Known Member

Active Member

Active Member