Welcome to Tesla Motors Club
Discuss Tesla's Model S, Model 3, Model X, Model Y, Cybertruck, Roadster and More.
Register

Solving the "WiFi Connectivity" problem...

evanevery

Member
Feb 5, 2016
53
64
Waukesha, WI
(I posted this elsewhere in another "WiFi thread", but in hindsight, I'm not sure it was the best place for visibility...)

This is going to be a long post
but please consider it a PSA. I'm pretty sure that Tesla is not going to tell anyone any of this, but maybe it will help some of the growing number of folks who are experiencing WiFi connectivity issues in the newer vehicles. This post contains a lot of information (and background) about what I have found, and how I have specifically resolved, MY "WiFi connectivity" issue with my Model X. I have been a certified Network Engineer (CNE, MCSE) building and managing networks since the early 1980's. Fortunately I have enough tools (software), and understand how to use them, in order for me to identify and resolve this issue I was having. I'm also lucky my network equipment (Access Points and Routers) are robust enough to be able to provide me the data needed to identify the very specific source of issues which was causing me "WiFi" connectivity issues (and maybe the same as others are having as well). I'll try to be brief and precise but this story bears some background discussion. So here goes...

First: Tesla should be ashamed. The issues many of us are seeing are due to lazy network programming. Additionally, this is not a "WiFi" issue as many have surmised (simply cause the car connects via WiFi), it is a security issue. To make matters worse, the error message the car presents ("Can't connect to WiFi because it doesn't support a login page") is completely inaccurate and misleading. This is not a "WiFi" problem...

How I got here: I brought in my 2016 Model X for an MCU "upgrade" last week. Not because I really wanted one, but because it was the only way I could get the eMMC recall issue mitigated. Our SiriusXM function kept disappearing and I was told it was because of eMMC corruption. I have been waiting for that recall for about a year and a half now (and experiencing continued problems) but was told they simply could not do the recall because of the "chip shortage". However, it seems if you are willing to cough up $2000 (MCU + Radio) they don't have any problems getting chips! Its like magic! Its only if you want Tesla to honor their recall at THEIR expense that there seems to be a shortage. ANYWAY... The original MCU had no problems connecting to my network but the very next day when we brought the car home from service, the new MCU refused to connect. Same network, very next day, different MCU. So now I paid $2000 for this "upgrade" and my car is complaining that it "can't connect to my network"!

I'm not going to go through the run-around I got from service in trying to address this problem (I had to solve it myself), but here is the gist of things:

1. Here is the error message which the car reports after I provide it a valid SSID and PWD:

Could Not Join {SSID}

This network tried to present a login page which
may require authentication, payment or
acceptance of terms and conditions which is not
currently supported.

2. This is a misleading message. The car actually WAS "joining" my network just fine!

3. After getting the MAC address for the WiFi NIC in my car (from the console), I downloaded the transaction logs from my WiFi access points (Ubiquiti) and saw nothing odd in the transactions when the car tried to connect. According to the AP's everything looked fine and the car WAS connecting...

4. Furthermore, I checked my DHCP server and I saw that an IP Address was being issued to the car. So the car WAS connecting to the WiFi Network and it WAS getting an IP Address on the network. (So much for the erroneous error message). The car WAS talking on the network. So what was it saying?

5. So, now I set up a packet capture on both the inside interface and the outside interface on my router/firewall. I ran the WiFi connection exercise again from inside the car and and asked for all the packets to be captured as the conversation passed through my router (Watchguard M370).

6. The car was making an HTML page request to "connman.vn.tesla.services" (Cloudfront) and it was getting a single reply back. I compared the packets on both sides of the router/firewall, loaded both sets of captures into WireShark, and noticed the HTML reply being sent back was being filtered as it passed through my firewall.

7. The HTML reply from Tesla (back to the car) contained four HTML headers which were being stripped out by my router: X-ConnMan, X-Cache, X-Amz-Cf-Pop, and X-Amz-CF-Id. The car was getting a reply to its query, except those four headers (but not others) was being stripped out. Apparently the car didn't like this so it simply gave up and posted a misleading and inaccurate error message. So, why was the HTTP proxy on my firewall filtering the reply?

8. According to the Internet Engineering Task Force (IETF) in RFC6648, the use of "X-" HTML headers was deprecated back in 2012! These were being stripped by default in my router/firewall's HTTP proxy by default for security reasons.

rfc6648

Why we need to deprecate x prefix for HTTP headers?

*** So the router/firewall was doing EXACTLY what it was supposed to do but unfortunately Tesla listens to nobody and decided to go their own way on this. Its also unfortunate that many folks are referring this to a WiFi problem, when in fact, it is more specifically a router/firewall issue. ...and its being doen for YOUR protection. I guess it doesn't help that many folks have their WiFi Access Point and Router built into the same piece of equipment. But, it it IS a security issue, and this is TESLA's FAULT.

9. So I set up a packet filter (wide open) for the IP being assigned to my Model X so it didn't get subject to all that pesky security nonsense being imposed on all the other devices on my network which use HTML just fine. Basically, punching a hole in my router, just for my MX, solved the problem! Nice, huh?

10. After confirming this, I created a separate/dedicated HTML Proxy rule just for use by the IP assigned to my Model X (which provides no HTML Header Filtering at all). At least this gives me some level of protection rather than leaving that hole wide open (despite the Tesla Network Engineers). I'm lucky my equipment provides me the ability to diagnose these types of issues and also the granularity to address them. But what does Tesla expect the average consumer do? ...Turn off all the security on their network (cause they may not have the granularity of control to mitigate this on an individual IP or service basis)? ...or simply keep trying new network equipment until they find one "unsophisticated" enough that it doesn't provide this level of security protection at all?

The original MCU and the new MCU obviously behave differently. I had no issues with the original one yet the new one was violating the default policy established on my router. So, something has changed in the way their network protocols are functioning (or not) between the two devices. Needless to say, no one at Tesla has replied to tell me what changed in their protocols/services between the two MCU's.

Although my Service Center said they forwarded my findings on to Tesla, its not likely they are going to step up and tell everyone about how they are ignoring RFC6648. ...and what do you think the odds are that they will fix it? However, if you are having issues with "WiFi connectivity" and your Tesla, its likely because the device at the boundary of your network is doing the job it was originally designed to do.
 

luckyj

Member
Dec 8, 2016
434
509
Northern Virginia, USA
Good finding! Hate when error messages don't give any real details, just some generic message that could mean a million things. Some thoughts on your post:

I wonder if some of those X headers could be from cloudflare servers vs something Tesla did? Unfortunately a lot of HTTP server responses still include X headers, though I'm not sure how many apps would actually break if they didn't get through.

I'm guessing a fair number of folks don't have a firewall/router with a proxy feature, just NAT.

Interesting that it sounds like you observed unencrypted HTTP versus encrypted HTTPS requests. Sounds like maybe the MCU doing some initial lightweight "ping" over unencrypted HTTP, because I would think normal communication would be HTTPS.
 

Twiglett

Single pedal driver
Oct 3, 2014
3,134
3,289
Austin
Brilliant bit of sleuthing there :D

Thing is that "x-" is only deprecated, not removed so they are still a valid header and are used all over the place and not that easy to get rid of.
Deprecation just means you need to plan on migrating to whatever replaces your specific header.
For instance - the wildly popular x-forward-for currently has no replacement and is used in most proxies and load balancers which would fall apart without it.
This means that firewalls deciding to remove x- headers are being rather premature to say the least.
Sooner or later the various headers will be replaced with something other the x-, but in the meantime, networking hardware should support it or at least not default to removing it.
 
Last edited:

evanevery

Member
Feb 5, 2016
53
64
Waukesha, WI
Yeah, I get it. But they WERE deprecated and I expect there was SOME reason that they were specifically NOT on the list of the default headers that my security device would allow to pass. There is a pretty lengthy list of headers which are specifically allowed, and these were not on that. I gotta believe the guys who set up the default proxy rules had a reason to specifically not to include them. They were likely not passing for a reason. Nothing else seems broken on any other devices because of this... ...and as many of the security devices on the market use a common code base at their foundation, I expect they might also share (copy?) some of the default proxy behavior as well. If this is happening on WatchGuard Routers/Firewalls devices, it wouldn't surprise me if its happening on other popular edge-devices as well...

Its also curious to note that my Router/Firewall allows pretty much ANY HTML header to pass outbound (only two are specifically stripped), while inbound headers are only allowed if specifically authorized. IOW: outbound headers are allowed by default whereas inbound headers are stripped by default unless specifically allowed. My firewall provides individual control of the outbound and inbound traffic and set up the individual header authorizations any way I see fit. But the aforementioned is the default. Although any "X-" header is allowed to flow outbound in the page request, the only inbound X-Headers allowed in the reply are X-Dig-XMLPipe-Status, X-Pad, X-Powered-By, and x-server-ip-address. (As the "X-Forward-For" is typically sent by the client in the page request, it would not have been stripped).

Rather than adding these four headers and ending up playing whack-a-mole with any OTHER headers which were used in future communications, I simply white listed the Tesla servers for any/all http proxy filtering.

As you might guess I have a whole myriad of different devices which all play well on my networks (both at home and in the office) and the Tesla is the first one which has required any type of digital investigation to determine why it wasn't working right out of the box.

These headers could very well be Cloudflare related, but there really isn't anything else in the html message which seems to have any unique identifying information. I'm guessing they are using these headers to pass some sort of serialization and identification back and forth and without them the car was not very happy. ...and there are a million ways to do this without having to use deprecated HTM headers which can (are obviously ARE) being filtered by some equipment.

...and yes, curious it was simple http (and not https). Once I figured out what was causing my "WiFi Unable to Join Network" message, and found a suitable workaround, I haven't really looked back. I should run another packet trace to see if it ever switches over to https (or something else). Of course, there really isn't any benefit to start with http in the first place if you plan on transitioning later. Why not use https right out of the gate?

Regardless, this is obviously a functional change in the way the old MCU operated vs the current MCU.
 

Products we're discussing on TMC...

About Us

Formed in 2006, Tesla Motors Club (TMC) was the first independent online Tesla community. Today it remains the largest and most dynamic community of Tesla enthusiasts. Learn more.

Do you value your experience at TMC? Consider becoming a Supporting Member of Tesla Motors Club. As a thank you for your contribution, you'll get nearly no ads in the Community and Groups sections. Additional perks are available depending on the level of contribution. Please visit the Account Upgrades page for more details.


SUPPORT TMC
Top