Solving the "WiFi Connectivity" problem...

[evanevery]

(I posted this elsewhere in another "WiFi thread", but in hindsight, I'm not sure it was the best place for visibility...)

This is going to be a long post but please consider it a PSA. I'm pretty sure that Tesla is not going to tell anyone any of this, but maybe it will help some of the growing number of folks who are experiencing WiFi connectivity issues in the newer vehicles. This post contains a lot of information (and background) about what I have found, and how I have specifically resolved, MY "WiFi connectivity" issue with my Model X. I have been a certified Network Engineer (CNE, MCSE) building and managing networks since the early 1980's. Fortunately I have enough tools (software), and understand how to use them, in order for me to identify and resolve this issue I was having. I'm also lucky my network equipment (Access Points and Routers) are robust enough to be able to provide me the data needed to identify the very specific source of issues which was causing me "WiFi" connectivity issues (and maybe the same as others are having as well). I'll try to be brief and precise but this story bears some background discussion. So here goes...

First: Tesla should be ashamed. The issues many of us are seeing are due to lazy network programming. Additionally, this is not a "WiFi" issue as many have surmised (simply cause the car connects via WiFi), it is a security issue. To make matters worse, the error message the car presents ("Can't connect to WiFi because it doesn't support a login page") is completely inaccurate and misleading. This is not a "WiFi" problem...

How I got here: I brought in my 2016 Model X for an MCU "upgrade" last week. Not because I really wanted one, but because it was the only way I could get the eMMC recall issue mitigated. Our SiriusXM function kept disappearing and I was told it was because of eMMC corruption. I have been waiting for that recall for about a year and a half now (and experiencing continued problems) but was told they simply could not do the recall because of the "chip shortage". However, it seems if you are willing to cough up $2000 (MCU + Radio) they don't have any problems getting chips! Its like magic! Its only if you want Tesla to honor their recall at THEIR expense that there seems to be a shortage. ANYWAY... The original MCU had no problems connecting to my network but the very next day when we brought the car home from service, the new MCU refused to connect. Same network, very next day, different MCU. So now I paid $2000 for this "upgrade" and my car is complaining that it "can't connect to my network"!

I'm not going to go through the run-around I got from service in trying to address this problem (I had to solve it myself), but here is the gist of things:

1. Here is the error message which the car reports after I provide it a valid SSID and PWD:

Could Not Join {SSID}

This network tried to present a login page which
may require authentication, payment or
acceptance of terms and conditions which is not
currently supported.

2. This is a misleading message. The car actually WAS "joining" my network just fine!

3. After getting the MAC address for the WiFi NIC in my car (from the console), I downloaded the transaction logs from my WiFi access points (Ubiquiti) and saw nothing odd in the transactions when the car tried to connect. According to the AP's everything looked fine and the car WAS connecting...

4. Furthermore, I checked my DHCP server and I saw that an IP Address was being issued to the car. So the car WAS connecting to the WiFi Network and it WAS getting an IP Address on the network. (So much for the erroneous error message). The car WAS talking on the network. So what was it saying?

5. So, now I set up a packet capture on both the inside interface and the outside interface on my router/firewall. I ran the WiFi connection exercise again from inside the car and and asked for all the packets to be captured as the conversation passed through my router (Watchguard M370).

6. The car was making an HTML page request to "connman.vn.tesla.services" (Cloudfront) and it was getting a single reply back. I compared the packets on both sides of the router/firewall, loaded both sets of captures into WireShark, and noticed the HTML reply being sent back was being filtered as it passed through my firewall.

7. The HTML reply from Tesla (back to the car) contained four HTML headers which were being stripped out by my router: X-ConnMan, X-Cache, X-Amz-Cf-Pop, and X-Amz-CF-Id. The car was getting a reply to its query, except those four headers (but not others) was being stripped out. Apparently the car didn't like this so it simply gave up and posted a misleading and inaccurate error message. So, why was the HTTP proxy on my firewall filtering the reply?

8. According to the Internet Engineering Task Force (IETF) in RFC6648, the use of "X-" HTML headers was deprecated back in 2012! These were being stripped by default in my router/firewall's HTTP proxy by default for security reasons.

rfc6648

Why we need to deprecate x prefix for HTTP headers?

*** So the router/firewall was doing EXACTLY what it was supposed to do but unfortunately Tesla listens to nobody and decided to go their own way on this. Its also unfortunate that many folks are referring this to a WiFi problem, when in fact, it is more specifically a router/firewall issue. ...and its being doen for YOUR protection. I guess it doesn't help that many folks have their WiFi Access Point and Router built into the same piece of equipment. But, it it IS a security issue, and this is TESLA's FAULT.

9. So I set up a packet filter (wide open) for the IP being assigned to my Model X so it didn't get subject to all that pesky security nonsense being imposed on all the other devices on my network which use HTML just fine. Basically, punching a hole in my router, just for my MX, solved the problem! Nice, huh?

10. After confirming this, I created a separate/dedicated HTML Proxy rule just for use by the IP assigned to my Model X (which provides no HTML Header Filtering at all). At least this gives me some level of protection rather than leaving that hole wide open (despite the Tesla Network Engineers). I'm lucky my equipment provides me the ability to diagnose these types of issues and also the granularity to address them. But what does Tesla expect the average consumer do? ...Turn off all the security on their network (cause they may not have the granularity of control to mitigate this on an individual IP or service basis)? ...or simply keep trying new network equipment until they find one "unsophisticated" enough that it doesn't provide this level of security protection at all?

The original MCU and the new MCU obviously behave differently. I had no issues with the original one yet the new one was violating the default policy established on my router. So, something has changed in the way their network protocols are functioning (or not) between the two devices. Needless to say, no one at Tesla has replied to tell me what changed in their protocols/services between the two MCU's.

Although my Service Center said they forwarded my findings on to Tesla, its not likely they are going to step up and tell everyone about how they are ignoring RFC6648. ...and what do you think the odds are that they will fix it? However, if you are having issues with "WiFi connectivity" and your Tesla, its likely because the device at the boundary of your network is doing the job it was originally designed to do.

 

[luckyj]

Good finding! Hate when error messages don't give any real details, just some generic message that could mean a million things. Some thoughts on your post:

I wonder if some of those X headers could be from cloudflare servers vs something Tesla did? Unfortunately a lot of HTTP server responses still include X headers, though I'm not sure how many apps would actually break if they didn't get through.

I'm guessing a fair number of folks don't have a firewall/router with a proxy feature, just NAT.

Interesting that it sounds like you observed unencrypted HTTP versus encrypted HTTPS requests. Sounds like maybe the MCU doing some initial lightweight "ping" over unencrypted HTTP, because I would think normal communication would be HTTPS.

 

[Twiglett]

Brilliant bit of sleuthing there

Thing is that "x-" is only deprecated, not removed so they are still a valid header and are used all over the place and not that easy to get rid of.
Deprecation just means you need to plan on migrating to whatever replaces your specific header.
For instance - the wildly popular x-forward-for currently has no replacement and is used in most proxies and load balancers which would fall apart without it.
This means that firewalls deciding to remove x- headers are being rather premature to say the least.
Sooner or later the various headers will be replaced with something other the x-, but in the meantime, networking hardware should support it or at least not default to removing it.

 

[evanevery]

Yeah, I get it. But they WERE deprecated and I expect there was SOME reason that they were specifically NOT on the list of the default headers that my security device would allow to pass. There is a pretty lengthy list of headers which are specifically allowed, and these were not on that. I gotta believe the guys who set up the default proxy rules had a reason to specifically not to include them. They were likely not passing for a reason. Nothing else seems broken on any other devices because of this... ...and as many of the security devices on the market use a common code base at their foundation, I expect they might also share (copy?) some of the default proxy behavior as well. If this is happening on WatchGuard Routers/Firewalls devices, it wouldn't surprise me if its happening on other popular edge-devices as well...

Its also curious to note that my Router/Firewall allows pretty much ANY HTML header to pass outbound (only two are specifically stripped), while inbound headers are only allowed if specifically authorized. IOW: outbound headers are allowed by default whereas inbound headers are stripped by default unless specifically allowed. My firewall provides individual control of the outbound and inbound traffic and set up the individual header authorizations any way I see fit. But the aforementioned is the default. Although any "X-" header is allowed to flow outbound in the page request, the only inbound X-Headers allowed in the reply are X-Dig-XMLPipe-Status, X-Pad, X-Powered-By, and x-server-ip-address. (As the "X-Forward-For" is typically sent by the client in the page request, it would not have been stripped).

Rather than adding these four headers and ending up playing whack-a-mole with any OTHER headers which were used in future communications, I simply white listed the Tesla servers for any/all http proxy filtering.

As you might guess I have a whole myriad of different devices which all play well on my networks (both at home and in the office) and the Tesla is the first one which has required any type of digital investigation to determine why it wasn't working right out of the box.

These headers could very well be Cloudflare related, but there really isn't anything else in the html message which seems to have any unique identifying information. I'm guessing they are using these headers to pass some sort of serialization and identification back and forth and without them the car was not very happy. ...and there are a million ways to do this without having to use deprecated HTM headers which can (are obviously ARE) being filtered by some equipment.

...and yes, curious it was simple http (and not https). Once I figured out what was causing my "WiFi Unable to Join Network" message, and found a suitable workaround, I haven't really looked back. I should run another packet trace to see if it ever switches over to https (or something else). Of course, there really isn't any benefit to start with http in the first place if you plan on transitioning later. Why not use https right out of the gate?

Regardless, this is obviously a functional change in the way the old MCU operated vs the current MCU.

 

[UnknownName]

I spent the last 2 hours connecting a Plaid Model S to a dedicated Unifi IOT network, and found a different problem.

Tesla isn't supporting Protected Management Frames, even on their latest cars and software. At least, the Model S refuses to join a network that has PMF set to "Required".

Via the Wi-fi.org:

Protected Management Frames (PMF) provide protection for unicast and multicast management action frames. Unicast management action frames are protected from both eavesdropping and forging, and multicast management action frames are protected from forging. They augment privacy protections already in place for data frames with mechanisms to improve the resiliency of mission-critical networks. PMF is required for all new certified devices.

A new Unifi Wifi network, by default, sets "Protected Management Frames" (PMF) to "Required". You've got to set this to "Optional" or "Disabled" in the wifi security settings configuration in Unifi for the Tesla to join. This setting applies to both 2.4ghz and 5ghz networks.

 

[Twiglett]

I spent the last 2 hours connecting a Plaid Model S to a dedicated Unifi IOT network, and found a different problem.

Tesla isn't supporting Protected Management Frames, even on their latest cars and software. At least, the Model S refuses to join a network that has PMF set to "Required".

Via the Wi-fi.org:


A new Unifi Wifi network, by default, sets "Protected Management Frames" (PMF) to "Required". You've got to set this to "Optional" or "Disabled" in the wifi security settings configuration in Unifi for the Tesla to join. This setting applies to both 2.4ghz and 5ghz networks.

I had the same thing on my UniFi Network and like you ended up putting the car on my IOT network where I have PMF set optional. So many IOT devices need the same thing. I also ended up creating a firewall rule with a group for the devices that are allowed internet traffic.
The car seems happier on the 2.4g network as well and certainly gets a better signal in the garage when compared to 5g

 

[Cal1]

We have no problems connecting our plaid at home. But when we do it screws up our network. To the point that our other devices can't get on anymore. Spent a significant amount of time troubleshooting to narrow it down to our new car. In retrospect it was pretty obvious as our streaming/wireless internet problems began shortly after taking delivery. We never had problems with updates. But even after 4 updates the car kept accessing and bogging our system down.

So now I forget our network unless we have an update that needs to be downloaded. We've done 2 without reconnecting, just letting it download itself driving around town. Later that evening we'd do the install. Our last update 2021.44.6 did send a notification asking that we connect to wifi to download. Must have been too big to download via 3/5G. After the download and install our network went crazy and I had to forget our home system again. Tesla is so busy that I figured it would be a waste of time trying to get them to look into it.

We are in a really rural area and CenturyLink is pretty worthless. We lucky if we get about 10-20Mbps. Another reason to not complain. They would just tell us it's our crappy service. Ordered Starlink over a year ago.

 

[dazzap]

I have a different WiFi connectivity problem with my 2022 MY v11 S/W

Car just doesn’t see available networks most of the time. By putting my tp-link deco on an extension lead outside the house it connected fine and worked.

I moved the tpkink back indoors and it continued to work.. until I went for a drive and after coming back it doesn’t see the network. At this point the car is parked maybe 12ft from the AP which is indoors.

My iPhone sees it just fine with 2 bars. 3 days after originally getting it working it’s not worked at all.

Weirdly the car doesn’t always see my personal hotspot when I’m sitting in it.

I’m parking with charging port facing the house but don’t know where the cars radio is located.

This is only really an issue because we just happen to have quite poor mobile signal so the car is only getting 3G.

 

[mostroad]

I have a different WiFi connectivity problem with my 2022 MY v11 S/W

Car just doesn’t see available networks most of the time. By putting my tp-link deco on an extension lead outside the house it connected fine and worked.

I moved the tpkink back indoors and it continued to work.. until I went for a drive and after coming back it doesn’t see the network. At this point the car is parked maybe 12ft from the AP which is indoors.

My iPhone sees it just fine with 2 bars. 3 days after originally getting it working it’s not worked at all.

Weirdly the car doesn’t always see my personal hotspot when I’m sitting in it.

I’m parking with charging port facing the house but don’t know where the cars radio is located.

This is only really an issue because we just happen to have quite poor mobile signal so the car is only getting 3G.

I have EXACTLY the same issue on my Model Y. Rebooting the car solves the issues, but when I leave the house and come back, it will not see or reconnect tot eh network. I also have a TP-Link setup, and wondering if there is some sort of settings issue between the Tesla and the TP-Link router, and perhaps a recent update for one of the two caused these issues. I just tried assigning the Tesla a fixed IP, and moving that IP to be a DMZ device on the network, which seems to be working over the past 3 days. Time will tell.