LTE Throttling since 11/3/17

[Happychaos]

We have a car with 3g and another with LTE and both have issues. Maps load fine, but the streaming is terrible now, and if you do a speed test off of the browser it's not even 1Mbps.

Obviously being throttled - got the firmware 50 update and the LTE car isn't perfect but a lot better. 3g car still has issues, not sure if it's buffering fast enough or has inadequate coverage vs LTE. Frustrating.

Perhaps other 3g cars don't have the issue but ours definitely has the issue unfortunately.

 

[HankLloydRight]

as far as I know, there's no way to disable LTE and force 3G without hacking the car in some fashion.

 

[Skipdd]

I've read through all the posts on this thread and (I think) all of the others dealing with this topic (thanks @MasterT ). I recently joined this chorus when I upgraded my 85D to a S100 D. I had 3G in the 85D and while there were occasional issues, it was hugely better than in the new 100 D.

What I've noticed though, is that the issue is much worse during the day - like Slacker is unusable. Whereas, in the evening it is much better. I've oberved this pattern across multiple days. Am curious if others have had similar experience? It could be a fluke, dunno.

If it is a common observation, then does that help us futher isolate the source of the issue, other than reinforcing that it is related to LTE and involves some sort of congestion issue (maybe bandwidth, maybe software)?

Short of it being fixed, I'm left to keep speculating..

 

[jdw]

It may well be a combination of Tesla backend services *and* AT&T LTE bandwidth given HankLloydRight's comment about 3G users not having the bandwidth issue and my own experience with unplayable content but no LTE bandwidth issues.

 

[Skipdd]

It may well be a combination of Tesla backend services *and* AT&T LTE bandwidth given HankLloydRight's comment about 3G users not having the bandwidth issue and my own experience with unplayable content but no LTE bandwidth issues.

Makes sense. Combo issues are usually the hardest to solve.

 

[ahkahn]

Yeah, that's what I'm wondering. If all of the traffic is being filtered through Tesla's servers, it sounds more like a bandwidth issue on the Tesla end rather than an AT&T issue. I'd imagine there is some core pipeline coming in from AT&T which is being oversaturated.

If someone could hack into their car and start doing traceroutes, that would probably explain quite a bit.

 

[HankLloydRight]

Yeah, that's what I'm wondering. If all of the traffic is being filtered through Tesla's servers, it sounds more like a bandwidth issue on the Tesla end rather than an AT&T issue. I'd imagine there is some core pipeline coming in from AT&T which is being oversaturated.

I don't know much about cell tower infrastructure, but I would assume that cell traffic hitting the same tower, either LTE or 3G, gets onto the internet backbone not far from there. (Or even two adjacent towers if only one tower supports 3G). And once the traffic hits the AT&T internet entry point, it doesn't matter if it's from an 3G or LTE car. The traffic then goes to Jasper Networks, then to Tesla, then back out to the internet.

Since very few 3G owners are experiencing these problems, and all Tesla traffic is routed through Jasper Networks, again, I don't think this is a problem there or with Tesla's end of the connection. A friend with a 3G car we tested was getting 2.5mbps on fast.com, and at the same time, parked right next to him, I was getting 200kbps on my LTE.

At this point, I'm thinking that AT&T updated the firmware in their LTE towers that somehow is incompatible with the LTE radios in our cars. Hopefully that's fixable with a FW update of the LTE firmware in the car. I sure hope it didn't render the LTE hardware obsolete!

 

[jdw]

At this point, I'm thinking that AT&T updated the firmware in their LTE towers that somehow is incompatible with the LTE radios in our cars. Hopefully that's fixable with a FW update of the LTE firmware in the car. I sure hope it didn't render the LTE hardware obsolete!

Something like this, specific to AT&T, makes sense given that many international users do not have an LTE bandwidth problem running the same hardware/software as US users and that many US users seem to be OK over 3G. It does leave the (for me) missing and unplayable content over LTE, which works over WiFi, pointing at something additional on Tesla's end.

 

[Pollux]

Buffer management problem latent within(*) or introduced into Tesla's LTE driver?

Accounts for:
+ LTE-only, not WiFi, not 3G
+ Significant, orders-of-magnitude throughput decrease (can't keep the pipe full), time delays, latency degradation
+ Could be AT&T-specific, given complexity of LTE protocol means literally identical implementation across multiple vendors is unlikely
+ All of the vehicle's Internet-dependent apps would be affected in possibly-varying degrees with all kinds of funny symptoms
+ If the buffer pool in question were shared outside the LTE driver, other OS services could be affected, e.g., OS crash/reboot due to memory exhaustion
+ If a leak, there would be time-dependent / load-dependent failure modes
+ The occasionally-reported, Tesla-attributed claim that the problem will be fixed with a vehicle firmware update
+ Does not require third-party involvement, AT&T infrastructure problems, tired data plans that will be rejuvenated on January 1(**)

Figured I'd lay my marker down before the possibly-magic January 1 date.

Alan

(*)Latent problem probably would require an external trigger, e.g., by a change in AT&T's infrastructure
(**)I admit, I'm looking forward to checking out my car's Slacker behavior tomorrow!

 

[ahkahn]

I don't know much about cell tower infrastructure, but I would assume that cell traffic hitting the same tower, either LTE or 3G, gets onto the internet backbone not far from there. (Or even two adjacent towers if only one tower supports 3G). And once the traffic hits the AT&T internet entry point, it doesn't matter if it's from an 3G or LTE car. The traffic then goes to Jasper Networks, then to Tesla, then back out to the internet.

Since very few 3G owners are experiencing these problems, and all Tesla traffic is routed through Jasper Networks, again, I don't think this is a problem there or with Tesla's end of the connection. A friend with a 3G car we tested was getting 2.5mbps on fast.com, and at the same time, parked right next to him, I was getting 200kbps on my LTE.

At this point, I'm thinking that AT&T updated the firmware in their LTE towers that somehow is incompatible with the LTE radios in our cars. Hopefully that's fixable with a FW update of the LTE firmware in the car. I sure hope it didn't render the LTE hardware obsolete!

I'm a former network engineer, so I can take a loosely educated guess at how it's set up...

Jasper Networks is Tesla's ISP. They probably have a monster pipeline to the Internet through Jasper. All of the vehicles have a connection to the Internet (ATT LTE, ATT 3G, or foreign carrier) and then have a VPN connection over the Internet INTO Tesla. Tesla is then routing the inbound traffic from the vehicles to it's targeted destination (ie. out to Slacker through Internet, Google, or any other browser data, or route it internally for the telematics, etc).

From reading the others' testing, either the LTE data is routed through a different VPN router on Tesla's end than the 3G and foreign, and perhaps that router has issues, OR something might be wrong with the LTE provisioning on the AT&T side (highly doubt that), or perhaps the hardware drivers or VPN stack on the vehicles have an issue.

This is definitely not a tower issue. Most towers have both LTE and 3G equipment on them. It could also still be an AT&T throttling issue...

 

[HankLloydRight]

From reading the others' testing, either the LTE data is routed through a different VPN router on Tesla's end than the 3G and foreign,

I keep coming back to this -- once the traffic gets to Jasper/Tesla it's all just IP traffic -- I doubt they have a layer 1 or layer 2 switch to shunt LTE sourced traffic to a different network. What would be the point of that?

Also, if that's what they ARE doing, it seems like it would be an easy fix to correct whatever is throttling the data on the Tesla proxy server end.

 

[bcalfee]

I don't know anything about the tech stuff. What we see is that both our Tesla 2015 S 70D USA and our ATT phones have equally bad service for Data... I was assuming it was an ATT problem...

 

[ahkahn]

I keep coming back to this -- once the traffic gets to Jasper/Tesla it's all just IP traffic -- I doubt they have a layer 1 or layer 2 switch to shunt LTE sourced traffic to a different network. What would be the point of that?

Also, if that's what they ARE doing, it seems like it would be an easy fix to correct whatever is throttling the data on the Tesla proxy server end.

Not necessarily. If the VPN traffic from each vehicle (based on their hardware configuration or other specification) is directed towards a different server, it is totally possible. For instance, if the firmware "sees" that there is an LTE modem, it routes the traffic to server A. If it "sees" a 3G modem, it routes it to server B. Unlikely they're doing this, but totally possible. Why would they do this? An attempt to load balance, but not having the foresight to see that eventually one will outnumber the other.

But you're right - if they are doing this, it would be pretty easy to fix it. You'd have to either set up another VPN server and redirect a chunk of the vehicles to it, or beef up your current VPN server to handle the load.

 

[wolfson292]

Appears things are being capped at 256kb/s based on CURL download tests on the vehicle itself.

 

[Wrkent100]

Agree... that is the high end of what I get.

 

[HankLloydRight]

Appears things are being capped at 256kb/s based on CURL download tests on the vehicle itself.

How about a traceroute?

 

[Skipdd]

Someone posted awhile back that Tesla claimed Slacker changed their encryption and that was the cause - was that ever refuted? Asking, because I have encountered situations where encryption changed packet size, which affected underlying network (in one case frames were bigger than MTU size which caused application latency). Traceroute found that problem..

 

[ahkahn]

I would be interested in doing a traceroute to something simple, like Google.com, and then start pinging backwards to try Jasper's outbound gateway, Tesla.com, and some other basic internal IP addresses. I'd be looking for latency numbers to see if this is a latency issue or simply a bandwidth issue. Obviously a latency issue can appear as a bandwidth issue. If latency is great throughout (even to Google.com) then it is most certainly a bandwidth throttling issue. Now, that could be throttling on the Tesla end, or possibly the AT&T end.

It would also confirm whether or not the Wifi routes through the VPN or how it is handled differently for different services.

If there is a fairly easy way to root into the car, I'd be willing to try it... but I don't want to have to fabricate cables or anything dramatic like that.

 

[Wrkent100]

I would be interested in doing a traceroute to something simple, like Google.com, and then start pinging backwards to try Jasper's outbound gateway, Tesla.com, and some other basic internal IP addresses. I'd be looking for latency numbers to see if this is a latency issue or simply a bandwidth issue. Obviously a latency issue can appear as a bandwidth issue. If latency is great throughout (even to Google.com) then it is most certainly a bandwidth throttling issue. Now, that could be throttling on the Tesla end, or possibly the AT&T end.

It would also confirm whether or not the Wifi routes through the VPN or how it is handled differently for different services.

If there is a fairly easy way to root into the car, I'd be willing to try it... but I don't want to have to fabricate cables or anything dramatic like that.

Whatever the case, three months is ample time for 99% of companies to triage and resolve this, put simply... tesla doesn’t give a damn!

 

[DamageInc]

Whatever the case, three months is ample time for 99% of companies to triage and resolve this, put simply... tesla doesn’t give a damn!

Now that my theory of 2017 throttling is refuted, I am confident this is the reason!