LTE Throttling since 11/3/17

[ahkahn]

I'm sure it's not that they don't give a damn, but it's probably a priority issue. I'd hate to see their laundry list of priorities and to-do's...

 

[verygreen]

How about a traceroute?

It appears to be filtered so you only get a small part of the picture at the start and that's it.

(from my old stash of data)

# traceroute -n google.com
traceroute to google.com (172.217.0.142), 30 hops max, 60 byte packets
 1  172.24.0.3  42.379 ms  41.894 ms  41.369 ms
 2  100.123.28.7  41.979 ms  41.642 ms  41.450 ms
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  172.217.0.142  89.070 ms * *
root@# traceroute -n yahoo.com
traceroute to yahoo.com (206.190.39.42), 30 hops max, 60 byte packets
 1  172.24.0.3  36.599 ms  36.238 ms  35.621 ms
 2  100.123.28.7  36.554 ms  36.408 ms  36.281 ms
 3  * * *
 4  * 208.185.160.185  35.515 ms *
 5  * * *
 6  * * 64.125.31.168  69.871 ms
 7  * * *
 8  * * *
 9  206.126.236.2  107.641 ms * *
10  * * *
11  * * *
12  * * 216.115.96.62  178.580 ms
13  * * *
14  67.195.37.73  178.035 ms * *
15  * * 67.195.1.81  189.774 ms
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

If there is a fairly easy way to root into the car, I'd be willing to try it... but I don't want to have to fabricate cables or anything dramatic like that.

you can buy premade cables.

Or you can just plug bare wires in - good enough for a couple of tries only

 

[ahkahn]

It appears to be filtered so you only get a small part of the picture at the start and that's it.

(from my old stash of data)

# traceroute -n google.com
traceroute to google.com (172.217.0.142), 30 hops max, 60 byte packets
 1  172.24.0.3  42.379 ms  41.894 ms  41.369 ms
 2  100.123.28.7  41.979 ms  41.642 ms  41.450 ms
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  172.217.0.142  89.070 ms * *
root@# traceroute -n yahoo.com
traceroute to yahoo.com (206.190.39.42), 30 hops max, 60 byte packets
 1  172.24.0.3  36.599 ms  36.238 ms  35.621 ms
 2  100.123.28.7  36.554 ms  36.408 ms  36.281 ms
 3  * * *
 4  * 208.185.160.185  35.515 ms *
 5  * * *
 6  * * 64.125.31.168  69.871 ms
 7  * * *
 8  * * *
 9  206.126.236.2  107.641 ms * *
10  * * *
11  * * *
12  * * 216.115.96.62  178.580 ms
13  * * *
14  67.195.37.73  178.035 ms * *
15  * * 67.195.1.81  189.774 ms
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

you can buy premade cables.

Or you can just plug bare wires in - good enough for a couple of tries only

Interesting. So it's 36-42ms to get to the first hop, which is a private IP address. It could be an AT&T private IP, or a Tesla private IP. I would presume that since this is alegedly going through a VPN, the 172.24.0.3 IP address is a Tesla private IP (probably the VPN server) and then the next IP in the list is the gateway out to the Internet from Tesla. I tried looking that up (100.123.28.7), and it appears to be a hidden IP address, so not sure who owns that (perhaps Jasper?).

That first hop would indicate the actual LTE service, and at 36-42ms, that's not unreasonable for a VPN connection over wireless. The next hop (since the time is essentially the same) tells me that it is essentially a direct plugin from the VPN server to the outbound gateway.

I don't think this is a latency issue, which would be due to overcrowding of the network. Seems definitely like a throttling issue, provided @verygreen 's traceroute came from a point in time where the slowdown existed. Now, it's not necessarily a throttling by AT&T, as it could be Tesla throttling bandwidth on their end to each vehicle for whatever reasoning they may have - perhaps to prevent overcrowding of the network.

Thanks for posting the traceroute, @verygreen!

 

[verygreen]

Interesting. So it's 36-42ms to get to the first hop, which is a private IP address. It could be an AT&T private IP, or a Tesla private IP. I would presume that since this is alegedly going through a VPN, the 172.24.0.3 IP address is a Tesla private IP (probably the VPN server) and then the next IP in the list is the gateway out to the Internet from Tesla. I tried looking that up (100.123.28.7), and it appears to be a hidden IP address, so not sure who owns that (perhaps Jasper?).

That first hop would indicate the actual LTE service, and at 36-42ms, that's not unreasonable for a VPN connection over wireless. The next hop (since the time is essentially the same) tells me that it is essentially a direct plugin from the VPN server to the outbound gateway.

I don't think this is a latency issue, which would be due to overcrowding of the network. Seems definitely like a throttling issue, provided @verygreen 's traceroute came from a point in time where the slowdown existed. Now, it's not necessarily a throttling by AT&T, as it could be Tesla throttling bandwidth on their end to each vehicle for whatever reasoning they may have - perhaps to prevent overcrowding of the network.

Thanks for posting the traceroute, @verygreen!

regular traffic does NOT go into the Tesla vpn.

my traceroute is from a time when the throttling existed. I measured it to be ~170kbps at the time the trace was taken.

 

[ahkahn]

regular traffic does NOT go into the Tesla vpn.

my traceroute is from a time when the throttling existed. I measured it to be ~170kbps at the time the trace was taken.

If that's the case, I'm not sure where that 100.123.28.7 address is. It's clearly the gateway, but it's not tracing back to anything. Typically the AT&T IP scheme would come back as owned by AT&T, but this one is not... A Traceroute should be going by the way of regular traffic, but who knows how their VPN routing table is set up. They would have had to differentiate what traffic (either by port or by destination IP) goes through the VPN or not.

 

[HankLloydRight]

regular traffic does NOT go into the Tesla vpn.

But all traffic while on 3G/LTE is routed through the Jasper Networks proxy, no?

 

[Wrkent100]

Not sure why we feel the need to trouble-shoot this when we should be pressuring Tesla to identify and remediate the issue. After all, these are high end very expensive cars. I understand the "fun" aspect of trying to figure it out but in the end, it must be Tesla who assesses and fixes the issue. That said, my opinion is we should be discussing ways to force Tesla into much more of a focused effort on resolving this. I called my local dealership and let them know I will be opening a better business bureau formal case if I don't get a satisfactory answer around status and timeline. I encourage others to do the same.

 

[HankLloydRight]

Not sure why we feel the need to trouble-shoot this

Because the majority of people here are either highly technical, or have technical backgrounds, and trouble-shooting these things is what we do everyday. Instead of just complaining to Tesla 'X doesn't work' -- it could be much more helpful for Tesla to report 'X doesn't work and here's EXACTLY why'. It helps cut through the BS typically given by Level 1 customer support people ("have you rebooted?") or other lame excuses that we KNOW are not the cause.

When I find a bug or problem on a website I need to report, if I didn't provide extra info, they'll just write it off as "user error" or "user configuration error" or some such BS. So instead, I typically report that there is Y problem which appears on Mac, PC, and Linux, and in Chrome, Firefox, Opera, Vivaldi and any other browser. So it's not my computers or OS or browser. Then sometimes they listen to me and accept that there's a problem on THEIR end.

Extra information is always helpful. That's why we do it.

 

[Wrkent100]

Because the majority of people here are either highly technical, or have technical backgrounds, and trouble-shooting these things is what we do everyday. Instead of just complaining to Tesla 'X doesn't work' -- it could be much more helpful for Tesla to report 'X doesn't work and here's EXACTLY why'. It helps cut through the BS typically given by Level 1 customer support people ("have you rebooted?") or other lame excuses that we KNOW are not the cause.

When I find a bug or problem on a website I need to report, if I didn't provide extra info, they'll just write it off as "user error" or "user configuration error" or some such BS. So instead, I typically report that there is Y problem which appears on Mac, PC, and Linux, and in Chrome, Firefox, Opera, Vivaldi and any other browser. So it's not my computers or OS or browser. Then sometimes they listen to me and accept that there's a problem on THEIR end.

Extra information is always helpful. That's why we do it.

Hundreds if not thousands of people have reported this to dealers. I have spoken with the owner of the local dealership who tells me they have owner meetings weekly with other owners and then collectively escalate their concerns to the mother ship.

Tesla has the brightest minds in the world who are changing the essence of how cars are made. They have brilliant engineers who know much more about the inner workings of their cars than anyone on this forum ever will. Now if we invoke the concept of Occam’s razor, what is more plausible:

A) after 3 months the greatest minds at tesla who are familiar with the cars still don’t know the root cause or a fix and it’s going to be figured out by forum members or

B) tesla knows the problem but not a priority to fix given model 3 delays.

B is by far the clear choice. The logical next step is to apply pressure and force tesla to prioritize the issue, not never-ending technical guesswork by forum members.

 

[pdxgibby]

Tesla knows about the issue and I'm sure they know how to prioritize it. In the meantime, if people have the ability to find more clues by debugging the network connections / latency / bandwidth to satisfy themselves, it doesn't hurt anything. If it adds value to Tesla R&D, great. If not, who cares.

Collectively putting pressure on Tesla to raise the priority of this issue above other unknown issues does nothing but distract from the ultimate goal of fixing other issues AND this one.

 

[HankLloydRight]

Hundreds if not thousands of people have reported this to dealers. I have spoken with the owner of the local dealership

Tesla doesn't have 'dealerships.'

Tesla has the brightest minds in the world who are changing the essence of how cars are made. They have brilliant engineers who know much more about the inner workings of their cars than anyone on this forum ever will.

This is arguably not true. There are plenty of smart people at Tesla. But they don't have a monopoly on every smart person. If they all were brilliant engineers, then I'd like to meet the "brilliant" engineer(s) responsible for the media player debacle. What a complete piece of crap software nowhere near as good and feature complete as my Rio Riot from 15 years ago. Yeah, those are the brilliant engineers at Tesla.

There are plenty of people on this very forum who do not work for Tesla that know more about the cars than most of the engineers at Tesla.

Also, this particular problem has almost nothing to do with building cars. It's a computer/networking problem where there are no shortage of smart people outside of Tesla who are better qualified to diagnose and fix this problem.

And just because they're smart people doesn't mean they aren't able fully diagnose problems likely within the AT&T network. I would guess there aren't many cellular/LTE network engineers at Tesla.

A) after 3 months the greatest minds at tesla

Who says the "greatest minds" at Tesla are even working on this problem?

B) tesla knows the problem but not a priority to fix given model 3 delays.

This is the same lame excuse "model 3 delays" brought up for every other unaddressed problem across Tesla. Like every single person at Tesla from the top down are shunted away from their regular work to work on M3. There are many different groups within Tesla not working on Model 3. I would think one or two of their "brilliant" engineers could locate, identify, and fix this problem without impacting Model 3. And Model 3 production seems to be increasing at a fever pace, so any real roadblocks to M3 production should be gone now.

A) after 3 months the greatest minds at tesla who are familiar with the cars still don’t know the root cause or a fix and it’s going to be figured out by forum members or

I don't think anyone here is going to "figure it out" for Tesla, but as mentioned, the more precise information and testing we can do to help them narrow down the problem is a bonus and shortens their path to a fix. If you put three of us together, with access to the network within Tesla, I bet we could figure it out in a couple of days. And then come up with a fix.

The fact it's taking this long leads me to believe it's possibly a political or contractual issue between AT&T and Tesla, that requires lawyers and new contracts to resolve.

Last year the Newark, DE supercharger was down for three of four days. It wasn't a technical problem, but simply someone didn't pay the power bill, and there was rumors of lots of finger pointing between Tesla and the Newark rest stop over who was responsible. It took days to figure out (read: lawyers), and eventually the power was restored.

So no, I don't believe Tesla knows the problem and it's not a priority.

 

[Wrkent100]

Tesla doesn't have 'dealerships.'

I concur with the political or contractual issue potential. That said, as someone else pointed out...a handful of average skilled network engineers could identify the issue within a couple of weeks, networking is a fairly binary science. If it is a contractual issue, shame on Tesla legal department for getting us into this bind and I fear for the future of autonomous driving legislation if they can't properly insert the correct clauses to avoid these type contractual issues.



This is arguably not true. There are plenty of smart people at Tesla. But they don't have a monopoly on every smart person. If they all were brilliant engineers, then I'd like to meet the "brilliant" engineer(s) responsible for the media player debacle. What a complete piece of crap software nowhere near as good and feature complete as my Rio Riot from 15 years ago. Yeah, those are the brilliant engineers at Tesla.

There are plenty of people on this very forum who do not work for Tesla that know more about the cars than most of the engineers at Tesla.

Also, this particular problem has almost nothing to do with building cars. It's a computer/networking problem where there are no shortage of smart people outside of Tesla who are better qualified to diagnose and fix this problem.

And just because they're smart people doesn't mean they aren't able fully diagnose problems likely within the AT&T network. I would guess there aren't many cellular/LTE network engineers at Tesla.



Who says the "greatest minds" at Tesla are even working on this problem?



This is the same lame excuse "model 3 delays" brought up for every other unaddressed problem across Tesla. Like every single person at Tesla from the top down are shunted away from their regular work to work on M3. There are many different groups within Tesla not working on Model 3. I would think one or two of their "brilliant" engineers could locate, identify, and fix this problem without impacting Model 3. And Model 3 production seems to be increasing at a fever pace, so any real roadblocks to M3 production should be gone now.



I don't think anyone here is going to "figure it out" for Tesla, but as mentioned, the more precise information and testing we can do to help them narrow down the problem is a bonus and shortens their path to a fix. If you put three of us together, with access to the network within Tesla, I bet we could figure it out in a couple of days. And then come up with a fix.

The fact it's taking this long leads me to believe it's possibly a political or contractual issue between AT&T and Tesla, that requires lawyers and new contracts to resolve.

Last year the Newark, DE supercharger was down for three of four days. It wasn't a technical problem, but simply someone didn't pay the power bill, and there was rumors of lots of finger pointing between Tesla and the Newark rest stop over who was responsible. It took days to figure out (read: lawyers), and eventually the power was restored.

So no, I don't believe Tesla knows the problem and it's not a priority.

 

[Unpilot]

“The fact it's taking this long leads me to believe it's possibly a political or contractual issue between AT&T and Tesla, that requires lawyers and new contracts to resolve.”

Using the inverse of Occams razor this is probably it.

 

[verygreen]

But all traffic while on 3G/LTE is routed through the Jasper Networks proxy, no?

When you are not connected to wifi, all traffic goes into the lte network, how it's handled on that side is up to the AT&T and then WISP (Jasper?) that buys bandwidth or whatever from them.

Not sure why we feel the need to trouble-shoot this when we should be pressuring Tesla to identify and remediate the issue.

A pretty plausible theory I heard is this:
Some smart guy at Tesla said something like "hey, we only allow up to 128kbps max bitrate on our streaming media, so if we throttle the bandwidth to 2x that, nobody would notice and we will save a bunch of money!" and so they proceeded to do this ignoring all the other parts of the car being data hogs too at times.
Either that or their WISP decided to do the throttling for some other (likely also monetary-related) reasons.

 

[AMPUP]

It’s funny that 3g isn’t impacted, I leave my work underground lot each day and for the first 10 mins only get 3G, at 2.1 MBPS!! Then it picks up LTE and I get 160kbps!!!!!!

I paid for the LTE upgrade too, wish I could select or go back to 3G!

 

[ahkahn]

When you are not connected to wifi, all traffic goes into the lte network, how it's handled on that side is up to the AT&T and then WISP (Jasper?) that buys bandwidth or whatever from them.



A pretty plausible theory I heard is this:
Some smart guy at Tesla said something like "hey, we only allow up to 128kbps max bitrate on our streaming media, so if we throttle the bandwidth to 2x that, nobody would notice and we will save a bunch of money!" and so they proceeded to do this ignoring all the other parts of the car being data hogs too at times.
Either that or their WISP decided to do the throttling for some other (likely also monetary-related) reasons.

I am assuming that Jasper is Tesla's ISP and AT&T is really just the carrier that the VPN connection goes through.

Vehicle -> VPN Client -> ATT (Internet) -> Tesla VPN Server -> Internet (Jasper)

All Internet-destined data seems to go through VPN out through Jasper.

I'd put money on your plausible theory. Get an overzealous network engineer and exactly that would happen. It would potentially prevent congestion down the road.

 

[bak_phy]

I paid for the LTE upgrade too, wish I could select or go back to 3G!

Call and demand your money back.

 

[DamageInc]

There's now a post on Reddit about this same issue. It seems as though this issue is only affecting those in the eastern US for whatever reason.

 

[pdxgibby]

Build date: 12/2017
Delivered: 12/7/2017
VIN: 228xxx
Firmware: 2017.50.3 f3425a1
LTE: single bar in garage, wifi disconnected
Speed: 2.0Mbps

 

[JaredBanyard]

Hey guys, I have been getting some pretty good test results on Reddit. Anecdotally, it seems the throttling is limited to the Eastern US. I would really like some more feedback. Here is my comment on my post on Reddit keeping track: LTE being heavily throttled for some owners right now. Could be actual cause of poor Slacker/TuneIn performance. I tested multiple times across a few miles getting 100-200kbps. • r/teslamotors

Please test using either Internet Speed Test or the 3G test at Simple speed test | DSLReports, ISP Information and let me know if you are throttled and what your location is. I think this is regional.