Why does Spectrum Backbone route all durango colorado traffic to seattle wa?

whichwaywazzit

I'm located in Durango CO, spectrum customer, what I've noticed is that all traffic from SW colorado (Durango) spectrum customers is routed up through Seattle WA. This inefficent routing adds a ton of additional latency for latency sensitive applications(online gaming, voip calls)

This is especially bad when you need to reach resources on the east coast. I get somewhere near 160ms to eastcoast destinations.

current traceroute showing destinations are routed Durango->Grand Junction->Missoula->Seattle:

% traceroute 8.8.8.8

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets

 1 192.168.1.1 (192.168.1.1) 2.373 ms 0.933 ms 1.009 ms

 2 * * *

 3 lag-63.durncocu02h.netops.charter.com (69.146.239.174) 10.198 ms 11.212 ms 16.384 ms

 4 lag-10.durncocu01h.netops.charter.com (69.144.94.94) 10.631 ms 12.353 ms 11.304 ms

 5 lag-27.gdjtco0602h.netops.charter.com (96.34.76.114) 12.842 ms 13.541 ms 70.935 ms

 6 lag-24.msslmt33zp0.netops.charter.com (69.144.131.142) 44.256 ms 54.198 ms 53.881 ms

 7 lag-702.bbr02sttlwa.netops.charter.com (69.144.130.200) 67.354 ms 60.284 ms 60.103 ms

 8 lag-802.prr01sttlwa.netops.charter.com (96.34.3.39) 64.304 ms 74.736 ms 64.673 ms

 9 72.14.194.48 (72.14.194.48) 73.851 ms

  096-034-158-069.biz.spectrum.com (96.34.158.69) 68.127 ms 58.285 ms

10 * * *

11 dns.google (8.8.8.8) 63.751 ms 56.662 ms

  142.251.50.244 (142.251.50.244) 55.187 ms

What I would expect is spectrum routes traffic to Denver as it is a larger hub site for internet and would vastly improve latency for customers in Durango.

What's really interesting is that I have seen the path via Denver as recent as evening of Nov 18 and took a traceroute at that time.

path that is preferred:

% traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets
 1  192.168.1.1 (192.168.1.1)  1.664 ms  1.162 ms  1.039 ms
 2  * * *
 3  lag-63.durncocu02h.netops.charter.com (69.146.239.174)  11.335 ms  11.732 ms  13.375 ms
 4  lag-10.durncocu01h.netops.charter.com (69.144.94.94)  12.150 ms  12.420 ms  11.587 ms
 5  lag-27.gdjtco0602h.netops.charter.com (96.34.76.114)  21.865 ms  13.019 ms  13.520 ms
 6  lag-10.gdjtco0601h.netops.charter.com (69.146.239.100)  28.136 ms  16.170 ms  14.123 ms
 7  lag-24.dnvtco56zp0.netops.charter.com (69.144.131.140)  18.789 ms  20.290 ms  18.042 ms
 8  lag-702.bbr01dnvrco.netops.charter.com (69.144.130.202)  19.498 ms  20.669 ms  23.499 ms
 9  lag-801.prr02dnvrco.netops.charter.com (96.34.173.241)  19.224 ms  19.143 ms  19.626 ms
10  096-034-157-003.biz.spectrum.com (96.34.157.3)  18.434 ms
    096-034-157-001.biz.spectrum.com (96.34.157.1)  22.201 ms
    096-034-157-003.biz.spectrum.com (96.34.157.3)  17.897 ms
11  * * *
12  dns.google (8.8.8.8)  17.949 ms
    142.251.61.180 (142.251.61.180)  18.790 ms
    dns.google (8.8.8.8)  21.737 ms

But since then the non-ideal path has taken over as stated above. Users in Durango would greatly appreciate some insight on why the more efficient path is not the default path for day-to-day use.

Thanks!

James_M

Hi & welcome!

Generally speaking, internet traffic is routed on the path of least resistance. There are multiple reasons that traffic may be routed on a specific path as opposed to another path, and the most common is simple congestion.

Think of it like driving on the highway. While there may be a more direct path from A to B, congestion in the form of higher than usual traffic, or a disabled vehicle may mean that the direct path may not be the most efficient path. So going the long way around the block to get the corner may simply be faster.

In terms of lag, generally anything less than 100ms is acceptable and 40-60ms is optimal. Anything less than 40ms is excellent.

The trace routes provided above do not show anything of concern. If you are experiencing an issue on a regular basis, please provide three trace routes that domonstrate the issue (from the command prompt ie: H:\>tracert google.com) to three different websites and if there is an issue, we can forward it to out network engineers for investigation.

Thanks for the question!

HT_Greenfield

Maybe the network topology and traffic routing is largely driven by where the server farms are at for the highest volume type of traffic which is TV streaming and maybe there's a big ole TV streaming server farm hosted by AWS there in Seattle. Sorry, man, but TV streaming is the big man on campus now of days and other types of traffic are just going to have to go with the flow.

whichwaywazzit

Hey James,

To re-use your example, the current path is akin to driving from Durango to Denver, but having to drive to Denver via Seattle, which adds 1000s miles to the path, when the 'road' or fiber path clearly already exists between Durango and Denver (as seen in my optimal traceroute). I fail to understand how the more circuitous path is less congested than the shorter path, if congestion is the case, then I would expect the ideal path during low/non-peak hours. In this case the non-optimal path is taken all the time. Can you explain why things were routed much more optimally on Nov18 and changed the next day?

Since this is a daily pain point for me I would greatly appreciate more technical details as to why this occurring and would happily discuss further with network engineers (I am one myself).

Thanks!

James_M

Unfortunately, our network engineers are not customer facing. If you are experiencing an issue with your service, please provide the trace routes menioned previously so we can investigate further.

Otherwise, we will happily forward your feedback and suggestion.

whichwaywazzit

Here's a current traceroute show casing how circuitous the routing is at 155+ms to US-EAST-1 (Amazon Virginia destination). This is well above your threshold of 100ms provided.

$ traceroute 23.23.255.255

traceroute to 23.23.255.255 (23.23.255.255), 30 hops max, 60 byte packets

 1 RT-N66U-C050 (192.168.1.1) 0.482 ms 0.643 ms 0.692 ms

 2 * * *

 3 lag-63.durncocu02h.netops.charter.com (69.146.239.174) 11.346 ms 12.394 ms 12.365 ms

 4 lag-10.durncocu01h.netops.charter.com (69.144.94.94) 12.660 ms 12.625 ms 12.674 ms

 5 lag-27.gdjtco0602h.netops.charter.com (96.34.76.114) 14.815 ms 14.776 ms 14.748 ms

 6 lag-24.msslmt33zp0.netops.charter.com (69.144.131.142) 46.189 ms 46.150 ms 46.072 ms

 7 lag-702.bbr02sttlwa.netops.charter.com (69.144.130.200) 69.793 ms 56.475 ms 58.623 ms

 8 lag-805.bbr02snjsca.netops.charter.com (96.34.0.198) 89.699 ms 90.540 ms 100.363 ms

 9 * lag-2.bbr01dnvrco.netops.charter.com (96.34.0.3) 100.824 ms 333.762 ms

10 lag-800.bbr02dnvrco.netops.charter.com (96.34.0.205) 108.119 ms 100.088 ms 102.604 ms

11 lag-805.bbr01euclwi.netops.charter.com (96.34.1.83) 143.246 ms 140.474 ms 150.416 ms

12 lag-801.prr01mplsmn.netops.charter.com (96.34.3.65) 136.473 ms 145.269 ms 146.160 ms

13 99.83.94.194 (99.83.94.194) 136.256 ms 99.82.176.238 (99.82.176.238) 138.445 ms 136.780 ms

14 150.222.205.69 (150.222.205.69) 146.290 ms 52.93.61.68 (52.93.61.68) 156.052 ms 52.93.61.84 (52.93.61.84) 150.422 ms

15 52.93.61.31 (52.93.61.31) 136.984 ms 52.93.61.125 (52.93.61.125) 146.437 ms 52.93.61.49 (52.93.61.49) 146.015 ms

16 * * *

17 * * *

18 * * *

19 * * *

20 * * *

21 * * *

22 52.93.28.208 (52.93.28.208) 165.115 ms 52.93.28.228 (52.93.28.228) 164.643 ms 52.93.28.226 (52.93.28.226) 165.436 ms

23 * * *

24 * * *

25 * * *

26 * * *

27 * * *

28 * * *

29 ec2-23-23-255-255.compute-1.amazonaws.com (23.23.255.255) 166.642 ms * *

 ping -c 10 23.23.255.255

PING 23.23.255.255 (23.23.255.255) 56(84) bytes of data.

64 bytes from 23.23.255.255: icmp_seq=1 ttl=230 time=154 ms

64 bytes from 23.23.255.255: icmp_seq=2 ttl=230 time=154 ms

64 bytes from 23.23.255.255: icmp_seq=3 ttl=230 time=155 ms

64 bytes from 23.23.255.255: icmp_seq=4 ttl=230 time=155 ms

64 bytes from 23.23.255.255: icmp_seq=5 ttl=230 time=154 ms

64 bytes from 23.23.255.255: icmp_seq=6 ttl=230 time=154 ms

64 bytes from 23.23.255.255: icmp_seq=7 ttl=230 time=155 ms

64 bytes from 23.23.255.255: icmp_seq=8 ttl=230 time=156 ms

64 bytes from 23.23.255.255: icmp_seq=9 ttl=230 time=155 ms

64 bytes from 23.23.255.255: icmp_seq=10 ttl=230 time=155 ms

--- 23.23.255.255 ping statistics ---

10 packets transmitted, 10 received, 0% packet loss, time 9012ms

rtt min/avg/max/mdev = 154.006/154.705/155.674/0.515 ms

James_M

Thanks. I was able to locate your account using your registration information. I am seeing some signal issues that would need to be addressed first with a service call. If you would like assistance setting up a service call with a technician, please let us know.

whichwaywazzit

The issue I'm describing would not be fixed with a truck roll to my place; the issue I'm pointing out is related to how all traffic in the town is routed and not specific to my address.

RAIST5150

Hate to say it, but may have to wait until the next round of BGP announcements are made... not every network runs on the same refresh cycles either.

It is an unfortunate downside to some of the automation that takes place... stale metrics in the tables when there is a sudden shift in traffic patterns. It can sometimes take a while for things to right the ship again.

Keep in mind that some of the routing decisions may eventually fall out of Spectrum's hands. Once you get handed off to a peering partner or someone's edge networks, it is in someone else's wheelhouse and you are then subject to the metrics saved in their tables. This can sometimes happen pretty early in the route, not always midway like when you got handed off to Amazon at hop 13 (99.83.94.194 and 99.82.176.238).

In the short term, Can opt to try a VPN to get around it. Sometimes you can get lucky if you power down the modem for a long while (like over night, while out of town for a trip or something). If it manages to switch to a different enough subnet, the routing may latch a better route that puts you into a different region for that next hop out of your area that otherwise points you to a more optimal peering assignment or entry point to the endpoint's perimeter network.

May be a bigger problem in general with east/west traffic atm. Poked around a looking glass from two locations in Denver to that aws address, and the IGP results they use were not pretty...

RAIST5150

Should also note that icmp echo is far from a perfect diagnosis tool... one of the lower priority items when a node starts to get busy. Some are even set to outright ignore them as a rule.

Pretty much what the general public has to work with though. Usually good enough to point tier3 and such in the general direction.... just have learned to not take them as definitive proof of where a problem lies.

Might want to poke around with the cloudping tool:

https://www.cloudping.cloud/aws

There are more tools out there where you can get more info... EC2 reachability and additional diagnostics. Some games have their own suite of tools you can use as well through their support portals.

May want to take them up on the tech visit... signs of more localized bad jitter (at times as bad as 60%).

Whatever is the root cause of that could be compounding things.

The process has to start somewhere... isolate and address each problem you can to get things rooted out properly.

whichwaywazzit

Yes I took them up on a truck roll since customer service said I was due for modem replacement anyway (upgrade to docsis 3.1).

Long story short, modem was replaced, signal checked, etc. All of that is/was fine.

Of course the backbone issue still exists because it has nothing to do with my local modem connection. This is an issue with how spectrum backbone routes sources of this network segment towards the internet.

Also note my test towards 23.23.255.255  IS an ec2-reachability ip from http://ec2-reachability.amazonaws.com/

All us-east-1 destinations will get routed this non-optimal way until spectrum can sort out their backbone routing to leverage Denver peering.