Seeing random latency spikes and packet loss consistently

nute89

I am seeing consistent latency spikes and packet loss. For testing I used Google DNS 8.8.8.8. Trace routes indicates the first hop seems to be at least part of the culprit. I posted the pings to Google DNS, the trace route and pings to the first hop. If this is not the appropriate audience any direction for assistance with this would be appreciated.

Reply from 8.8.8.8: bytes=32 time=27ms TTL=58
Reply from 8.8.8.8: bytes=32 time=63ms TTL=58
Reply from 8.8.8.8: bytes=32 time=43ms TTL=58
Reply from 8.8.8.8: bytes=32 time=30ms TTL=58
Reply from 8.8.8.8: bytes=32 time=28ms TTL=58
Reply from 8.8.8.8: bytes=32 time=56ms TTL=58
Request timed out.
Reply from 8.8.8.8: bytes=32 time=188ms TTL=58
Reply from 8.8.8.8: bytes=32 time=26ms TTL=58
Reply from 8.8.8.8: bytes=32 time=36ms TTL=58
Reply from 8.8.8.8: bytes=32 time=33ms TTL=58
Reply from 8.8.8.8: bytes=32 time=30ms TTL=58
Reply from 8.8.8.8: bytes=32 time=25ms TTL=58
Request timed out.
Reply from 8.8.8.8: bytes=32 time=37ms TTL=58
Request timed out.
Reply from 8.8.8.8: bytes=32 time=25ms TTL=58
Reply from 8.8.8.8: bytes=32 time=99ms TTL=58
Reply from 8.8.8.8: bytes=32 time=62ms TTL=58
Reply from 8.8.8.8: bytes=32 time=43ms TTL=58
Reply from 8.8.8.8: bytes=32 time=26ms TTL=58
Reply from 8.8.8.8: bytes=32 time=28ms TTL=58
Reply from 8.8.8.8: bytes=32 time=25ms TTL=58
Reply from 8.8.8.8: bytes=32 time=25ms TTL=58
Reply from 8.8.8.8: bytes=32 time=24ms TTL=58
Request timed out.
Reply from 8.8.8.8: bytes=32 time=26ms TTL=58
Reply from 8.8.8.8: bytes=32 time=81ms TTL=58
Reply from 8.8.8.8: bytes=32 time=25ms TTL=58
Reply from 8.8.8.8: bytes=32 time=38ms TTL=58
Reply from 8.8.8.8: bytes=32 time=25ms TTL=58
Reply from 8.8.8.8: bytes=32 time=34ms TTL=58
Reply from 8.8.8.8: bytes=32 time=33ms TTL=58
Reply from 8.8.8.8: bytes=32 time=27ms TTL=58
Reply from 8.8.8.8: bytes=32 time=31ms TTL=58
Reply from 8.8.8.8: bytes=32 time=31ms TTL=58
Reply from 8.8.8.8: bytes=32 time=32ms TTL=58
Reply from 8.8.8.8: bytes=32 time=36ms TTL=58
Reply from 8.8.8.8: bytes=32 time=126ms TTL=58
Reply from 8.8.8.8: bytes=32 time=29ms TTL=58
Reply from 8.8.8.8: bytes=32 time=32ms TTL=58
Reply from 8.8.8.8: bytes=32 time=25ms TTL=58
Reply from 8.8.8.8: bytes=32 time=27ms TTL=58
Reply from 8.8.8.8: bytes=32 time=54ms TTL=58
Reply from 8.8.8.8: bytes=32 time=57ms TTL=58

Request timed out.
Reply from 8.8.8.8: bytes=32 time=29ms TTL=58
Reply from 8.8.8.8: bytes=32 time=32ms TTL=58
Reply from 8.8.8.8: bytes=32 time=35ms TTL=58
Reply from 8.8.8.8: bytes=32 time=147ms TTL=58
Reply from 8.8.8.8: bytes=32 time=25ms TTL=58
Reply from 8.8.8.8: bytes=32 time=153ms TTL=58
Reply from 8.8.8.8: bytes=32 time=33ms TTL=58
Reply from 8.8.8.8: bytes=32 time=30ms TTL=58
Reply from 8.8.8.8: bytes=32 time=22ms TTL=58
Reply from 8.8.8.8: bytes=32 time=103ms TTL=58
Reply from 8.8.8.8: bytes=32 time=54ms TTL=58

Tracing route to dns.google [8.8.8.8]
over a maximum of 30 hops:

1 <1 ms 1 ms <1 ms 192.168.1.1
2 155 ms * 14 ms syn-142-254-218-097.inf.spectrum.com [142.254.218.97]
3 * * 36 ms lag-63.irndnyaf01h.netops.charter.com [24.58.232.233]
4 159 ms 55 ms 63 ms lag-52.mcr11hnrtnyaf.netops.charter.com [24.58.52.24]
5 * 131 ms 35 ms lag-28.rcr01albynyyf.netops.charter.com [24.58.32.70]
6 * 172 ms * lag-16.nycmny837aw-bcr00.netops.charter.com [66.109.6.74]
7 269 ms 45 ms 58 ms 74.125.50.134
8 60 ms * 59 ms 142.251.225.91
9 25 ms 34 ms 28 ms 142.251.65.103
10 36 ms 32 ms 25 ms dns.google [8.8.8.8]

Pinging 142.254.218.97 with 32 bytes of data:
Reply from 142.254.218.97: bytes=32 time=18ms TTL=254
Reply from 142.254.218.97: bytes=32 time=20ms TTL=254
Reply from 142.254.218.97: bytes=32 time=36ms TTL=254
Reply from 142.254.218.97: bytes=32 time=8ms TTL=254
Reply from 142.254.218.97: bytes=32 time=16ms TTL=254
Reply from 142.254.218.97: bytes=32 time=21ms TTL=254
Reply from 142.254.218.97: bytes=32 time=15ms TTL=254
Reply from 142.254.218.97: bytes=32 time=9ms TTL=254
Reply from 142.254.218.97: bytes=32 time=7ms TTL=254
Request timed out.
Reply from 142.254.218.97: bytes=32 time=12ms TTL=254
Reply from 142.254.218.97: bytes=32 time=52ms TTL=254
Reply from 142.254.218.97: bytes=32 time=10ms TTL=254
Reply from 142.254.218.97: bytes=32 time=24ms TTL=254
Reply from 142.254.218.97: bytes=32 time=25ms TTL=254
Reply from 142.254.218.97: bytes=32 time=41ms TTL=254
Reply from 142.254.218.97: bytes=32 time=8ms TTL=254
Reply from 142.254.218.97: bytes=32 time=25ms TTL=254
Reply from 142.254.218.97: bytes=32 time=46ms TTL=254
Reply from 142.254.218.97: bytes=32 time=11ms TTL=254
Reply from 142.254.218.97: bytes=32 time=8ms TTL=254
Reply from 142.254.218.97: bytes=32 time=17ms TTL=254
Request timed out.
Reply from 142.254.218.97: bytes=32 time=69ms TTL=254

RAIST515O

Try running pathping instead of tracert…may provide some insight into how things are going between hops.

May be that congestion is causing things to intermittently get held in queue or policies are causing hops to ignore the ICMP ECHO request because of high utilization--but otherwise they still get forwarded, though possibly a bit delayed.

RAIST515O

Here is a sample pathping report (from the Microsoft page.about it) that demonstrates the difference.

https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/pathping

D:>pathping /n contoso1
Tracing route to contoso1 [10.54.1.196]
over a maximum of 30 hops:
0 172.16.87.35
1 172.16.87.218
2 192.168.52.1
3 192.168.80.1
4 10.54.247.14
5 10.54.1.196
computing statistics for 125 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct address
0 172.16.87.35
0/ 100 = 0% |
1 41ms 0/ 100 = 0% 0/ 100 = 0% 172.16.87.218
13/ 100 = 13% |
2 22ms 16/ 100 = 16% 3/ 100 = 3% 192.168.52.1
0/ 100 = 0% |
3 24ms 13/ 100 = 13% 0/ 100 = 0% 192.168.80.1
0/ 100 = 0% |
4 21ms 14/ 100 = 14% 1/ 100 = 1% 10.54.247.14
0/ 100 = 0% |
5 24ms 13/ 100 = 13% 0/ 100 = 0% 10.54.1.196
Trace complete.

The formatting gets a bit wierd when you copy/paste it, tried to edit it a little to line things up, but it ignored the trailing spaces... don't remember if this web app supports using code or html tags, so it still looks a bit weird.

¯\_(ツ)_/¯

What you see here is 13 packets actually getting dropped (did not appear to make it to the endpoint), while some hops are just intermittently timing out on processing the reply requests—they are still forwarding those packets to the next hop.

nute89

Thanks for the suggestion. If I understand correctly I believe these results point to congestion. Is that correct?

Tracing route to 8.8.8.8 over a maximum of 30 hops

0 192.168.1.16
1 192.168.1.1
2 142.254.218.97
3 24.58.232.233
4 24.58.52.24
5 24.58.32.70
6 66.109.6.74
7 74.125.50.134
8 142.251.225.91
9 142.251.65.103
10 8.8.8.8

Computing statistics for 250 seconds...
Source to Here This Node/Link
Hop RTT Lost/Sent = Pct Lost/Sent = Pct Address
0 192.168.1.16
0/ 100 = 0% |
1 0ms 0/ 100 = 0% 0/ 100 = 0% 192.168.1.1
3/ 100 = 3% |
2 32ms 5/ 100 = 5% 2/ 100 = 2% 142.254.218.97
0/ 100 = 0% |
3 61ms 3/ 100 = 3% 0/ 100 = 0% 24.58.232.233
0/ 100 = 0% |
4 75ms 4/ 100 = 4% 1/ 100 = 1% 24.58.52.24
0/ 100 = 0% |
5 103ms 3/ 100 = 3% 0/ 100 = 0% 24.58.32.70
0/ 100 = 0% |
6 101ms 3/ 100 = 3% 0/ 100 = 0% 66.109.6.74
1/ 100 = 1% |
7 99ms 9/ 100 = 9% 5/ 100 = 5% 74.125.50.134
0/ 100 = 0% |
8 86ms 4/ 100 = 4% 0/ 100 = 0% 142.251.225.91
0/ 100 = 0% |
9 70ms 4/ 100 = 4% 0/ 100 = 0% 142.251.65.103
0/ 100 = 0% |
10 67ms 4/ 100 = 4% 0/ 100 = 0% 8.8.8.8

RAIST515O

Most likely the case further down the line (breaking 100ms spikes midway)

The flag goes up on the first hops off your network though. Something is odd more locally for sure.

Seeing 3-5% rejection with 1-3% flux between Spectrum hops is a head scratcher too.

This is good data for them to pass on to Tier3 though. Thanks for sharing it.

Could just be an unfortunate side effect of bandwidth limitations that <hopefully> gets resolved as the DOCSIS upgrades complete. May not have that answer until a significant amount of the 3.0 services upgrade though (no one will ever admit to an oversold market condition, but it happens naturally).