Continuing troubleshooting intermittent packet loss

Filivanili
Filivanili Posts: 16 Participant
edited April 2023 in Gaming & Gaming Related

Continuing troubleshooting from another discussion: https://community.spectrum.net/discussion/comment/179854#Comment_179854

The next step I was asked to do is plug my PC directly into the modem. I did this, powercycled the modem, and ran some traceroutes to show the latency/packet loss problems I'm experiencing

Traceroute to google.com


Traceroute to nypl.org (New York Public Library), no problems


Traceroute to sfpl.org (San Francisco Public Library)


Traceroute to nypl.org Traceroute to nypl.org (New York Public Library) # 2 (problem)


Traceroute to sfpl.org (San Francisco Public Library) #2


What's next to try?

Best Answers

  • Filivanili
    Filivanili Posts: 16 Participant
    Answer ✓

    So clearly the answer is 'yes' to can you monitor your network connection to me, looks like support chat is going to send a tech out FYI


    I'll provide an update after that happens

  • RAIST5150
    RAIST5150 Posts: 918 Contributor
    Answer ✓

    Can't help but wonder how much of it is simply tied to congestion issues.


    Have you checked it via a VPN that shifts the traffic through a different region?

Answers

  • Filivanili
    Filivanili Posts: 16 Participant

    Next I tried ping tests to the hops where the trouble seemed to be and saw evidence of packet loss consistently with Orlando1.Level13.net


  • RAIST5150
    RAIST5150 Posts: 918 Contributor

    One thing very important to remember about ICMP ECHO requests... replying to them is a low priority task, potentially one of the lowest.

    In other words, not uncommon to delay replies or even not respond at all to preserve resources for more important activities (like a hop's primary focus of forwarding packets to the next hop). Doesn't automatically follow your packets are getting dropped along the way.

  • William_M
    William_M Posts: 1,021 ✅ Verified Employee Moderator

    Hi @Filivanili, sorry you are still having trouble.

    What @RAIST5150 is saying is correct. The primary function of the hop is to forward data to the next hop, not respond to ping requests. Your tests actually all show a healthy connection, you always make it to the next hop and without any spikes in response time.

    Will you run a ping test on the destination server instead of a hop, and run one to 8.8.8.8, to see if you are measuring any loss there?

  • Filivanili
    Filivanili Posts: 16 Participant

    Here you go - you can see the intermittent nature of the latency I'm experiencing



  • Filivanili
    Filivanili Posts: 16 Participant

    Here's what it looks like when gaming (Rocket League). Notice how I'm the only one with the latency problem (I'm FiDO).


  • William_M
    William_M Posts: 1,021 ✅ Verified Employee Moderator

    Thank you.

    When you play Rocket League is it like this constantly, or is it just for a second every few minutes like in your ping test?

    If you run the ping test while playing do you notice the connection error occuring at the same time as the ping spike? If you ping a different IP in a second command prompt does a spike occur on that at the same time?

    I know you ran the earlier traceroutes connected directly to the modem, but have you tried playing Rocket League direct to the modem? Any firewall, antivirus, or other software that might impact the connection that can be paused while you do this?

    Do you have this issue playing different games, or on different devices?

  • Filivanili
    Filivanili Posts: 16 Participant

    Latency in Rocket League is same as the ping test - intermittently for 1-4 seconds and then goes away, then comes back again. I have tried playing direct to modem and the same problem occurs. No firewall, no antivirus, and bear in mind that I'll play without a problem for hours and then suddenly, boom, intermittent latency. It happens in other games, like Hunt Showdown as well.


    I've provided screenshots with dates and timestamps - is there any monitoring on your side where you can see my modem's performance during those times I've provided?

  • RAIST5150
    RAIST5150 Posts: 918 Contributor
    edited January 2023

    Just for clarification, what regional servers are you using?

    I ask because in the earlier snapshots, I am seeing hops assigned to Florida and California.

    If you are reaching that far for a game, there are going to be packet delivery snafus, and if the games are using UDP in addition to or instead of TCP that is gonna get messy (no error correction/retransmit/etc.)


    EDIT: Should probably clarify it a bit myself. Not uncommon to see east/west coast loaded latency factors hitting triple digits, especially in higher traffic times of day (streaming, gaming, etc.). Loaded latency is a different animal than your normal ping stats... run the Netflix speed test for a demonstration (fast.com). What this means is the developers have to run funky lag compensation on their servers to try to give gameplay the appearance of being smooth and in synch (in reality, it can be a hot mess). Different tricks are used... some will integrate UDP, which while it has some nice pros, there is a big con--it is a best effort protocol. Once on the wire, it just goes... somewhere, gets there whenever. If the route is clean, it zips through great--but if it gets delayed, gets corrupted, etc.--no mechanism to correct the data stream. If the server doesn't like what it gets for whatever reason, it just gets discarded. When latency crosses the 100ms mark (just over 3 frames at 30fps, 6 at 60fps... much worse at 120/144hz rate), it can be a challenge keeping things in line even with TCP's corrective measures. With UDP, it can get pretty ugly... fast. Any old school Destiny 1 player can attest to that.

  • Filivanili
    Filivanili Posts: 16 Participant

    Yeah, I'm always selecting US East or North America East every time, every game. I usually get 30-50 ms latency in nearly every game. Is there a way for you to monitor your network to see if higher demand in the evening with elevated usage causes stress leading to latency like I'm experiencing? Seems a likely explanation and should be fixable. Bear in mind, I've had a tech out already who replaced the coax from the street all the way in, replaced the modem, and recommended I change my router which I did. The problem persists - almost always 6 PM to 7 PM and sometimes later.

    I ran two ping tests to two different IPs and sure enough, timeout happens at the same time:


    Appreciate your help, very frustrated

  • Filivanili
    Filivanili Posts: 16 Participant

    Tech #2 came out, stated that Tech #1 did a great job, everything was clean, tightly connected, and a spot check showed green on his scope both at the modem and at the street. However, the same intermittent problem persists.


    What would you suggest next?

  • Jaleesa_F
    Jaleesa_F Posts: 317 ✅ Verified Employee Moderator

    Good morning @Filivanili

    If the tech has verified that modem is working properly, the issue may be with the router. Have you tried bypassing your router and playing the game hardwired directly to the modem?

  • Filivanili
    Filivanili Posts: 16 Participant

    Yes I have, scroll way up, that was one of the first things I did. Here's what I'm experiencing again. That's really really bad.



  • William_M
    William_M Posts: 1,021 ✅ Verified Employee Moderator

    I recommend testing with a different device and a different ethernet cable. Make sure there are no electronic devices close to the coaxial cable or ethernet cables. Interference while using cables is usually not a problem but weakly shielded cables (like the thin flat ethernet cables) or extremely close electronics/radio signals can still cause intermittent problems. Check if the cables might be getting pinched in any doorways or under furniture. If you can, connect the modem to a different power outlet, use a surge protector or even better one with a battery backup to make sure there aren't power issues.

    Is there anything else that happens around 6-7 PM, such as sprinklers that come on automatically or large electrical appliances use around that time? Is it worse on nights that see larger temperature drops at the end of the day?

    You mentioned the problem is always 6 PM to 7 PM but it looks like we have been sending technicians in the morning, it may be worth having another technician come around the time the problem is normally occurring. Intermittent issues can be tough to fix especially if the problem isn't present when the technician is testing.

  • Filivanili
    Filivanili Posts: 16 Participant

    Thanks for those suggestions. The only electronic equipment near our coax or ethernet cables are: the modem, the router (TP-link AX11000), and our gaming PCs (2 towers, 2 monitors).

    Though I joke that the signal degrades when the wind blows, in reality there's no correlation, either to weather or temperature. 6-7PM is likely when neighbors are starting to use the internet so my guess is there's insufficient bandwidth or other factor resulting in connection degradation as usage increases during this time. The problem isn't unique to this time period, I have experienced sudden lag/packet loss at other times of the day, (e.g. dropped from work Zoom calls around 9 or 10 AM during the week a few times), but 6-7:30 PM seems particularly problematic - daily problem.

    I agree that having a tech come out right around 7PM would allow them to experience my problem, but would they be able to diagnose it? Seems to me your company should have sufficient equipment monitoring in place to diagnose appliance problems or network connectivity issues at different nodes, e.g. from an individual user's modem, to the street, neighborhood, district, etc. At this point, I'd really be interested to learn what Spectrum is seeing in my area. I asked a Spectrum tech that was visiting my street (not visiting me) if he was fixing something, and he stated, "we activated our new hub and it's now seeing 17-18 problems in this area that weren't detected before".

    What can you tell me about larger, street or subdivision level issues?

  • William_M
    William_M Posts: 1,021 ✅ Verified Employee Moderator

    They would be much more likely to be able to find the source of the problem if it is occuring when they test, but even so intermittent issues do still sometimes take multiple appointments to resolve. Repeat appointments for the same problem within a month are automatically escalated, I definitely recommend persistence if an issue ever remains unresolved.

    What we can see remotely is mostly limited to information that equipment like the modem and cable receivers are reporting back to us, such as transmit and receive signal strength for each used frequency, ping, up time, timeouts, etc. Even when we see an issue it does not generally tell us where exactly the issue is coming from. Our system can automatically detect some area problems such as when multiple modems all go offline at the same time or all report the same signal deviation, but some problems may not be noticed until we receive reports from multiple customers having the same issue. Both would require a technician going out in person and testing at various points to find where the problem originates. If you know anyone else having the exact same problem you should encourage them to call in and report it, and schedule their own appointments as needed. Even when technicians are looking to fix an individual issue it may still lead back to a problem affecting everyone.

    Have you tested this with both of your computers and different ethernet cables?

  • RAIST5150
    RAIST5150 Posts: 918 Contributor

    Unless your modem has one built in, they need to be on site to run their spectrum analyzer that actually shows the signal degradation in the waveform.

    Only a handful of modems have this feature built in, and some ISP's have actually disabled it (they aren't the greatest, tbh... at best, they may help you identify the rolloff pattern indicative of an old school choke still on the line from when the system was mostly analog tv).


    Barring ingress from somewhere on the street (like 600mhz LTE from the likes of TMobile or Walmart Family Mobile), this could actually be an upstream problem as well, which could technically wind up outside the scope of Spectrum's direct influence. But that could take escalation to multiple tiers of technicians and engineers, which takes time and multiple calls to get past the line on your street.

  • RAIST5150
    RAIST5150 Posts: 918 Contributor

    Just for clarification, this is the kind of thing they are not likely to see remotely, and are not likely to see locally unless it is happening at that time:


  • Filivanili
    Filivanili Posts: 16 Participant

    Thanks, good info all. Yes I did try both machines and different ethernet cables, no change.

    Waiting on the tech to arrive, but in the meantime, I've been running PingPlotter. It's doing a great job of capturing my intermittent issues. You can see an example here just before 2 PM today, where a problem occurred:


    And then there's lot of "good green" in between these short episodes of "bad red":

    I'll provide an update after the tech visit - thx

  • Filivanili
    Filivanili Posts: 16 Participant

    3rd tech came out, in spite of intermittent latency issues I was able to show him using PingPlotter, his scope showed healthy signal. He changed out the small yellow ethernet cable that connects the modem to my router with a longer gray one. He confirmed wiring was good, including the connection at the street. At the street, he identified a problem with one of the 4 upload channels and called it in. A different truck came after the tech left and worked on the appliance at the street.

    I'm still evaluating the impact of these adjustments. Anecdotally, the severity of latency when it happens seems to be reduced, however intermittent latency still happens. Will be monitoring and will provide an update again.

  • Filivanili
    Filivanili Posts: 16 Participant

    Sadly the intermittent latency & packet loss continues.



  • Jaleesa_F
    Jaleesa_F Posts: 317 ✅ Verified Employee Moderator

    Good morning @Filivanili

    I'm sorry to hear that you're still experiencing connectivity issues. I do see here that maintenance was completed on 1/24. There were no further issues reported by the maintenance technicians. I would advise further troubleshooting your wireless home network.

  • Filivanili
    Filivanili Posts: 16 Participant

    I haven't tried VPN but am concluding that congestion/insufficient network bandwidth to handle peak usage hours is the culprit. Most recent tech suggested swapping out modems to try to see if we get a different experience, but when the latency issues recur at the same times of the day, every day, and don't occur outside those times, clearly the equipment is functioning just fine. Thanks for your help to this point, but I'm going to stop troubleshooting.

This discussion has been closed.