Sys-net slow + dropping packets after recent updates

[groxx]

As of ~a few weeks ago (sorry, can’t really be more precise), while doing fairly frequent updates (usually within a day or two of the updater notifying me), my sys-net sometimes gets into a state where it drops tons of packets and responds very slow. It is nearly always in this state immediately after my machine boots up.

E.g. here’s from a personal qube connected to sys-firewall, and the gap marks where I rebooted only sys-net:

`ping 1.1.1.1` results for a couple minutes, with it dropping something like 75% of packets at 120ms until a reboot brings it to perfect at 20ms

user@personal:~$ ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=7 ttl=53 time=120 ms
64 bytes from 1.1.1.1: icmp_seq=16 ttl=53 time=138 ms
64 bytes from 1.1.1.1: icmp_seq=20 ttl=53 time=131 ms
64 bytes from 1.1.1.1: icmp_seq=21 ttl=53 time=105 ms
64 bytes from 1.1.1.1: icmp_seq=24 ttl=53 time=116 ms
64 bytes from 1.1.1.1: icmp_seq=28 ttl=53 time=132 ms
64 bytes from 1.1.1.1: icmp_seq=32 ttl=53 time=68.5 ms
64 bytes from 1.1.1.1: icmp_seq=38 ttl=53 time=110 ms
64 bytes from 1.1.1.1: icmp_seq=40 ttl=53 time=133 ms
64 bytes from 1.1.1.1: icmp_seq=66 ttl=53 time=120 ms
64 bytes from 1.1.1.1: icmp_seq=71 ttl=53 time=144 ms
64 bytes from 1.1.1.1: icmp_seq=79 ttl=53 time=136 ms
64 bytes from 1.1.1.1: icmp_seq=80 ttl=53 time=111 ms
64 bytes from 1.1.1.1: icmp_seq=82 ttl=53 time=17.4 ms
64 bytes from 1.1.1.1: icmp_seq=89 ttl=53 time=111 ms
64 bytes from 1.1.1.1: icmp_seq=95 ttl=53 time=111 ms
64 bytes from 1.1.1.1: icmp_seq=97 ttl=53 time=119 ms
64 bytes from 1.1.1.1: icmp_seq=103 ttl=53 time=142 ms
64 bytes from 1.1.1.1: icmp_seq=104 ttl=53 time=21.4 ms
64 bytes from 1.1.1.1: icmp_seq=105 ttl=53 time=124 ms
64 bytes from 1.1.1.1: icmp_seq=115 ttl=53 time=118 ms
64 bytes from 1.1.1.1: icmp_seq=116 ttl=53 time=140 ms
64 bytes from 1.1.1.1: icmp_seq=117 ttl=53 time=123 ms
64 bytes from 1.1.1.1: icmp_seq=122 ttl=53 time=134 ms
64 bytes from 1.1.1.1: icmp_seq=126 ttl=53 time=142 ms
64 bytes from 1.1.1.1: icmp_seq=128 ttl=53 time=142 ms
64 bytes from 1.1.1.1: icmp_seq=143 ttl=53 time=134 ms
64 bytes from 1.1.1.1: icmp_seq=145 ttl=53 time=142 ms
64 bytes from 1.1.1.1: icmp_seq=148 ttl=53 time=134 ms
64 bytes from 1.1.1.1: icmp_seq=157 ttl=53 time=134 ms
64 bytes from 1.1.1.1: icmp_seq=166 ttl=53 time=166 ms
64 bytes from 1.1.1.1: icmp_seq=170 ttl=53 time=136 ms
64 bytes from 1.1.1.1: icmp_seq=174 ttl=53 time=111 ms
64 bytes from 1.1.1.1: icmp_seq=181 ttl=53 time=127 ms
64 bytes from 1.1.1.1: icmp_seq=187 ttl=53 time=112 ms
64 bytes from 1.1.1.1: icmp_seq=189 ttl=53 time=136 ms
64 bytes from 1.1.1.1: icmp_seq=196 ttl=53 time=128 ms
64 bytes from 1.1.1.1: icmp_seq=198 ttl=53 time=136 ms
64 bytes from 1.1.1.1: icmp_seq=213 ttl=53 time=143 ms
64 bytes from 1.1.1.1: icmp_seq=214 ttl=53 time=117 ms
64 bytes from 1.1.1.1: icmp_seq=219 ttl=53 time=119 ms
64 bytes from 1.1.1.1: icmp_seq=224 ttl=53 time=119 ms

64 bytes from 1.1.1.1: icmp_seq=250 ttl=53 time=16.4 ms
64 bytes from 1.1.1.1: icmp_seq=251 ttl=53 time=12.1 ms
64 bytes from 1.1.1.1: icmp_seq=252 ttl=53 time=12.1 ms
64 bytes from 1.1.1.1: icmp_seq=253 ttl=53 time=28.3 ms
64 bytes from 1.1.1.1: icmp_seq=254 ttl=53 time=24.6 ms
64 bytes from 1.1.1.1: icmp_seq=255 ttl=53 time=24.7 ms
64 bytes from 1.1.1.1: icmp_seq=256 ttl=53 time=23.7 ms
64 bytes from 1.1.1.1: icmp_seq=257 ttl=53 time=15.7 ms
64 bytes from 1.1.1.1: icmp_seq=258 ttl=53 time=26.7 ms

The behavior looks the same from within sys-net, sometimes it’s normal and sometimes it drops tons of packets like above. After a sys-net reboot (but ~never on initial machine boot) it recovers immediately, and usually it remains “good” for the day (at least a couple hours - I have no S3 support, so I power it off frequently). So far I’ve seen it “break” only once after a sys-net-reboot fixed it, only a few minutes later.
(ironically this occurred while trying to submit this error. so I had to re-type it all. drafts would be nice?)

none of the sys-* qubes look resource constrained during this (~4% cpu, top doesn’t seem to imply memory pressure, and I’ve raised memory to test but no changes), so it seems more likely some kind of flaky failure state that it can’t get out of.

I try to keep my dom0 and sys-{usb,net,firewall} as close to unmodified as I can manage, and I follow updates frequently, so it seems fairly likely this was introduced by an update to sys-net ~2-4 weeks ago. But I’m not sure which one that would be, nor how I might figure that out, or what info might be useful for troubleshooting - I haven’t done much with linux networking yet.

Would anyone be willing to point me towards stuff to try?

[barto]

You didn’t include some important info, like the template used for sys-net, amount of memory, type of connection from sys-net (WiFi? wired?).

If you say that it worked before, you can try using a different (older) kernel on sys-net. There are instructions for Fedora and Debian in the official documentation.

But just as a starting point, look in the sys-net logs for driver errors or anything else out of order: /var/log/xen/console/guest-sys-net.log (older entries are gzipped in the same dir).

[qubist]

@groxx

Would anyone be willing to point me towards stuff to try?

I would start with cables, then network adapter testing.
A thread with similar complaint:

Slow Internet Speed Due to Sys-Net

Today I noticed that my internet is slow. I ran some speed tests. Instead of 100, I’m only getting about 10 Mbps. I’ve ruled out all possible sources of error. I restarted the router. I restarted sys-firewall and sys-net. I measured the speed using another computer. Everything is fine there. I switched the router ports. I also ran a speed test on sys-net. The problem exists directly on sys-net as well. Where can I look for more information? I saw with ip -s link show that there are a lot…