Recently network speed slowdown - apparently happening between appvm / sys-net

[curbs94]

In the last two weeks give or take I have noticed a drastic slow down in network transfer speeds over wifi and wired (but rarely use the latter) using both Debian 12 and Fedora 41 templates

I am at a bit of a loss as to why.

I am running

• Novacustoms nv41
• Qubes: 4.2.4
  • Global settings kernel: 6.12.18-1.fc37
• Debian 12 AppVM
  • kernel: 6.12.18-1.qubes.fc37.x86_64
• Fedora 41 AppVM
  • kernel: 6.12.18-1.qubes.fc37.x86_64
• sys-net is Fedora 41
• sys-firewall (disposable) is Fedora41

This first was noticed as I have to rsync over ssh to my NAS daily, this used to transfer files around 80-90 MB/s but in the last two weeks this has dropped down to 2-4 MB/s.

this goes

qubes -> pfsense -> NAS.

in Qubes:

debian: debian appVM -> sys-firewall -> sys-net
fedora: fedora appvm -> net-VPN (vpn off) -> sys-firewall -> sys-net

Playing around with iperf3 I noticed that it looks like sys-net gets full speed on the network but the latency happens somewhere on or after my appVMs in Qubes

(results were the same when destination was qubes directly to the NAS)

sys-net to pfsense

$ iperf3 -c 10.20.30.1
Connecting to host 10.20.30.1, port 5201
[  5] local 10.20.40.101 port 57618 connected to 10.20.30.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  46.0 MBytes   386 Mbits/sec    0    506 KBytes       
[  5]   1.00-2.00   sec  48.3 MBytes   405 Mbits/sec    0    506 KBytes       
[  5]   2.00-3.00   sec  46.9 MBytes   393 Mbits/sec    0    506 KBytes       
[  5]   3.00-4.00   sec  49.8 MBytes   418 Mbits/sec    0    506 KBytes       
[  5]   4.00-5.00   sec  49.4 MBytes   414 Mbits/sec    0    506 KBytes       
[  5]   5.00-6.00   sec  49.4 MBytes   414 Mbits/sec    0    506 KBytes       
[  5]   6.00-7.00   sec  47.5 MBytes   398 Mbits/sec    0    506 KBytes       
[  5]   7.00-8.00   sec  46.8 MBytes   393 Mbits/sec    0    506 KBytes       
[  5]   8.00-9.00   sec  49.0 MBytes   411 Mbits/sec    0    506 KBytes       
[  5]   9.00-10.00  sec  46.4 MBytes   389 Mbits/sec    0    506 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   479 MBytes   402 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   478 MBytes   400 Mbits/sec                  receiver

sys-firewall to pfsense

$ iperf3 -c 10.20.30.1
Connecting to host 10.20.30.1, port 5201
[  5] local 10.138.20.249 port 57298 connected to 10.20.30.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.88 MBytes  40.8 Mbits/sec  174   4.28 KBytes       
[  5]   1.00-2.00   sec  2.12 MBytes  17.8 Mbits/sec  122   4.28 KBytes       
[  5]   2.00-3.00   sec  1.25 MBytes  10.5 Mbits/sec   94   4.28 KBytes       
[  5]   3.00-4.00   sec  1.75 MBytes  14.7 Mbits/sec   98   4.28 KBytes       
[  5]   4.00-5.00   sec  2.12 MBytes  17.8 Mbits/sec  120   5.70 KBytes       
[  5]   5.00-6.00   sec  1.75 MBytes  14.7 Mbits/sec  118   4.28 KBytes       
[  5]   6.00-7.00   sec  1.25 MBytes  10.5 Mbits/sec  101   5.70 KBytes       
[  5]   7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec   94   4.28 KBytes       
[  5]   8.00-9.00   sec  1.75 MBytes  14.7 Mbits/sec  100   4.28 KBytes       
[  5]   9.00-10.00  sec  1.75 MBytes  14.7 Mbits/sec  118   4.28 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  19.9 MBytes  16.7 Mbits/sec  1139             sender
[  5]   0.00-10.00  sec  19.0 MBytes  15.9 Mbits/sec                  receiver

debian appvm to pfsense

$ iperf3 -c 10.20.30.1 
Connecting to host 10.20.30.1, port 5201
[  5] local 10.137.0.10 port 34102 connected to 10.20.30.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   516 KBytes  4.23 Mbits/sec   18   27.1 KBytes       
[  5]   1.00-2.00   sec  19.0 MBytes   159 Mbits/sec    0   68.4 KBytes       
[  5]   2.00-3.00   sec  19.0 MBytes   159 Mbits/sec    0   68.4 KBytes       
[  5]   3.00-4.00   sec  6.03 MBytes  50.6 Mbits/sec   45   2.85 KBytes       
[  5]   4.00-5.00   sec  1.23 MBytes  10.3 Mbits/sec   96   4.28 KBytes       
[  5]   5.00-6.00   sec  1.68 MBytes  14.1 Mbits/sec  114   4.28 KBytes       
[  5]   6.00-7.00   sec  1.23 MBytes  10.3 Mbits/sec   88   4.28 KBytes       
[  5]   7.00-8.00   sec  1.07 MBytes  8.99 Mbits/sec   88   8.55 KBytes       
[  5]   8.00-9.00   sec  1.38 MBytes  11.6 Mbits/sec   97   4.28 KBytes       
[  5]   9.00-10.00  sec  1.07 MBytes  8.99 Mbits/sec   77   2.85 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  52.1 MBytes  43.7 Mbits/sec  623             sender
[  5]   0.00-10.00  sec  51.8 MBytes  43.4 Mbits/sec                  receiver

fedora appvm to pfsense

iperf3 -c 10.20.30.1
Connecting to host 10.20.30.1, port 5201
[  5] local 10.137.0.25 port 47468 connected to 10.20.30.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  13.2 MBytes   111 Mbits/sec  270   2.85 KBytes       
[  5]   1.00-2.00   sec  1.00 MBytes  8.39 Mbits/sec  142   4.28 KBytes       
[  5]   2.00-3.00   sec   896 KBytes  7.34 Mbits/sec  108   2.85 KBytes       
[  5]   3.00-4.00   sec  2.00 MBytes  16.8 Mbits/sec  102   4.28 KBytes       
[  5]   4.00-5.00   sec  1.00 MBytes  8.39 Mbits/sec   88   2.85 KBytes       
[  5]   5.00-6.00   sec  1.00 MBytes  8.39 Mbits/sec  107   1.43 KBytes       
[  5]   6.00-7.00   sec  1.00 MBytes  8.39 Mbits/sec   65   2.85 KBytes       
[  5]   7.00-8.00   sec   896 KBytes  7.34 Mbits/sec   98   2.85 KBytes       
[  5]   8.00-9.00   sec  1.00 MBytes  8.39 Mbits/sec   96   4.28 KBytes       
[  5]   9.00-10.00  sec  2.00 MBytes  16.8 Mbits/sec   83   2.85 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  24.0 MBytes  20.1 Mbits/sec  1159             sender
[  5]   0.00-10.00  sec  21.1 MBytes  17.7 Mbits/sec                  receive

[renehoj]

I’m using the Debian 12 minimal template, I don’t have any issues with speed from my appVMs.

[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0301 sec  1.11 GBytes   947 Mbits/sec

What is your CPU and memory load in sys-firewall when running iperf?

My sys-firewall does consume a lot of CPU when iperf is testing the network speed.

[curbs94]

It doesn’t really break a sweat

sys-firewall top

top - 23:27:54 up 1 day,  6:13,  2 users,  load average: 0.51, 0.22, 0.07
Tasks: 172 total,   2 running, 170 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.0 us,  1.4 sy,  0.0 ni, 94.7 id,  0.0 wa,  0.5 hi,  0.2 si,  0.2 st 
MiB Mem :   3926.7 total,   2460.7 free,    614.9 used,    865.9 buff/cache     
MiB Swap:   1024.0 total,   1011.9 free,     12.1 used.   3311.8 avail Mem

[curbs94]

Update:

bypassing sys-firewall and routing the appVMs directly through sys-net results in the same latency.

[slcoleman]

Whenever I have an unexplained and sudden reduction in network speed in Qubes itself the first thing I look at is the DNS configuration. When DNS is not answering properly/promptly everything slows way down due to timeouts in resolving addresses. There won’t be spikes in CPU because it is all timed delays built into the protocol.

Try adding a third party DNS resolver (e.g. 8.8.8.8) and see if this resolves your problem.

[ChrisA]

Hi curbsv94

Is it somewhat the same issue as:

Slow upload to same network as sys-net

This is a new one for me. There were a slew of updates to dom0 and somewhere along the line, my setup started having slow upload speeds to my internal network. Both sys-net and internal host are on the same wifi net, nothing in between. sys-net → internal host = 33mb/sec AppVm → sys-net → internal host = 3mb/sec And this was after I changed the sys-net to a fedora 40 template (was debian 12) and also using an older kernel (6.6.77). With debian 12 and the latest kernel (6.12.18-1) transfer sp…

? – and if so, does the solution from that, work for you?

[curbs94]

Thank you. This was it…

Downgraded sys-net kernel to 6.6.77-1.qubes.fc37.x86_64 and speeds went back up to what I was previously getting prior to the issues.