Slow upload to same network as sys-net

This is a new one for me. There were a slew of updates to dom0 and somewhere along the line, my setup started having slow upload speeds to my internal network. Both sys-net and internal host are on the same wifi net, nothing in between.

sys-net → internal host = 33mb/sec
AppVm → sys-net → internal host = 3mb/sec

And this was after I changed the sys-net to a fedora 40 template (was debian 12) and also using an older kernel (6.6.77). With debian 12 and the latest kernel (6.12.18-1) transfer speed would be around 500kb and would stall often with these types of errors from strace

read(3, “\34\220:\311V\211/E,r\304\325\2\30\237\276\4s6G\334\271\340\352"\311\244\347\324.\244\215”…, 261120) = 261120
writev(4, [{iov_base=“\0\3\374\31”, iov_len=4}, {iov_base=“\6\0\0\0001\0\0\0\4\0\0\0\0\0\0\0\0\0\257P\0\0\3\374\0\34\220:\311V\211/”…, iov_len=261145}], 2) = 73088
— SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} —
alarm(1) = 0
rt_sigreturn({mask=}) = 73088
getpgrp() = 1726
ioctl(1, TIOCGPGRP, [1726]) = 0
tempo 0% 255KB 167.0KB/s 29:54 ETA) = 190
writev(4, [{iov_base=“de\303 \224\310\262k\367\37\f\3350w\267\331\344\271]\22\375\331@\376s\202\f\3427\263\377\31”…, iov_len=188061}], 1) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
— SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} —
alarm(1) = 0
rt_sigreturn({mask=}) = -1 EINTR (Interrupted system call)
getpgrp() = 1726
ioctl(1, TIOCGPGRP, [1726]) = 0
tempo 0% 255KB 150.3KB/s 33:13 ETA) = 190

Prior to the slew of changes to dom0, I was able to get better speeds from the AppVm to the internal host used to get great speeds. Maybe something in the latest updates of dom0 causes this? I can help in debugging if required. Just need to know what to give you guys as info to assist.

Edit: Qubes OS version 4.2.4

I have zero problems with upload speeds to the internet from appvm → sys-firewall → sys-net, this gets the max of my internet profile. Just the transfer speed to this internal host from an AppVM

Some stats:

Download speed shows close to 43MB/s

wget
–2025-04-09 13:32:55--
Connecting to internal:8000… connected.
HTTP request sent, awaiting response… 200 OK
Length: 307200000 (293M) [application/octet-stream]
Saving to: ‘tempo1’

tempo1 100%[===================>] 292.97M 42.8MB/s in 7.4s

2025-04-09 13:33:02 (39.7 MB/s) - ‘tempo1’ saved [307200000/307200000]

Upload is 10x slower and it shows about 3.2MB/s

curl -X POST -F ‘files=@tempo2’ > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
46 292M 0 0 46 136M 0 3208k 0:01:33 0:00:43 0:00:50 3175k

The upload speed is very very slow regardless of protocol used (scp, http, smb, etc).

Yes, same here and it “probably” only appears in internal networks and on uploads only! Looks (for me) as a bug in Kernel 6.12 (appeared on QubesOS the first time, after dom0 was updated to Kernel 6.12-1) and also is under investigation by QubesOS staff. But somethings tells me, it could be a non-Qubes issue…

You can currently fix this by returning to Kernel 6.6 in sys-net (settings > advanced) ONLY! No need to change anything on sys-firewall or on your AppVMs.

Even with downgrading the kernel, the performance is still not good. At least it’s stable and doesn’t flake out anymore but speeds should be way higher. 3mb/s from AppVM while from sys-net I get 33mb/s is a huge performance hit for simply going through sys-net. Glad the Qubes OS team is aware of the issue. They can reach me if they need any logs / traces / etc and I’ll be happy to help them if they can’t reproduce in house

is there a GitHub issue?

I couldn’t find one when I looked this morning but I’m not super good with issue trackers in general

Not yet! Marek just had a look on it, but assume he’s busy with all his other prioritíes.

On a whim, I increased the ram for sys-net. It was at 425M which I believe is the default as I never adjusted it after install. So, with fedora-40, kernel 6.6.77-1 and 550M of RAM for sys-net, performance for my internal network is back to normal.

1 Like

and what is with Kernel 6.6.77-1 and 425M of RAM for sys-net ? Still the mentioned issue? Or just a few less of “normal” for the internal network (around 22-25MB/s)?

with 6.6.77-1 and 425M, I did have the very slow speed issue. I haven’t reduced to size to retest it. I can if required

this is weird, went back to 425M (fedora-40 and 6.6.77-1) and I cannot reproduce the issue. There was an update to the fedora-40 template this morning so this might have solved the issue. This was a really weird one for me. Hopefully something upstream in fedora fixed the issue

No! Just change back to Kernel 6.12… in sys-net and the issue will be there again…

Just to add, I had the same experience

Downgraded sys-net (Fedora 41) kernel to 6.6.77-1.qubes.fc37.x86_64 and speeds went back up to what I was previously getting prior to kernel 6.12.18-1.qubes.fc37.x86_64

My sys-net is from a Debian 12 template btw.

for me I tried it with both Fedora 41 and Debian 12 templates. Issue only got better when I downgraded to older Kernel.

That leaves me wonder what next, as eventually that kernel will get auto remove from update

In my case, even with kernel 6.6.77, my debian-12-xfce template for sys-net makes it very slow. I know, I should not use an xfce template with sys-net but I just want to highlight that it might be a bit more than just a kernel related issue. For the time being, I’m sticking to fedora-40 and kernel 6.6.77 for my sys-net as this is what is currently working

Same here with Fedora-41. Switched to 6.74 in sys-net and all went back to normal.

I stupidly updated the fedora-40 template this morning without taking a backup. That update once again broke the upload speeds to lan (maximum of 2.9mb/s). This was with fedora-40 and kernel 6.6.77. I have switched to fedora-41 and will not update this one as whatever is causing the issue is now propagating everywhere. This is what was updated this morning:

dnf history info 12
Transaction ID : 12
Begin time : Sun Apr 20 08:07:43 2025
Begin rpmdb : 87c8dee45ce5ef90992e9a51753d07664d306bfd3de7584a7d2a3731596a951b
End time : Sun Apr 20 08:08:35 2025 (52 seconds)
End rpmdb : 49d69c333c7d2e25efc80959f6ad7e9ff42c2829b396e5787a2b3a3ab3799a23
User : Super User
Return-Code : Success
Releasever : 40
Command Line : qubes-vm-update
Comment :
Packages Altered:
Install kernel-6.13.11-100.fc40.x86_64 @updates
Install kernel-core-6.13.11-100.fc40.x86_64 @updates
Install kernel-modules-6.13.11-100.fc40.x86_64 @updates
Install kernel-modules-core-6.13.11-100.fc40.x86_64 @updates
Upgrade bluez-5.81-2.fc40.x86_64 @updates
Upgraded bluez-5.79-1.fc40.x86_64 @@System
Upgrade bluez-cups-5.81-2.fc40.x86_64 @updates
Upgraded bluez-cups-5.79-1.fc40.x86_64 @@System
Upgrade bluez-libs-5.81-2.fc40.x86_64 @updates
Upgraded bluez-libs-5.79-1.fc40.x86_64 @@System
Upgrade bluez-obexd-5.81-2.fc40.x86_64 @updates
Upgraded bluez-obexd-5.79-1.fc40.x86_64 @@System
Upgrade dnf-4.23.0-1.fc40.1.noarch @updates
Upgraded dnf-4.23.0-1.fc40.noarch @@System
Upgrade dnf-data-4.23.0-1.fc40.1.noarch @updates
Upgraded dnf-data-4.23.0-1.fc40.noarch @@System
Upgrade glib-networking-2.80.1-1.fc40.x86_64 @updates
Upgraded glib-networking-2.80.0-1.fc40.x86_64 @@System
Upgrade glibmm2.68-2.80.1-1.fc40.x86_64 @updates
Upgraded glibmm2.68-2.80.0-1.fc40.x86_64 @@System
Upgrade gnome-calculator-46.2-1.fc40.x86_64 @updates
Upgraded gnome-calculator-46.0-1.fc40.x86_64 @@System
Upgrade gnome-control-center-46.8-1.fc40.x86_64 @updates
Upgraded gnome-control-center-46.5-1.fc40.x86_64 @@System
Upgrade gnome-control-center-filesystem-46.8-1.fc40.noarch @updates
Upgraded gnome-control-center-filesystem-46.5-1.fc40.noarch @@System
Upgrade hwdata-0.394-1.fc40.noarch @updates
Upgraded hwdata-0.393-1.fc40.noarch @@System
Upgrade javascriptcoregtk4.1-2.48.1-2.fc40.x86_64 @updates
Upgraded javascriptcoregtk4.1-2.48.0-1.fc40.x86_64 @@System
Upgrade javascriptcoregtk6.0-2.48.1-2.fc40.x86_64 @updates
Upgraded javascriptcoregtk6.0-2.48.0-1.fc40.x86_64 @@System
Upgrade libsolv-0.7.32-4.fc40.x86_64 @updates
Upgraded libsolv-0.7.31-1.fc40.x86_64 @@System
Upgrade libxcrypt-4.4.38-7.fc40.x86_64 @updates
Upgraded libxcrypt-4.4.38-6.fc40.x86_64 @@System
Upgrade libxcrypt-devel-4.4.38-7.fc40.x86_64 @updates
Upgraded libxcrypt-devel-4.4.38-6.fc40.x86_64 @@System
Upgrade netpbm-11.10.00-1.fc40.x86_64 @updates
Upgraded netpbm-11.09.00-2.fc40.x86_64 @@System
Upgrade python3-dnf-4.23.0-1.fc40.1.noarch @updates
Upgraded python3-dnf-4.23.0-1.fc40.noarch @@System
Upgrade upower-1.90.9-1.fc40.x86_64 @updates
Upgraded upower-1.90.8-1.fc40.x86_64 @@System
Upgrade upower-libs-1.90.9-1.fc40.x86_64 @updates
Upgraded upower-libs-1.90.8-1.fc40.x86_64 @@System
Upgrade webkit2gtk4.1-2.48.1-2.fc40.x86_64 @updates
Upgraded webkit2gtk4.1-2.48.0-1.fc40.x86_64 @@System
Upgrade webkitgtk6.0-2.48.1-2.fc40.x86_64 @updates
Upgraded webkitgtk6.0-2.48.0-1.fc40.x86_64 @@System
Reason Change kernel-6.13.9-100.fc40.x86_64 @updates
Removed kernel-6.10.7-200.fc40.x86_64 @@System
Reason Change kernel-core-6.13.9-100.fc40.x86_64 @updates
Removed kernel-core-6.10.7-200.fc40.x86_64 @@System
Reason Change kernel-modules-6.13.9-100.fc40.x86_64 @updates
Removed kernel-modules-6.10.7-200.fc40.x86_64 @@System
Reason Change kernel-modules-core-6.13.9-100.fc40.x86_64 @updates
Removed kernel-modules-core-6.10.7-200.fc40.x86_64 @@System
Scriptlet output:
1 dracut[E]: Module ‘systemd-pcrphase’ depends on ‘tpm2-tss’, which can’t be installed
2 dracut[E]: Module ‘systemd-cryptsetup’ depends on ‘crypt’, which can’t be installed

For some reason, I was not able to rollback the changes (then again, I’m not very familiar with fedora…)

Can confirm that something in the last update causes the issue. I was able to undo the last updates in my fedora-40 template and speeds to the lan are back to normal.