Since one of the last updates, Qubes R4.3-rc1 fails to start sys-net and sys-firewall, which are set to start automatically on boot. This does not depend on the VMs themselves, because it happens for sys-net, no matter which template it is based on: Fedora-42, Debian-12-minimal, or Debian-13-minimal. The VMs have not changed since the last time they started automatically on Qubes boot.
Has anyone observed a similar behavior?
I haven’t come across this issue with r4.3-rc1, personally.
Since the VMs are started as systemd services, what does sudo systemctl status tell you?
You see any error on screen, log or by try to start them manualy?
There are no error messages in journalctl, but sys-net and sys-firewall do not appear in the systemctl status listing under qubesd.service, as is to be expected in this situation.
There is a log entry X connection to :0.0 broken (explicit kill or server shutdown) in /var/log/qubes/guid.sys-net that disappears as soon as sys-net is started manually or by starting a VM needing it.
Starting these VMs occurs without an error message, and they run as they should.
I found a possible explanation, but so far, I could not verify it: With the Qubes update in W32, a new version of mgmt-salt-dom0-virtual-machines was provided, which, according to @alimirjamali, "If a USB keyboard is used during installation, sys-usb startup will get priority after installation. "My Qubes installation is running from a USB-SSD and therefore has no sys-usb. So I suppose that on booting, sys-net and sys-firewall may simply be waiting for the non-existing sys-usbto come up. Maybe this was caused by upgrading this system from a copy of an installation that originally ran with a sys-usb?
Now my question is: Where is this dependence of the boot startup sequence configured? So far, I have not found that.
This is kinda interesting. First to confirm, sys-firewall starts but sys-net does not? (if you try to start them manually).
If the answer is yes, then it shows that there is a problem with HVMs. And in that case, what happens if you hold shift key while grub loads and select an older version of Xen/Kernel (we had a very recent update of Xen).
And if the problem does not exist with the older version of Xen/Kernel, what is the hardware specification?
No, I don’t think that it’s a problem with HVMs or the hardware, because sys-net can be started without problems manually or by starting some VM that needs it; for instance, starting untrusted starts sys-firewall and sys-net, and they all run as expected.
So, I guess that it is just this dependence on the (non-existing) sys-usb that should run first when booting - and, as that VM never comes up, the others wait indefinitely.
After a reboot, with sys-net still not started, does the output of sudo systemctl status qubes-vm@sys-net.service in dom0 contain anything interesting?
My assumption that the cause was a missing sys-usb, on which the other VMs were waiting, seems to be wrong. After creating a dummy sys-usb, which has no functionality but was set to autostart, this dummy started well on boot, but sys-net and sys-firewall still did not start.
So I had a look at the service states. sudo systemctl status qubes-vm@sys-net.service resulted in:
? qubes-vm@sys-net.service - Start Qubes VM sys-net
Loaded: loaded (/usr/lib/systemd/system/qubes-vm@.service; disabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/service.d
??10-timeout-abort.conf, 50-keep-warm.conf
/etc/systemd/system/qubes-vm@.service.d
??50_autostart.conf
Active: inactive (dead)
This did not change at all when the two VMs were started. On the other hand, looking at this state in a Qubes R4.2.4 installation, where the VMs are started correctly, showed the following state, which is quite different:
● qubes-vm@sys-net.service - Start Qubes VM sys-net
Loaded: loaded (/usr/lib/systemd/system/qubes-vm@.service; enabled; preset: disabled)
Active: active (exited) since Sat 2025-08-23 10:46:37 CEST; 4min 15s ago
Process: 1709 ExecStart=/usr/bin/qvm-start --skip-if-running sys-net (code=exited, status=0/SUCCESS)
Main PID: 1709 (code=exited, status=0/SUCCESS)
CPU: 84ms
Aug 23 10:46:18 dom0 systemd[1]: Starting qubes-vm@sys-net.service - Start Qubes VM sys-net...
Aug 23 10:46:37 dom0 systemd[1]: Finished qubes-vm@sys-net.service - Start Qubes VM sys-net.
Therefore, I looked into sudo systemctl status qubes-vm@sys-net.service:
Failed to enable unit: Refusing to operate on template unit qubes-vm@.service when destination unit multi-user.target is a non-template unit
Trying the same in Qubes R4.2.4 showed the following, which I don’t understand at all:
Failed to get properties: Unit name qubes-vm@.service is neither a valid invocation ID nor unit name.
So I had a look at the state of the multi-user target, which is more or less the same in both versions of Qubes QS.
— multi-user.target - Multi-User System
Loaded: loaded (/usr/lib/systemd/system/multi-user.target; static)
Active: active since Sat 2025-08-23 11:17:06 CEST; 3min 30s ago
Invocation: 6eee5e194dba49ec837a2d654cdfe992
Docs: man:systemd.special(7)
Aug 23 11:17:06 dom0 systemd[1]: Reached target multi-user.target - Multi-User System.
— multi-user.target - Multi-User System
Loaded: loaded (/usr/lib/systemd/system/multi-user.target; static)
Active: active since Sat 2025-08-23 10:56:44 CEST; 10min ago
Docs: man:systemd.special(7)
Aug 23 10:56:44 dom0 systemd[1]: Reached target multi-user.target - Multi-User System.
Where can I look now???
So far, I have done my Tests with Xen 4.19-3 and Linux 6.15.7-1. Going forward to Linux 6.15.10-1 and backward to 61.12.42-1 did not change the behavior. Trying to go back to Xen 4.19.2 just crashed the boot process with “file not found” in Grub.
The configuration is in /etc/systemd/system/qubes-vm@.service.d/50_autostart.conf and in /etc/systemd/system/qubes-vm@sys-usb.service.d/50_autostart.conf but those are purely orderings, which shouldn’t cause other VM autostart units to fail if qubes-vm@sys-usb.service isn’t enabled or doesn’t start successfully.
I get the second message on R4.2.4 if I run systemctl status qubes-vm@.service rather than systemctl status qubes-vm@sys-net.service.
BTW qubes-vm@foo.service units are only ever enabled and started because the VM foo explicitly has autostart set to True. It’s normal that the units for other VMs that are started manually or as a dependency of foo remain inactive, as far as systemd is concerned.
Some more observations:
mgmt-salt-dom0-virtual-machines. Going back to the previous version 4.3.3-1 or forward to the newest version 4.3.8-1 does not change the situation.sys-usb service by changing the “After” entry in /etc/systemd/system/qubes-vm@.service.d/50_autostart.conf accordingly has no effect either.qubes-vm@sys-net.service manually via sudo systemctl startworks without error.untrusted, to autostart causes that VM and, in consequence, sys-net and sys-firewall, to start when booting Qubes.So, it’s just that the network does not start on its own at booting, unless it is required by some other autostarting VM to be available, and then it starts without error. If it does not start, there is no error either.
Oh okay, looks like for some reason enabling autostart for sys-net did not create the corresponding /etc/systemd/system/multi-user.target.wants/qubes-vm@sys-net.service symlink. This a new R4.3-rc1 installation then, not an upgrade from R4.2.x?
Can you try to untick autostart in the sys-net Qube Settings, apply the change, tick and apply again? That should change the systemctl status qubes-vm@sys-net.service output to ; enabled; in that line.
Same for sys-firewall.
sudo journalctl | grep -E -C5 '/etc/systemd/system/multi-user.target.wants/qubes-vm@sys-(net|firewall).service'
might still have a record of the error.
You found it!
After removing and recreating the autostart option for sys-net, this VM started at the next boot, but sys-firewall did not, as should be expected. Applying the same to sys-firewall lets this VM start at boot, too. The symlinks are now both present and point to /usr/lib/systemd/qubes-vm@.service.
The installation was an in-place upgrade from R4.2.4. There, both symlinks existed and were dated 18-MAY-2025. The symlinks in R4.3 are now dated as changed on 09-AUG-2025 02:00:00, but have the old creation date.
Here are the log files of journalctl:
sys-firewall.log (4.6 KB)
sys-net.log (4.5 KB)
Something seems to have happened at 31-JUL-2025. It could be that that was the date of performing the Qubes upgrade, but I am not sure.
Apparently the symlinks were successfully (re?)created back then. Strange that they disappeared in the meantime.
Wait, that sounds odd. Can you post the output of Never mind, 09-AUG-2025 02:00:00 must be the modification date of the symlink target in your timezone. So that part isn’t odd.stat /etc/systemd/system/multi-user.target.wants/qubes-vm@sys-{net,firewall}.service
I checked now the time of the upgrade to R4.3-rc1: 14-AUG-2025. So that was after the last change to the symlinks, which were in order before and after the upgrade. Something must have happened after that date to lose the symlinks, but nothing in the logs shows that.
I guess it is a riddle and remains so, and it is not worthwhile to put more effort into it.