Qubes OS broken after update

[QubesParanoia]

Yesterday I updated dom0, namely dom0 only. It seems to have updated the kernel, Qubes-dom0-core or something and kernel. But after restarting the OS, even on the old kernel version, most cubes are broken and do not start, this applies especially to those based on fedora41. Since a couple of Debian-based appvm cubes continue to work. But even template, standalonevm, many do not start. The same Whonix-gateway template starts fine, but the same error in any appvm, even created from scratch specifically for testing from template. How, why… I do not know. But now neither sys-usb, nor sys-net, nor sys-firewall can be started, they all have the same error. I tried to change the kernel version in qube, I tried to increase and decrease RAM, but everything is useless. Affected all modes (pvh/hvm)

dom0 qubesd[2756]: vm.Game: Start failed: внутренняя ошибка: libxenlight не удалось создать домен «Game»
апр 02 21:57:20 dom0 qubesd[2756]: unhandled exception while calling src=b’dom0’ meth=b’admin.vm.Start’ dest=b’Game’ arg=b’’ len(untrusted_payload)=0
апр 02 21:57:20 dom0 qubesd[2756]: Traceback (most recent call last):
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib/python3.11/site-packages/qubes/vm/qubesvm.py”, line 1205, in start
апр 02 21:57:20 dom0 qubesd[2756]: self.libvirt_domain.createWithFlags(
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib/python3.11/site-packages/qubes/app.py”, line 107, in wrapper
апр 02 21:57:20 dom0 qubesd[2756]: return getattr(self._vm, attrname)(*args, **kwargs)
апр 02 21:57:20 dom0 qubesd[2756]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib64/python3.11/site-packages/libvirt.py”, line 1409, in createWithFlags
апр 02 21:57:20 dom0 qubesd[2756]: raise libvirtError(‘virDomainCreateWithFlags() failed’)
апр 02 21:57:20 dom0 qubesd[2756]: libvirt.libvirtError: внутренняя ошибка: libxenlight не удалось создать домен «Game»
апр 02 21:57:20 dom0 qubesd[2756]: During handling of the above exception, another exception occurred:
апр 02 21:57:20 dom0 qubesd[2756]: Traceback (most recent call last):
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib/python3.11/site-packages/qubes/api/init.py”, line 297, in respond
апр 02 21:57:20 dom0 qubesd[2756]: response = await self.mgmt.execute(
апр 02 21:57:20 dom0 qubesd[2756]: ^^^^^^^^^^^^^^^^^^^^^^^^
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib/python3.11/site-packages/qubes/api/admin.py”, line 864, in vm_start
апр 02 21:57:20 dom0 qubesd[2756]: await self.dest.start()
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib/python3.11/site-packages/qubes/vm/qubesvm.py”, line 1222, in start
апр 02 21:57:20 dom0 qubesd[2756]: await self.fire_event_async(‘domain-start-failed’,
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib/python3.11/site-packages/qubes/events.py”, line 227, in fire_event_async
апр 02 21:57:20 dom0 qubesd[2756]: sync_effects, async_effects = self._fire_event(event,
апр 02 21:57:20 dom0 qubesd[2756]: ^^^^^^^^^^^^^^^^^^^^^^^
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib/python3.11/site-packages/qubes/events.py”, line 164, in _fire_event
апр 02 21:57:20 dom0 qubesd[2756]: effect = func(self, event, **kwargs)
апр 02 21:57:20 dom0 qubesd[2756]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib/python3.11/site-packages/qubes/api/admin.py”, line 70, in vm_handler
апр 02 21:57:20 dom0 qubesd[2756]: self.send_event(subject, event, **kwargs)
апр 02 21:57:20 dom0 qubesd[2756]: File “/usr/lib/python3.11/site-packages/qubes/api/init.py”, line 374, in send_event
апр 02 21:57:20 dom0 qubesd[2756]: self.transport.write(‘{}\0{}\0’.format(k, str(v)).encode(‘ascii’))
апр 02 21:57:20 dom0 qubesd[2756]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
апр 02 21:57:20 dom0 qubesd[2756]: UnicodeEncodeError: ‘ascii’ codec can’t encode characters in position 7-16: ordinal not in range(128)

[corny]

Can you switch back to fedora 40 ? Or try switching all the service qubes to debian ?

[QubesParanoia]

I’m afraid it won’t work to return to fedora40 for all templates, most likely, since all these appvm and standalone were created on fedora41 initially.

I tried to transfer some of the cubes to Debian as a template (change the appvm settings), but it didn’t help at all. It seems like it broke the vm-pool for all appvm and some of the templates? Because I don’t understand why, but Whonix Qubes also broke, although they have nothing to do with fedora. I really hope that this is just some kind of error in dom0, and not a serious breakdown of all data storage for all VMs

[corny]

You should be able to move the service qubes back to fedora 40 with no issues. Probably the appvms too, not the standalones obviously.

If it was me, I’d want the service apps to work as the first step. Otherwise how are you going to get internet access to update or revert the erroneous update that has messed things up

If an update created this issue then I would redo the update or revert to previous one that was working. You could try loading an older kernel in grub to test if that fixes any issues

Look at some of the later cli commands in this thread as a way to fix the dom0 update issues

Can't use the latest kernel version after interrupted dom0 update

Hello everyone, when my dom0 was updating, I had to power off my computer, and therefore its update (update was about kernel and its microcode) was interrupted. And after I rebooted my system, then I wasn’t able to use latest kernel version for any vm. I had to set my default kernel to a lower kernel version (Qube settings → Advanced → Kernel) to be able to open my qubes. Now, if I try to open any qube with latest kernel version, I get “Cannot Cannot to Qrexec Agent for 60 Seconds” error and re…

[QubesParanoia]

But after restarting the OS, even on the old kernel version, most cubes are broken and do not start, this applies especially to those based on fedora41

I don’t understand what it means to move back. They were never in fedora40, from the very beginning after installing Qubes OS they were in fedora41. I understand that there are topics on the forum where they updated 4.1->4.2 and got the same error. And I read about the kernel. I deleted this version of the kernel via rpm without any problems and returned the old one, but it didn’t help at all. At the same time, many of my cubes contained important data, programs and I am scared by the thought that I have to delete everything now and re-download the template and create everything from them again.

[QubesParanoia]

Although I don’t understand how it happened, but reinstalling dom0 via dnf qubes-core-dom0 qubes-core-dom0-linux and qubes-core-dom0-Linux-install mostly solved the problem. Now all cubes can be launched, although for launching it is necessary to install the kernel from dom0. That is, provided by qube leads to the same error