Recovery for failed 4.3 upgrade on step 3

User Support General
1

I forgot to reboot my machine before manually running the upgrade on step 3.

It failed on the removal of the qubes-vm package

Then I made my second mistake I rebooted because I could not get a network connection (my Ethernet controller did not properly get attached)

So now I am stuck with a qubes install that’s kinda bricked. While dom0 starts all qubes daemons fail to do so (qubesd…).
I did make a backup right before the upgrade but sadly not of everything so I would really like to at least get some VMs restored.

My only choice is probably to somehow manually move the VMs to another install or manually run some of the post install scripts.
Could somebody pls help me RIP.

I have access to another qubes install that I successfully upgraded to 4.3

1 Like
2

Somehow getting qubesd to start again would probably be enough to make a quick backup of everything I need

I would also be fine if somebody could tell me how I can access the files in the home volume of the VMs when I manually mount the whole disk on another machine.

3

I can’t help you but I can push you towards the right direction. On Arch Wiki there is a page about Logical Volume Manager (LVM). Which is what Qubes set up for me. You should be able to use ArchISO and external hard drive to mount the Qubes partitions and move them onto external hard drive. From there you can manually extract important data from them

4

Same issue here, journalctl tells me it is some issue with virtlogd.socket. I will investigate further and if I find a fix, i’ll let you know.

Any help from other readers is welcome.

You should be able to recover any files with these instructions: Rescue Re-Install the EFI and Boot Grub initramfs Partitions

5

Another idea, from someone who doesn’t really know…

Did you try some things like (guessing…) systemctl status qubesd or systemctl restart qubesd ?

Maybe there is even a more low level command for launching the daemon?

6

@phceac

qubesd.service

Dez 27 09:18:01 dom0 systemd[1]: Starting qubesd.service - Qubes OS daemon...
Dez 27 09:18:03 dom0 qubesd[4463]: libvirt: XML-RPC error : Cannot recv data: Connection reset by peer
Dez 27 09:18:03 dom0 qubesd[4463]: ERROR: Task was destroyed but it is pending!
Dez 27 09:18:03 dom0 qubesd[4463]: task: <Task pending name='Task-1' coro=<virEventAsyncIOImpl._ff_callback() running at /usr/lib64/python3.13/site-packages/libvirtaio.py:329>>
Dez 27 09:18:03 dom0 qubesd[4463]: /usr/lib64/python3.13/asyncio/base_events.py:750: RuntimeWarning: coroutine 'virEventAsyncIOImpl._ff_callback' was never awaited
Dez 27 09:18:03 dom0 qubesd[4463]:   self._ready.clear()
Dez 27 09:18:03 dom0 qubesd[4463]: RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Dez 27 09:18:03 dom0 qubesd[4463]: Traceback (most recent call last):
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/bin/qubesd", line 5, in <module>
Dez 27 09:18:03 dom0 qubesd[4463]:     sys.exit(main())
Dez 27 09:18:03 dom0 qubesd[4463]:              ~~~~^^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/tools/qubesd.py", line 46, in main
Dez 27 09:18:03 dom0 qubesd[4463]:     args = parser.parse_args(args)
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/tools/__init__.py", line 426, in parse_args
Dez 27 09:18:03 dom0 qubesd[4463]:     namespace.app = qubes.Qubes(
Dez 27 09:18:03 dom0 qubesd[4463]:                     ~~~~~~~~~~~^
Dez 27 09:18:03 dom0 qubesd[4463]:         namespace.app, offline_mode=namespace.offline_mode
Dez 27 09:18:03 dom0 qubesd[4463]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dez 27 09:18:03 dom0 qubesd[4463]:     )
Dez 27 09:18:03 dom0 qubesd[4463]:     ^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/app.py", line 1040, in __init__
Dez 27 09:18:03 dom0 qubesd[4463]:     self.load(lock=lock)
Dez 27 09:18:03 dom0 qubesd[4463]:     ~~~~~~~~~^^^^^^^^^^^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/app.py", line 1164, in load
Dez 27 09:18:03 dom0 qubesd[4463]:     vm.fire_event("domain-load")
Dez 27 09:18:03 dom0 qubesd[4463]:     ~~~~~~~~~~~~~^^^^^^^^^^^^^^^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/events.py", line 200, in fire_event
Dez 27 09:18:03 dom0 qubesd[4463]:     sync_effects, async_effects = self._fire_event(
Dez 27 09:18:03 dom0 qubesd[4463]:                                   ~~~~~~~~~~~~~~~~^
Dez 27 09:18:03 dom0 qubesd[4463]:         event, kwargs, pre_event=pre_event
Dez 27 09:18:03 dom0 qubesd[4463]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dez 27 09:18:03 dom0 qubesd[4463]:     )
Dez 27 09:18:03 dom0 qubesd[4463]:     ^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/events.py", line 169, in _fire_event
Dez 27 09:18:03 dom0 qubesd[4463]:     effect = func(self, event, **kwargs)
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/ext/block.py", line 327, in on_domain_init_load
Dez 27 09:18:03 dom0 qubesd[4463]:     device_attachments = self.get_device_attachments(vm)
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/ext/block.py", line 357, in get_device_attachments
Dez 27 09:18:03 dom0 qubesd[4463]:     xml_desc = lxml.etree.fromstring(vm.libvirt_domain.XMLDesc())
Dez 27 09:18:03 dom0 qubesd[4463]:                                      ^^^^^^^^^^^^^^^^^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/vm/adminvm.py", line 147, in libvirt_domain
Dez 27 09:18:03 dom0 qubesd[4463]:     self._libvirt_domain = self.app.vmm.libvirt_conn.lookupByID(0)
Dez 27 09:18:03 dom0 qubesd[4463]:                            ^^^^^^^^^^^^^^^^^^^^^^^^^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/app.py", line 219, in libvirt_conn
Dez 27 09:18:03 dom0 qubesd[4463]:     self.init_vmm_connection()
Dez 27 09:18:03 dom0 qubesd[4463]:     ~~~~~~~~~~~~~~~~~~~~~~~~^^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/app.py", line 210, in init_vmm_connection
Dez 27 09:18:03 dom0 qubesd[4463]:     self._libvirt_conn = VirConnectWrapper(
Dez 27 09:18:03 dom0 qubesd[4463]:                          ~~~~~~~~~~~~~~~~~^
Dez 27 09:18:03 dom0 qubesd[4463]:         qubes.config.defaults["libvirt_uri"],
Dez 27 09:18:03 dom0 qubesd[4463]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dez 27 09:18:03 dom0 qubesd[4463]:         reconnect_cb=self._libvirt_reconnect_cb,
Dez 27 09:18:03 dom0 qubesd[4463]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dez 27 09:18:03 dom0 qubesd[4463]:     )
Dez 27 09:18:03 dom0 qubesd[4463]:     ^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib/python3.13/site-packages/qubes/app.py", line 122, in __init__
Dez 27 09:18:03 dom0 qubesd[4463]:     self._conn = libvirt.open(uri)
Dez 27 09:18:03 dom0 qubesd[4463]:                  ~~~~~~~~~~~~^^^^^
Dez 27 09:18:03 dom0 qubesd[4463]:   File "/usr/lib64/python3.13/site-packages/libvirt.py", line 335, in open
Dez 27 09:18:03 dom0 qubesd[4463]:     raise libvirtError('virConnectOpen() failed')
Dez 27 09:18:03 dom0 qubesd[4463]: libvirt.libvirtError: Cannot recv data: Connection reset by peer
Dez 27 09:18:03 dom0 systemd[1]: qubesd.service: Main process exited, code=exited, status=1/FAILURE
Dez 27 09:18:03 dom0 systemd[1]: qubesd.service: Failed with result 'exit-code'.
Dez 27 09:18:03 dom0 systemd[1]: Failed to start qubesd.service - Qubes OS daemon.
Dez 27 09:18:04 dom0 systemd[1]: qubesd.service: Scheduled restart job, restart counter is at 1.

libvirtd.service

○ libvirtd.service - libvirt legacy monolithic daemon
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: inactive (dead)
TriggeredBy: ○ libvirtd-ro.socket
             ○ libvirtd-admin.socket
             × libvirtd.socket
       Docs: man:libvirtd(8)
             https://libvirt.org/

libvirtd.socket

× libvirtd.socket - libvirt legacy monolithic daemon socket
     Loaded: loaded (/usr/lib/systemd/system/libvirtd.socket; enabled; preset: disabled)
     Active: failed (Result: resources) since Sat 2025-12-27 09:18:03 CET; 5min ago
   Duration: 2.493s
 Invocation: 3d8d69693ca44814ba5868a0059506f0
   Triggers: ● libvirtd.service
     Listen: /run/libvirt/libvirt-sock (Stream)

Dez 27 09:18:01 dom0 systemd[1]: Listening on libvirtd.socket - libvirt legacy monolithic daemon socket.
Dez 27 09:18:03 dom0 systemd[1]: libvirtd.socket: Failed to queue service startup job: Unit virtlogd.socket not found.
Dez 27 09:18:03 do

I can’t start virtlogd.service or .socket, just found this, sounds similar:

○ virtlockd.service - libvirt locking daemon
     Loaded: loaded (/usr/lib/systemd/system/virtlockd.service; disabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: inactive (dead)
TriggeredBy: ● virtlockd.socket
             ● virtlockd-admin.socket
       Docs: man:virtlockd(8)
             https://libvirt.org/
7

Accessing the data from another system is no issue at all.

I though I would have to deal with some qubes magic but it’s just normal LVM.

8

After trying to work round a different upgrade problem, I think that it is probably not useful to try to debug.

I think that the implied advice of @tokaso80 and @Mirai, and the upgrade tool, is useful:

Make sure to have effective back up copies of all your data, before trying to upgrade. Use a live boot system to make one if you ignored the warnings.

9

Another user ran into a similar issue with qubesd and libvirt and they continued from the middle of Stage 3 by manually editing the upgrade script: