Qubes-dist-upgrade 4.2 -> 4.3: qubesd and dom0 borked after failed dist-upgrade and reboot

User Support
,
1

Having a long painful journey with trying to belatedly in-place upgrade from EoL 4.2 to 4.3,qubes-dist-upgrade still appears broken…

Following Qubes 4.3 upgrade docs and doing each stage separately.

First I will note the snags I hit and steps I took to proceed, ending with where I am currently stuck. Unfortunately I don’t have shell history retained so some steps are from memory, missing and incomplete. Some other logs and data are retained and can be provided on request. System is currently in broken state.

Pre-upgrade state

  • Qubes R4.2 - dom0 and templates updated to latest
  • updatevm: sys-whonix
    • whonix-gateway-18
  • netvm: disp-sys-net
    • connecting over wpa-psk wifi so requires manual reconnection on each start
    • normally, all internet access is done via sys-whonix which connects via bridge on local network. netvm and sys-firewall do not have direct internet access.
  • qvm-pool storage: zfs
  • dom0 /: LUKS + lvm + ext4
    • /var/lib/qubes mounted as ZFS dataset (same zpool as qvm-pool dataset)
    • 40+GiB free on / so I think lacking storage space shouldn’t be an issue.

Repository snags 1: Disabling use of private repo mirrors

I would normally not use sys-whonix as updatevm, but instead a janky combination of bengrande/qusal sys-cacher (actual updatevm) and redirecting to repo mirrors hosted on local network via sys-firewall. This caused issues with stage 3 of dist-upgrade as the yum mirror configuration would be overwritten with incorrect values on each invocation

Eventually I just gave up and made everything use sys-whonix which made things progress over tor and I could proceed downloading packages for stage 3.

Repository snags 2: Inconsistent updates

  • The script updated my fedora-updates.repo to f41 but left fedora.conf intact which resulted in it initially trying to do a “mixed upgrade” with 37+41. I had to manually go edit fedora.conf to fetch the new repos.

Stage 3 package installation issues

I hit the exact same issue(s) reported here: Qubes-dist-upgrade script error 4.3.rc-2 - #35 by Tee :

Problem: The operation would result in removing the following protected packages: dnf followed by the same and other conflict errors as in that thread.

So the mentioned patch does not seem to solve that issue.

Repository snags 3: switching updatevm template

As noted in thread above, at least some of the errors above were related to use of sys-whonix as updatevm. Switching to sys-firewall, I had to configure that to use a proxy on the local network to get access to internet (as noted above was unable to figure out how to make it actually fetch from my local mirror instead). Here I hit a snag with not being able to figure out a proper way to set proxy= in dnf.conf in updatevm - it was being reset on every call with contents of dom0 /etc/dnf.conf and putting it there would make the subsequent step fail instead due to incorrect use of proxy. I ended up overcoming this with some ugly hack of config generation in python script.

Failing on kernel removal seems to cause inconsistent state

Still during stage 3 I got similar/same issue as reported here: Error during 4.3 upgrade

After manually uninstalling some problematic packages causing conflicts (breeze-icons I recall) I was able to get a dnf --releasever=41 distro-sync which proceeded with apparently full upgrade.

Then towards the end I hit this: 4.3 upgrade failed during the Stage 3 dnf distro-sync due to old kernel-qubes-vm - #2 by stage3help

And moved forward similarly to stage3help.

Reboot woopsies

I had to to a hard reboot once or twice because I inadvertently had the script kill my USB VM (disabling keyboard) and netvm (which became inoperable even after restart, as keyboard input required to do network auth after netvm start). As power-button-press only caused the XFCE logout prompt to show, the only way out was powercycling. I only did so after seeing it idling for an extended time so I don’t believe these reboots would have contributed to inconsistencies etc.

Remaining inconsistencies

After reboot, I noticed I still somehow had a bunch of fc37 packages installed.

  • Removed some that aren’t necessary and probably installed manually at some point
  • Replaced some by pulling rpm files manually from Qubes 4.3.1 ISO and Fedora repos.
    • (I think I) had to explicitly remove libproxy-update to allow upgrading duktapeand polkit.
  • I still have these fc37 packages installed: qubes-desktop-linux-menu, nim-srpm-macros.

At this point, qubesd.service fails to start with No such file or directory being raised by qubesdb.QubesDB(). So no qubes. I am stuck here.

libvirt also failed to start due to a missing virtlogd.sock. This file should be provided by libvirt-daemon-log package, which for some reason was missing. I installed that rpm manually from Qubes 4.3.1 iso and after that libvirtd.service starts successfully.

I also note that while all dom0 qubes- packages are now on f41, it seems to have installed a mix of 4.2 and 4.3 versions. These packages are currently 4.2 versions (rest on 4.3) and match the rpms shipped in 4.3.1 iso:

qubes-dist-upgrade             qubes-dom0-cached
qubes-libvchan-xen             <unknown>
qubes-mgmt-salt                <unknown>
qubes-mgmt-salt-admin-tools    <unknown>
qubes-mgmt-salt-config         <unknown>
qubes-mgmt-salt-dom0           <unknown>
xfce4-settings-qubes           <unknown>

The corresponding python modules are populating /usr/lib/python3.11/site-packages instead of /usr/lib/python3.13/site-packages:

# ls /usr/lib/python3.11/site-packages
jaraco/
qubes_config/
qubesmanager/
qubes_menu/
qubes_menu-0.1-py3.11.egg-info/
qubes_menu_settings/
qubessalt/
qui/
salt/
vmupdate/

I guess that at least some of those need upgrade to 4.3 but unrelated to my error?

qubesd missing qubesdb?

At this point I am stuck with the failing qubesd and not sure exactly what it’s missing (as it’s calling directly into .pyc.so)…

File "/usr/lib/python3.13/site-packages/qubes/vm/__init.py", line 632, in start_qdb_watch
  self._qdb_connection_watch = qubesdb.QubesDB(self.name)
                               ~~~~~~~~~~~~~~~^^^^^^^^^^^
qubesdb.Error: (2, 'No such file or directory')

The zfs dataset used for qvm-pool is available and unlocked, and /var/lib/qubes is mounted.

2

No real progress and still in the dark but some minor additions:

  • I verified that installed qubes- packages still on 4.2 versions actually match those shipped in 4.3.1 iso so I guess those are in order.

  • I dnf reinstalled them anyway in order to rule out package postinstall hooks/triggers. They installed without error and behaviour is still the same.

3

Maybe my case could help you

Please note that I ran each stage manually, both pre-boot and post-boot, and those commands were very long since many qubes are involved in that for different reasons.

Maybe you should try the same again, while stating here commands you applied.

4

@corporateblush I see we had some of the same notes there. As noted in post, I did also run into the missing virtlog.sock initially and worked around it in similar way you did. The No such file or directory I see now doesn’t look to be related to missing systemd unit but I can’t say that for certain… Your post may be helpful for others but unfortunately seems like something is different here. :confused:

Do you remember if you had something like that also?

As qubesd can’t start, I’m not yet at the point were I can interact with Qubes/templates at all.

5

So just as I was going to take a closer look at qubesd issue by sideloading strace and gdb, it started (apparently randomly) working after another reboot (not the first). I don’t think I actually did anything to “fix” that last error besides rebooting multiple times after addressing the already mentioned.

My qubes are now running with 4.3 and fc41 dom0 :smiling_face: