[qubes-users] (Template)upgrade Debian r4.0 to r4.1 breaks (on r4.1 DOM0)

Hello everyone,

I "upgraded" my Qubes machine from r4.0 to r4.1 via fresh install, and am seeing difficulties now.
My system is on a btrfs filesystem, with a subvolume for the template/guest VMs. I did a fresh DOM0
install with formatting / , but kept the complete /var/lib/qubes folder with the templates and guest VMs,
and mounted it in-place while in system setup.
The system then booted up without problems, and listed and started the old VMs too.

1) (fixed)
My first problem was that (DOM0- and template?) updates failed. For people stumbling upon this:

cannot open /etc/qubes-rpc/qubes.TemplateSearch: No such file
ERROR: qrexec call 'qubes.TemplateSearch' failed.

This can be fixed with using a "fresh", r4.1 VM as the DOM0 update VM in Qube-Managers 'global settings'.
Then follow [1] to manually remove old templates and reinstall them via 'qvm-template-gui', for example.

2) (template upgrades fail)
However, I would prefer to upgrade the templates to r4.1 instead of removing and recreating them, as I make
use of quite a few templates.
The root cause is that the old templates still have r4.0 in their sources, in both Fedora and Debian.
Debian only has generic "bullseye" references in sources.list, which seems fine. In

/etc/apt/sources.list.d/qubes-r4.list

only one line is active,

deb [arch=amd64] Index of /r4.0/vm/ bullseye main

I edited this line from "r4.0" to "r4.1", and reloaded the package lists.

Then, hoping to upgrade the template:

apt dist-upgrade

quits with an error:

Unpacking qubes-vm-dependencies (4.1.20-1+deb11u1) over >(4.0.12-1+deb11u1) ...
Errors were encountered while processing:
/tmp/apt-dpkg-install-YMgGgX/006-xen-utils-guest_4.14.5-10+deb11u1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

It does not seem to be possible to fix this state with

apt --fix-broken install

If instead upgrading with

apt-get upgrade

it finishes without errors, but with fewer packages actually updated.

In both cases (after apt or apt-get) however the template is not able to boot up any more:

"cannot connect to qrexec agent for 60 seconds"

the mentioned logfile

/var/log/xen/console/<qube>.log

shows no errors.

I found a few mentions on the forums that this behavior did show up before, but as far as I can see those
cases were always solved with recreating fresh templates.
I believe those problems would show also when backing up the VMs and restoring those then outdated VMs
into a new (r4.1) DOM0.

Is there any fix for this state? I have working r4.0 templates now, none is edited for r4.1 repositories
and borked yet. Cloning VMs, borking them and deleting them within minutes really helps here!
Sorry if this is out-of-scope for Qubes and more of a general dependencyhell topic. In that case, maybe
the hints on how to get DOM0 to update again in the first place helps someone.

Thank you in advance,

Stickstoff

[1] Qube troubleshooting | Qubes OS