[qubes-users] In-place upgrade of QubesOS

Hi!

As announced, here are more details of the problem I saw:

First I was surprised that some packages will be downgraded:

Downgrading:
device-mapper x86_64 1.02.199-4.fc41 qubes-dom0-cached 133 k
device-mapper-event x86_64 1.02.199-4.fc41 qubes-dom0-cached 31 k
device-mapper-event-libs x86_64 1.02.199-4.fc41 qubes-dom0-cached 28 k
device-mapper-libs x86_64 1.02.199-4.fc41 qubes-dom0-cached 174 k
lvm2 x86_64 2.03.25-4.fc41 qubes-dom0-cached 1.5 M
lvm2-libs x86_64 2.03.25-4.fc41 qubes-dom0-cached 996 k
python3-kickstart noarch 3.58-3.fc41 qubes-dom0-cached 492 k
qubes-desktop-linux-menu noarch 1.2.10-1.fc41 qubes-dom0-cached 120 k

Then there were some strange errors before reboot:

Running scriptlet: qubes-mgmt-salt-dom0-qvm-4.3.5-1.fc41.noarch 1264/2592
Invalid chars in property name: {'=', ' '}
...

Invalid chars in property name: {'=', ' '}

Running scriptlet: qubes-mgmt-salt-dom0-virtual-machines-4.3.13-1.fc41.noarch 1265/2592
Upgrading : qubes-mgmt-salt-dom0-virtual-machines-4.3.13-1.fc41.noarch 1265/2592
Running scriptlet: qubes-mgmt-salt-dom0-virtual-machines-4.3.13-1.fc41.noarch 1265/2592
Invalid chars in property name: {' ', '='}
...

During boot I saw these messages (Lenovo ThinkPad P16 Gen 2, i9-13980HX with Nvidia Ada 2000):

...

Dec 22 17:17:46 dom0 kernel: installing Xen timer for CPU 22
Dec 22 17:17:46 dom0 kernel: installing Xen timer for CPU 23
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 2: APIC ID mismatch. CPUID: 0x0002 APIC: 0x0008
Dec 22 17:17:46 dom0 kernel: cpu 2 spinlock event irq 201
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 4: APIC ID mismatch. CPUID: 0x0004 APIC: 0x0010
Dec 22 17:17:46 dom0 kernel: cpu 4 spinlock event irq 202
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 6: APIC ID mismatch. CPUID: 0x0006 APIC: 0x0018
Dec 22 17:17:46 dom0 kernel: cpu 6 spinlock event irq 203
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 8: APIC ID mismatch. CPUID: 0x0008 APIC: 0x0020
Dec 22 17:17:46 dom0 kernel: cpu 8 spinlock event irq 204
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 10: APIC ID mismatch. CPUID: 0x000a APIC: 0x0028
Dec 22 17:17:46 dom0 kernel: cpu 10 spinlock event irq 205
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 12: APIC ID mismatch. CPUID: 0x000c APIC: 0x0030
Dec 22 17:17:46 dom0 kernel: cpu 12 spinlock event irq 206
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 14: APIC ID mismatch. CPUID: 0x000e APIC: 0x0038
Dec 22 17:17:46 dom0 kernel: cpu 14 spinlock event irq 207
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 16: APIC ID mismatch. CPUID: 0x0010 APIC: 0x0040
Dec 22 17:17:46 dom0 kernel: cpu 16 spinlock event irq 208
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 17: APIC ID mismatch. CPUID: 0x0011 APIC: 0x0042
Dec 22 17:17:46 dom0 kernel: cpu 17 spinlock event irq 209
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 18: APIC ID mismatch. CPUID: 0x0012 APIC: 0x0044
Dec 22 17:17:46 dom0 kernel: cpu 18 spinlock event irq 210
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 19: APIC ID mismatch. CPUID: 0x0013 APIC: 0x0046
Dec 22 17:17:46 dom0 kernel: cpu 19 spinlock event irq 211
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 20: APIC ID mismatch. CPUID: 0x0014 APIC: 0x0048
Dec 22 17:17:46 dom0 kernel: cpu 20 spinlock event irq 212
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 21: APIC ID mismatch. CPUID: 0x0015 APIC: 0x004a
Dec 22 17:17:46 dom0 kernel: cpu 21 spinlock event irq 213
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 22: APIC ID mismatch. CPUID: 0x0016 APIC: 0x004c
Dec 22 17:17:46 dom0 kernel: cpu 22 spinlock event irq 214
Dec 22 17:17:46 dom0 kernel: [Firmware Bug]: CPU 23: APIC ID mismatch. CPUID: 0x0017 APIC: 0x004e
Dec 22 17:17:46 dom0 kernel: cpu 23 spinlock event irq 215
Dec 22 17:17:46 dom0 kernel: installing Xen timer for CPU 1
Dec 22 17:17:46 dom0 kernel: installing Xen timer for CPU 3
Dec 22 17:17:46 dom0 kernel: installing Xen timer for CPU 5
...

(some systemd warning about the LUKS device not being an absolute path)

...

Dec 22 17:17:47 dom0 kernel: pci 10000:e0:1d.4: bridge window [mem 0xae100000-0xae1fffff]
Dec 22 17:17:47 dom0 kernel: pcieport 10000:e0:06.0: can't derive routing for PCI INT D
Dec 22 17:17:47 dom0 kernel: pcieport 10000:e0:06.0: PCI INT D: no GSI
Dec 22 17:17:47 dom0 kernel: pcieport 10000:e0:06.0: PME: Signaling with IRQ 297
Dec 22 17:17:47 dom0 kernel: pcieport 10000:e0:06.0: AER: enabled with IRQ 297
Dec 22 17:17:47 dom0 kernel: pcieport 10000:e0:1d.4: can't derive routing for PCI INT A
Dec 22 17:17:47 dom0 kernel: pcieport 10000:e0:1d.4: PCI INT A: no GSI
Dec 22 17:17:47 dom0 kernel: pcieport 10000:e0:1d.4: PME: Signaling with IRQ 298
Dec 22 17:17:47 dom0 kernel: pcieport 10000:e0:1d.4: AER: enabled with IRQ 298
Dec 22 17:17:47 dom0 kernel: vmd 0000:00:0e.0: Bound to PCI domain 10000
...

Dec 22 17:17:48 dom0 kernel: usb 1-4: new high-speed USB device number 4 using xhci_hcd
Dec 22 17:17:48 dom0 kernel: tmpfs: Unsupported parameter 'huge'
Dec 22 17:17:48 dom0 kernel: i915 0000:00:02.0: [drm] Transparent Hugepage support is recommended for optimal performance on this platform!
...

Dec 22 17:17:48 dom0 kernel: pci 0000:01:00.0: optimus capabilities: enabled, status dynamic power, hda bios codec supported
Dec 22 17:17:48 dom0 kernel: VGA switcheroo: detected Optimus DSM method \_SB_.PC00.PEG1.PEGP handle
Dec 22 17:17:48 dom0 kernel: nouveau: detected PR support, will not use DSM
Dec 22 17:17:48 dom0 kernel: nouveau 0000:01:00.0: enabling device (0000 -> 0003)
Dec 22 17:17:48 dom0 kernel: xen: registering gsi 16 triggering 0 polarity 1
Dec 22 17:17:48 dom0 kernel: Already setup the GSI :16
Dec 22 17:17:48 dom0 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
Dec 22 17:17:48 dom0 kernel: sd 0:0:0:0: [sda] Spinning up disk...
Dec 22 17:17:48 dom0 kernel: nouveau 0000:01:00.0: NVIDIA AD107 (197000a1)
Dec 22 17:17:48 dom0 kernel: nouveau 0000:01:00.0: gsp ctor failed: -2
Dec 22 17:17:48 dom0 kernel: nouveau 0000:01:00.0: probe with driver nouveau failed with error -2

...

## why does Dom0 try to set up WiFi?

Dec 22 17:18:09 dom0 kernel: platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
Dec 22 17:18:09 dom0 kernel: cfg80211: failed to load regulatory.db
Dec 22 17:18:09 dom0 kernel: i2c i2c-11: Successfully instantiated SPD at 0x53
Dec 22 17:18:09 dom0 kernel: input: PC Speaker as /devices/platform/pcspkr/input/input16
Dec 22 17:18:09 dom0 kernel: Intel(R) Wireless WiFi driver for Linux
...

Now the problems after reboot (and after having fixed the WLAN issues):

root@dom0 ~]# qubes-dist-upgrade --releasever=4.3 --all-post-reboot
INFO: Please wait while running pre-checks...
WARNING: /!\ MAKE SURE YOU HAVE MADE A BACKUP OF ALL YOUR VMs AND dom0 DATA /!\
-> Launch upgrade process? [y/N] y
---> (STAGE 4) Upgrade templates and standalone VMs to R4.3 repository...
----> Upgrading debian-13-xfce...
+ [[ 0 -ne 0 ]]
+ enable_current_testing=
+ '[' '' = --enable-current-testing ']'
+ '[' -e /etc/fedora-release ']'
+ '[' -e /etc/debian_version ']'
++ awk -F. '{print $1}' /etc/debian_version
+ releasever=13
+ '[' 13 -lt 12 ']'
+ cp /etc/apt/sources.list.d/qubes-r4.list /etc/apt/sources.list.d/qubes-r4.list.bak
+ sed -i s/r4.2/r4.3/g /etc/apt/sources.list.d/qubes-r4.list
+ sed -i 's/\[arch=amd64\( signed-by=.*\)\?\]/[arch=amd64\ signed-by=\/usr\/share\/keyrings\/qubes-archive-keyring-4.3.gpg]/g' /etc/apt/sources.list.d/qubes-r4.list
+ export DEBIAN_FRONTEND=noninteractive
+ DEBIAN_FRONTEND=noninteractive
+ '[' '' ']'
+ apt-get clean
+ apt-get update -o Dpkg::Options::=--force-confdef
Ign:1 Index of /debian trixie InRelease
Ign:2 https://deb.debian.org/debian-security trixie-security InRelease
Get:3 Index of /r4.3/vm/ trixie InRelease [4,735 B]
Get:4 Index of /r4.3/vm/ trixie/main amd64 Packages [38.6 kB]
Ign:2 https://deb.debian.org/debian-security trixie-security InRelease
Ign:1 Index of /debian trixie InRelease
Ign:2 https://deb.debian.org/debian-security trixie-security InRelease
Err:2 https://deb.debian.org/debian-security trixie-security InRelease
SSL connection failed: error:0A000126:SSL routines::unexpected eof while reading / Success [IP: 127.0.0.1 8082]
Err:1 Index of /debian trixie InRelease
SSL connection failed: error:0A000126:SSL routines::unexpected eof while reading / Success [IP: 127.0.0.1 8082]
Fetched 43.4 kB in 8s (5,531 B/s)
Reading package lists...
E: Failed to fetch https://deb.debian.org/debian/dists/trixie/InRelease SSL connection failed: error:0A000126:SSL routines::unexpected eof while reading / Success [IP: 127.0.0.1 8082]
E: Failed to fetch https://deb.debian.org/debian-security/dists/trixie-security/InRelease SSL connection failed: error:0A000126:SSL routines::unexpected eof while reading / Success [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.
ERROR: A general error occurred while upgrading debian-13-xfce (exit code 100).
-> Continue? [y/N]
REVERTING template to pre-ugrade state
[root@dom0 ~]#

A similar problem occurs for Fedora.

FWIW, I successfully performed an ‘in-place upgrade’ on my system this afternoon …

I performed the upgrade in two steps, using the ‘–all-pre-reboot’ & ‘–all-post-reboot’ option.

The only visible difference from Ulrich’s report is that I did the upgrade as ‘[vr@dom0 ~ ]’ …

The upgrade succeeded with DL13, FL42 & WN17 Template & App VMs successfully being upgraded !

Having retried it just a moment ago, it worked!
However what I saw was:

---> (STAGE 5) Updating PCI device IDs
Storing device id for 00_14.3 (sys-net)
---> (STAGE 5) Enabling minimal-netvm / minimal-usbvm services
Enabling minimal-netvm for sys-net
---> (STAGE 5) Cleaning up salt
Error on ext_pillar interface qvm_prefs is expected
local:
True
[CRITICAL] Specified ext_pillar interface qvm_features is unavailable
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
[CRITICAL] Specified ext_pillar interface qvm_tags is unavailable
local: