Can't use the latest kernel version after interrupted dom0 update

Hello everyone, when my dom0 was updating, I had to power off my computer, and therefore its update (update was about kernel and its microcode) was interrupted. And after I rebooted my system, then I wasn’t able to use latest kernel version for any vm. I had to set my default kernel to a lower kernel version (Qube settings → Advanced → Kernel) to be able to open my qubes.

Now, if I try to open any qube with latest kernel version, I get “Cannot Cannot to Qrexec Agent for 60 Seconds” error and related log file ends with “Waiting for /dev/xvdd device” line.

But before (at first reboots after dom0 update interruption) I was having issue about libxl. Maybe now it gives another error (qrexec agent error), since I’ve updated dom0 a few times after it.

Is there a way to verify the integrity of kernel, or maybe a chance to reinstall it like it is done in dom0 updates?

Have you tried telling it to reinstall ?

sudo qubes-dom0-update kernel-6.6.77-1.qubes.fc37

1 Like

I tried it now, and it successfully reinstalled the kernel, but still having the same issue. But this time I’m getting qrexec agent error three times like it runs it and then fails again .

Does this mean that when you start your computer, none of the autostart qubes correctly run ?

Perhaps your power failure happened during the fedora update ? If you had really messed up dom0 update I’m doubtful the grub update would have been run and thus new kernel wouldn’t work at all at boot

Try changing to older default kernel and then running fedora template update

I don’t have any autostart qubes but when I try to run any qube with latest kernel version manually, then I receive the error. Power failure happened during dom0 update, and I’m not sure if fedora update was involved (but if used kernel is from fedora, then it should be effected). And I’m already running my qubes with an older kernel version right now.

Note: I didn’t install fedora templates when setting up my Qubes OS, and only have debian and whonix templates.

Perhaps your issue is with the 2nd kernel ? You could rerun that part ?

The package(s) you would betterreinstall and verify are:

kernel-lates-qubes-vm and kernel-qubes-vm. And it would be necessary to look at /var/lib/qubes/vm-kernels. There might be some left over there.

1 Like

Thanks for your reply,

My /var/lib/qubes/vm-kernels file included:

6.6.65-1.fc37/ 6.6.68-1.fc37/ 6.6.74-1.fc37/ 6.6.77-1.fc37/ 

and /var/lib/qubes/vm-kernels/6.6.77-1.fc37/ included:

default-kernelopts-common.txt  memory-hotplug-supported
initramfs                      vmlinuz

Then I proceeded with these steps:

I’ve tried to reinstall kernel-qubes-vm

$ sudo qubes-dom0-update --action=reinstall kernel-qubes-vm
Using sys-whonix as UpdateVM for Dom0
Downloading packages. This may take a while...
Unable to detect release version (use '--releasever' to specify release version)
Unable to detect release version (use '--releasever' to specify release version)
Unable to detect release version (use '--releasever' to specify release version)
Fedora 37 - x86_64                              237 kB/s |  70 MB     05:02    
Fedora 37 - x86_64 - Updates                    225 kB/s |  43 MB     03:17    
Qubes Host Repository (updates)                 159 kB/s | 4.2 MB     00:27    
Last metadata expiration check: 0:00:18 ago on Fri Feb 28 16:27:13 2025.
Error: 
 Problem 1: conflicting requests
 Problem 2: package kernel-qubes-vm-1000:6.6.68-1.qubes.fc37.x86_64 requires kernel-modules = 6.6.68-1.qubes.fc37.x86_64, but none of the providers can be installed
  - conflicting requests
 Problem 3: package kernel-qubes-vm-1000:6.6.68-1.qubes.fc37.x86_64 requires kernel-devel = 6.6.68-1.qubes.fc37.x86_64, but none of the providers can be installed
  - conflicting requests
(try to add '--skip-broken' to skip uninstallable packages)

Then I’ve tried to do it with –skip-broken

$ sudo qubes-dom0-update --skip-broken --action=reinstall kernel-qubes-vm
Using sys-whonix as UpdateVM for Dom0
Downloading packages. This may take a while...
Unable to detect release version (use '--releasever' to specify release version)
Unable to detect release version (use '--releasever' to specify release version)
Unable to detect release version (use '--releasever' to specify release version)
Fedora 37 - x86_64                              4.2 kB/s | 6.0 kB     00:01    
Fedora 37 - x86_64 - Updates                    1.8 kB/s | 5.9 kB     00:03    
Qubes Host Repository (updates)                 647  B/s | 2.7 kB     00:04    
Error: 
 Problem 1: conflicting requests
 Problem 2: package kernel-qubes-vm-1000:6.6.68-1.qubes.fc37.x86_64 requires kernel-modules = 6.6.68-1.qubes.fc37.x86_64, but none of the providers can be installed
  - conflicting requests
 Problem 3: package kernel-qubes-vm-1000:6.6.68-1.qubes.fc37.x86_64 requires kernel-devel = 6.6.68-1.qubes.fc37.x86_64, but none of the providers can be installed
  - conflicting requests

Then I’ve tried to install kernel-latest-qubes-vm, since I didn’t have it before:

$ sudo qubes-dom0-update kernel-latest-qubes-vm
Using sys-whonix as UpdateVM for Dom0
Downloading packages. This may take a while...
Unable to detect release version (use '--releasever' to specify release version)
Unable to detect release version (use '--releasever' to specify release version)
Unable to detect release version (use '--releasever' to specify release version)
Fedora 37 - x86_64                              1.5 kB/s | 6.0 kB     00:03    
Fedora 37 - x86_64 - Updates                    1.6 kB/s | 5.9 kB     00:03    
Qubes Host Repository (updates)                 795  B/s | 2.7 kB     00:03    
Dependencies resolved.
================================================================================
 Package              Arch   Version                   Repository          Size
================================================================================
Installing:
 kernel-latest-qubes-vm
                      x86_64 1000:6.12.11-1.qubes.fc37 qubes-dom0-current  19 M
Installing dependencies:
 kernel-latest-devel  x86_64 1000:6.12.11-1.qubes.fc37 qubes-dom0-current  15 M
 kernel-latest-modules
                      x86_64 1000:6.12.11-1.qubes.fc37 qubes-dom0-current  84 M
Removing:
 kernel-qubes-vm      x86_64 1000:6.6.65-1.qubes.fc37  @System            518 M

Transaction Summary
================================================================================
Install  3 Packages
Remove   1 Package

Total download size: 118 M
DNF will only download packages for the transaction.
Downloading Packages:
(1/3): kernel-latest-devel-6.12.11-1.qubes.fc37  29 kB/s |  15 MB     08:40    
(2/3): kernel-latest-qubes-vm-6.12.11-1.qubes.f  33 kB/s |  19 MB     10:06    
(3/3): kernel-latest-modules-6.12.11-1.qubes.fc  70 kB/s |  84 MB     20:28    
--------------------------------------------------------------------------------
Total                                            98 kB/s | 118 MB     20:32     
Complete!
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Qubes OS Repository for Dom0                                                                            2.9 MB/s | 3.0 kB     00:00    
Qubes OS Repository for Dom0                                                                             10 MB/s | 198 kB     00:00    
Dependencies resolved.
========================================================================================================================================
 Package                             Architecture        Version                                   Repository                      Size
========================================================================================================================================
Installing:
 kernel-latest-qubes-vm              x86_64              1000:6.12.11-1.qubes.fc37                 qubes-dom0-cached               19 M
Installing dependencies:
 kernel-latest-devel                 x86_64              1000:6.12.11-1.qubes.fc37                 qubes-dom0-cached               15 M
 kernel-latest-modules               x86_64              1000:6.12.11-1.qubes.fc37                 qubes-dom0-cached               84 M
Removing:
 kernel-qubes-vm                     x86_64              1000:6.6.65-1.qubes.fc37                  @@System                       518 M

Transaction Summary
========================================================================================================================================
Install  3 Packages
Remove   1 Package

Total size: 118 M
Is this ok [y/N]: y

I guess it installed the kernel 6.12.11-1.qubes.fc37.x86_64, and removed kernel-qubes-vm-1000:6.6.65-1.qubes.fc37.x86_64, and now the freshly installed kernel (6.12.11-1) works well.

But as a result I’m confused with some problems and questions:

  • The kernel I was having issue (kernel-6.6.77-1.qubes.fc37) is still not working on AppVMs.
  • If there was higher kernel versions like kernel 6.12.11-1.qubes.fc37.x86_64, then why my dom0 updates didn’t fetch them and why was I using old kernels like 6.6.68 that were offered to my by default?
  • In this operation, kernel-qubes-vm is removed and kernel-latest-qubes-vm, but you said I have to verify/install these packages. If I can’t have them both, then which one should I have in my system?

Good. The above are the latest (of current stable for dom0, see below).

Not good. modules.img is missing.

Confirmed. Kernel is installed. Its modules is not installed. Install it:

sudo qubes-dom0-update kernel-modules`

With --action=reinstall if necessary.

No. Both kernels could be present on the system at the same time (see below).

Because its modules are missing. Maybe they were not installed due to system crash or power loss

There are different line of kernels. ancient LTS (5.15), old stable (6.6), latest stable (6.12), Development, Debug…
And distro policy or user preference is different on what should be shipped by default. Maybe user has an ancient hardware in sys-net which only works with LTS kernel. I personally prefer to have all of them installed. They might come handy :slight_smile:

No. kernel-qubes-vm is not removed. kernel-lates-qubes-vm is installed besides it. Both are present. And you can have them both (or even more). Just repair the broken ones. Set the default via qubes-prefs default_kernel to the one which works best for you (I use latest). Assure you have not set kernel for individual qubes to a broken one (qvm-prefs VM kernel ...). If yes, reset them to default (qvm-prefs VM kernel -D).

Look and find out all individual kernels and modules available for dom0 and VMs. In dom0:

dnf list 'kernel*'
1 Like

Thanks for your clarification, now I’m starting to understand the logic behind the kernel architecture of Qubes OS.

Like you said, modules.img is missing, and it is actually the main part of kernel. And now I’ve tried to reinstall it with

sudo qubes-dom0-update --action=reinstall kernel-modules

After it installed a relatively huge amount of data, I checked the /var/lib/qubes/vm-kernels/6.6.77-1.fc37/, and it was still only 18MB due to the lack of modules.img.

So, I decided to remove 6.6.77-1.fc37 completely and reinstall it manually, but then I saw this:

$ sudo dnf remove kernel*6.6.77*
Dependencies resolved.
========================================================================================================================================
 Package                        Architecture          Version                                   Repository                         Size
========================================================================================================================================
Removing:
 kernel                         x86_64                1000:6.6.77-1.qubes.fc37                  @qubes-dom0-cached                 40 M
 kernel-devel                   x86_64                1000:6.6.77-1.qubes.fc37                  @qubes-dom0-cached                 65 M
 kernel-modules                 x86_64                1000:6.6.77-1.qubes.fc37                  @qubes-dom0-cached                427 M
 kernel-qubes-vm                x86_64                1000:6.6.77-1.qubes.fc37                  @qubes-dom0-cached                518 M

Transaction Summary
========================================================================================================================================
Remove  4 Packages

Freed space: 1.0 G
Is this ok [y/N]:  

Here, it tells that 6.6.77-1 has the size total 1GB, which means that it is already present in my system, but for an unknown reason I only see 18MB in /var/lib/qubes/vm-kernels/6.6.77-1.fc37/, since modules.img is missing.

So, I didn’t remove it. Should I still remove it and then install it manually, or is it possible to fix it in a way such that my system recognize this kernel version?

This is wrong. Not only it will remove kernel-qubes-vm which is what you see at /var/lib/qubes/vm-kernels, but also it will remove the same version of the kernel for dom0 (the on at /boot). You only need to remove the former. NOT the latter.

Only remove or reinstall kernel-qubes-vm. I just double checked and assured modules.img is a part of that package.

Thanks for your help, I’ve solved the issue:

Basically, removed the kernel-qubes-vm package of broken kernel version and reinstalled it.

Followed these steps:

sudo dnf remove kernel-qubes-vm*6.6.77-1*
sudo qubes-dom0-update kernel-qubes-vm
sudo dnf install kernel-qubes-vm-6.6.77-1.qubes.fc37
2 Likes