Can only boot older Qubes .img after inproper shutdown

User Support General
1

[SOLVED]
Problem: kernel version 48 doesn’t work on my system, I was under the impression that it did, but that it stopped after an inproper shutdown, but it seems like I may have been using version 42 prior to that shutdown, and that the session prior to that shutdown was when version 48 was added to my system.

I have an NVIDIA gpu and looking at apparatus’ first reply, this is probably what makes the most sense. I’m gonna keep using version 42 for now and mark this case as solved. Thanks for the help

I have an issue with booting Qubes.

I get to grub, and simply clicking the Qubes option at the top brings me to a long GRUB loading screen, which leads to rescue mode eventually.

What I have to do is to go to advanced, and select 6.6.42.1-qubes.fc37.x86_64 and below (so not anything higher than 42, ie 48, 54).

What happened was that I was on 6.6.48.1 one day, and after having my computer on over the night, when I went to turn off the computer, the bootdown process was extremely long. It was reaching 10 hours of just shutting down, so I held the power button to kill my computer.

Next time I boot, the grub loading problem occurred, where it tries to load the image like I explained, but just never does, and eventually goes into rescue mode.

And since then I’ve had to go to advanced (in GRUB), select 6.6.42-qubes.fc37.x86_64 or 6.6.38, and boot that way.

Yesterday after running the Qubes updater it seems that a new .img has been added to my system. 6.6.54.

I went to try the new .img, but the problem that I had with 6.6.48 occurred with this new one, 6.6.54. Which I found weird, why would the updated .img file be corrupted. Doesn’t make sense, it’s got to be something else, and that’s why I’m here.

I was under the impression that holding the power button corrupted the 6.6.48 image but maybe it just messed up some GRUB config. I have no clue.

After this update where the new 6.6.54 image has been added to my system, 6.6.38 has been removed. So I figured I would try to have this problem fixed before 6.6.42 is removed too, and my system becomes “unbootable” since I can’t boot from 6.6.48 or 6.6.54, which are the only remaining two, and I guess I wouldn’t be able to launch from any future image since 6.6.54 doesn’t work either, for some reason.

2

What GPU do you use in dom0? Is it NVIDIA?
If yes then maybe it’s this issue:

But if that’s the case, then I think it shouldn’t be going into GRUB rescue mode.

3

Thanks, I’ll try it out and get back to you. Although I doubt that it’s because of this since I’ve been able to run 6.6.48 dozens of times before, until I did the weird shutdown, and then it stopped working. But I’ll try

4

Tried on both 54 and 48, didn’t change anything.

5

Do you have enough free space in the /boot and /boot/efi partitions?
Check the output of this command in dom0:

df -h /boot /boot/efi
6

dev sdb1
size 974m
used 239m
available 668m
use 27%
mounted on /boot

It says this twice, so /boot and /boot/efi gives identical results

7

Check if you have enough free space in the dom0 root directory:

df -h /

Try to regenerate initramfs:

sudo dracut -f --kver 6.6.54-1.qubes.fc37.x86_64

And check if you’ll have any errors in the output.

Try to reinstall the kernel in dom0:

sudo qubes-dom0-update --action=reinstall <package>

If it won’t help then check the boot log when you boot with kernel 6.6.54 and search the errors there.

8

I have 50% unused space or 10gb free in dom0 root.

did sudo dracut thing, returns:
dracut: The provided directory where to look for kernel modules (ver)
dracut: does not match the kernel version set for initramfs (DRACUT_KMODDIR_OVERRIDE=1).
dracut: Set DRACUT_KMODDIR_OVERRIDE=1 to override this check.

I did the update and it reinstalled all 3 on my system (42, 48, and 54), and it also remade some grub files.

Edit:

forgot to mention, the last command printed some errors.
Heres the entirety of what it printed:

[user2@dom0 ~]$ sudo qubes-dom0-update --action=reinstall kernel
Using sys-whonix as UpdateVM for Dom0
Downloading packages. This may take a while...
Unable to detect release version (use '--releasever' to specify release version)
Fedora 37 - x86_64                              3.9 kB/s | 6.0 kB     00:01    
Fedora 37 - x86_64 - Updates                    4.9 kB/s | 5.9 kB     00:01    
Qubes Host Repository (updates)                 3.1 kB/s | 2.7 kB     00:00    
Dependencies resolved.
================================================================================
 Package   Arch      Version                        Repository             Size
================================================================================
Reinstalling:
 kernel    x86_64    1000:6.6.42-1.qubes.fc37       qubes-dom0-current     12 M
 kernel    x86_64    1000:6.6.48-1.qubes.fc37       qubes-dom0-current     12 M
 kernel    x86_64    1000:6.6.54-1.qubes.fc37       qubes-dom0-current     12 M

Transaction Summary
================================================================================

Total size: 36 M
Installed size: 119 M
DNF will only download packages for the transaction.
Downloading Packages:
[SKIPPED] kernel-6.6.42-1.qubes.fc37.x86_64.rpm: Already downloaded            
[SKIPPED] kernel-6.6.48-1.qubes.fc37.x86_64.rpm: Already downloaded            
[SKIPPED] kernel-6.6.54-1.qubes.fc37.x86_64.rpm: Already downloaded            
Complete!
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Qubes OS Repository for Dom0                                                                                               2.9 MB/s | 3.0 kB     00:00    
Qubes OS Repository for Dom0                                                                                                94 kB/s | 1.6 kB     00:00    
Dependencies resolved.
===========================================================================================================================================================
 Package                      Architecture                 Version                                           Repository                               Size
===========================================================================================================================================================
Reinstalling:
 kernel                       x86_64                       1000:6.6.42-1.qubes.fc37                          qubes-dom0-cached                        12 M
 kernel                       x86_64                       1000:6.6.48-1.qubes.fc37                          qubes-dom0-cached                        12 M
 kernel                       x86_64                       1000:6.6.54-1.qubes.fc37                          qubes-dom0-cached                        12 M

Transaction Summary
===========================================================================================================================================================

Total size: 36 M
Installed size: 119 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                   1/1 
  Reinstalling     : kernel-1000:6.6.54-1.qubes.fc37.x86_64                                                                                            1/6 
  Reinstalling     : kernel-1000:6.6.48-1.qubes.fc37.x86_64                                                                                            2/6 
  Reinstalling     : kernel-1000:6.6.42-1.qubes.fc37.x86_64                                                                                            3/6 
  Running scriptlet: kernel-1000:6.6.54-1.qubes.fc37.x86_64                                                                                            4/6 
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.6.48-1.qubes.fc37.x86_64
Found initrd image: /boot/initramfs-6.6.48-1.qubes.fc37.x86_64.img
Found linux image: /boot/vmlinuz-6.6.42-1.qubes.fc37.x86_64
Found initrd image: /boot/initramfs-6.6.42-1.qubes.fc37.x86_64.img
Adding boot menu entry for UEFI Firmware Settings ...
done

  Cleanup          : kernel-1000:6.6.54-1.qubes.fc37.x86_64                                                                                            4/6 
  Running scriptlet: kernel-1000:6.6.48-1.qubes.fc37.x86_64                                                                                            5/6 
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.6.42-1.qubes.fc37.x86_64
Found initrd image: /boot/initramfs-6.6.42-1.qubes.fc37.x86_64.img
Adding boot menu entry for UEFI Firmware Settings ...
done

  Cleanup          : kernel-1000:6.6.48-1.qubes.fc37.x86_64                                                                                            5/6 
  Running scriptlet: kernel-1000:6.6.42-1.qubes.fc37.x86_64                                                                                            6/6 
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done

  Cleanup          : kernel-1000:6.6.42-1.qubes.fc37.x86_64                                                                                            6/6 
  Running scriptlet: kernel-1000:6.6.54-1.qubes.fc37.x86_64                                                                                            6/6 
Error: kernel image argument /boot/vmlinuz-6.6.54-1.qubes.fc37.x86_64 not a file
warning: %posttrans(kernel-1000:6.6.54-1.qubes.fc37.x86_64) scriptlet failed, exit status 1

Error in POSTTRANS scriptlet in rpm package kernel
  Running scriptlet: kernel-1000:6.6.48-1.qubes.fc37.x86_64                                                                                            6/6 
Error: kernel image argument /boot/vmlinuz-6.6.48-1.qubes.fc37.x86_64 not a file
warning: %posttrans(kernel-1000:6.6.48-1.qubes.fc37.x86_64) scriptlet failed, exit status 1

Error in POSTTRANS scriptlet in rpm package kernel
  Running scriptlet: kernel-1000:6.6.42-1.qubes.fc37.x86_64                                                                                            6/6 
Error: kernel image argument /boot/vmlinuz-6.6.42-1.qubes.fc37.x86_64 not a file
warning: %posttrans(kernel-1000:6.6.42-1.qubes.fc37.x86_64) scriptlet failed, exit status 1

Error in POSTTRANS scriptlet in rpm package kernel
  Verifying        : kernel-1000:6.6.42-1.qubes.fc37.x86_64                                                                                            1/6 
  Verifying        : kernel-1000:6.6.42-1.qubes.fc37.x86_64                                                                                            2/6 
  Verifying        : kernel-1000:6.6.48-1.qubes.fc37.x86_64                                                                                            3/6 
  Verifying        : kernel-1000:6.6.48-1.qubes.fc37.x86_64                                                                                            4/6 
  Verifying        : kernel-1000:6.6.54-1.qubes.fc37.x86_64                                                                                            5/6 
  Verifying        : kernel-1000:6.6.54-1.qubes.fc37.x86_64                                                                                            6/6 

Reinstalled:
  kernel-1000:6.6.42-1.qubes.fc37.x86_64             kernel-1000:6.6.48-1.qubes.fc37.x86_64             kernel-1000:6.6.54-1.qubes.fc37.x86_64            

Complete!
9

I don’t have this warning when running this command:

sudo dracut -f --kver 6.6.54-1.qubes.fc37.x86_64

I don’t know why do you have it.
Did you run the exact same command?
And not something like this instead:

sudo dracut -f -k 6.6.54-1.qubes.fc37.x86_64

I’m not sure what this error is about.
What’s the output of this command in dom0?

ls -la /boot
10

Yeah its the exact same command, its just that I’m booted into 6.6.42 in order to actually be able to use the system, so whilst running that command in other words, which might be why. Idk though. Cause I’m guessing you’re booted in version 54 whilst running that dracut command for the 54 version? I could try running it but for 42, since thats what im booted in, or maybe even 48. Ill try 48 first, so that I dont mess up 42 since that’s the only one working for me.

[user2@dom0 ~]$ ls -la /boot
total 1656
dr-xr-xr-x.  5 root root    4096 Nov  4 17:21 .
dr-xr-xr-x. 19 root root    4096 Feb 15  2024 ..
drwxr-xr-x   3 root root    4096 Feb 15  2024 efi
drwx------   6 root root    4096 Nov  4 17:21 grub2
drwx------.  2 root root   16384 Feb 15  2024 lost+found
-rw-r--r--   1 root root  185363 Aug  3 02:00 symvers-6.6.42-1.qubes.fc37.x86_64.gz
-rw-r--r--   1 root root  185464 Sep  3 02:00 symvers-6.6.48-1.qubes.fc37.x86_64.gz
-rw-r--r--   1 root root  185618 Oct  5 02:00 symvers-6.6.54-1.qubes.fc37.x86_64.gz
-rw-r--r--   1 root root    3484 Oct 19 02:00 xen-4.17.5.config
-rw-r--r--   1 root root 1092334 Oct 19 02:00 xen-4.17.5.gz

Edit: Just tried the dracut command but for 6.6.48-1 and 42 likewise, and they both return the same as 54 did.

I made a typo in my previous reply though.

I said that it returns:

dracut: The provided directory where to look for kernel modules (ver)
dracut: does not match the kernel version set for initramfs (DRACUT_KMODDIR_OVERRIDE=1).
dracut: Set DRACUT_KMODDIR_OVERRIDE=1 to override this check.

but it actually returns (the middle line is different):

dracut: The provided directory where to look for kernel modules (ver)
dracut: does not match the kernel version set for initramfs (6.6.42-1.qubes.fc37.x86_64).
dracut: Set DRACUT_KMODDIR_OVERRIDE=1 to override this check.

And does so for all these commands:

sudo dracut -f --kver 6.6.54-1.qubes.fc37.x86_64

sudo dracut -f --kver 6.6.48-1.qubes.fc37.x86_64

sudo dracut -f --kver 6.6.42-1.qubes.fc37.x86_64
11

I’m not using kernel 6.6.54 in dom0.
I’ve just checked and it seems that you’ve used wrong command:

[user@dom0 ~]$ sudo dracut -f -k 6.6.54-1.qubes.fc37.x86_64
dracut: The provided directory where to look for kernel modules (6.6.54-1.qubes.fc37.x86_64)
dracut: does not match the kernel version set for the initramfs (6.11.2-1.qubes.fc37.x86_64).
dracut: Set DRACUT_KMODDIR_OVERRIDE=1 to ignore this check.

You need to use this command:

sudo dracut -f --kver 6.6.54-1.qubes.fc37.x86_64

Is that all you have in /boot directory? No initramfs/vmlinuz/etc files?

12

No I haven’t used -f -k, only -f -kver. Idk where you copied that from. Im user2 not “user”

I have edited my reply before this to include some clarification. Apologies for being unclear

13

That is everything in my /boot directory

14

It’s the output of running this command in my dom0.
Here’s an example of wrong and right commands output:

[user@dom0 ~]$ sudo dracut -f -k 6.6.54-1.qubes.fc37.x86_64
dracut: The provided directory where to look for kernel modules (6.6.54-1.qubes.fc37.x86_64)
dracut: does not match the kernel version set for the initramfs (6.11.2-1.qubes.fc37.x86_64).
dracut: Set DRACUT_KMODDIR_OVERRIDE=1 to ignore this check.
[user@dom0 ~]$ sudo dracut -f --kver 6.6.54-1.qubes.fc37.x86_64
[user@dom0 ~]$

First command fails with the same error that you see but second command succeeds.
It looks to me that you’re using the wrong command or have an error somewhere in it.

Then I don’t know how did you even boot your Qubes OS.
What do you have in GRUB config file /boot/grub2/grub.cfg?

15

The issue was that I was typing -kver, and not --kver. Anyway, typing this command

sudo dracut -f --kver 6.6.54-1.qubes.fc37.x86_64

returns a new error

dracut: /lib/modules/6.6.54-1.qubes.fc37.x86_64//modules.dep is missing. Did you run depmod?
16 
[user2@dom0 ~]$ sudo cat /boot/grub2/grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -f ${config_directory}/grubenv ]; then
  load_env -f ${config_directory}/grubenv
elif [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/08_fallback_counting ###
insmod increment
# Check if boot_counter exists and boot_success=0 to activate this behaviour.
if [ -n "${boot_counter}" -a "${boot_success}" = "0" ]; then
  # if countdown has ended, choose to boot rollback deployment,
  # i.e. default=1 on OSTree-based systems.
  if  [ "${boot_counter}" = "0" -o "${boot_counter}" = "-1" ]; then
    set default=1
    set boot_counter=-1
  # otherwise decrement boot_counter
  else
    decrement boot_counter
  fi
  save_env boot_counter
fi
### END /etc/grub.d/08_fallback_counting ###

### BEGIN /etc/grub.d/10_reset_boot_success ###
# Hiding the menu is ok if last boot was ok or if this is a first boot attempt to boot the entry
if [ "${boot_success}" = "1" -o "${boot_indeterminate}" = "1" ]; then
  set menu_hide_ok=1
else
  set menu_hide_ok=0 
fi
# Reset boot_indeterminate after a successful boot
if [ "${boot_success}" = "1" ] ; then
  set boot_indeterminate=0
# Avoid boot_indeterminate causing the menu to be hidden more then once
elif [ "${boot_indeterminate}" = "1" ]; then
  set boot_indeterminate=2
fi
# Reset boot_success for current boot 
set boot_success=0
save_env boot_success boot_indeterminate
### END /etc/grub.d/10_reset_boot_success ###

### BEGIN /etc/grub.d/12_menu_auto_hide ###
if [ x$feature_timeout_style = xy ] ; then
  if [ "${menu_show_once}" ]; then
    unset menu_show_once
    save_env menu_show_once
    set timeout_style=menu
    set timeout=60
  elif [ "${menu_auto_hide}" -a "${menu_hide_ok}" = "1" ]; then
    set orig_timeout_style=${timeout_style}
    set orig_timeout=${timeout}
    if [ "${fastboot}" = "1" ]; then
      # timeout_style=menu + timeout=0 avoids the countdown code keypress check
      set timeout_style=menu
      set timeout=0
    else
      set timeout_style=hidden
      set timeout=1
    fi
  fi
fi
### END /etc/grub.d/12_menu_auto_hide ###

### BEGIN /etc/grub.d/14_menu_show_once ###
if [ x$feature_timeout_style = xy ]; then
  if [ "${menu_show_once_timeout}" ]; then
    set timeout_style=menu
    set timeout="${menu_show_once_timeout}"
    unset menu_show_once_timeout
    save_env menu_show_once_timeout
  fi
fi
### END /etc/grub.d/14_menu_show_once ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
if [ "$grub_platform" = "efi" ]; then
	menuentry 'UEFI Firmware Settings' $menuentry_id_option 'uefi-firmware' {
		fwsetup
	}
fi
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/35_fwupd ###
### END /etc/grub.d/35_fwupd ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg
fi
### END /etc/grub.d/41_custom ###
17 
[user2@dom0 ~]$ sudo dracut -f --kver 6.6.54-1.qubes.fc37.x86_64
dracut: /lib/modules/6.6.54-1.qubes.fc37.x86_64//modules.dep is missing. Did you run depmod?
[user2@dom0 ~]$ sudo dracut -f --kver 6.6.48-1.qubes.fc37.x86_64
dracut: /lib/modules/6.6.48-1.qubes.fc37.x86_64//modules.dep is missing. Did you run depmod?
[user2@dom0 ~]$ sudo dracut -f --kver 6.6.42-1.qubes.fc37.x86_64
dracut: /lib/modules/6.6.42-1.qubes.fc37.x86_64//modules.dep is missing. Did you run depmod?
18

Try to remove the kernel and install it again:

sudo dnf remove kernel-6.6.54-1.qubes.fc37
sudo qubes-dom0-updates 6.6.54-1.qubes.fc37

Your grub config is missing all boot entries.
Maybe the kernel reinstall failed somehow and it messed up the files in /boot along with grub config.

I suggest you to make a backup just in case and not to reboot before it because you may not be able to boot anymore.