Qubes 4.2-rc5 update did not generate the initramfs images in the system partition /EFI/qubes. Broken installation, I can’t boot anymore

I had a Qubes 4.2-rc5 installation with full disk encryption, which was working fine. The drive was partitioned by the Qubes OS installer automatically. I updated the Qubes OS through its GUI a couple of days ago, and I can’t boot anymore. The UEFI boot process stops in the early stages (before showing the Xen boot menu or asking for the LUKS password), and terminates in the grub shell.

I checked the partition layout for Qubes 4.1x and 4.2. In 4.1x, Qubes with FDE creates these partitions:

  • an unencrypted system partition that contains EFI and grub config files and some initramfs images.

  • An unencrypted boot partition, that contains some of the same files, more config files, initramfs images and Linux kernel images vmlinuz.

  • An encrypted LUKS OS and data partition

For Qubes 4.2, the layout is different, and more is encrypted:

  • an unencrypted system partition that contains EFI and grub config files (in my installation, there are no initramfs images in /EFI/qubes directory, they seem to be missing)

  • An encrypted LUKS swap partition

  • A LUKS encrypted partition that contains grub config files, initramfs images, Linux kernel images vmlinuz, operating system, VM templates, and users data

It seems that the initramfs images are missing from my EFI directory in Qubes 4.2, not created during the update. There is no fallback boot directory in Qubes 4.2.

Is there a way to regenerate the initramfs images (and other required boot files)?

It seems I could chroot from another OS to a particular directory, and install or run grub. Are there tutorials?

A similar issue is reported by another user in the post linked below

More on the issue that I encountered and what I did so far with the help of the user @apparatus in the link below. I regenerated the grub entry for Qubes a few times using the instructions in the link below, pointing to grub.cfg, Xen.cfg or grubx64.cfg, and they all end up with a grub shell saying, please load the kernel.

Are you sure it’s a separate swap partitions?
I think by default you should have your swap inside your LUKS partition and have separate unencrypted /boot partition.
Are you sure that you’re looking at the right disk? Maybe it’s your other disk with another Linux OS and your Qubes OS disk is another one?

Sounds like the grub.cfg has an error of some sort (or GRUB simply fails to find it) … the initramfs shouldn’t be loaded until after GRUB has selected which kernel/image/config should be used.

I’m about to give 4.2-RC5 a go myself, so I’ll know more/better in a few days … did you select a “all default” partitioning … no custom layout?

Yes, I’m careful to select the right disk. I mount it and see the qubes file, as noted.

The partition /dev/nvme0n1p2 is system partition.

The partition /dev/nvme0n1p2/ has one sub-partition “cryptoswap” with the type “crypt [swap]”. It’s not mountable. This looks like a standard swap partition.

The partition /dev/nvme0n1p3, has Qubes and user data.

Is it encrypted as well?
If it is then I’m not sure where is your /boot partition, because it should be present if you used default Qubes OS installer partitioning.
Check the UUID in your Qubes OS grub config e.g. in your config you have:

### BEGIN /etc/grub.d/20_linux_xen ###
menuentry 'Qubes, with Xen hypervisor' --class qubes --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-simple-/dev/mapper/qubes_dom0-root' {
	insmod part_gpt
	insmod ext2
	set root='hd0,gpt2'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  4cf1ce08-ceeb-49eb-bae4-f2e5c2bb4d89
	else
	  search --no-floppy --fs-uuid --set=root 4cf1ce08-ceeb-49eb-bae4-f2e5c2bb4d89
	fi
	echo	'Loading Xen 4.14.6 ...'
        if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
            xen_rm_opts=
        else
            xen_rm_opts="no-real-mode edd=off"
        fi
	multiboot2	/xen-4.14.6.gz placeholder  console=none dom0_mem=min:1024M dom0_mem=max:4096M ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096 ${xen_rm_opts}
	echo	'Loading Linux 6.1.43-1.qubes.fc32.x86_64 ...'
	module2	/vmlinuz-6.1.43-1.qubes.fc32.x86_64 placeholder root=/dev/mapper/qubes_dom0-root ro rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles rd.driver.pre=btrfs rhgb quiet 
	echo	'Loading initial ramdisk ...'
	module2	--nounzip   /initramfs-6.1.43-1.qubes.fc32.x86_64.img
}

Then partition with UUID 4cf1ce08-ceeb-49eb-bae4-f2e5c2bb4d89 is your /boot partition.
Check if it’s present with this command:

blkid

Here is my grub.cfg file

search --no-floppy --fs-uuid --set=dev 2e55c6e-1fe1-4a9e-570c-bad1eab5d9d0
set prefix=($dev)/grub2

export $prefix
configfile $prefix/grub.cfg

The initramfs images should be unencrypted so that the file system is understood and LUKS partition can be unlocked.

Here are the partitions on the Qubes drive. P1 is the system partition. Lsblk indicates that p2 is a crypt swap partition. I can’t mount p2 to see what is inside. Logically , this should be a boot partition, but I don’t know how to mount it and see. P3 is encrypted and can be mounted with correct password: it includes dom0 and virtual machine images

fdisk -l /dev/nvme0n1

Device                      Size	 Type
/dev/nvme0n1p1      600M	 EFI System
/dev/nvme0n1p2      1G 		 Linux     filesystem
/dev/nvme0n1p3      429.9G 	 Linux filesystem

lsblk  /dev/nvme0n1p2

NAME          	  RM SIZE RO TYPE      MOUNTPOINTS
 nvme0n1p2         0   1G  0 part  
└─cryptoswap      0   1G  0 crypt [SWAP]

I have copied the grub.cfg in my previous reply (there is also grub.cfg.swaprpm). The UUID mentioned there does not match any UIUD returned by blkid command.

I don’t see any image file in unencrypted partitions.

It seems that you’ve created swap partition over your /boot partition.
Did you make any changes regarding swap in your system since last time you’ve boot your Qubes OS 4.2?
Or did you maybe install a new OS and selected your Qubes OS /boot partition to be the swap partition?

No, I have not installed a new OS. I have simply updated the Qubes OS from GUI. I upgraded the RAM, and Qubes booted and was snappy. In a subsequent boot, it stopped booting.

I think you might be right. P2 is expected to be an unencrypted boot partition. After I decrypt P3, I see an LVM partition there named qubes_dom0-swap which is 4G. So there should be a swap in P3.

Somehow the boot partition is not recognized and can’t be mounted (I enounter an error message that it’s already mounted or the mount is busy). Not sure what kind of partition it is.

It might be easier to reinstall it.

It’s possible to restore the /boot partition but it’s not easy as you’ll need to format the /boot partition to ext4 again, mount all Qubes OS partitions for chroot, download the kernel package from Qubes OS repository, copy it to your Qubes OS root partition, chroot into it, change UUID of /boot partition in grub.conf in EFI partition, in fstab and maybe somewhere else and then reinstall the kernel package (it should install vmlinuz and regenerate initramfs with dracut and regenerate grub config automatically).

For those who come across this post, the problem was something else, not related to Qubes. Ubuntu installed on another disk, automatically over-writes the boot partition of Qubes:

This seems to be a bug.