Cannot boot Qubes with detached boot and header after update - need rescue

User Support 4.1
#1

Hello all.
I find myself in a fairly unfortunate situation:

Some time ago, I reinstalled Qubes with a detached encrypted boot and luks header following this guide:

All was good up until yesterday when I performed updates of dom0. I believe some package was updated that tried to mess with initramfs but since the boot disk was detached it wasn’t able to do it properly.

At startup the boot is decrypted, then the classic Qubes decryption screen appears (obviously without the password prompt, as that’s usually handled with a saved key) and then black screen with the following:

Warning: /dev/mapper/qubes_dom0-root does not exist
Warning: /dev/qubes_dom0/root does not exist
Warning: /dev/qubes_dom0/swap does not exist

Generating "/run/initramfs/rdsosreport.txt"

Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot

Give root password for maintenance
(or press Control-D to continue):

As I am not familiar with the grub rescue, I would normally boot from a live system, mount everything and chroot into the host but as the luks header is detached from my Qubes installation, I’m only able to mount /boot.

I am lost, please advise. ANY help will be greatly appreciated.

#2

First, don’t panic. I am almost sure everything is there and it’s just about regenerating grub and I am sure @51lieal could do his best to help you (sorry for tagging you).
Just consider changing the topic to something more attractive, like
Cannot boot Qubes with detached boot and header
to its better visibility, and also try some search. These are my limits regarding the issue.

1 Like
#3

first boot into qubes os rescue, skip to shell and decrypt your root partition.
cryptsetup open /dev/xxxroot luks --header /dev/xxxheader

mount root to mnt.
mount /dev/mapper/luks /mnt

then examine are there any /boot folder there? if yes delete it.
rm -rf /boot

#4

I’m able to decrypt it with the --header parameter but when I try to mount it I get the following error:

# mount /dev/mapper/luks /mnt
mount: /mnt: unknown filesystem type 'LVM2_member'.

I was instead able to mount the LV:

mount /dev/qubes_dom0/root /mnt

There was indeed a /boot folder with the following structure:

#ls -a /mnt/boot/
. .. efi  xen-4.14.4.config  xen-4.14.4.gz

I removed it, rebooted but the issue persisted.
Qubes still does NOT boot

#5

Ah yes my bad, this right.

After decrypting /boot, you should be showed a grub menu, press ‘e’ then check how it would boot, or you can take a image and post it here.

#6

About that… During the installation of Qubes, when chrooted I set the following params in order to speed up and already excruciating slow boot:

GRUB_TIMEOUT=0
GRUB_TIMEOUT_STYLE=hidden

So now pressing e does nothing.
However I decrypted the detached boot partition and checked /boot/grub2/ and to my surprise I did not find grub.cfg.

Should I decrypt everything on live, mount /boot into /dev/qubes_dom0/root and generate a new grub.cfg while chrooted? If so, could you indicate me the best way?

#7

Your boot config is in EFI partition (/boot/efi/EFI/qubes/grub.cfg), not in /boot/grub2,
So after you decrypt your /boot and mount it to /mnt, you should mount your efi to /mnt/efi.

I’ve did this once and resolving this wouldn’t be that hard.
Can you confirm /boot folder in /qubes_dom0/root already removed ?

Try boot again and if you still entering dracut emergency, please provide /run/initramfs/rdsosreport.txt it actually tell us how it boot so it could give a hint of what happen.

No, remember we are doing detached boot, so It’s should be nothing in /boot folder.
Everything is done in your flashdrive.

1 Like
#8

Yes, I removed it already

What I meant was mounting /dev/qubes_dom0/root to /mnt then mounting /dev/mapper/luksboot (from the decrypted boot partition of the flashdrive) to /mnt/boot:
mount -o rw /dev/mapper/luksboot /mnt/boot.
But as I now found grub.cfg in the efi partition this should no longer be necessary.

I’ll do it now and report back

#9

Here’s the final portion when it started showing errors:
https://psty.io/r?q=69d98

The original is very long, let me know if it’s needed (if that’s the case, I’d rather send it privately as it contains more identifiers).

#10

I didn’t bring my laptop today, so i’m not sure how it would resolve for your case.
I’ll publish my notes tomorrow.

1 Like
#11

I think there’s a problem in your sd card based on the image you share, I’ve confirm that upgrading kernel with or without detached drive will not affect anything, except that you already installed kernel.

Can you share the full log on the dracut emergency ?

2 Likes
#12

Wait what exactly made you think that, because this was the right call. I restored a backup I had made of the sd card and just like that the system booted up correctly.
Seriously, thanks a lot for the help and the time you dedicated me.

But now I wonder, what’s the right way to update grub and initramfs with a detached boot?

2 Likes
#13

Because everything is boot as normal, first boot it called your efi configuration, second it asking you to decrypt the password (in that case, its absolutely your configuration), then in the log i’ve seen that it was looking for /dev/mmcblk.

Try using good quality of sd card, as far i know, 4 - 8 gb using low grade chip.
and for your backup, it’s okay to have a backup in the /dev/qubes_dom0/root, no need to add additional drive.

There’s no easy way, just make sure sys-usb is not automatically started when you want to upgrade kernel.

1 Like
#14

What I meant is, since a /boot exists when the system is on, would I need to delete it, mount the sd card on /boot and then update grub/initramfs? Or is there another procedure?

#15

if your detached boot drive never been unmounted, then just do as normal, and you don’t need to update grub / initramfs, it would be done automatically.

2 Likes
#16

Got it, thanks a lot

#17

I’ve re read your situation in the #1 perhaps you using sys-usb ? remember that when you using sys-usb, it’s passing the usb controller to sys-usb and that would umount your detached boot drive. That’s why i said you need to turn off auto start sys-usb when you want to upgrade kernel.

And for your situation now if you want to upgrade kernel,

  1. boot as normal.
  2. umount your detached boot.
  3. make sure it already removed.
  4. uninstall recently installed kernel, then remove everything on the /boot rm -rf /boot/*
  5. turn off auto start sys-usb then reboot.
  6. boot as normal, install new kernel then turn on auto start sys-usb and reboot (or you can reboot first to use newer kernel, remove old kernel, then turn on auto start sys-usb and reboot).

I really keep it clean, so i always switch to new kernel then remove old kernel.

2 Likes
#18

I do use sys-usb but the sdcard controller is not attached to it, it’s attached to dom0. I’ve been highly reluctant to attach it to sys-usb as I’m afraid it’ll make my system unbootable, since I have to decrypt the root partition with the sd card.

And thank you lots for the step-by-step