Grub error after installing Qubes 4.2 on external SSD

savi · November 18, 2023, 1:15am

Hello. Friend asked me to install the Qubes system on an external drive, I had the test version 4.2 rc.1 and it was an error. After the installation, the grub of My Qubes was corrupted on my PC and after installation it did not work what i show later.

After reinstalling on external SSD to stable version 4.1.2, Qubes already works on his external disk but now my is damaged.
When boot i have this announcement :
error : no such device ( and here number id … this external disk not mine )
error : unknow filesystem
grub resc

whats goin on ?

please help I need it quickly my qubes.

apparatus · November 18, 2023, 3:53am

Read this:

github.com/QubesOS/qubes-issues

Document UEFI drive removal behavior and workarounds (if drive is unplugged, UEFI removes associated boot entries)

opened 09:26AM - 02 Jun 22 UTC

mehrexe

T: enhancement C: doc P: default

[How to file a helpful issue](https://www.qubes-os.org/doc/issue-tracking/) #…## Qubes OS release Latest as of posting, 4.1.0 ### Brief summary I manually unplugged every drive on my computer except for the target drive for the installation, which was all free space unpartitioned. Installer ran typically, chose automatic for the installation type. Booted into the OS' configuration and then the full OS. GRUB is verified to have worked because I managed to see it when initially booting for configuration and because of a later restart due to a display artificating and then system lockup problem I had that I attribute to Nvidia. After using another drive and then unplugging it, and plugging the Qubes drive back, there was no detectable boot source. I booted a live OS and found that the partition structure was intact but didn't investigate further and reinstalled Qubes. I then got the idea to see if this issue happened if I BIOS disabled drives instead of unplugging them. I disabled my Qubes drive and enabled another, and booted into it fine. I then disabled my other drive and reenabled the Qubes drive and the same issue happened again where my system does not see any boot entries. The system does see the drive, and switching on CSM and legacy boot I can boot from the drive but get an error screen saying to boot from proper media. This second time, I _kind of_ fixed it but I must do these steps every time I remove the drive or disable it in BIOS. I lose the option to boot without the Xen hypervisor and the boot hiccups. I essentially have to boot my install media, run Anaconda rescue and: 1. Mount the bootloader partition 2. Copy the contents of /mnt/EFI/qubes/ to /mnt/EFI/BOOT/ 3. Rename grubx64.efi to bootx64.efi 4. Rename grub.cfg to bootx64.cfg 5. efibootmgr -v -c -u -L Qubes2 -l /EFI/BOOT/bootx64.efi -d /dev/sda -p 1 ### Steps to reproduce I believe the best way to reproduce this issue is to do a fresh install of Qubes on hardware similar to mine: Asus Prime Z-390-A Motherboard Intel i7-9700K CPU 16GB DDR4 RAM 1TB HDD Most notably, this motherboard. I will be filing a compatibility report soon, but I believe this motherboard may have something to do with how it reads the GRUB bootloader as it's configured for Qubes. Further, these are the only non-stock BIOS settings: - Disabling the Intel LAN Controller (it causes a PCI reset error and ends the last step of the configuration, doesn't matter since I don't use it, and I believe the fact that it's unplugged is the reason this issue happens) - Enabling Virtualization - Enabling Vt-d - Disabling all other drives ### Expected behavior The GRUB bootloader is supposed to appear after the BIOS initializes regardless of whether the drive was previously unplugged from the system or not (safely of course). ### Actual behavior Please read above. After unplugging the drive or disabling it in BIOS, the entry for GRUB is missing.

github.com/QubesOS/qubes-issues

Raw disk backup of Qubes EFI is unbootable

opened 02:53PM - 15 Jul 23 UTC

adrelanos

T: bug P: default needs diagnosis affects-4.1 C: boot

### Qubes OS release R4.1 and R4.2 ### Brief summary When making a raw …disk backup from a Qubes installed to an internal hard drive to an external hard drive, the external hard drive is unbootable. raw disk backup means a backup using `dd` or 1 to 1 exact copy. ### Steps to reproduce 1. install Qubes normally on a computer that only support EFI booting on the internal harddrive 2. reboot 3. brief test that Qubes is working normally (yes) 4. shutdown Qubes 5. boot from an external drive, boot a live DVD or live USB such as Debian Live 6. perform a raw disk backup from the internal disk to the external disk 7. unplug that disk and try to boot from it in a different computer (or the same Qubes computer) ([example instructions for raw disk backups](https://www.kicksecure.com/wiki/Raw_Disk_Backup)) ### Expected behavior The raw disk backup of Qubes is bootable. ### Actual behavior The raw disk backup of Qubes is unbootable. ### Additional information According to my research that might be because of missing entries in the EFI firmware's NVRAM which is stored on the motherboard. Unfortunately, the EFI boot process doesn't seem by default to be self-contained on 1 harddrive but require extra settings stored outside harddrives on the motherboard (EFI NVRAM). Using `grub2-install` with options `--removable` / `--force-extra-removable` during Qubes installation might help? > [--removable](https://manpages.debian.org/bookworm/grub2-common/grub-install.8.en.html#removable) > the installation device is removable. This option is only available on EFI. > [--force-extra-removable](https://manpages.debian.org/bookworm/grub2-common/grub-install.8.en.html#force~2) > force installation to the removable media path also. This option is only available on EFI. Qubes does not have good support for multiboot support anyhow: * https://github.com/QubesOS/qubes-issues/issues/8351 * https://www.qubes-os.org/faq/#can-i-install-qubes-os-together-with-other-operating-system-dual-bootmulti-boot * It's "patches welcome". (And that's okay.) This ticket is not a feature request to improve multiboot support. Why do I mention this? Because otherwise, when considering options `--removable` / `--force-extra-removable` one *might* argue "but that breaks mutliboot support". I would argue that being able to boot a raw disk backup of Qubes is more important than mutliboot support. [Why do I like full raw disk backups? See this link.](https://forum.qubes-os.org/t/how-do-you-organize-your-backups/3986/16?u=adrelanos) Manually updating the NVRAM can be challenging: * It's vendor dependent because of many different EFI BIOS versions. * Some BIOS don't even have such an option. * Might require booting into an operating system installed on USB and running console commands to fix it. * Difficult, impossible for far most users to do or even to find instructions for it. Qubes with legacy BIOS booting as far as I remember didn't have this issue. However, never notebooks sometimes (or often, dunno) don't even support legacy BIOS booting anymore. Therefore "use legacy BIOS booting" is a non-solution. Also not a good long term solution in either case because of Qubes planned Secure Boot support. This bug might also make Qubes "non-portable". Meaning, Qubes installed to an external drive such as a USB SSD might be bootable on the computer where it was installed but unbootable when attempting to boot the same Qubes USB SSD on another computer. I didn't test this very part mentioned in this very chapter. Anaconda by Fedora might be using `grub2-install` with options `--removable` / `--force-extra-removable` already when installing to external devices (USB) instead of internal harddrive. Where is the Qubes code which defines bootloader / grub2 / EFI installation? Or is this currently all done by upstream's Anaconda?

savi · November 18, 2023, 1:34pm

I read it and honestly I don’t understand much, so I know it’s not a random error, but with other systems, mainly Fedora, there has never been a problem, so I didn’t check it before installation.

I don’t have access to dom0 here, only have grub rescue
because If i understand correctly that these names need to be changed there (or should it be done before install Qubes on external drive )? :

Copy files from /boot/efi/EFI/qubes/ to /boot/efi/EFI/BOOT/

sudo cp -r /boot/efi/EFI/qubes/* /boot/efi/EFI/BOOT/

Change filename grubx64.efi → BOOTX64.efi.

sudo mv /boot/efi/EFI/BOOT/grubx64.efi /boot/efi/EFI/BOOT/BOOTX64.efi

Change filename grub.cfg → BOOTX64.cfg

sudo mv /boot/efi/EFI/BOOT/grub.cfg /boot/efi/EFI/BOOT/BOOTX64.cfg

can do it from the qubes terminal on an external drive ? it seems that in my case it will be sda1 which has 1 GB sda2 is the main qubes partition (but i see it havent /boot like for sda1 like it is on sdb1 1GB (external drive with qubes) , but how to choose the purpose so that the changes are only on the sda1 partition, what should I enter in the terminal earlier… isn’t that the point? or maybe i should reinstall grub but the same question how do it step by step from cli for this case

apparatus · November 18, 2023, 5:03pm

Do you have Qubes OS installed using UEFI or Legacy BIOS mode?
If it’s UEFI and assuming your Qubes OS drive is sda then sda1 should be EFI System partition, sda2 should be boot partition and sda3 should be main Qubes OS partition.
You can boot from any LiveUSB OS or from Qubes OS installed, mount your Qubes OS EFI System partition (sda1) there and add the required changes.

savi · November 18, 2023, 7:47pm

This is the Legacy BIOS mode, so the disk of my Qubes is SDA and there are only two partitions :
sda1 1GB (without mountpoint like : /boot) and sda2 main partition 500GB

I’ve read more or less on the forums but I don’t want to screw anything up so as not to have another problem to fix, so if you would be so kind as to tell me in such a scenario that I’m already on the live USB system, I run the terminal and what next?

apparatus · November 18, 2023, 8:09pm

The issues that I linked are only applicable for the EFI systems so that problem is not the issue that you’re facing.

You can try to fix up your like this:
First boot into any LiveUSB OS or in Qubes OS installer, mount your system (either manually or automatically if you’re booting from Qubes OS installer in rescue mode), chroot into it and then fix grub config:

savi · November 19, 2023, 12:11am

sudo mount --bind /dev/sdY /mnt/boot when enter this command : mount point does not exist

apparatus · November 19, 2023, 12:26am

Did this command succeed?

sudo mount /dev/sdX /mnt

Where /dev/sdX is your Qubes OS main partition /dev/sdb2 if /dev/sdb is your Qubes OS drive.
You can check the Qubes OS disk name using output of this command:

sudo fdisk -l

savi · November 19, 2023, 12:38am

Did this command succeed?

sudo mount /dev/sdX /mnt

Yes i have info
mount: /mnt: /dev/sda1 already mounted on /run/media/user/(someting numbers)

sudo fdisk -l
yes i know that sda1 i boot partition and sda2 main parition Qubes

but next command sudo mount --bind /dev/sda1 /mnt/boot when enter this command : mount point does not exist

apparatus · November 19, 2023, 1:04am

Is there a typo and it should be /dev/sda2?

Try it like this

sudo umount /run/media/user/(someting numbers)
sudo mount /dev/sda2 /mnt
sudo mount /dev/sda1 /mnt/boot

savi · November 19, 2023, 2:48am

I previously tried to run it from another disk with fedora39 from grub like “Windows 10” (dev/sda1) and :
error …/…/grub-core.commands/search.c.315:no such device: (number like 6857ED75 etc.)
setting partition type to 0x83
error: …/…grub-core/loader/i386/pc/chainloader.c213: invalid signature

I’m tired of this … do you see this program boot-repair-disk / Home / Home like live usb as a quicker alternative to fixing this?

But is it safe ( For BIOS ? ) Do you know this program? Thanks for your answers, I appreciate it

replying to your post
sudo mount /dev/sda2 /mnt :
mount: /mnt: special device /dev/sda2 does not exist.

sudo mount /dev/sda1 /mnt/boot :
mount: /mnt/boot : mount point does not exist

apparatus · November 19, 2023, 5:27am

It won’t work.

You’ve said before that:

Change sda to some other drive name, e.g. sdb if it’s the correct Qubes OS drive name.

savi · November 19, 2023, 12:43pm

Still these errors.Looks like I won’t do anything about it anymore, it’s really screw up, it’s a shame to waste your time and you’ve sacrificed so much, thanks

it seems like the only thing left is to decrypt the disk, copy it to another one and reinstall the system … ?

savi · November 23, 2023, 2:17pm

Okay, someone tell me if it makes sense and is possible to create an image of an encrypted disk with Qubes, then reinstall cleanly Qubes and then replace the new partitions with this created image. Will this work well ?
Because I don’t consider it safe to copy all from a decrypted disk to another (used) USB and do it on new Qubes.

apparatus · November 23, 2023, 6:07pm

It’s possible but for it to work you’ll need to change the UUIDs of your encrypted disk, dom0 root and boot partitions in dom0 root and boot partition configs and have the same kernel boot files (initramfs, vmlinuz) in old boot partition that you have in new Qubes dom0.