How to restore dom0 from backup after kernel panic

What happened: I was updating dom0 (qubes-dom0-update) and it seemed to freeze after installing everything. It was erasing old kernels, step 12/12. Eventually I ctl-c a few times to get out of there. I ran qubes-dom0-update again and it ran with no errors, showing the items I’d just installed but didn’t seem to do anything. At this point I decided to reboot and now I get a kernel panic and cannot boot into Qubes.

What I have: A backup of my system including dom0, with old versions of other qubes. My current system that won’t boot.

My plan: My thought was to take the back up dom0 and restore it to my current system to try to fix this. Let me know if you think there’s a better way.

  1. recover my old dom0 using this: https://www.qubes-os.org/doc/backup-emergency-restore-v4/

  2. copy it over to my current system

I have not done either step, but I’m assuming the instructions in step 1 will work.

I’m trying to figure out how/if i can do step 2. If I open up my encrypted backup I get

qubes_dom0-pool00
qubes_dom0-pool00_tdata
qubes_dom0-pool00_tmeta
qubes_dom0-pool00-tpool
qubes_dom0-root
qubes_dom0-swap

and all my vm’s. I’m guessing qubes_dom0-root is where I would restore my dom0 backup to? If so, can I just rsync everything from the restored backup dom0 to there?

I’m also concerned that my grub config might be a problem, but not sure how to check on that.

Thanks for reading and any tips you can offer. Hoping to get this figured out today :confused:

1 Like

In step 1 of the emergency restore I linked to above it says:

To extract only specific VMs: Each VM in the backup file has its path listed in qubes.xml.000.enc . Decrypt it. (In this example, the password is password .)

Is the password the password you used to encrypt backup? Because this doesn’t seem to be working.

Here is the specific message i’m getting:

Kernel Panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

And I can find help like this: https://askubuntu.com/questions/41930/kernel-panic-not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0

Problem is I don’t have a live Qubes cd (because it doesn’t exist), so how to get the correct in initramfs installed on there. Can I do it in any live cd? and if so how do i determine the correct version to update to?

From what you write, here are my observations:

  1. You probably killed the initramfs generation during updates. This will cause the error you see (unable to mount root). Never kill the updates!

  2. Your backups won’t help you recover your old system.

  3. Your backups only contain user data…the error you have affects your system data.

  4. Your backups however will help you rebuild a new system.

  5. You may not need to do this if the only thing wrong is your /boot partition

What you want to do is download any linux live CD, or the Qubes installer if you want to download 4 gigabytes all over again. Then you want to get to a shell after booting with the live CD. No need to worry about disk encryption.

If you normally boot with EFI:

  1. Mount your EFI partition, will be a FAT filesystem with EFI/qubes directory in it
  2. See how much free space is in that partition. Is it low?
  3. ls -l initramfs* in [mountpoint]/EFI/qubes/
  4. Do any of them look less than 30MB?
  5. Whichever is the largest one, remember the kernel version
  6. Update xen.cfg, set the default= line to point to that kernel version
  7. unmount, reboot from hard drive

If you legacy boot with GRUB:

Try to select any other kernels from the menu.

If you aren’t able to recover the system, then you would reinstall Qubes, and use the Restore Backup function to restore the private volumes of your qubes.

1 Like

Ensure the directory name is included. If you are decrypting vm24/private.img.012.enc, the file name portion of the password is vm24/private.img.012.

The decryption passphrase = BACKUP_ID!FILENAME!USER_PASSPHRASE

So if your password is password123, the scrypt decryption passphrase would be something like 20201031T123456-7890!vm24/private.img.012!password123.

1 Like

Good tip, however I was just trying to decrypt qubes.xml.000.enc

I was just able to test this using Qubes Restore and it seems I don’t have the right password or the header is corrupt if I try to verify this backup! I’ll have to be more careful about verifying my backups in the future to avoid this.

Thanks icequbes1, you pointed me down the right road. I solved this, but without using a Qubes restore. I didn’t quite use your suggestions though I adapted the ubuntu link I posted above. Here are some highlights/differences:

First I have an luks encrypted disk on a RAID0 with 2 disks. So it looked like:

/dev/sdb1 - boot partition
/dev/sdb2 - luks partition
/dev/sdc1 - boot partition mirror
/dev/sdc2 - luks partition mirror

So I decrypted the /dev/sdb2 and mounted it, then used the “mount --bind” commands in the ubuntu link and chroot’ed into that.

Then I mounted the boot partition /dev/sdb1 separately.

In there I could see that I was missing the initramfs for 4.19.152-1 as you suggested. all other files for that kernel version were present. And all files for 2 other kernels were present.

In the chroot I checked /lib/modules - in there I could see that I had 4.19.152-1 kernel, among the others.

so I did a “mount --bind” to give access to the /dev/sdb1 boot partition within the chroot

And in the boot partition within the chroot I ran dracut (instead of initramfs-update in ubuntu since dom0 is fedora) to create the initramfs for 4.19.152-1

Then I checked in boot/grub2 to see I had a grub.cfg and noted the size and contents and made a backup. Again, I had to find the fedora application, which was

grub2-mkconfig -o grub.cfg

This created a larger, updated grub.cfg.

Finally I copied the grub.cfg and the initramfs for 4.19.152-1 to /dev/sdc1 just in case (maybe not necessary? not sure.)

After that I can now boot as normal.

And yes, do not ever kill in the middle of an update. Also don’t rush things when you’re tired and trying to sleep.

1 Like

Hold on. Those emergency backup recovery instructions are intended to be used when no Qubes system is available (e.g., because you have no Qubes ISO and no internet access). That’s why the title of the page is “Emergency Backup Recovery without Qubes (v4)” and the description says, “In the event a Qubes system is unavailable, you can access your data on any GNU/Linux system with the following procedure.”

Can you use these instructions even when a Qubes system is available to you? Sure, but you might be making things harder than they need to be.

The simplest way forward in your situation is to:

  1. Reinstall Qubes.
  2. Restore from your backup.

That’s it. You never have to touch the command line, if you don’t want to.

If you prefer to do something else, that’s fine. I just want to make sure you understand that this is the standard restore procedure.

Edit: Oh, I replied without reading the whole thread. Glad to hear you already got things working. Hopefully the above is still helpful for future reference. :slightly_smiling_face:

1 Like

I was considering reinstalling. The problem was my backup was somewhat dated. So the dom0 backup should have been fine, but I would have lost some significant data in my other VMs/Qubes. So I really needed to get back into the system that was not booting.

2 Likes

Great work! Disaster recovery is always fun…when you’re able to recover!

1 Like

Good question and excellent answers, thanks! I have something to learn and find out from this topic. :+1: :+1:

Oddly enough i am investigating a similar problem. I had rebooted after a Kernel update last Wednesday and all was fine. Then a power outage a day later somehow managed to zero file size the last Kernel update files (initramfs, vmlinuz and xen.cfg) in the efi folder.

Unfortunately i do not understand much about the whole efi boot process and a new xen.cfg with previous kernel that is a non zero file size file, does not boot up Qubes.
Going to check out if i can hopefully rebuild the kernel like described above.

What error are you getting on boot? Does Xen even (attempt to) start?

While I did restore my system above, I struggled getting my RAID working again. I’ll save you the details, but I got to the point where I needed to reinstall from a backup. My thought was this would solve my RAID issue in the process as I could take care of that priort to install, but no such luck. Using the new 4.0.4 installer, it crashes with an error.

Based on the advice on the top of this page: Installation Troubleshooting | Qubes OS I removed all partitions from the drives. I was able to get past the first page of the installer this time. But I switched to the console and created a RAID with my two disks before proceeding. The installer did not recognize the RAID as such and the install failed with the same error (something related to python I did not document).

I apparently installed Qubes on a RAID before, but cannot figure out how I did it. Are there any guides for installing Qubes on a RAID (1)?

For now I’m back to installing on one disk and trying to figure out the RAID after.

Which brings me back to my problem of restoring dom0. I’m restoring my qubes from backup, seems to be working so far. But restoring dom0 merely creates a directory in home/ wtih a copy of the old home/ directory in there.

My Question: Can you restore other parts of dom0 besides home? Specifically stuff in /etc/ like qubes-rpc/policy and menu preferences, etc? Do I have to use Emergency Backup Recovery (v4) | Qubes OS to access those files?