Software RAID (mdadm) Qubes Installation Procedure (WIP)

I see comments that people are using software raid (md rescue disk, set up the EFI partition (which you have already allocated) and install grub on thadm), but I don’t see any good instructions for how to do it. I just finished a software raid install, so I’m putting instructions here for the next person.

Notes:

  • Please mention any information/comments you have to add to this procedure
  • This procedure is written for qubes 4.2 installs
  • This procedure will assume you have 2 hard drives. If you have more, then whenever it says to do something “for the 2nd hard drive”, do it for each of the hard drives that are left
  • This procedure assumes the drives are blank, and do not already have a partition table
  • This is a work in progress. Currently it’s probably about half done.

Graphical Software Raid installation procedure:

  1. Do the standard steps listed in Installation guide | Qubes OS , up to the part about configuring the “Installation Destination” (after the anchor: Installation guide | Qubes OS)
  2. Select “Installation Device”
  3. Select all media
  4. Select “advanced custom”

NOTE: There is the CLI way to set up the partitions, and the graphical way to set up the partitions. Below is the graphical way, but see “If it breaks during the install” section at the end of this document for the beginning of the discussion of CLI related ways of doing things.

Create the (blank) partition tables

  1. Click on disk 1
  2. Right click on “Free space” → Edit → Set partition table
  3. Set it to GPT
  4. Click on disk 2
  5. Right click on “Free space” → Edit → Set partition table
  6. Set it to GPT

Create the partitions

Note: One of the problems that we will run into is that once we create the raid device is that we wont be able to just say “use the new raid device and do your automatic partitioning thing using that”. We will have to make partitions manually.

Create the MBR partition (optional)

The first 1 meg on a drive should normally be reserved for MBR related things. You may end up needing this space if UEFI does not go well for you.
Some partition editors automatically reserve the first meg for you in some way. The current version of the blivet-gui partition editor that Qubes installer uses appears to set the default for the starting block of new partitions to be after the first meg, so as long as you leave the defaults, you don’t need to create a 1 meg partition for MBR (and then do nothing with it, just so nothing else uses it)

Create the EFI filesystem

  1. Click on disk 1

  2. Right click “Free space” → New

  3. Set the following:

    Field Setting
    Device type partition
    size 1 Gib
    Filesystem “EFI System partition”
    label EFI
    mountpoint /boot/efi
    encryption no. do not encrypt the EFI partitions
  4. Then do the same for the 2nd drive, but dont give it a mountpoint)
    (It would be neat to mount it to /boot/efi2, but during the attempt that actually worked i did not have a mountpoint set)

Create the boot filesystem

  1. Click on empty space of one of the drives

  2. Right click → new

  3. Select type “software raid”

  4. See the list of drives spring up and add the 2nd drive

  5. Set raid level to “raid1”

  6. Set the following:

    Field Setting
    Device type partition
    size 2 Gib
    Filesystem “ext4”
    label boot
    name boot
    mountpoint /boot
    encryption no. do not encrypt the boot partition

Create the root filesystem

  1. Click on empty space of one of the drives

  2. Right click → new

  3. Select type “software raid”

  4. See the list of drives spring up and add the 2nd drive

  5. Set raid level to “raid1”

  6. Set the following:

    Field Setting
    Device type partition
    size Take all the space
    Filesystem “physical volume LVM”
    name set name to a random string
    encryption yes! if you want encryption, enable it here

Create the volume group

  1. Right click on empty space of the new “luks raid physical volume” → new
  2. Set the name to “qubes_dom0”

Create the swap volume

  1. Right click on empty space of the new volume group → new

  2. Set the following:

    Field Setting
    Device type “LVM2 Logical Volume”
    size 10 Gib
    Filesystem swap
    label swap
    name swap
    encryption no. do not encrypt (already encrypted)

Create the thinpool

  1. Right click on empty space of the new volume group → new
  2. Select device type: “LVM2 ThinPool”
  3. Set size to: take it all
  4. Set name to “pool00”
  5. do not encrypt (already encrypted)

Create the root volume

  1. Right click on new lvmthinpool → New

  2. Set the following:

    Field Setting
    Device type “LVM2 Logical Volume”
    size take it all
    Filesystem ext4
    label root
    name root
    mountpoint /
    encryption no. do not encrypt (already encrypted)

At this point click “done”. it should complain at the bottom of the screen. You can view the issues, and it should say something like: “/boot” is on a RAID volume, but EFI is not! If your boot drive goes down you wont be able to boot.

EFI filesystem is not intended for RAID, so thats not a real option. And undoing raid for /boot won’t help the situation. So you can ignore that.

However, if it says something else, like 'you forgot to put a root mountpoint for “/” ', then you should go fix that.

After all problems are resolved:

  1. Click “done”
  2. If nothing happens, Click “done” again
  3. Start the installation

WARNING:

  1. It did not save the partitions you just set up. If you reboot now it will be gone.
  2. Going back into the “Installation Device” will clear out all the work you just did.

If it breaks during the install (or if you did it the CLI way)

If it breaks during the install, it probably write the partitions, so after the reboot:

  1. Hit control-alt-F2
  2. If you can’t read the screen because the text is too small on a high-res screen, type: setfont solar24x32{enter}

Note: Many other CLI instructions that can be typed here are available from Custom installation | Qubes OS

  1. cat /proc/mdstat to view what raid drives are currently active
    (it will say none are active)
  2. do: mdadm --assemble --scan
    (hopefully it will find the raid drives)
  3. cat /proc/mdstat to view what raid drives are currently active
    (this time it should show both as active)
  4. Hit Alt-F6 (to get back to the graphical screen)

Now the fun part. If you hit “rescan” to get the graphical partitioner to acknowledge the RAID drives, the “rescan” process will deactivate the RAID drives!

Basically switch back and fourth between doing “mdadm --assemble --scan” and being in different partitioning screens until something works. I believe what worked for me was doing “mdadm --assemble --scan” after clicking “Installation Device” and before clicking “advanced custom - blivet-gui”, then “Done” (to get to the advanced screen). Whatever I did, when i got to blivet-gui, it recognized the raid disks and I was able to reassign the mount points and then start the install (which worked this time)

After install works and it tries to reboot into the new qubes system

You’ll get the “out of box” configuration options. Under “Advanced configuration” select “use existing LVM thin pool” with “qubes_dom0” (the only option) and LVM thin pool “Pool00” (the only option).

This might mean we did not need to manually create the thin-pool lvm thing from the partition editor??

CLI Software Raid installation procedure:

This section is not complete. For now, see Custom installation | Qubes OS for ideas

Aftermath

Things that would be cool

Once it’s installed, it would be cool to mount the 2nd efi partition as /boot/efi2 and adjust the grub installer to install to both partitions. As it stands, if your boot drive goes out, then you will have to boot to a rescue disk, set up the EFI partition (which you have already allocated) and install grub on the other disk

1 Like

Dear @ddevz ,
After giving your post a well deserved :heart:, I decided I must thank you “in person” since your guide just saved me a HUGE headache trying to figure out how to install Qubes with RAID1. Thank you, Thank you, Thank you :heart_eyes:
ps. the installer use the abbreviation “Gib”, not Gig"

@redmind You are welcome! It’s good to know people get actual use out of what I write :slight_smile:

(also, changed s/Gig/Gib/)

Something strange during this part:

after step 1, it shows that we can choose anywhere between 0G and 230G. However, after setting type “LVM2 ThinPool”, it suddenly says it can only be between 0G and 190G (or something like that). After creating the maximum thinpool (190G), it says there is still 40G of free space. I can create another thinpool, but that still leaves space free.

Anyone know what’s going on here?

Actually, anyone know why root is thin provisioned to take everything? (hopefully dom0 runs trim on root regularly). (I did it that way cause that’s how qubes installer with default options seems to do it)

@ddevz I had that happen to me as well, but after trying a few times and making sure it wasn’t something I did wrong, I figured this space must be reserved on purpose and just moved on. If anyone more knowledgeable than myself can point to the explanation it would be nice.