I see comments that people are using software raid (md rescue disk, set up the EFI partition (which you have already allocated) and install grub on thadm), but I don’t see any good instructions for how to do it. I just finished a software raid install, so I’m putting instructions here for the next person.
Notes:
- Please mention any information/comments you have to add to this procedure
- This procedure is written for qubes 4.2 installs
- This procedure will assume you have 2 hard drives. If you have more, then whenever it says to do something “for the 2nd hard drive”, do it for each of the hard drives that are left
- This procedure assumes the drives are blank, and do not already have a partition table
- The graphical install procedure seems to fully work for the stated case, but this will always be a work in progress as i’d like people to add information that can help optimize the procedure, and there is always more that can be added (like adding a real /boot/efi2)
What is mdadm?
There are 3 known ways to do software raid on qubes.
- Mdadm (the traditional way)
- zfs (a really cool way, with a strange licensing quirk)
- and btrfs (tries to do zfs like things, but without the licensing quirk).
- This article documents the mdadm way.
If you want to try the zfs way, you can start by looking here: ZFS in Qubes OS
If you want to try the btrfs way, you can try starting here: BTRFS redundant disk setup (RAID alternative)
Graphical Software Raid installation procedure:
- Do the standard steps listed in Installation guide | Qubes OS , up to the part about configuring the “Installation Destination” (after the anchor: Installation guide | Qubes OS)
- Select “Installation Destination”
- Select all media
- Select “advanced custom Blivet-GUI”
- Click “Done”
NOTE: There is the CLI way to set up the partitions, and the graphical way to set up the partitions. Below is the graphical way, but see “If it breaks during the install” section at the end of this document for the beginning of the discussion of CLI related ways of doing things.
Create the (blank) partition tables
- Click on disk 1
- Right click on “Free space” → Edit → Set partition table
- Set it to GPT
- Click on disk 2
- Right click on “Free space” → Edit → Set partition table
- Set it to GPT
Create the partitions
Note: One of the problems that we will run into is that once we create the raid device is that we wont be able to just say “use the new raid device and do your automatic partitioning thing using that”. We will have to make partitions manually.
Create the MBR partition (optional)
The first 1 meg on a drive should normally be reserved for MBR related things. You may end up needing this space if UEFI does not go well for you.
Some partition editors automatically reserve the first meg for you in some way. The current version of the blivet-gui partition editor that Qubes installer uses appears to set the default for the starting block of new partitions to be after the first meg, so as long as you leave the defaults, you don’t need to create a 1 meg partition for MBR (and then do nothing with it, just so nothing else uses it)
Create the EFI filesystem
-
Click on disk 1
-
Right click “Free space” → New
-
Set the following:
Field Setting Device type partition size 1 Gib Filesystem “EFI System partition” label EFI mountpoint /boot/efi encryption no. do not encrypt the EFI partitions -
Then do the same for the 2nd drive, but instead of /boot/efi as the mountpoint, use /boot/efi2 as the mountpoint
(Note: This procedures still does not use /boot/efi2 for anything, but maybe someday we can have it set up /boot/efi2 so the 2nd drive can be booted from as well (I.E. without a rescue disk))
Create the boot filesystem
-
Click on “Free space” of one of the drives
-
Right click → new
-
Select type “software raid”
-
See the list of drives spring up and add the 2nd drive
-
Set raid level to “raid1”
-
Set the following:
Field Setting Device type Software RAID size 2 Gib Filesystem “ext4” label boot name boot mountpoint /boot encryption no. do not encrypt the boot partition -
Click “OK” when done
Create the root filesystem
-
Click on “Free space” of one of the drives
-
Right click → new
-
Select type “software raid”
-
See the list of drives spring up and add the 2nd drive
-
Set raid level to “raid1”
-
Set the following:
Field Setting Device type Software RAID size Take all the space Filesystem “physical volume LVM” name pv-encrypted encryption yes! if you want encryption, enable it here
Create the volume group
- On the left hand side, under “RAID” (instead of under "Disks), select the new pv-encrypted device
- Right click in the area to the right that normally said “free space” of the new pv-encrypted volume → new
- Set the name to “qubes_dom0”
Create the swap volume
-
Right click on empty space of the new volume group → new
-
Set the following:
Field Setting Device type “LVM2 Logical Volume” size 10 Gib Filesystem swap label swap name swap encryption no. do not encrypt (already encrypted)
Create the thinpool
(Note: if we do not create the thinpool, do we not have to select “use pre-existing pool” at the end and it automatically creates one? I have no idea but itd be interesting to investigate)
- Right click on empty space of the new volume group → new
- Select device type: “LVM2 ThinPool”
- Set size to: take it all
- Set name to “pool00”
- do not encrypt (already encrypted)
Create the root volume
-
Right click on new lvmthinpool → New
-
Set the following:
Field Setting Device type “LVM2 Thin Logical Volume” size take it all Filesystem ext4 label root name root mountpoint / encryption no. do not encrypt (already encrypted)
At this point click the “done” button in the upper left hand corner. it should complain at the bottom of the screen. You can view the issues, and it should say something like: “/boot” is on a RAID volume, but EFI is not! If your boot drive goes down you wont be able to boot.
EFI filesystem is not intended for RAID, so thats not a real option. And undoing raid for /boot won’t help the situation. So you can ignore the message.
However, if it says something else, like 'you forgot to put a root mountpoint for “/” ', then you should go fix that.
After all problems (other then the one you are ignoring) are resolved:
- Click “done”
- If nothing happens, Click “done” again
- Start the installation
WARNING:
- It did not save the partitions you just set up. If you reboot now it will be gone.
- Going back into the “Installation Device” will clear out all the work you just did.
- You are not done! The installer will install stuff, then when it reboots you will still need to do one last thing! (shown in the very next line on this page)
- If it breaks during install, jump to section “If it breaks during the install”
After install works and it tries to reboot into the new qubes system
You’ll get the “Out Of Box” configuration options. Under “Advanced configuration” select “use existing LVM thin pool” with “qubes_dom0” (the only option) and LVM thin pool “Pool00” (the only option).
This might mean we did not need to manually create the thin-pool lvm thing from the partition editor??
If it breaks during the install (or if you did it the CLI way)
If it breaks during the install, it probably write the partitions, so after the reboot:
- Hit control-alt-F2
- If you can’t read the screen because the text is too small on a high-res screen, type: setfont solar24x32{enter}
Note: Many other CLI instructions that can be typed here are available from Custom installation | Qubes OS
- cat /proc/mdstat to view what raid drives are currently active
(it will say none are active) - do: mdadm --assemble --scan
(hopefully it will find the raid drives) - cat /proc/mdstat to view what raid drives are currently active
(this time it should show both as active) - Hit Alt-F6 (to get back to the graphical screen)
Now the fun part. If you hit “rescan” to get the graphical partitioner to acknowledge the RAID drives, the “rescan” process will deactivate the RAID drives!
Basically switch back and fourth between doing “mdadm --assemble --scan” and being in different partitioning screens until something works. I believe what worked for me was doing “mdadm --assemble --scan” after clicking “Installation Device” and before clicking “advanced custom - blivet-gui”, then “Done” (to get to the advanced screen). Whatever I did, when i got to blivet-gui, it recognized the raid disks and I was able to reassign the mount points and then start the install (which worked this time)
CLI Software Raid installation procedure:
This section is not complete. For now, see Custom installation | Qubes OS for ideas
Aftermath
How to tell if a drive fails
You can type cat /proc/mdstat to check if both drives are working. (Note: there is a good chance it’s still syncing the drive at this moment)
You will probably want to write a script that checks it and notifies you if there is a problem, and then put it the script in cron. something like this:
NUMBER_OF_RAIDS=2
#NOTE: You can run a test of this, to make sure nootifications are working by setting NUMBER_OF_RAIDS to more then you have
if [ $NUMBER_OF_RAIDS != `grep '[[]UUU*[]]' /proc/mdstat | wc -l` ]
then
notify-send --expire-time=360000 "RAID issue!" "A drive may have failed, or other raid issue. (or you have NUMBER_OF_RAIDS set wrong). do: cat /proc/mdstat to see details"
exit
fi
exit
Things that would be cool
Once it’s installed, it would be cool to mount the 2nd efi partition as /boot/efi2 and adjust the grub installer to install to both partitions. As it stands, if your boot drive goes out, then you will have to boot to a rescue disk, set up the EFI partition (which you have already allocated) and install grub on the other disk
, I decided I must thank you “in person” since your guide just saved me a HUGE headache trying to figure out how to install Qubes with RAID1. Thank you, Thank you, Thank you 
