Unable to get fully functional system when installing Qubes 4.1 on a RAID1

scallyob · April 9, 2022, 3:40am

My first attempts at installing Qubes 4.1 on a RAID1 left me with a system where I could not run backups while any of the qubes being backed up were running. This was a bug documented here:

github.com/QubesOS/qubes-issues

Qubes Backup hangs and never finishes if any of the target qubes are running

opened 05:00PM - 02 Apr 22 UTC

scallyob

T: bug C: installer P: major diagnosed C: storage

[How to file a helpful issue](https://www.qubes-os.org/doc/issue-tracking/) #…## Qubes OS release 4.1rc3 and 4.1 final release ### Brief summary Qubes Backup hangs after creating 30K file if qube being backed up is still running. If multiple qubes being backed up and some are running it will create a bigger file (presumably of the non-running ones) before hanging. ### Steps to reproduce 1. Create a new standalone qube based on fedora-34 called *backuptest* 2. Start *backuptest* 3. Run Qubes Backup 4. select just *backuptest* to be backedup 5. "Compress backup" is checked 6. click "Next>" 7. Select external drive and directory 8. set password for encryption 9. uncheck save settings as default backup profile 10. leave turn computer off unchecked 11. click "Next>" 12. it warns you that it will backup the state prior to starting the qube 13. click the button to start backup 14. watch the file in terminal ### Expected behavior If I shutdown *backuptest* and run the steps above I see the progress bar move on the Backup tool until it reaches 100%. This creates a 3.7GB file that I can verify with the Restore Backup tool. I would expect these same results when *backuptest* is running. ### Actual behavior progress bar stays at 0% in Backup tool file never exceeds 30K and timestamp doesn't change after that in Terminal backup never completes, have to cancel the backup tool

The solution proposed there was to try BTRFS instead of EXT4 for the file system for root.

Took me a while to figure this out in the installer, so I will document the steps I used to do this:

on the “Device Selecting” screen I checked the two 1TB drives I had used for my RAID1 install
I clicked “Custom” for the partitioning
In “New Qubes OS R4.1.0 Installation” I left the existing EXT4 RAID1 partition for the boot and mapped it to /boot and deleted the other EXT4 RAID1 partition for root that was most of the disk
I clicked “+” to add a partition
set mount point as “/” (root) and capacity as 920GB (for a 1TB drive) and click “Add”
on the right hand portion of screen I select Device Type = BTRFS
click the “Modify” button (this was the secret that took me a while to figure out)
In the new pop-up; I checked “encrypt”, select Raid Level=RAID1 and select Size Policy= “as large as possible”

After that I re-installed Qubes OS 4.1

When I run
dom0$] lsblk

It shows /dev/sda1 and /dev/sdb1 as MOUNTPOINT = /boot
However, while /dev/sda2 MOUNTPOINT = /
/dev/sdb2 does not show a MOUNTPOINT

If I mount /dev/sdb2 it appears to be a mirror of /dev/sda2. Maybe RAID1 is working? but lsblk does not indicate that it is a RAID.

While I was happy to find that backup tool now works as expected. My system load is constantly getting very high. I am referring to “Load: 2.52” in the bottom bar on i3. From what I read this indicates the wait for CPU resources.

If I run ‘xentop’ in dom0 my CPU% use looks similar to what it was on EXT4. But the “load” is constantly ramping up in the 2.5-5 range, and often into the double digits if starting or stopping a qube. This means updates and backups are a real drag on the system. Conducting normal operations that were no problem on EXT4 lead to things like music player pausing every few minutes and slow scrolling on browser windows in BTRFS.

Most of what I’ve read about BTRFS and Qubes is that it improves performance.
However, in the link to github above, DemiMarie mentions that " I/O performance can be unstable" with BTRFS. Perhaps this is what they were referring to?

Any suggestions for a more usable Qubes install with RAID1 redundancy?

fepitre · April 9, 2022, 9:30am

Just a feedback, I’ve installed Qubes OS on a server with two NVMe and did a software RAID1. I admit I had to fight a bit with the installer and we definitively have some bugs. I’ve not selected BTRFS but standard ext4 for the root volume. I think this is a setup we need to implement and validate because some of those bugs prevent me to install Qubes on another machine. On other servers, I’m relying on hardware RAID card directly.