I’m currently evaluating the possibility of moving my workstation to Qubes OS and ran into a problem.
It’s using an SSD for Qubes OS, but I want to use HDDs with RAID6 to hold qubes data.
I’ve been using the secondary storage guide and slightly altered it for RAID6:
I ran “sudo cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdx” for 5 test HDDs.
Then I added their UUID to /etc/crypttab and rebooted.
After than I ran “sudo pvcreate /dev/mapper/luks-[UUID]” for the 5 drives.
I created a volume group with “sudo vgcreate qubeshd0 /dev/mapper/luks-UUID1 /dev/mapper/luks-UUID2 …”
Then I did this:
sudo lvcreate -i 3 --type raid6 -L 10G -n poolhd0 qubeshd0
sudo lvconvert --thinpool qubeshd0/poolhd0
sudo lvextend -l +100%FREE qubeshd0/poolhd0
This seems to give me a RAID6 thin pool with the correct size. I created VMs and used this pool for data storage and everything seems to work fine. So far so good.
After that I wanted to go through some other scenarios like adding HDDs to the RAID (growing it) or replacing faulty hdds.
I was able to add a new HDD to the volume group but after that I got stuck, since I was unable to add the new free space to the RAID6.
Everything I tried gave me an error. One of them was that thin pool did not support the operation. I also tried it without converting to thin pool but still was unable to extend the RAID6.
Things like this are the reason why I wanted to simulate/evaluate everything first before actually moving everything.
How can I extend the 5 hdd RAID6 to 6 hdds (and higher)? Over time the final target would be to grow the RAID to around 10-12 HDDs.
Was that even correct how I implemented it? What would be the best way to accomplish it?
I'm currently evaluating the possibility of moving my workstation to Qubes OS and ran into a problem.
It's using an SSD for Qubes OS, but I want to use HDDs with RAID6 to hold qubes data.
I've been using the secondary storage guide and slightly altered it for RAID6:
I ran "sudo cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdx" for 5 test HDDs.
I'm not an expert on this, but why do you cryptsetup each HDD instead of cryptsetup the RAID device? The data on HDD still should be encrypted. 
Then I added their UUID to /etc/crypttab and rebooted.
After than I ran "sudo pvcreate /dev/mapper/luks-[UUID]" for the 5 drives.
I created a volume group with "sudo vgcreate qubeshd0 /dev/mapper/luks-UUID1 /dev/mapper/luks-UUID2 ...."
Then I did this:
sudo lvcreate -i 3 --type raid6 -L 10G -n poolhd0 qubeshd0
sudo lvconvert --thinpool qubeshd0/poolhd0
sudo lvextend -l +100%FREE qubeshd0/poolhd0
This seems to give me a RAID6 thin pool with the correct size. I created VMs and used this pool for data storage and everything seems to work fine. So far so good.
After that I wanted to go through some other scenarios like adding HDDs to the RAID (growing it) or replacing faulty hdds.
I was able to add a new HDD to the volume group but after that I got stuck, since I was unable to add the new free space to the RAID6.
You want to extend the RAID, , i.e. the PV, not add a single HDD as PV, right?
Everything I tried gave me an error. One of them was that thin pool did not support the operation. I also tried it without converting to thin pool but still was unable to extend the RAID6.
I think you must reserve "RAID slots" (number of disks to expect" when creating the RAID. Maybe you can tune that later, but I'm unsure.