ABSTRACT: When verifying integrity of a 200GB Qubes OS backup on USB SSD, performance degradation occurs due to at least 200GB HDD writes.
A Qubes OS VM backup file in versions R4.0, R4.1, and R4.2 is essentially a tarball of encrypted compressed VM segments.
Recently, I encountered a situation where I needed to back up my laptop running Qubes OS, which had a slow HDD. I expected the backup process is slow, since the backup data must be read from my hard disk. Nevertheless I decided to use a brand new fast USB SSD as my backup medium hoping that backup process is stable.
Upon completing the backup, I decided to test the integrity of the backup file using the dry run option (“Verify backup integrity, do not restore the data”), which verifies the backup without restoring the data. To my surprise, this dry run option significantly utilized my HDD and was also slow in operation.
After some investigation, I discovered that the restore logic, whether dry or not, operates as follows based on my observations in the dom0 environment:
- A specific directory is designated to store temporary data (dom0:
/var/tmpin R4.0~R4.1, dom0:/home/${USER}/QubesIncomingin R4.2). - Thread A: The backup file is untarred, and the backup segments are sequentially dumped onto the specified directory. If the filesystem where the directory resides is close to being full (around 75% capacity), the process waits for a while and retries. Otherwise, it continues to retrieve data.
- Thread B, C, and so on: These threads decrypt and unzip each segment, subsequently removing them.
The issue lies in the fact that both the encrypted and plaintext scratch data ends up stored on the hard disk. Notably, step 2 continuously writes to the dom0 disk (in the default setup) until the disk reaches approximately 75% capacity. This continuous writing causes a flood of write operations that may never end in any foreseeable future. Probably only after the entire backup data is read into the dom0 root will the dom0 kernel schedule the gunzip+scrypt+dd operations, provided the user is patient enough.
During the actual restore process, on an SSD, I guess this problem result in a 3x write magnification (cipherdata, plaindata, and VM disk). On an HDD (which I am facing), the endless write flood directly renders the system unresponsive. Additionally, the costs of LUKS encryption, LVM/BTRFS CoW (Copy-on-Write), and compression should not be overlooked.
To restore system responsiveness, I did a small workaround by mounting a tmpfs filesystem onto the backup directory. The following commands demonstrate how I achieved this:
(For R4.1)
sudo mount -t tmpfs tmpfs1 /var/tmp
(For R4.2)
sudo mount -t tmpfs tmpfs1 /home/${USER}/QubesIncoming
Nevertheless, it does not make sense for the backup segments to be stored on disk, considering both performance and security considerations. It appears to be an oversight in the implementation of the backup restorer that the backup segments are stored on the dom0 disk in the first place.
By the way, from a security perspective, it would be more appropriate for the scrypt+gzip process to be performed in a management VM (default-mgmt-vm?). I am uncertain whether there are plans to refactor this aspect.