Qubes in tmpfs 🤫

qstateless · November 7, 2023, 2:52pm

News… Qubes Stateless (RAM-based Qubes in tmpfs/zram, like Tails OS) is working in Qubes 4.2.

Easy and full Step-by-Step instructions provided here in this post to implement a basic Qubes Stateless mode in 4.1 or 4.2. Super credit to @xuy!

Qubes Stateless: Overview...

Qubes Stateless runs your entire Qubes Dom0 environment as Disposable / Amnesic / Ephemeral / Anti-Forensic in RAM, where, unless you purposefully save things in specific alternative ways, your Qubes environment may reset your changes upon restart. This is similar to how Tails OS works, but with the more powerful security & capabilities of Qubes OS.

At each boot, you have the option to run your Dom0 session in Stateless mode or Persistent (normal) mode. Qubes Stateless runs Dom0 from RAM, and you can optionally create and run other user qubes in RAM using the “varlibqubes” storage pool. Your persistent boot storage drive and other attached persistent storage drives will also be available and writable to you during a Stateless session, including any pre-existing user qubes stored on these drives.

Qubes Stateless is an “advanced hybrid state system” where you have both Stateless and Persistent spaces to work with in the same OS environment, and you can optionally control where individual qube volumes operate from (Stateless storage or Persistent storage). This gives us great power and flexibility for controlling the level of statelessness of our qubes.

For those wondering how much hardware system RAM is required to run Qubes Stateless, I’d say 8GB minimum to work and 16GB for basic comfort. Here are my general feelings on amounts of RAM:

8GB RAM = Minimal Running
16GB RAM = Basic Comfort - A basic Qubes Stateless system can run comfortably on 16GB RAM if number and intensity of AppQubes remains towards lower end.
32GB RAM = Extra Comfort - Only needed if your number or intensity of AppQubes demands more.
64GB RAM = Luxury Comfort - Only needed if your number or intensity of AppQubes demands more.

Below are the exact step-by-step commands & instructions for a basic RAM-based Qubes Stateless, primarily derived from @xuy’s posts…

These steps implement the following aspects & components:

Basic RAM-based Qubes from xuy’s post #1.
Dracut Automation Module from xuy’s post #13.
Disable Dom0 Swap Permanently in /etc/fstab.
Disable Dom0 Swap Temporarily from xuy’s post #18.
ZRAM Compression from xuy’s post #28.
Unified grub.cfg location for both UEFI and legacy boot (#7985) → R4.2 Release Notes.
Minor command & code style adjustments for greater newbie ease.

Qubes Stateless: Step-by-Step…

WARNING: Although these Qubes Stateless instructions are made easy enough for almost any Qubes user to follow, Qubes Stateless does involve the modification of your Dom0 system files. Therefore, as a precaution, it is highly recommended that you first perform a full backup of your qubes and data before proceeding, in case your system accidentally gets corrupted or won't boot. No warranty is provided.

In Dom0 of Qubes OS 4.1 or 4.2, open a command line Dom0 Terminal, and perform the following commands & instructions:

+ Step 0. Make a Save Qube:

Optional (Recommended):

qvm-create -l blue --prop=netvm= save

Further explanation:

To give yourself a convenient option for permanently saving data files while operating in RAM-based Stateless mode, I suggest you create a persistent vault AppQube named “save”, so that you can easily copy/move files that you want to save, to the “save” qube, before your computer shuts down, restarts, crashes, etc, and you lose important files while working in the RAM-based Stateless mode.

If you create a “save” qube while in the normal persistent boot mode, then this “save” qube will be available and permanently retain any files sent to it, on your Qubes storage drive, even before, during, and after you are in the RAM-based Stateless mode.

WARNING: If you don’t understand what you’re doing with an advanced RAM-based stateless configuration of Qubes OS, then it can be very easy to lose data. Ensure that you understand and get familiar with (by testing) the different modes and data saving capabilities of this RAM-based system, and take extra precaution to setup and test reliable methods for saving your important data beyond a stateless session.

+ Step 1. Disable Dom0 Swap:

sudo nano /etc/fstab

Using nano, type a # character in front of the line containing “/dev/mapper/qubes_dom0-swap”, to comment it out and disable it, like this:

#/dev/mapper/qubes_dom0-swap ...

Press Ctrl + O to save fstab file.
Press Ctrl + X to exit nano editor.

That will permanently disable your Dom0 Swap after restart, which is likely best for operating Qubes Dom0 in Stateless mode.

If you still want to frequently use Qubes in Persistent mode and have Dom0 Swap enabled, then you can alternatively skip this permanent disable step, and rather temporarily disable Dom0 Swap when you startup Qubes Stateless by remembering (each and every time) to manually run the following Dom0 command…

sudo swapoff -a

+ Step 2. Edit GRUB to Increase Dom0's Max RAM:

sudo nano /etc/default/grub

Within the GRUB_CMDLINE_XEN_DEFAULT line, make a partial modification of this line, from dom0_mem=max:4096M to now be dom0_mem=max:10240M → Just change the text “4096M” to “10240M” (without quotes).

Press Ctrl + O to save grub file.
Press Ctrl + X to exit nano editor.

Note: If you have something like 32GB of RAM or greater in your system, and you want to run several fully stateless user qubes within Dom0 RAM space, then you may want to consider increasing this “10240M” Dom0 RAM maximum value, in order to allow for more Dom0 RAM space where you can store more user qubes as fully stateless. For example, with 32GB of system RAM, you may choose to allocate “20480M” (20GB) or even greater if you wish.

+ Step 3. Regenerate with New GRUB Configuration:

If you are using Qubes 4.2, you should run this single command:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

If you are using Qubes 4.1, you should instead run these two commands:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo grub2-mkconfig -o /boot/efi/EFI/grub.cfg

+ Step 4. Make New Directory for Dracut Automation Module:

sudo mkdir /usr/lib/dracut/modules.d/01ramboot

+ Step 5. Make New Dracut Script File module-setup.sh:

sudo touch /usr/lib/dracut/modules.d/01ramboot/module-setup.sh
sudo nano /usr/lib/dracut/modules.d/01ramboot/module-setup.sh

Edit the newly created module-setup.sh file and type-in exactly the following 11 lines of code:

#!/usr/bin/bash
check() {
return 0
}
depends() {
return 0
}
install() {
inst_simple "$moddir/tmpfs.sh" "/usr/bin/tmpfs"
inst_hook cleanup 00 "$moddir/pass.sh"
}

Press Ctrl + O to save module-setup.sh file.
Press Ctrl + X to exit nano editor.

Perform the following verification steps for your module-setup.sh file:

a. Verify that your module-setup.sh file has the exact correct contents with a file hash checksum…

sudo sha256sum /usr/lib/dracut/modules.d/01ramboot/module-setup.sh

…should show output of:

cb3e802e9604dc9b681c844d6e8d72a02f2850909ede9feb7587e7f3c2f11b8a /usr/lib/dracut/modules.d/01ramboot/module-setup.sh

If this output shows as incorrect, run the previous nano edit commands to correct the file contents and then re-verify this file again. Repeat until correct. Note that only an exact duplication, character by character, of the same file contents from these instructions will produce a successful file hash match, where even extra blank spaces or extra blank lines will cause a changed file hash mismatch. Being very precise counts!

b. Verify that your module-setup.sh file is owned by root…

sudo ls -l /usr/lib/dracut/modules.d/01ramboot/module-setup.sh

…should show output of: “root root” for /usr/lib/dracut/modules.d/01ramboot/module-setup.sh.

If this output shows as incorrect, run the following command and then re-verify this file again…

sudo chown root:root /usr/lib/dracut/modules.d/01ramboot/module-setup.sh

+ Step 6. Make New Dracut Script File pass.sh:

sudo touch /usr/lib/dracut/modules.d/01ramboot/pass.sh
sudo chmod 755 /usr/lib/dracut/modules.d/01ramboot/pass.sh
sudo nano /usr/lib/dracut/modules.d/01ramboot/pass.sh

Edit the newly created pass.sh file and type-in exactly the following 6 lines of code:

#!/usr/bin/bash
command -v ask_for_password >/dev/null || . /lib/dracut-crypt-lib.sh
PROMPT="Boot to RAM? (y/n)"
CMD="/usr/bin/tmpfs"
TRY="3"
ask_for_password --cmd "$CMD" --prompt "$PROMPT" --tries "$TRY" --ply-cmd "$CMD" --ply-prompt "$PROMPT" --ply-tries "$TRY" --tty-cmd "$CMD" --tty-prompt "$PROMPT" --tty-tries "$TRY" --tty-echo-off

Press Ctrl + O to save pass.sh file.
Press Ctrl + X to exit nano editor.

Perform the following verification steps for your pass.sh file:

a. Verify that your pass.sh file has the exact correct contents with a file hash checksum…

sudo sha256sum /usr/lib/dracut/modules.d/01ramboot/pass.sh

…should show output of:

a2750fa31c216badf58d71abbc5b92097e8be21da23bbae5779d9830e2fdd144 /usr/lib/dracut/modules.d/01ramboot/pass.sh

If this output shows as incorrect, run the previous nano edit commands to correct the file contents and then re-verify this file again. Repeat until correct. Note that only an exact duplication, character by character, of the same file contents from these instructions will produce a successful file hash match, where even extra blank spaces or extra blank lines will cause a changed file hash mismatch. Being very precise counts!

b. Verify that your pass.sh file is owned by root and is executable…

sudo ls -l /usr/lib/dracut/modules.d/01ramboot/pass.sh

…should show output of: “-rwxr-xr-x” and “root root” for /usr/lib/dracut/modules.d/01ramboot/pass.sh.

If this output shows as incorrect, run the following commands and then re-verify this file again…

sudo chown root:root /usr/lib/dracut/modules.d/01ramboot/pass.sh
sudo chmod 755 /usr/lib/dracut/modules.d/01ramboot/pass.sh

+ Step 7. Make New Dracut Script File tmpfs.sh:

sudo touch /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh
sudo chmod 755 /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh
sudo nano /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh

Edit the newly created tmpfs.sh file and type-in exactly the following 21 lines of code:

#!/usr/bin/bash
read line
case "${line:-Nn}" in
[Yy]* )
mkdir /mnt
umount /sysroot
mount /dev/mapper/qubes_dom0-root /mnt
modprobe zram
echo 10G > /sys/block/zram0/disksize
/mnt/usr/sbin/mkfs.ext2 /dev/zram0
mount /dev/zram0 /sysroot
cp -a /mnt/* /sysroot
exit 0
;;
[Nn]* )
exit 0
;;
* )
exit 1
;;
esac

Press Ctrl + O to save tmpfs.sh file.
Press Ctrl + X to exit nano editor.

Perform the following verification steps for your tmpfs.sh file:

a. Verify that your tmpfs.sh file has the exact correct contents with a file hash checksum…

sudo sha256sum /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh

…should show output of:

d9e85c06c3478cc0cf65a4e017af1a4f9f9dd4ad87c71375e8d4604399f5217d /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh

If this output shows as incorrect, run the previous nano edit commands to correct the file contents and then re-verify this file again. Repeat until correct. Note that only an exact duplication, character by character, of the same file contents from these instructions will produce a successful file hash match, where even extra blank spaces or extra blank lines will cause a changed file hash mismatch. Being very precise counts!

b. Verify that your tmpfs.sh file is owned by root and is executable…

sudo ls -l /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh

…should show output of: “-rwxr-xr-x” and “root root” for /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh.

If this output shows as incorrect, run the following commands and then re-verify this file again…

sudo chown root:root /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh
sudo chmod 755 /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh

+ Step 8. Make New Dracut Config File ramboot.conf:

sudo touch /etc/dracut.conf.d/ramboot.conf
sudo nano /etc/dracut.conf.d/ramboot.conf

Edit the newly created ramboot.conf file and type-in exactly the following 2 lines of code:

add_drivers+=" zram "
add_dracutmodules+=" ramboot "

Press Ctrl + O to save ramboot.conf file.
Press Ctrl + X to exit nano editor.

Perform the following verification steps for your ramboot.conf file:

a. Verify that your ramboot.conf file has the exact correct contents with a file hash checksum…

sudo sha256sum /etc/dracut.conf.d/ramboot.conf

…should show output of:

60d69ee8f27f68a5ff66399f63a10900c0ea9854ea2ff7a77c68b2a422df4bef /etc/dracut.conf.d/ramboot.conf

If this output shows as incorrect, run the previous nano edit commands to correct the file contents and then re-verify this file again. Repeat until correct. Note that only an exact duplication, character by character, of the same file contents from these instructions will produce a successful file hash match, where even extra blank spaces or extra blank lines will cause a changed file hash mismatch. Being very precise counts!

b. Verify that your ramboot.conf file is owned by root…

sudo ls -l /etc/dracut.conf.d/ramboot.conf

…should show output of: “root root” for /etc/dracut.conf.d/ramboot.conf.

If this output shows as incorrect, run the following command and then re-verify this file again…

sudo chown root:root /etc/dracut.conf.d/ramboot.conf

+ Step 9. Regenerate with New Dracut Automation Module:

sudo dracut --verbose --force

This command may take a couple minutes to complete. When done, the last line will read something like: "dracut: *** Creating initramfs image file '/boot/initramfs-6.1.56-1.qubes.fc37.x86_64.img' done ***".

+ Step 10. Shutdown, Restart, Test Qubes Stateless:

Shutdown & Restart your computer. You are ready to test Qubes Stateless!

After normally decrypting your Qubes boot drive, you should be prompted with:

Boot to RAM? (y/n):

You can enter “y” (without quotes) to boot & use your Qubes Dom0 to Stateless mode in RAM. Your pre-existing persistent TemplateQubes, AppQubes, Data Files, etc, will also be accessible to you while operating in Stateless mode. Note that it can take a few minutes to load Qubes Dom0 into RAM once you enter “y”.
You can enter “n” (without quotes) to boot & use your Qubes Dom0 normally to Persistent mode from your storage drive. Here you can set and save any permanent state changes you’d like for the next time you boot to Stateless mode. For example, use this normal Persistent mode for: installing Qubes OS updates, installing/configuring TemplateQubes, creating new persistent AppQubes, ServiceQubes, etc, where your next boot into Stateless mode can then take advantage of such persistent changes, improvements, updates, etc.

Notes on Using Qubes Stateless:

Booting into Persistent mode is where you can make any permanent changes/settings and updates to your Dom0/TemplateQubes, etc, which will then be available in your next Stateless mode session.

You’ll most likely want to keep your Qubes OS Xen, Dom0, and TemplateQubes up to date by semi-regularly restarting into Persistent mode and installing updates there.

Booting into Stateless mode is where you can operate a stateless Dom0 environment in RAM (which resets its state, upon shutdown, to your last saved Persistent state) and where you also have simultaneous Persistent read/write access to your boot storage drive and secondary storage drives.

When operating in Stateless mode, you can still access and save data to your pre-existing user qubes that were created in Persistent mode or on a secondary storage drive, as well as create and use new user qubes and set their storage volumes (root, private, volatile) to operate fully stateless in Dom0 RAM space using the “varlibqubes” storage pool.

You can create new user qubes while in Stateless mode, however, they may or may not persist and be accessible after shutdown (Dom0 metadata is wiped). In the “Create new qube” GUI tool, you can use the “Advanced” tab to specify a “Storage pool”. The “vm-pool” pool is located on your boot storage drive by default and the “varlibqubes” pool is located in your Dom0 filesystem at /var/lib/qubes as traditional file-backed volumes (runs in Dom0 RAM space during Stateless mode). However, if you choose “varlibqubes”, depending on some various conditions and settings, all data volumes of the qube may not be stored in RAM and require extra special configuration for running fully stateless user qubes.

As a precaution to losing important data, assume all of your new changes and new data may be lost by default after your stateless session ends, unless you send the data to a pre-existing persistently created “save” qube (see Step #0) or otherwise really know what you are doing to save your data effectively.

WARNING: If you don’t understand what you’re doing with an advanced RAM-based stateless configuration of Qubes OS, then it can be very easy to lose data. Ensure that you understand and get familiar with (by testing) the different modes and data saving capabilities of this RAM-based system, and take extra precaution to setup and test reliable methods for saving your important data beyond a Stateless session.

WARNING: For those relying upon Qubes Stateless for anti-forensic use of Qubes OS (which is possible), there are some major caveats to learn about. By default, Qubes Stateless only runs the Dom0 environment as stateless in RAM (take care to disable Dom0 Swap). Other user qubes (such as AppQubes, ServiceQubes, TemplateQubes, etc) might retain some state in their storage volumes if not specifically setup to operate fully stateless in Dom0 RAM space. Other user qubes have multiple storage volumes (root, private, volatile) which can each operate either as Persistent or Stateless depending on various conditions and settings.

Qubes Stateless is an “advanced hybrid state system” where you have both Stateless and Persistent spaces to work with in the same OS environment, and you can optionally control where individual qube volumes operate from (Stateless storage or Persistent storage). The various storage drives, storage pools, and storage volumes involved can be confusing but very relevant to the degree of statelessness you need or desire for each individual qube.

Some useful commands for managing your statelessness:

qubes-prefs

qvm-pool

qvm-volume

qvm-prefs

lsblk

ls -lsa /var/lib/qubes

Qubes Stateless: Conclusion...

A massive THANK YOU to the ultimate legend @xuy who brought Qubes in tmpfs/zram to life for us!

Everyone feel free to enjoy Qubes Stateless and post any feedback, improvements, tweaks, critiques, issues, ideas, etc.

Enjoy being stateless.

Edit Updates:

#1. 2023-11-11

Corrected malformed curly/angled-quotes to now be plain straight-quote characters ("), by removing improper HTML code tags that did not work well with Qubes Forum software (Discourse), which was preventing the stateless boot software from running properly if users had copied the malformed quote characters verbatim.
Added Qubes 4.1 GRUB commands.
Added user backup warning/recommendation.
Minor edits.

#2. 2023-11-12

Added Disable Dom0 Swap step.
Added file verification sub-steps.
Minor edits.

#3. 2023-11-12

Added links to cited resources.
Minor edits.

#4. 2023-11-12

Fixed error in tmpfs.sh that caused stateless boot prompt to be skipped.

#5. 2023-11-12

Corrected number of lines listed for tmpfs.sh file.

#6. 2023-12-08

Added permanent disable of Dom0 Swap.

#7. 2023-12-08

Improved several explanations throughout post.
Minor edits.