Qubes in tmpfs 🤫

[xuy]

You can use your QubesOS 𝚜𝚝𝚊𝚝𝚎𝚕𝚎𝚜𝚜 just like TailsOS, with persistent storage for VMs. That is pretty simple! It takes 6Gb of extra 𝚁𝙰𝙼 (for store root filesystem files).

The steeps:

1. Install QubesOS, boot to it and make base configuration: screen resolution, keyboard layout, etc.

2. Edit kernel parameters variables at grub settings file /etc/default/grub:

GRUB_CMDLINE_LINUX="... 𝚛𝚍.𝚋𝚛𝚎𝚊𝚔"
GRUB_CMDLINE_XEN_DEFAULT="... dom0_mem=max:10240M ..."

The 𝚛𝚍𝚋𝚛𝚎𝚊𝚔 option - drop to a shell at the end (see 𝚖𝚊𝚗 𝚍𝚛𝚊𝚌𝚞𝚝).

3. Generate grub2 config and reboot PC:

sudo grub2-mkconfig | sudo tee /boot/efi/EFI/grub.cfg
reboot

4. “𝙿𝚛𝚎𝚜𝚜 𝙴𝚗𝚝𝚎𝚛 𝚏𝚘𝚛 𝚖𝚊𝚒𝚗𝚝𝚒𝚗𝚊𝚗𝚌𝚎”, then copy files from SSD to 𝚁𝙰𝙼:

umount /sysroot
mkdir /mnt
mount /dev/mapper/qubes_dom0-root /mnt
mount -t 𝚝𝚖𝚙𝚏𝚜 -o size=100% none /sysroot
cp -a /mnt/* /sysroot

• Press Ctrl-D to continue QubesOS boot-up.

  Hooya! Your 𝚀𝚞𝚋𝚎𝚜𝙾𝚂 𝚠𝚘𝚛𝚔𝚒𝚗𝚐 𝚒𝚗 𝚁𝙰𝙼.

You can create a 𝚍𝚛𝚊𝚌𝚞𝚝 module to automate the steep four, if that makes sense.

Then the volume

1. Mount a special 𝚑𝚒𝚍𝚍𝚎𝚗 partition to /opt

2. Create qubes VMs files at varlibqubes pool

qvm-create -P varlibqubes --class TemplateVM --label black debian-10-pool
qvm-create -P varlibqubes --template debian-10-pool --label blue darknet-i2p

3. Change the path via symlinks to access the VMs:

sudo rm -Rf /var/lib/qubes/vm-templates/ ; ln -s /opt/vm-templates/ /var/lib/qubes/
sudo rm -Rf /var/lib/qubes/appvms/ ; ln -s /opt/appvms /var/lib/qubes/

• In the /opt directory should be a VM files created earlier, with identical to current VMs names.

You should like to configure the system

1. Add bash aliases

echo '
alias qvm-clone="qvm-clone -P varlibqubes"
alias qvm-create="qvm-create -P varlibqubes"
' >> $HOME/.bashrc

2. Configure AppVMs

lspci
qvm-pci attach --persistent --verbose vmname dom0:06_00.0

qvm-prefs --set vmname ip 10.137.0.81
qvm-prefs --set vmname netvm none
qvm-prefs --set vmname provides_network true
qvm-prefs --set vmname memory 800
qvm-prefs --set vmname maxmem 8000

Good luck!

NOTE: Install QubesOS updates from normal persistent mode (not from 𝚁𝙰𝙼 mode).

References

1. Linux - Load your root partition to 𝚁𝙰𝙼 and boot it - Tutorials - reboot.pro:
   _Linux - Load your root partition to RAM and boot it - Tutorials - reboot.pro
   _Linux - Load your root partition to RAM and boot it - Tutorials - reboot.pro
2. 𝙳𝚛𝚊𝚌𝚞𝚝 Wiki: _Home · dracutdevs/dracut Wiki · GitHub
3. Deniable encryption · Issue #2402 · QubesOS/qubes-issues · GitHub: _Deniable encryption · Issue #2402 · QubesOS/qubes-issues · GitHub
4. AMD Memory Encryption — The Linux Kernel documentation: _14. AMD Memory Encryption — The Linux Kernel documentation

[xuy]

One little detail, for sync the app shorcuts with a template:
qvm-sync-appmenus debian-10-pool

[alzer89]

This could be written as a systemd service that could read the kernelopts for something like rd.qubes.run_in_ram.

I will post a sample systemd service file for automating this by the end of next week.

This could prove very handy when paired with a veracrypt drive.

Fantastic job!

[deeplow]

This is awesome! Thank you for writing this!

Could this be used to revive the Qubes live USB project?

[51lieal]

I think it could be included in initramfs, would be great for testing, thanks.

[alzer89]

@deeplow

Could this be used to revive the Qubes live USB project ?

Why yes, yes it could

---

An update on getting this to work. Those commands that wrote seem to be for a specific hardware configuration and software personal preference.

I’m trying to get it to parse an existing Qubes OS configuration, so that it’ll recreate the entire configuration in RAM, as opposed to just a “default config”.

This might take longer than a week, but I’ll post up what I have by the end of the week anyway

[deeplow]

Awesome!

[deeplow]

(renamed the title for it to be ascii. Otherwise people won’t find it by searching)

[enmus]

This is so interesting, thanks.

Since the OP compared it to TailOS, what about the footprins on the SSD? I’m not sure I can perceive it.

[51lieal]

I think for the live usb project is not simply as above, the op guide is just for “already” installed qubes, if you want to open an enhancement issue I will join the discussion.

[qstateless]

Wow, “Qubes Stateless” is insanely great! Thanks to @xuy for this simple but powerful solution!

I just tested Qubes Stateless out, using a fresh Qubes 4.1. Dom0 does run stateless from RAM. It works!

This provides a way to do up-to-date and useful stateless amnesic Qubes + Whonix now! Better than Tails, like @ adw’s cheeky “TaiQuWhonDo” in qubes-issues #2024.

Until a true read-only Qubes Live is built by someone, I plan to now use this Qubes Stateless method in real world scenarios going forward. Thank you @xuy!

Here’s some further observations, instructions, thoughts:

GRUB Instructions Fix and Boot Process Description:

Step #3 did not work for me, which was:

sudo grub2-mkconfig | sudo tee /boot/efi/EFI/grub.cfg

When I rebooted, the GRUB boot config had not changed and Qubes OS just did a typical normal boot.

Some researching led me to find that @xuy’s instruction was maybe only for UEFI booted systems, where my system boots via Legacy BIOS.

So I fixed this step by changing step #3 into these two commands:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo grub2-mkconfig -o /boot/efi/EFI/grub.cfg

I believe this changes the GRUB boot config to simultaneously work for both types of systems (both UEFI and Legacy BIOS).

This fix caused my next reboot to break out of the typical normal Qubes OS boot process, and prompts you to “Press Enter for maintenance”, which then drops you into a Dracut command line.

Note that this boot process break happens AFTER entering the Qubes OS disk decryption passphrase. So everything boots normally until after you enter your Qubes OS disk decryption passphrase (if you have full disk encryption enabled).

After decrypting the disk, and you “Press Enter for maintenance”, the system drops into a Dracut shell and you proceed by entering the exact commands @xuy provided in Step #4:

umount /sysroot
mkdir /mnt
mount /dev/mapper/qubes_dom0-root /mnt
mount -t tmpfs -o size=100% none /sysroot
cp -a /mnt/* /sysroot

Note that the last “cp” (copy) step can take a little while (1-4 minutes) to complete, as it is copying all of Dom0 into RAM (I believe).

Pressing Ctrl + D exits you out of the Dracut command line and continues booting into Qubes Stateless Dom0 from RAM.

Once booted up and logged in, I noticed some interesting things…

Lower RAM Optimization:

First, I noticed that the Dom0 RAM partition only takes up ~3.5GB for me initially, which grew to ~5.5GB with some use. Maybe it would grow further, but I didn’t seem to need 10GB of RAM allocated to Dom0, so I rebooted into normal Qubes OS mode, and repeated Step #2 & Step #3, and lowered the Dom0 RAM partition in the GRUB boot config to 6GB (dom0_mem=max:6000M). I rebooted into Qubes Stateless mode and it seemed to still work fine with only 6GB allocated to Dom0.

This means that one can explore and choose a higher or lower Dom0 RAM partition size for better RAM efficiency on lower RAM systems. If you only have 4GB of total system RAM, then you could still probably bootup Qubes Stateless. However, the additional RAM you’ll then need depends on the number and type of additional qubes and apps you want to run once booted up. Note that not all system RAM should be allocated to Dom0. RAM used by other qubes is not to be confused with the Dom0 RAM partition size that you set in the GRUB boot config with Step #2.

VM Pool on Disk is still Accessible & Writable:

The follow-on volume instructions provided @xuy show you one way to configure file-backed storage for your VMs, in the “varlibqubes” pool, which is located in the Dom0 directory path /var/lib/qubes.

However, in testing, I didn’t have any secondary storage setup with such pre-existing VM storage loaded. So I instead played with some other options.

You can simply create new qubes and they will exist and work temporarily in RAM, but then you will loose them by default upon shutdown. That may be okay if you don’t want to save anything from your work session.

When creating new qubes in RAM, you can choose from a couple storage pools, which work differently.

If you create a new qube in the “varlibqubes” pool, and you haven’t implemented @xuy’s secondary storage symlinks, then your new qube’s storage will be file-backed in RAM at /var/lib/qubes and will be completely lost upon shutdown by default.

If you create a new qube in the “vm-pool” pool, then your new qube’s storage will be written to your boot drive’s qubes storage pool, and the data will be saved to your boot drive, but the qube in Qube Manager will disappear upon shutdown, due to the metadata (in qubes.xml I believe) being located in the Dom0 RAM partition and being wiped. However, if you try to recreate a qube with the same name using the same storage pool in the future, Qube Manager throws an error but then seems to overwrite the old qube data if you try again, so be careful and don’t assume your data will survive this method unless you really know what you are doing and take advanced precautions.

Using Qubes Stateless and saving your data while using it is best done with adding a secondary storage location as @xuy describes as file-backed, or Qubes documentation describes as pool-backed, or via online cloud storage if that works with your threat model. But, for advanced users, I just wanted to point out that RAM storage and boot drive storage of your qubes data technically do work and these are available for expert data management use cases, or alternatively if one just does not care to save any local data while using Qubes Stateless.

Qubes Stateless Does NOT Seem to Work with a Read-Only Boot Drive:

I tried booting Qubes Stateless on a read-only USB connected drive and some errors did not allow it to work. The boot process of Qubes OS in general seems to demand write access in the initial stages of startup.

First, the LUKS FDE (full drive encryption) seems to demand write access to pass, which can be bypassed by simply not using drive encryption (not ideal).

Second, in @xuy’s Step #4, the following command seems to throw an error when using a read-only boot drive, but seems to work when write access is enabled to the boot drive:

mount /dev/mapper/qubes_dom0-root /mnt
THROWS ERROR:
mount: /mnt: can’t read superblock on /dev/mapper/qubes_dom0-root.

When booting the drive with write access, I was able to pull out the USB drive once Dom0 booted up to the user login screen, seemingly without encountering any unexpected errors after that.

One time, during some boot testing I was doing, I accidentially borked my Dom0 (i forget how, but it seemed unexpected), and had to reinstall from scratch. So a true read-only Qubes Live boot drive would be much more ideal for protection and security, compared to needing to give write access to the boot drive.

Dual Boot Qubes and Qubes Stateless:

It was implied by @xuy’s post, but just wanted to mention this insight for those who may have misunderstood this concept, that:

Qubes Stateless kind of creates an optional “dual boot” mode for Qubes. One, the normal fully stateful mode. The other, the new stateless dom0 mode.

The normal stateful mode becomes the “base” for configuring each live boot of Qubes Stateless. You can reboot into Qubes normal mode, change and reconfigure your Qubes environment, then reboot into Qubes Stateless and have your newly configured Qubes environment running as Qubes Stateless.

This “dual mode” or “dual boot” for Qubes is pretty neat, although somewhat less ideal for security, compared to a true read-only Qubes Live implementation.

Automated Qubes Stateless Booting Considered Desirable:

Automating the inconvenient manual boot Step #4 would be great!

These commands:
umount /sysroot
mkdir /mnt
mount /dev/mapper/qubes_dom0-root /mnt
mount -t tmpfs -o size=100% none /sysroot
cp -a /mnt/* /sysroot

@ xuy mentions the idea of implementing this automation as a Dracut Module.

@ alzer89 mentions the idea of implementing this automation as a SystemD Service.

However best, it would be awesome to have the code made available for automating this Qubes Stateless boot process, as it gets bothersome when repeating this manually multiple times per day.

Qubes Stateless to Qubes Live:

It would be great to see a truly read-only Qubes Live implementation come out of this innovative Qubes Stateless method somehow!

This Qubes Stateless implementation seems very very close already to a read-only Qubes Live, suitable for everyday use at least from advanced users initially, except for the read-only drive errors during boot and the needed automation of the tmpfs steps.

I’d love to be able to boot from a read-only Qubes Live using read-only media.

@51lieal and All Other Community Members - YES! - as we now seem quite close, let’s start a technical discussion about extending this excellent initial work of Qubes Stateless into a truly working read-only Qubes Live! Would this thread, a new forum thread, or a GitHub issue be best?

[qstateless]

Can a Qubes moderator please review and approve my previous post that was auto censored by akismet? Thank you!

[xuy]

The “𝚀𝚞𝚋𝚎𝚜 𝚒𝚗 𝚝𝚖𝚙𝚏𝚜” will be like a green wig feature for LiveUSB. At least till, that becames a mainstram, hah!

Continue, the automation:
This boot scripts will ask you “𝙱𝚘𝚘𝚝 𝚝𝚘 𝚁𝙰𝙼? (𝚗)” question at graphical boot splash, just after the “𝙳𝚒𝚜𝚔 𝚙𝚊𝚜𝚜𝚠𝚘𝚛𝚍” screen. You can just press Enter if 𝚗𝚘.

The boot to 𝚁𝙰𝙼 automation with a 𝚍𝚛𝚊𝚌𝚞𝚝 module

See the man dracut.modules

1. Create foder for the boot module

cd /usr/lib/dracut/modules.d/
sudo mkdir 01ramboot

2. module-setup.sh: the module main script

echo '#!/usr/bin/bash

check() {
    return 0
}

depends() {
    return 0
}

install() {
        inst_simple "$moddir/tmpfs.sh" "/usr/bin/tmpfs"
        inst_hook cleanup 00 "$moddir/pass.sh"
} 
' | sudo tee 01ramboot/module-setup.sh

3. pass.sh: ask question script

echo '#!/usr/bin/bash 

command -v ask_for_password >/dev/null || . /lib/dracut-crypt-lib.sh

PROMPT="Boot to RAM? (n)"
CMD="/usr/bin/tmpfs"
TRY="3"

ask_for_password \
    --cmd "$CMD" \
    --prompt "$PROMPT" \
    --tries "$TRY" \
    --ply-cmd "$CMD" \
    --ply-prompt "$PROMPT" \
    --ply-tries "$TRY" \
    --tty-cmd "$CMD" \
    --tty-prompt "$PROMPT" \
    --tty-tries "$TRY" \
    --tty-echo-off 
' | sudo tee 01ramboot/pass.sh

See the man plymouth.

4. tmpfs.sh: mount 𝚝𝚖𝚙𝚏𝚜 script

echo '#!/usr/bin/bash 

read line

case "${line:-Nn}" in
        [Yy]* )
                mkdir /mnt
                umount /sysroot
                mount /dev/mapper/qubes_dom0-root /mnt
                mount -t tmpfs -o size=100% none /sysroot
                cp -a /mnt/* /sysroot
                exit 0
                ;;
        [Nn]* )
                exit 0
                ;;
        * )
                exit 1
                ;;
esac
' | sudo tee 01ramboot/tmpfs.sh

5. Make scritps executable:

sudo chmod 755 01ramboot/pass.sh 01ramboot/tmpfs.sh

6. Enable the module

echo 'add_dracutmodules+=" ramboot "' | sudo tee /etc/dracut.conf.d/ramboot.conf

See the man dracut.conf.

7. Regenerate the latest /boot/initramfs... image with ramboot

sudo dracut --verbose --force

sudo reboot

_
NOTE: This method can be modify with dracut 𝙷𝚘𝚘𝚔: 𝚌𝚖𝚍𝚕𝚒𝚗𝚎 if you need to have a special kernel boot string in the grub boot menu and not to use the graphical boot splash screen.

0. To remove the scripts

sudo rm -Rf /usr/lib/dracut/modules.d/01ramboot
sudo rm -f /etc/dracut.conf.d/ramboot.conf
sudo dracut --verbose --force

[51lieal]

Skipping the 2 and 3 step, then rebuild new initramfs with this module, then add new grub menu would be good too. So we can have 1 persistent + 1 tmpfs, the persistent aka normal boot would be only use for upgrading dom0, or configuring something that need to be persistent.

I’ve not tested how it would be good for others, since it need a lot of ram, I’ll try to upgrade my laptop to 64gb this week and see if my daily use is fine.

I still searching a good way to boot it as live os, at least we need a kickstart config and livecd-tools.

[BEBF738VD]

@51lieal

I’ve not tested how it would be good for others, since it need a lot of ram, I’ll try to upgrade my laptop to 64gb this week and see if my daily use is fine.

Have you tried it with less ram or not at all? If so, how much ram and how did it go?

Also, I’m assuming this to be compatible with a detached encrypted boot. Do you have any info?

[51lieal]

Yes its compatible, The detached boot is happen in the earlier stage, but my approach is not like in the script above, “it would do same” but not identical.

Actually even with 8gb ram it would run but not much you can do, and for just default install I think 16-32gb would be fine. Need to test first.

[tripleh]

Btw tmpfs can be swapped to disk.

ramfs cannot.

[xuy]

With ramfs there is no dynamic size limit. The ramfs size is severely cut off from RAM at the ramfs starts. So ramfs is not optimal memory costs. With 𝚝𝚖𝚙𝚏𝚜 an empty filesystem size doesn’t take RAM space.

And that is true, tmpfs can be swapped to disk. Therefore the swap desirable to be disabled. If you leave the swap enabled, some 𝚝𝚖𝚙𝚏𝚜 (root fs) data may be saved on swap. This data may contain a names of your AppVMs and other 𝚌𝚘𝚗𝚏𝚒𝚍𝚎𝚗𝚝𝚒𝚊𝚕 𝚒𝚗𝚏𝚘𝚛𝚖𝚊𝚝𝚒𝚘𝚗.

The simplest solution:

sudo swapoff -a

No swap, no problem.

_
Or if swap is necessary for a some reason, you need a separate physical SSD storage for swap.

• We can encrypt the standard QubesOS swap [LVM Logical] volume /dev/qubes_dom0/swap, but that will be a double encryption, because the /dev/qubes_dom0/swap is already encrypted with the underlying [LVM Physical] qubes_dom0 volume. That’s why we don’t do that.

Instead, we’ll encrypt the swap SSD with detached header, so no one knows it’s a 𝚜𝚠𝚊𝚙 𝚂𝚂𝙳 (moreover, it is an encrypted drive at all, just an unformatted disk with 𝚛𝚊𝚗𝚍𝚘𝚖 𝚍𝚊𝚝𝚊). The swap header file can be removed immediately, as well as the swap drive data is temporary and are not valuable.

1. Prepate a swap drive, let’s say 𝚜𝚍𝚇:

DRIVE=/dev/sdX

2. Fill the 𝚜𝚍𝚇 with uniform layer of random data:

dd if=/dev/urandom of=$DRIVE bs=4096 status=progress

3. Encrypt drive with detached header:

Generage keyfile (instead of a password):

cd /dev/shm
sudo dd bs=512 count=4 if=/dev/urandom of=swapkey.luks iflag=fullblock
sudo chmod 600 swapkey.luks

Format and Open the drive:

yes | sudo  cryptsetup luksFormat $DRIVE --key-file swapkey.luks --header swap-header.luks
sudo cryptsetup luksOpen --header swap-header.luks --key-file swapkey.luks $DRIVE swap

4. Create and mount a new swap:

sudo mkswap /dev/mapper/swap
sudo swapon /dev/mapper/swap

The swap-header.luks is the drive header file.

More detailed, look here:
https://wiki.archlinux.org/title/Dm-crypt/Device_encryption

[enmus]

@xuy

Or if swap is necessary for a some reason, you need a separate physical SSD storage for swap .

You don’t if you can spare a couple more gigs

How To Set And Use ZRAM swap

Continuing from the topic and making guide for those interested in, as well as a reminder for myself after clean install someday. Goals / Possible Benefits To minimize dedicated amounts of RAM To maximize utilizing total amount of RAM To extend lifespan of a SSD Hopefully to reduce Qubes footprint To increase overall performance To prevent performance hits to a level at least as without zram. Why? By default swappiness is set to 60. This means that when 40% of RAM is used, swapping is start…

[51lieal]

Have tested root to tmpfs yesterday, this is my notes:

• After rd.break, we need to mount everything manually which I don’t like so I did change the base dracut script instead of using rd.break which would boot same as the normal does.

• The difference in performance is not too significant (I install template and run it from varlibqubes), but still good since there’s no additional log from now on.

After researching for a few days I got many new thing that I don’t know, for the qubes os live project, I’ve seen joanna post about the disadvantage of using qubes live, and one of the reason is

3. We had to solve the problem of Qubes too easily triggering Out Of Memory
   condition in Dom0 when running as Live OS.

This last problem has been a result of Qubes using the copy-on-write backing for
the VM’s root filesystems, which is used to implement our cool Template-based
scheme [2]. Normally these are backed by regular files on disk – even though
these files are discardable upon VM reboots, they must be preserved during the
VM’s life span, and they can easily grow to a few tens of MBs per VM, sometimes
even more. Also, each of the VM’s private image, which essentially holds just
the user home directory, typically starts with a few tens of MBs for an “empty
VM”. Now, while these represent rather insignificant numbers on a disk-basked
system, in case of a LiveUSB scenarios all these files must be stored in RAM,
which is always a scare resource on any OS, and especially Qubes.

I will continue my report in next week about the progress.