Qubes in tmpfs 🀫

You can use your QubesOS πšœπšπšŠπšπšŽπš•πšŽπšœπšœ just like TailsOS, with persistent storage for VMs. That is pretty simple! It takes 6Gb of extra πšπ™°π™Ό (for store root filesystem files).

The steeps:

  1. Install QubesOS, boot to it and make base configuration: screen resolution, keyboard layout, etc.

  2. Edit kernel parameters variables at grub settings file /etc/default/grub:

GRUB_CMDLINE_LINUX="... πš›πš.πš‹πš›πšŽπšŠπš”"
GRUB_CMDLINE_XEN_DEFAULT="... dom0_mem=max:10240M ..."

The πš›πšπš‹πš›πšŽπšŠπš” option - drop to a shell at the end (see πš–πšŠπš— πšπš›πšŠπšŒπšžπš).

  1. Generate grub2 config and reboot PC:
sudo grub2-mkconfig | sudo tee /boot/efi/EFI/grub.cfg
reboot
  1. β€œπ™Ώπš›πšŽπšœπšœ π™΄πš—πšπšŽπš› πšπš˜πš› πš–πšŠπš’πš—πšπš’πš—πšŠπš—πšŒπšŽβ€, then copy files from SSD to πšπ™°π™Ό:
umount /sysroot
mkdir /mnt
mount /dev/mapper/qubes_dom0-root /mnt
mount -t πšπš–πš™πšπšœ -o size=100% none /sysroot
cp -a /mnt/* /sysroot
  • Press Ctrl-D to continue QubesOS boot-up.

    Hooya! Your πš€πšžπš‹πšŽπšœπ™Ύπš‚ πš πš˜πš›πš”πš’πš—πš πš’πš— πšπ™°π™Ό.

You can create a πšπš›πšŠπšŒπšžπš module to automate the steep four, if that makes sense.

Then the volume

  1. Mount a special πš‘πš’πšπšπšŽπš— partition to /opt

  2. Create qubes VMs files at varlibqubes pool

qvm-create -P varlibqubes --class TemplateVM --label black debian-10-pool
qvm-create -P varlibqubes --template debian-10-pool --label blue darknet-i2p
  1. Change the path via symlinks to access the VMs:
sudo rm -Rf /var/lib/qubes/vm-templates/ ; ln -s /opt/vm-templates/ /var/lib/qubes/
sudo rm -Rf /var/lib/qubes/appvms/ ; ln -s /opt/appvms /var/lib/qubes/
  • In the /opt directory should be a VM files created earlier, with identical to current VMs names.

You should like to configure the system

  1. Add bash aliases
echo '
alias qvm-clone="qvm-clone -P varlibqubes"
alias qvm-create="qvm-create -P varlibqubes"
' >> $HOME/.bashrc
  1. Configure AppVMs
lspci
qvm-pci attach --persistent --verbose vmname dom0:06_00.0

qvm-prefs --set vmname ip 10.137.0.81
qvm-prefs --set vmname netvm none
qvm-prefs --set vmname provides_network true
qvm-prefs --set vmname memory 800
qvm-prefs --set vmname maxmem 8000

Good luck!

NOTE: Install QubesOS updates from normal persistent mode (not from πšπ™°π™Ό mode).

References

  1. Linux - Load your root partition to πšπ™°π™Ό and boot it - Tutorials - reboot.pro:
    _https://reboot.pro/topic/14547-linux-load-your-root-partition-to-ram-and-boot-it/
    _https://web.archive.org/web/20220224235759/Linux - Load your root partition to RAM and boot it - Tutorials - reboot.pro
  2. π™³πš›πšŠπšŒπšžπš Wiki: _https://dracut.wiki.kernel.org/index.php/Main_Page
  3. Deniable encryption Β· Issue #2402 Β· QubesOS/qubes-issues Β· GitHub: _https://github.com/QubesOS/qubes-issues/issues/2402
  4. AMD Memory Encryption β€” The Linux Kernel documentation: _https://www.kernel.org/doc/html/v5.8/x86/amd-memory-encryption.html
10 Likes

One little detail, for sync the app shorcuts with a template:
qvm-sync-appmenus debian-10-pool

This could be written as a systemd service that could read the kernelopts for something like rd.qubes.run_in_ram.

I will post a sample systemd service file for automating this by the end of next week.

This could prove very handy when paired with a veracrypt drive.

Fantastic job!

3 Likes

This is awesome! Thank you for writing this!

Could this be used to revive the Qubes live USB project?

1 Like

I think it could be included in initramfs, would be great for testing, thanks.

Why yes, yes it could :thinking:


An update on getting this to work. Those commands that wrote seem to be for a specific hardware configuration and software personal preference.

I’m trying to get it to parse an existing Qubes OS configuration, so that it’ll recreate the entire configuration in RAM, as opposed to just a β€œdefault config”.

This might take longer than a week, but I’ll post up what I have by the end of the week anyway :slight_smile:

3 Likes

Awesome! :slight_smile:

(renamed the title for it to be ascii. Otherwise people won’t find it by searching)

This is so interesting, thanks.

Since the OP compared it to TailOS, what about the footprins on the SSD? I’m not sure I can perceive it.

I think for the live usb project is not simply as above, the op guide is just for β€œalready” installed qubes, if you want to open an enhancement issue I will join the discussion.

Wow, β€œQubes Stateless” is insanely great! Thanks to @xuy for this simple but powerful solution!

I just tested Qubes Stateless out, using a fresh Qubes 4.1. Dom0 does run stateless from RAM. It works!

This provides a way to do up-to-date and useful stateless amnesic Qubes + Whonix now! Better than Tails, like @ adw’s cheeky β€œTaiQuWhonDo” in qubes-issues #2024. :wink:

Until a true read-only Qubes Live is built by someone, I plan to now use this Qubes Stateless method in real world scenarios going forward. Thank you @xuy!

Here’s some further observations, instructions, thoughts:

GRUB Instructions Fix and Boot Process Description:

Step #3 did not work for me, which was:

sudo grub2-mkconfig | sudo tee /boot/efi/EFI/grub.cfg

When I rebooted, the GRUB boot config had not changed and Qubes OS just did a typical normal boot.

Some researching led me to find that @xuy’s instruction was maybe only for UEFI booted systems, where my system boots via Legacy BIOS.

So I fixed this step by changing step #3 into these two commands:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo grub2-mkconfig -o /boot/efi/EFI/grub.cfg

I believe this changes the GRUB boot config to simultaneously work for both types of systems (both UEFI and Legacy BIOS).

This fix caused my next reboot to break out of the typical normal Qubes OS boot process, and prompts you to β€œPress Enter for maintenance”, which then drops you into a Dracut command line.

Note that this boot process break happens AFTER entering the Qubes OS disk decryption passphrase. So everything boots normally until after you enter your Qubes OS disk decryption passphrase (if you have full disk encryption enabled).

After decrypting the disk, and you β€œPress Enter for maintenance”, the system drops into a Dracut shell and you proceed by entering the exact commands @xuy provided in Step #4:

umount /sysroot
mkdir /mnt
mount /dev/mapper/qubes_dom0-root /mnt
mount -t tmpfs -o size=100% none /sysroot
cp -a /mnt/* /sysroot

Note that the last β€œcp” (copy) step can take a little while (1-4 minutes) to complete, as it is copying all of Dom0 into RAM (I believe).

Pressing Ctrl + D exits you out of the Dracut command line and continues booting into Qubes Stateless Dom0 from RAM.

Once booted up and logged in, I noticed some interesting things…

Lower RAM Optimization:

First, I noticed that the Dom0 RAM partition only takes up ~3.5GB for me initially, which grew to ~5.5GB with some use. Maybe it would grow further, but I didn’t seem to need 10GB of RAM allocated to Dom0, so I rebooted into normal Qubes OS mode, and repeated Step #2 & Step #3, and lowered the Dom0 RAM partition in the GRUB boot config to 6GB (dom0_mem=max:6000M). I rebooted into Qubes Stateless mode and it seemed to still work fine with only 6GB allocated to Dom0.

This means that one can explore and choose a higher or lower Dom0 RAM partition size for better RAM efficiency on lower RAM systems. If you only have 4GB of total system RAM, then you could still probably bootup Qubes Stateless. However, the additional RAM you’ll then need depends on the number and type of additional qubes and apps you want to run once booted up. Note that not all system RAM should be allocated to Dom0. RAM used by other qubes is not to be confused with the Dom0 RAM partition size that you set in the GRUB boot config with Step #2.

VM Pool on Disk is still Accessible & Writable:

The follow-on volume instructions provided @xuy show you one way to configure file-backed storage for your VMs, in the β€œvarlibqubes” pool, which is located in the Dom0 directory path /var/lib/qubes.

However, in testing, I didn’t have any secondary storage setup with such pre-existing VM storage loaded. So I instead played with some other options.

You can simply create new qubes and they will exist and work temporarily in RAM, but then you will loose them by default upon shutdown. That may be okay if you don’t want to save anything from your work session.

When creating new qubes in RAM, you can choose from a couple storage pools, which work differently.

If you create a new qube in the β€œvarlibqubes” pool, and you haven’t implemented @xuy’s secondary storage symlinks, then your new qube’s storage will be file-backed in RAM at /var/lib/qubes and will be completely lost upon shutdown by default.

If you create a new qube in the β€œvm-pool” pool, then your new qube’s storage will be written to your boot drive’s qubes storage pool, and the data will be saved to your boot drive, but the qube in Qube Manager will disappear upon shutdown, due to the metadata (in qubes.xml I believe) being located in the Dom0 RAM partition and being wiped. However, if you try to recreate a qube with the same name using the same storage pool in the future, Qube Manager throws an error but then seems to overwrite the old qube data if you try again, so be careful and don’t assume your data will survive this method unless you really know what you are doing and take advanced precautions.

Using Qubes Stateless and saving your data while using it is best done with adding a secondary storage location as @xuy describes as file-backed, or Qubes documentation describes as pool-backed, or via online cloud storage if that works with your threat model. But, for advanced users, I just wanted to point out that RAM storage and boot drive storage of your qubes data technically do work and these are available for expert data management use cases, or alternatively if one just does not care to save any local data while using Qubes Stateless.

Qubes Stateless Does NOT Seem to Work with a Read-Only Boot Drive:

I tried booting Qubes Stateless on a read-only USB connected drive and some errors did not allow it to work. The boot process of Qubes OS in general seems to demand write access in the initial stages of startup.

First, the LUKS FDE (full drive encryption) seems to demand write access to pass, which can be bypassed by simply not using drive encryption (not ideal).

Second, in @xuy’s Step #4, the following command seems to throw an error when using a read-only boot drive, but seems to work when write access is enabled to the boot drive:

mount /dev/mapper/qubes_dom0-root /mnt
THROWS ERROR:
mount: /mnt: can’t read superblock on /dev/mapper/qubes_dom0-root.

When booting the drive with write access, I was able to pull out the USB drive once Dom0 booted up to the user login screen, seemingly without encountering any unexpected errors after that.

One time, during some boot testing I was doing, I accidentially borked my Dom0 (i forget how, but it seemed unexpected), and had to reinstall from scratch. So a true read-only Qubes Live boot drive would be much more ideal for protection and security, compared to needing to give write access to the boot drive.

Dual Boot Qubes and Qubes Stateless:

It was implied by @xuy’s post, but just wanted to mention this insight for those who may have misunderstood this concept, that:

Qubes Stateless kind of creates an optional β€œdual boot” mode for Qubes. One, the normal fully stateful mode. The other, the new stateless dom0 mode.

The normal stateful mode becomes the β€œbase” for configuring each live boot of Qubes Stateless. You can reboot into Qubes normal mode, change and reconfigure your Qubes environment, then reboot into Qubes Stateless and have your newly configured Qubes environment running as Qubes Stateless.

This β€œdual mode” or β€œdual boot” for Qubes is pretty neat, although somewhat less ideal for security, compared to a true read-only Qubes Live implementation.

Automated Qubes Stateless Booting Considered Desirable:

Automating the inconvenient manual boot Step #4 would be great!

These commands:
umount /sysroot
mkdir /mnt
mount /dev/mapper/qubes_dom0-root /mnt
mount -t tmpfs -o size=100% none /sysroot
cp -a /mnt/* /sysroot

@ xuy mentions the idea of implementing this automation as a Dracut Module.

@ alzer89 mentions the idea of implementing this automation as a SystemD Service.

However best, it would be awesome to have the code made available for automating this Qubes Stateless boot process, as it gets bothersome when repeating this manually multiple times per day.

Qubes Stateless to Qubes Live:

It would be great to see a truly read-only Qubes Live implementation come out of this innovative Qubes Stateless method somehow!

This Qubes Stateless implementation seems very very close already to a read-only Qubes Live, suitable for everyday use at least from advanced users initially, except for the read-only drive errors during boot and the needed automation of the tmpfs steps.

I’d love to be able to boot from a read-only Qubes Live using read-only media.

@51lieal and All Other Community Members - YES! - as we now seem quite close, let’s start a technical discussion about extending this excellent initial work of Qubes Stateless into a truly working read-only Qubes Live! Would this thread, a new forum thread, or a GitHub issue be best?

3 Likes

Can a Qubes moderator please review and approve my previous post that was auto censored by akismet? Thank you!

The β€œπš€πšžπš‹πšŽπšœ πš’πš— πšπš–πš™πšπšœβ€ will be like a green wig feature for LiveUSB. At least till, that becames a mainstram, hah!

Continue, the automation:
This boot scripts will ask you β€œπ™±πš˜πš˜πš 𝚝𝚘 πšπ™°π™Ό? (πš—)” question at graphical boot splash, just after the β€œπ™³πš’πšœπš” πš™πšŠπšœπšœπš πš˜πš›πšβ€ screen. You can just press Enter if πš—πš˜.

The boot to πšπ™°π™Ό automation with a πšπš›πšŠπšŒπšžπš module

See the man dracut.modules

1. Create foder for the boot module

cd /usr/lib/dracut/modules.d/
sudo mkdir 01ramboot

2. module-setup.sh: the module main script

echo '#!/usr/bin/bash

check() {
    return 0
}

depends() {
    return 0
}

install() {
        inst_simple "$moddir/tmpfs.sh" "/usr/bin/tmpfs"
        inst_hook cleanup 00 "$moddir/pass.sh"
} 
' | sudo tee 01ramboot/module-setup.sh

3. pass.sh: ask question script

echo '#!/usr/bin/bash 

command -v ask_for_password >/dev/null || . /lib/dracut-crypt-lib.sh

PROMPT="Boot to RAM? (n)"
CMD="/usr/bin/tmpfs"
TRY="3"

ask_for_password \
    --cmd "$CMD" \
    --prompt "$PROMPT" \
    --tries "$TRY" \
    --ply-cmd "$CMD" \
    --ply-prompt "$PROMPT" \
    --ply-tries "$TRY" \
    --tty-cmd "$CMD" \
    --tty-prompt "$PROMPT" \
    --tty-tries "$TRY" \
    --tty-echo-off 
' | sudo tee 01ramboot/pass.sh

See the man plymouth.

4. tmpfs.sh: mount πšπš–πš™πšπšœ script

echo '#!/usr/bin/bash 

read line

case "${line:-Nn}" in
        [Yy]* )
                mkdir /mnt
                umount /sysroot
                mount /dev/mapper/qubes_dom0-root /mnt
                mount -t tmpfs -o size=100% none /sysroot
                cp -a /mnt/* /sysroot
                exit 0
                ;;
        [Nn]* )
                exit 0
                ;;
        * )
                exit 1
                ;;
esac
' | sudo tee 01ramboot/tmpfs.sh

5. Make scritps executable:

sudo chmod 755 01ramboot/pass.sh 01ramboot/tmpfs.sh

6. Enable the module

echo 'add_dracutmodules+=" ramboot "' | sudo tee /etc/dracut.conf.d/ramboot.conf

See the man dracut.conf.

7. Regenerate the latest /boot/initramfs... image with ramboot

sudo dracut --verbose --force

sudo reboot

_
NOTE: This method can be modify with dracut π™·πš˜πš˜πš”: πšŒπš–πšπš•πš’πš—πšŽ if you need to have a special kernel boot string in the grub boot menu and not to use the graphical boot splash screen.

0. To remove the scripts

sudo rm -Rf /usr/lib/dracut/modules.d/01ramboot
sudo rm -f /etc/dracut.conf.d/ramboot.conf
sudo dracut --verbose --force
1 Like

Skipping the 2 and 3 step, then rebuild new initramfs with this module, then add new grub menu would be good too. So we can have 1 persistent + 1 tmpfs, the persistent aka normal boot would be only use for upgrading dom0, or configuring something that need to be persistent.

I’ve not tested how it would be good for others, since it need a lot of ram, I’ll try to upgrade my laptop to 64gb this week and see if my daily use is fine.

I still searching a good way to boot it as live os, at least we need a kickstart config and livecd-tools.

Have you tried it with less ram or not at all? If so, how much ram and how did it go?

Also, I’m assuming this to be compatible with a detached encrypted boot. Do you have any info?

Yes its compatible, The detached boot is happen in the earlier stage, but my approach is not like in the script above, β€œit would do same” but not identical.

Actually even with 8gb ram it would run but not much you can do, and for just default install I think 16-32gb would be fine. Need to test first.

1 Like

Btw tmpfs can be swapped to disk.

ramfs cannot.

1 Like

With ramfs there is no dynamic size limit. The ramfs size is severely cut off from RAM at the ramfs starts. So ramfs is not optimal memory costs. With πšπš–πš™πšπšœ an empty filesystem size doesn’t take RAM space.

And that is true, tmpfs can be swapped to disk. Therefore the swap desirable to be disabled. If you leave the swap enabled, some πšπš–πš™πšπšœ (root fs) data may be saved on swap. This data may contain a names of your AppVMs and other πšŒπš˜πš—πšπš’πšπšŽπš—πšπš’πšŠπš• πš’πš—πšπš˜πš›πš–πšŠπšπš’πš˜πš—.

The simplest solution:

sudo swapoff -a

No swap, no problem.

_
Or if swap is necessary for a some reason, you need a separate physical SSD storage for swap.

  • We can encrypt the standard QubesOS swap [LVM Logical] volume /dev/qubes_dom0/swap, but that will be a double encryption, because the /dev/qubes_dom0/swap is already encrypted with the underlying [LVM Physical] qubes_dom0 volume. That’s why we don’t do that.

Instead, we’ll encrypt the swap SSD with detached header, so no one knows it’s a πšœπš πšŠπš™ πš‚πš‚π™³ (moreover, it is an encrypted drive at all, just an unformatted disk with πš›πšŠπš—πšπš˜πš– 𝚍𝚊𝚝𝚊). The swap header file can be removed immediately, as well as the swap drive data is temporary and are not valuable.

1. Prepate a swap drive, let’s say πšœπšπš‡:

DRIVE=/dev/sdX

2. Fill the πšœπšπš‡ with uniform layer of random data:

dd if=/dev/urandom of=$DRIVE bs=4096 status=progress

3. Encrypt drive with detached header:

Generage keyfile (instead of a password):

cd /dev/shm
sudo dd bs=512 count=4 if=/dev/urandom of=swapkey.luks iflag=fullblock
sudo chmod 600 swapkey.luks

Format and Open the drive:

yes | sudo  cryptsetup luksFormat $DRIVE --key-file swapkey.luks --header swap-header.luks
sudo cryptsetup luksOpen --header swap-header.luks --key-file swapkey.luks $DRIVE swap

4. Create and mount a new swap:

sudo mkswap /dev/mapper/swap
sudo swapon /dev/mapper/swap

The swap-header.luks is the drive header file.

More detailed, look here:
https://wiki.archlinux.org/title/Dm-crypt/Device_encryption

You don’t if you can spare a couple more gigs

Have tested root to tmpfs yesterday, this is my notes:

  • After rd.break, we need to mount everything manually which I don’t like so I did change the base dracut script instead of using rd.break which would boot same as the normal does.

  • The difference in performance is not too significant (I install template and run it from varlibqubes), but still good since there’s no additional log from now on.

After researching for a few days I got many new thing that I don’t know, for the qubes os live project, I’ve seen joanna post about the disadvantage of using qubes live, and one of the reason is

  1. We had to solve the problem of Qubes too easily triggering Out Of Memory
    condition in Dom0 when running as Live OS.

This last problem has been a result of Qubes using the copy-on-write backing for
the VM’s root filesystems, which is used to implement our cool Template-based
scheme [2]. Normally these are backed by regular files on disk – even though
these files are discardable upon VM reboots, they must be preserved during the
VM’s life span, and they can easily grow to a few tens of MBs per VM, sometimes
even more. Also, each of the VM’s private image, which essentially holds just
the user home directory, typically starts with a few tens of MBs for an β€œempty
VM”. Now, while these represent rather insignificant numbers on a disk-basked
system, in case of a LiveUSB scenarios all these files must be stored in RAM,
which is always a scare resource on any OS, and especially Qubes.

I will continue my report in next week about the progress.

3 Likes