(renamed the title for it to be ascii. Otherwise people wonβt find it by searching)
This is so interesting, thanks.
Since the OP compared it to TailOS, what about the footprins on the SSD? Iβm not sure I can perceive it.
I think for the live usb project is not simply as above, the op guide is just for βalreadyβ installed qubes, if you want to open an enhancement issue I will join the discussion.
Wow, βQubes Statelessβ is insanely great! Thanks to @xuy for this simple but powerful solution!
I just tested Qubes Stateless out, using a fresh Qubes 4.1. Dom0 does run stateless from RAM. It works!
This provides a way to do up-to-date and useful stateless amnesic Qubes + Whonix now! Better than Tails, like @ adwβs cheeky βTaiQuWhonDoβ in qubes-issues #2024.
Until a true read-only Qubes Live is built by someone, I plan to now use this Qubes Stateless method in real world scenarios going forward. Thank you @xuy!
Hereβs some further observations, instructions, thoughts:
GRUB Instructions Fix and Boot Process Description:
Step #3 did not work for me, which was:
sudo grub2-mkconfig | sudo tee /boot/efi/EFI/grub.cfg
When I rebooted, the GRUB boot config had not changed and Qubes OS just did a typical normal boot.
Some researching led me to find that @xuyβs instruction was maybe only for UEFI booted systems, where my system boots via Legacy BIOS.
So I fixed this step by changing step #3 into these two commands:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo grub2-mkconfig -o /boot/efi/EFI/grub.cfg
I believe this changes the GRUB boot config to simultaneously work for both types of systems (both UEFI and Legacy BIOS).
This fix caused my next reboot to break out of the typical normal Qubes OS boot process, and prompts you to βPress Enter for maintenanceβ, which then drops you into a Dracut command line.
Note that this boot process break happens AFTER entering the Qubes OS disk decryption passphrase. So everything boots normally until after you enter your Qubes OS disk decryption passphrase (if you have full disk encryption enabled).
After decrypting the disk, and you βPress Enter for maintenanceβ, the system drops into a Dracut shell and you proceed by entering the exact commands @xuy provided in Step #4:
umount /sysroot
mkdir /mnt
mount /dev/mapper/qubes_dom0-root /mnt
mount -t tmpfs -o size=100% none /sysroot
cp -a /mnt/* /sysroot
Note that the last βcpβ (copy) step can take a little while (1-4 minutes) to complete, as it is copying all of Dom0 into RAM (I believe).
Pressing Ctrl + D exits you out of the Dracut command line and continues booting into Qubes Stateless Dom0 from RAM.
Once booted up and logged in, I noticed some interesting thingsβ¦
Lower RAM Optimization:
First, I noticed that the Dom0 RAM partition only takes up ~3.5GB for me initially, which grew to ~5.5GB with some use. Maybe it would grow further, but I didnβt seem to need 10GB of RAM allocated to Dom0, so I rebooted into normal Qubes OS mode, and repeated Step #2 & Step #3, and lowered the Dom0 RAM partition in the GRUB boot config to 6GB (dom0_mem=max:6000M). I rebooted into Qubes Stateless mode and it seemed to still work fine with only 6GB allocated to Dom0.
This means that one can explore and choose a higher or lower Dom0 RAM partition size for better RAM efficiency on lower RAM systems. If you only have 4GB of total system RAM, then you could still probably bootup Qubes Stateless. However, the additional RAM youβll then need depends on the number and type of additional qubes and apps you want to run once booted up. Note that not all system RAM should be allocated to Dom0. RAM used by other qubes is not to be confused with the Dom0 RAM partition size that you set in the GRUB boot config with Step #2.
VM Pool on Disk is still Accessible & Writable:
The follow-on volume instructions provided @xuy show you one way to configure file-backed storage for your VMs, in the βvarlibqubesβ pool, which is located in the Dom0 directory path /var/lib/qubes.
However, in testing, I didnβt have any secondary storage setup with such pre-existing VM storage loaded. So I instead played with some other options.
You can simply create new qubes and they will exist and work temporarily in RAM, but then you will loose them by default upon shutdown. That may be okay if you donβt want to save anything from your work session.
When creating new qubes in RAM, you can choose from a couple storage pools, which work differently.
If you create a new qube in the βvarlibqubesβ pool, and you havenβt implemented @xuyβs secondary storage symlinks, then your new qubeβs storage will be file-backed in RAM at /var/lib/qubes and will be completely lost upon shutdown by default.
If you create a new qube in the βvm-poolβ pool, then your new qubeβs storage will be written to your boot driveβs qubes storage pool, and the data will be saved to your boot drive, but the qube in Qube Manager will disappear upon shutdown, due to the metadata (in qubes.xml I believe) being located in the Dom0 RAM partition and being wiped. However, if you try to recreate a qube with the same name using the same storage pool in the future, Qube Manager throws an error but then seems to overwrite the old qube data if you try again, so be careful and donβt assume your data will survive this method unless you really know what you are doing and take advanced precautions.
Using Qubes Stateless and saving your data while using it is best done with adding a secondary storage location as @xuy describes as file-backed, or Qubes documentation describes as pool-backed, or via online cloud storage if that works with your threat model. But, for advanced users, I just wanted to point out that RAM storage and boot drive storage of your qubes data technically do work and these are available for expert data management use cases, or alternatively if one just does not care to save any local data while using Qubes Stateless.
Qubes Stateless Does NOT Seem to Work with a Read-Only Boot Drive:
I tried booting Qubes Stateless on a read-only USB connected drive and some errors did not allow it to work. The boot process of Qubes OS in general seems to demand write access in the initial stages of startup.
First, the LUKS FDE (full drive encryption) seems to demand write access to pass, which can be bypassed by simply not using drive encryption (not ideal).
Second, in @xuyβs Step #4, the following command seems to throw an error when using a read-only boot drive, but seems to work when write access is enabled to the boot drive:
mount /dev/mapper/qubes_dom0-root /mnt
THROWS ERROR:
mount: /mnt: canβt read superblock on /dev/mapper/qubes_dom0-root.
When booting the drive with write access, I was able to pull out the USB drive once Dom0 booted up to the user login screen, seemingly without encountering any unexpected errors after that.
One time, during some boot testing I was doing, I accidentially borked my Dom0 (i forget how, but it seemed unexpected), and had to reinstall from scratch. So a true read-only Qubes Live boot drive would be much more ideal for protection and security, compared to needing to give write access to the boot drive.
Dual Boot Qubes and Qubes Stateless:
It was implied by @xuyβs post, but just wanted to mention this insight for those who may have misunderstood this concept, that:
Qubes Stateless kind of creates an optional βdual bootβ mode for Qubes. One, the normal fully stateful mode. The other, the new stateless dom0 mode.
The normal stateful mode becomes the βbaseβ for configuring each live boot of Qubes Stateless. You can reboot into Qubes normal mode, change and reconfigure your Qubes environment, then reboot into Qubes Stateless and have your newly configured Qubes environment running as Qubes Stateless.
This βdual modeβ or βdual bootβ for Qubes is pretty neat, although somewhat less ideal for security, compared to a true read-only Qubes Live implementation.
Automated Qubes Stateless Booting Considered Desirable:
Automating the inconvenient manual boot Step #4 would be great!
These commands:
umount /sysroot
mkdir /mnt
mount /dev/mapper/qubes_dom0-root /mnt
mount -t tmpfs -o size=100% none /sysroot
cp -a /mnt/* /sysroot
@ xuy mentions the idea of implementing this automation as a Dracut Module.
@ alzer89 mentions the idea of implementing this automation as a SystemD Service.
However best, it would be awesome to have the code made available for automating this Qubes Stateless boot process, as it gets bothersome when repeating this manually multiple times per day.
Qubes Stateless to Qubes Live:
It would be great to see a truly read-only Qubes Live implementation come out of this innovative Qubes Stateless method somehow!
This Qubes Stateless implementation seems very very close already to a read-only Qubes Live, suitable for everyday use at least from advanced users initially, except for the read-only drive errors during boot and the needed automation of the tmpfs steps.
Iβd love to be able to boot from a read-only Qubes Live using read-only media.
@51lieal and All Other Community Members - YES! - as we now seem quite close, letβs start a technical discussion about extending this excellent initial work of Qubes Stateless into a truly working read-only Qubes Live! Would this thread, a new forum thread, or a GitHub issue be best?
Can a Qubes moderator please review and approve my previous post that was auto censored by akismet? Thank you!
The βπππππ ππ πππππβ will be like a green wig feature for LiveUSB. At least till, that becames a mainstram, hah!
Continue, the automation:
This boot scripts will ask you βπ±πππ ππ ππ°πΌ? (π)β question at graphical boot splash, just after the βπ³πππ πππππ πππβ screen. You can just press Enter if ππ.
The boot to ππ°πΌ automation with a ππππππ module
See the man dracut.modules
1. Create foder for the boot module
cd /usr/lib/dracut/modules.d/
sudo mkdir 01ramboot
2. module-setup.sh: the module main script
echo '#!/usr/bin/bash
check() {
return 0
}
depends() {
return 0
}
install() {
inst_simple "$moddir/tmpfs.sh" "/usr/bin/tmpfs"
inst_hook cleanup 00 "$moddir/pass.sh"
}
' | sudo tee 01ramboot/module-setup.sh
3. pass.sh: ask question script
echo '#!/usr/bin/bash
command -v ask_for_password >/dev/null || . /lib/dracut-crypt-lib.sh
PROMPT="Boot to RAM? (n)"
CMD="/usr/bin/tmpfs"
TRY="3"
ask_for_password \
--cmd "$CMD" \
--prompt "$PROMPT" \
--tries "$TRY" \
--ply-cmd "$CMD" \
--ply-prompt "$PROMPT" \
--ply-tries "$TRY" \
--tty-cmd "$CMD" \
--tty-prompt "$PROMPT" \
--tty-tries "$TRY" \
--tty-echo-off
' | sudo tee 01ramboot/pass.sh
See the man plymouth
.
4. tmpfs.sh: mount πππππ script
echo '#!/usr/bin/bash
read line
case "${line:-Nn}" in
[Yy]* )
mkdir /mnt
umount /sysroot
mount /dev/mapper/qubes_dom0-root /mnt
mount -t tmpfs -o size=100% none /sysroot
cp -a /mnt/* /sysroot
exit 0
;;
[Nn]* )
exit 0
;;
* )
exit 1
;;
esac
' | sudo tee 01ramboot/tmpfs.sh
5. Make scritps executable:
sudo chmod 755 01ramboot/pass.sh 01ramboot/tmpfs.sh
6. Enable the module
echo 'add_dracutmodules+=" ramboot "' | sudo tee /etc/dracut.conf.d/ramboot.conf
See the man dracut.conf
.
7. Regenerate the latest /boot/initramfs...
image with ramboot
sudo dracut --verbose --force
sudo reboot
_
NOTE: This method can be modify with dracut π·πππ: πππππππ
if you need to have a special kernel boot string in the grub boot menu and not to use the graphical boot splash screen.
0. To remove the scripts
sudo rm -Rf /usr/lib/dracut/modules.d/01ramboot
sudo rm -f /etc/dracut.conf.d/ramboot.conf
sudo dracut --verbose --force
Skipping the 2 and 3 step, then rebuild new initramfs with this module, then add new grub menu would be good too. So we can have 1 persistent + 1 tmpfs, the persistent aka normal boot would be only use for upgrading dom0, or configuring something that need to be persistent.
Iβve not tested how it would be good for others, since it need a lot of ram, Iβll try to upgrade my laptop to 64gb this week and see if my daily use is fine.
I still searching a good way to boot it as live os, at least we need a kickstart config and livecd-tools.
Have you tried it with less ram or not at all? If so, how much ram and how did it go?
Also, Iβm assuming this to be compatible with a detached encrypted boot. Do you have any info?
Yes its compatible, The detached boot is happen in the earlier stage, but my approach is not like in the script above, βit would do sameβ but not identical.
Actually even with 8gb ram it would run but not much you can do, and for just default install I think 16-32gb would be fine. Need to test first.
Btw tmpfs can be swapped to disk.
ramfs cannot.
With ramfs there is no dynamic size limit. The ramfs size is severely cut off from RAM at the ramfs starts. So ramfs is not optimal memory costs. With πππππ an empty filesystem size doesnβt take RAM space.
And that is true, tmpfs can be swapped to disk. Therefore the swap desirable to be disabled. If you leave the swap enabled, some πππππ (root fs) data may be saved on swap. This data may contain a names of your AppVMs and other ππππππππππππ πππππππππππ.
The simplest solution:
sudo swapoff -a
No swap, no problem.
_
Or if swap is necessary for a some reason, you need a separate physical SSD storage for swap.
- We can encrypt the standard QubesOS swap [LVM Logical] volume
/dev/qubes_dom0/swap
, but that will be a double encryption, because the/dev/qubes_dom0/swap
is already encrypted with the underlying [LVM Physical]qubes_dom0
volume. Thatβs why we donβt do that.
Instead, weβll encrypt the swap SSD with detached header, so no one knows itβs a ππ ππ πππ³ (moreover, it is an encrypted drive at all, just an unformatted disk with ππππππ ππππ). The swap header file can be removed immediately, as well as the swap drive data is temporary and are not valuable.
1. Prepate a swap drive, letβs say πππ
:
DRIVE=/dev/sdX
2. Fill the πππ
with uniform layer of random data:
dd if=/dev/urandom of=$DRIVE bs=4096 status=progress
3. Encrypt drive with detached header:
Generage keyfile (instead of a password):
cd /dev/shm
sudo dd bs=512 count=4 if=/dev/urandom of=swapkey.luks iflag=fullblock
sudo chmod 600 swapkey.luks
Format and Open the drive:
yes | sudo cryptsetup luksFormat $DRIVE --key-file swapkey.luks --header swap-header.luks
sudo cryptsetup luksOpen --header swap-header.luks --key-file swapkey.luks $DRIVE swap
4. Create and mount a new swap:
sudo mkswap /dev/mapper/swap
sudo swapon /dev/mapper/swap
The swap-header.luks
is the drive header file.
More detailed, look here:
https://wiki.archlinux.org/title/Dm-crypt/Device_encryption
You donβt if you can spare a couple more gigs
Have tested root to tmpfs yesterday, this is my notes:
-
After rd.break, we need to mount everything manually which I donβt like so I did change the base dracut script instead of using rd.break which would boot same as the normal does.
-
The difference in performance is not too significant (I install template and run it from varlibqubes), but still good since thereβs no additional log from now on.
After researching for a few days I got many new thing that I donβt know, for the qubes os live project, Iβve seen joanna post about the disadvantage of using qubes live, and one of the reason is
- We had to solve the problem of Qubes too easily triggering Out Of Memory
condition in Dom0 when running as Live OS.This last problem has been a result of Qubes using the copy-on-write backing for
the VMβs root filesystems, which is used to implement our cool Template-based
scheme [2]. Normally these are backed by regular files on disk β even though
these files are discardable upon VM reboots, they must be preserved during the
VMβs life span, and they can easily grow to a few tens of MBs per VM, sometimes
even more. Also, each of the VMβs private image, which essentially holds just
the user home directory, typically starts with a few tens of MBs for an βempty
VMβ. Now, while these represent rather insignificant numbers on a disk-basked
system, in case of a LiveUSB scenarios all these files must be stored in RAM,
which is always a scare resource on any OS, and especially Qubes.
I will continue my report in next week about the progress.
Did you find time to continue this?
havenβt try again, maybe in the next time.
Also, because Qubes manager will not show a in-RAM created VMs after PC reboot.
Therefore, change the default pool parameters to save new VMs as files (instead as LVM volumes) will be more user-friendly. This is an example parameters for opt00 file-pool:
qvm-pool add opt00 file -o dir_path=/opt
qubes-prefs --set default_pool opt00
qubes-prefs --set default_pool_private opt00
qubes-prefs --set default_pool_root opt00
qubes-prefs --set default_pool_volatile opt00
If you need to remove the wrongly created VMs as LVM Volumes you can use lvdisplay
, lvremove
commands. Itβs probably a way to clone (copy) a VM instead of removal. Anyway, itβs best to avoid a compromising names of your virtual machines.
I have tried this and after step 3 and reboot the system boots just as it did before.
On step 1 do I only need to add rd.break at the end of the line and leave the rest of the line as it is?
Try cat /proc/cmdline
to find out booted kernel command line parameters.
Try sudo find /boot -name "*.cfg"
to search the grub config location.
stant, you should edit the line as shown in the example in first post
I have been told 40GB ram is not enough for running qubes in ram.
The way the βQubes OS in tmpfsβ implies that AppVMs are simply stored on an encrypted disk (SSD). The main trick is the SSD is encrypted with a detached header. Without the detached header file, your disk will be the same as empty disk. The file is stored somewhere else from the SSD.
We can make the task easier and donβt boot Qubes in RAM to make traceless/stateless system:
You can just mount the SSD`s filesystem containing your secret AppVMs files above the default file system containing public AppVMs files. So you will get two versions of the same AppVMs: secret and public.
When youβve done your secret daily work, you should stop all you secret AppVMs and unmount the disk (SSD filesystem). Then the public (non-secret) AppVMs becomes available on the lower (layer) file system directory. So youβre not leaving any trace of your work outside the encrypted SSD. You just need to hide your SSD`s detached header file.
If human right violators will force you to decrypt the system, you will boot to your Qubes OS, show you AppVMs and say that (secret;) SSD doesnβt contain any data. You can store your detached header file (16MB size) in a cloud file hosting service or somewhere else.
In general terms you will need to:
- Encrypt you SSD with a detached header [
cryptsetup
] - Create a pool to store an AppVMs files (choose a storage directory) [
qmv-pool
] - Mount the encrypted SSD to the AppVMs files storage directory [
mount
] - Create an AppVMs in this pool [
qvm-create -P
]
All needed terminal commands are available on this thread.
This method doesnβt change the OS system requirements. You just need to use non-compromising names for your AppVMs. And disable the swap
, I guess.
I must warn you, there may be a slight difficulty in changing the parameters of both (secret and public) machines from Qubes Manager, private storage size, for example.