This update includes the addition of a Disable Dom0 Swap step and file verification sub-steps.
Edit Updates thus far:
#1. 2023-11-11
Corrected malformed curly/angled-quotes to now be plain straight-quote characters ("), by removing improper HTML code tags that did not work well with Qubes Forum software (Discourse), which was preventing the stateless boot software from running properly if users had copied the malformed quote characters verbatim.
Thanks a lot @qstateless and @xuy . You came up with a big breakthrough for Qubes IMHO. I am a big noob and I am concerned that I might mess up or misunderstand something, so as a good noob I went to a language model and asked to make a script based on the instructions of @qstateless .
#!/bin/bash
# Check if the script is running as root
if [ "$(id -u)" != "0" ]; then
echo "This script must be run as root" 1>&2
exit 1
fi
# Step 0: Create a Save Qube (optional)
echo "Creating a Save Qube..."
qvm-create -l blue --prop=netvm= save
# Step 1: Disable Dom0 Swap
echo "Disabling Dom0 Swap..."
sudo swapoff -a
# Step 2: Edit GRUB to Increase Dom0's Max RAM
echo "Editing GRUB Configuration..."
sed -i 's/dom0_mem=max:4096M/dom0_mem=max:10240M/' /etc/default/grub
# Step 3: Regenerate with New GRUB Configuration
echo "Updating GRUB Configuration..."
if [ -d /boot/efi/EFI/grub.cfg ]; then
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo grub2-mkconfig -o /boot/efi/EFI/grub.cfg
else
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
fi
# Step 4: Make New Directory for Dracut Automation Module
echo "Creating Dracut Automation Module directory..."
sudo mkdir -p /usr/lib/dracut/modules.d/01ramboot
# Step 5: Create Dracut Script File module-setup.sh
echo "Creating module-setup.sh..."
cat <<EOF > /usr/lib/dracut/modules.d/01ramboot/module-setup.sh
#!/usr/bin/bash
check() {
return 0
}
depends() {
return 0
}
install() {
inst_simple "\$moddir/tmpfs.sh" "/usr/bin/tmpfs"
inst_hook cleanup 00 "\$moddir/pass.sh"
}
EOF
sudo chmod 755 /usr/lib/dracut/modules.d/01ramboot/module-setup.sh
# Step 6: Create Dracut Script File pass.sh
echo "Creating pass.sh..."
cat <<EOF > /usr/lib/dracut/modules.d/01ramboot/pass.sh
#!/usr/bin/bash
command -v ask_for_password >/dev/null || . /lib/dracut-crypt-lib.sh
PROMPT="Boot to RAM? (y/n)"
CMD="/usr/bin/tmpfs"
TRY="3"
ask_for_password --cmd "\$CMD" --prompt "\$PROMPT" --tries "\$TRY" --ply-cmd "\$CMD" --ply-prompt "\$PROMPT" --ply-tries "\$TRY" --tty-cmd "\$CMD" --tty-prompt "\$PROMPT" --tty-tries "\$TRY" --tty-echo-off
EOF
sudo chmod 755 /usr/lib/dracut/modules.d/01ramboot/pass.sh
# Step 7: Create Dracut Script File tmpfs.sh
echo "Creating tmpfs.sh..."
cat <<EOF > /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh
#!/usr/bin/bash
read line
case "\${line:-Nn}" in
[Yy]* )
mkdir /mnt
umount /sysroot
mount /dev/mapper/qubes_dom0-root /mnt
modprobe zram
echo 10G > /sys/block/zram0/disksize
/mnt/usr/sbin/mkfs.ext2 /dev/zram0
mount /dev/zram0 /sysroot
cp -a /mnt/* /sysroot
exit 0
;;
[Nn]* )
exit 0
;;
esac
EOF
sudo chmod 755 /usr/lib/dracut/modules.d/01ramboot/tmpfs.sh
# Step 8: Make New Dracut Config File ramboot.conf
echo "Creating ramboot.conf..."
cat <<EOF > /etc/dracut.conf.d/ramboot.conf
add_drivers+=" zram "
add_dracutmodules+=" ramboot "
EOF
# Step 9: Regenerate with New Dracut Automation Module
echo "Regenerating Dracut..."
sudo dracut --verbose --force
echo "Setup completed. Please manually shut down and restart your system to test Qubes Stateless."
Is this any good? Would it work just moving it to dom0 and executing it as root?
I found the slight code error that is the culprit preventing your Qubes Stateless from running on boot.
There indeed was a code error preventing a successful run within the tmpfs.sh file.
The case logic in my posted tmpfs.sh file ended like this:
[Nn]* )
exit 0
;;
esac
However, it should have ended like this:
[Nn]* )
exit 0
;;
* )
exit 1
;;
esac
It was missing this slight portion of code near the end:
* )
exit 1
;;
This was causing the stateless “Boot to RAM? (y/n)” prompt not to show and be automatically skipped.
Somehow, between my development computer and my test computer, a code regression occurred, where my test computer received the good working code but my development computer was showing the bad non-working code making me misbelieve it was good.
Error is fixed now. Should be fully working. My apologies for the frustration.
Yeah, big thanks to @xuy for researching and delivering the innovation to us. I’ve been using it since last year and it is a game changer. I hope my step-by-step instructions can now make it accessible for more people in the Qubes community.
Couldn’t be sure without testing. A lot of it looks ok at a glance, but not sure.
I may be thinking of developing an automated Qubes Stateless installer script in the upcoming future to make this system even more easy.
Yes! My fault. Error found and corrected. See my prior post for explanation.
I am really eager to complete this and build my system around it so I already made the changes you suggested.
I get asked if I want to boot to RAM after inserting the decryption pass, but I tried to open a new tab in firefox on the personal domain and after restarting and booting selecting n I find the tab where it was . So I either did not understand how this works or something went wrong and the system is not running in RAM.
@Bob3 - Good to hear that the “Boot to RAM? (y/n)” prompt is showing up for you now.
Your stateless system is likely working now and there’s probably just a slight misunderstanding on how the system works…
If your personal qube was created in the Normal Persistent mode, then it will save anything written to it on your storage drive, even while using that personal qube during Stateless mode.
With the Qubes Stateless implementation, it is specifically the Dom0 environment that runs in RAM.
Your storage drive will still be accessible and writable, along with any pre-existing TemplateQubes, AppQubes, data files, etc that were created in the Normal Persistent mode.
Any AppQubes you create purely while in Stateless mode using the “varlibqubes” storage pool will only exist in RAM and be fully gone and wiped after restart, along with their data fully gone and wiped, unless you otherwise persistently save such data somewhere else (like a persistent “save” qube) while in a stateless session.
If you don’t want any changes saved within an AppQube, then you likely need to consider creating such qubes while in Stateless mode using the “varlibqubes” storage pool, which places them within Dom0 while running in RAM, which is fully wiped upon restart. Note the RAM-based storage space for “varlibqubes” is only limited to your free Dom0 memory.
The current implementation of Qubes Stateless is a “hybrid” environment, where Dom0 is running in RAM but what pre-exists on your storage drive is still accessible and writable (including any persistently created qubes). This is different than a full blown Qubes Live system, where everything is non-persistent in RAM by default for Qubes Live. There are some tradeoffs to each type of system, Qubes Stateless vs Qubes Live, but with some strategic configuration, you can likely make Qubes Stateless act how you want it to.
More info on these topics has been previously explored somewhat throughout this thread.
Feel free to ask further questions and provide further thoughts.
As far as I understood then I can create a “stateless qube” using “varlibqubes” storage pool while using the non stateless mode, set it up as I wish and them use the qube for daily work, everything that was done in that qube will be deleted at reboot every time I boot into “stateless mode”.
I haven’t personally made use of that approach, but I think it should work.
My understanding…
Create work qube using “varlibqubes” storage pool in Normal Persistent mode. This work qube will persistently exist as a file-backed qube in Dom0 within /var/lib/qubes.
Boot into Stateless mode. The work qube and its persistent files/configurations will load just as they were in your last Persistent mode session, but in RAM during your Stateless mode session.
Shutdown Stateless mode. Any and all changes in the work qube will be fully wiped and it will entireley revert back to its previous state from before your Stateless mode session began.
The “varlibqubes” storage pool is a file-backed (stored as traditional files) in Dom0 within the /var/lib/qubes directory. Since Dom0 loads into RAM during Stateless mode, all your file-backed qubes that use “varlibqubes” should not be able to persist any state from their operations in Statless mode.
A small caveat: If that work AppQube is based on a persistent TemplateQube and that persistent TemplateQube is updated or changed while in Stateless mode, then those TemplateQube changes will persist for all qubes that use this template, including the “stateless” work qube. However, template updates/changes usually are only for application packages, where your “Home” folder’s data files and app configurations should not be affected in the work qube itself. You can go deeper and customize just about anything to work non-typical in Qubes OS, but typical use of TemplateQubes should not likely interfere with your “stateless” work qube plans.
Different combinations of when, how, where you create & use various types of qubes & data can have different state outcomes. So, yeah, Qubes Stateless is an “advanced hybrid state system” like that.
Testing and confirming your expectations is always the key!
I did some further digging into and testing of Qubes Stateless operations.
It seems there are some lingering issues & caveats with achieving fully stateless (anti-forensic) use of Qubes Stateless.
1. Dom0 Still Swaps to Storage Drive
More info here on Wikipedia for those not familiar with what swap is.
It appears that Dom0 Swap is still enabled with each system restart, and that @xuy’s recommended command sudo swapoff -a in post #18 for turning off Dom0 Swap is only temporary for any given Persistent mode session or Stateless mode session, but doesn’t last beyond restart (or maybe even a logoff?).
We need to come up with a persistent way to disable Dom0 Swap. I haven’t researched this yet.
In the meantime, you should run the temporary command in the Dom0 Terminal at the beginning of every new Qubes Stateless session…
sudo swapoff -a
2. AppQubes Typically Created in ‘varlibqubes’ Still Partially Using Storage Drive
While testing in Qubes 4.2-RC4, I discovered that creating an AppQube that is based on a TemplateQube and setting the storage pool to ‘varlibqubes’ stores everything in Dom0’s /var/lib/qubes directory, EXCEPT for one thing I’ve found…
Running in Dom0 Terminal:
lsblk
lsblk | grep YOURQUBENAME
…reveals that your AppQube is still storing its ‘root’ volumes on your storage drive.
Here, you can clearly see volumes named ‘qubes_dom0-vm--YOURQUBENAME--root--snap’.
These ‘root’ volume can write sensitive data during your Qubes Stateless session to your persistent storage drive, which could potentially be recovered from the storage drive.
For a fully stateless workaround with AppQubes, see the approaches in “4. Workarounds for Fully Stateless AppQubes & DisposableQubes”.
3. DisposableQubes in Stateless Mode are Typically Not Fully Stateless
You should be aware that typical default configurations of DisposableQubes while in Stateless mode are not fully stateless and store their data on your storage drive.
For a fully stateless workaround with DisposableQubes, see specific approah “c” in “4. Workarounds for Fully Stateless AppQubes & DisposableQubes”.
4. Workarounds for Fully Stateless AppQubes & DisposableQubes
Thankfully, there seems to be a fully stateless workaround for the issues I described in “2. AppQubes Typically Created in ‘varlibqubes’ Still Partially Using Storage Drive” and “3. DisposableQubes in Stateless Mode are Typically Not Fully Stateless”, although quite costly in some RAM space…
You can take a few different approaches to resolve this fully stateless issue now:
a. Instead of using AppQubes based on TemplateQubes, you could alternatively create StandaloneQubes in the ‘varlibqubes’ storage pool, which appear to store ALL data in traditional image files within Dom0’s ‘/var/lib/qubes’ directory. This is very costly in Dom0 RAM, as it copies your entire TemplateQube’s OS into Dom0 RAM space for each StandaloneQube you make, whether it is actively running or not, which is usually multiple extra GBs per qube, in addition to the RAM it takes to store any user files and the RAM it takes to run and operate the qube’s OS & apps.
b. You could create a new TemplateQube from the previous TemplateQube you want to use, but store that new TemplateQube in the ‘varlibqubes’ storage pool. Then create a new AppQubes based on this new TemplateQube, and store this new AppQube in the ‘varlibqubes’ storage pool too. Now, when you use this new AppQube, it appears to store ALL data in traditional image files within Dom0’s ‘/var/lib/qubes’ directory. This is as costly as the other method for the first AppQube, but you do not have to copy & store the entire TemplateQube OS root filesystem for every AppQube you want to make with it, so this saves a lot of RAM space for using more than one qube.
c. Like “b”, you could create a new TemplateQube from the previous TemplateQube you want to use, but store that new TemplateQube in the ‘varlibqubes’ storage pool. Then create a Disposable Template by creating new AppQubes based on this new TemplateQube, and store this new AppQube in the ‘varlibqubes’ storage pool too. After creation, in the settings of this AppQube, under the “Advanced” tab, you can check to turn on “Disposable template” and after applying also select “Default disposable template” to either be “(none)” or that very same AppQube itself. Now, you can use both this TemplateQube and Disposable Template AppQube to create new AppQubes and DisposableQubes fully within the ‘varlibqubes’ storage pool. This is likely to generally be the most desirable approach for most people.
Here is an example implementation of approach “c”:
Let’s say you want to base some of your fully stateless qubes on the ‘debian-12-xfce’ persistent template.
In Persistent mode: Create & Configure a new TemplateQube named ‘debian-12-xfce-stateless’ based on ‘debian-12-xfce’ and choose to store it in storage pool ‘varlibqubes’ (Advanced tab).
In Persistent mode: Create & Configure a new AppQube named ‘debian-12-xfce-stateless-dvm’ based on ‘debian-12-xfce-stateless’ and choose to store it in storage pool ‘varlibqubes’ (Advanced tab).
In Persistent mode: After creation, for the AppQube ‘debian-12-xfce-stateless-dvm’, change the ‘Advanced’ tab setting ‘Disposable template’ to be checked as turned on (click Apply), then the ‘Default disposable template’ to either be ‘(none)’ or ‘debian-12-xfce-stateless-dvm’ itself.
In Persistent mode: Create & Configure any new AppQubes based on ‘debian-12-xfce-stateless’ that you want to exist across multiple stateless boot sessions and choose to store them in storage pool ‘varlibqubes’ (Advanced tab).
In Stateless mode: You are free to now use any AppQubes based on ‘debian-12-xfce-stateless’ and DisposableQubes based on ‘debian-12-xfce-stateless-dvm’, which appear to remain fully stateless by storing ALL data in traditional image files within Dom0’s ‘/var/lib/qubes’ directory (that directory gets wiped and reset back to match the state of your last persistent session once your stateless session is powered down).
One may need to think about re-creating more or all of their system’s various types of qubes to be fully stateless like this, if needing such levels of statelessness. It should be possible to make every single qube on one’s system be fully stateless in ‘varlibqubes’ storage pool.
With higher fully stateless RAM space demands, you may need a computer with higher amounts of hardware RAM and an increase to the ’ dom0_mem=max:10240M’ setting in my Qubes Stateless step-by-step instructions to be set meaningfully higher than ‘10240M’.
5. The Pull Out Method
One approach I experimented with last year was to boot from a USB drive, and once Qubes Stateless was at the login screen, I just pulled out the USB drive, and the system seemed to continue working fine.
I haven’t done further testing on this approach yet, but it could be a powerful hardware enforced method for ensuring you are fully stateless.
Qubes Stateless is an advanced hybrid state system, so controlling your level of statelessness can be a complex thing to manage, if desiring to be fully stateless.
Feel free to ask questions and provide further thoughts.
You’re welcome @qubesuser1234. Glad it’s working out for you and others. Awesome that you’re running a stateless Qubes OS Dom0 now! Super thanks to @xuy!
Is there anyway my account can be adjusted so I can edit that post beyond the 30 day limit?
Rather than continually “spamming” this thread with new big, long, repetitively new step-by-step instruction posts whenever there may be an instructions update, it seems more clean & correct to be able to edit the single instructions post and then make a brief notification/explanation post with a link to the “canonical” instructions post for those who may be further interested.
Thanks @gonzalo-bulnes! Edit ability for the post solves, so a “wiki” works I guess.
As far as making the post its own independent topic, I’m not sure about that yet. This version of instructions are primarily meant to be a step-by-step implementation of @xuy’s how-to posts within this thread. I may decide to fundamentally upgrade and re-implement the whole Qubes Stateless system in the future, so that would certainly deserve its own topic.
For now, let’s just keep this post in-place, and I will let you know if that should change.
This update includes an instruction to permanently disable Dom0 Swap.
This update also includes many re-written explanations to better explain what Qubes Stateless is and how it works. For those making use of Qubes Stateless, it may be worth re-reading.
Feel free to ask any questions or make any comments. Enjoy!
I read this post yesterday and read it closer today.
I can see where this would be useful for a live USB setup, but I think it would be better as a live DVD, a sort of "take your hard drive free laptop to DEFCON’ kinda thing, where you can leave it sitting somewhere and nobody can do anything that would survive a restart.
I am not sure what value there is to having dom0 running from ram on a system that does have a disk. If something in a VM can reach out and compromise dom0, it’s already compromised Xen, and it can fiddle with VMs on disk. I guess dom0 mods for persistence, something that kinks one or more VMs each time the system started, would be a potential problem?
But the “first, let’s assume dom0 is compromised” seems like an enormous stretch. A quick Google shows there have been four ‘game over’ Xen breakouts during the life of this project. Has there every been a proof of concept demonstrated?
Isn’t there a trusted boot process that would render persistence impossible?
From a hardcore security perspective, you are generally right that a traditional boot drive is still accessible and writable, even with Dom0 running from RAM. So a breakdown of Qubes OS security via a Xen VM breakout exploit could still compromise one’s system across restarts, if a persistent drive is used.
Detectable at least, with a TPM. I’ve read about things like UEFI Secure Boot, Heads, Anti Evil Maid, TrenchBoot (currently being integrated for Qubes AEM by 3mdeb), etc.
Yes. Using write-protected USB or read-only DVD boot media would certainly increase one’s ability to eliminate the possibility of persistent exploits that could otherwise survive a system restart.
I see a few general reasons for making a Qubes Dom0 boot session be disposable in RAM…
System Administration
Anti-Borking
Anti-Forensics
Anti-Fingerprinting
System Security
System Administration:
Having to account for and manage less state in less places can generally help ease sysadmin burdens. I personally take advantage of this benefit across a small fleet of multiple computers I administer. I’m planning to go further and use this to eliminate the use of local drives via RAM-based Qubes OS over PXE Network Boot.
Anti-Borking:
If one wants to run scripts, tests, etc that might bork or mess up one’s Dom0, then running from RAM allows for a quick reset to clean system state. I personally have a Qubes-based development test computer setup specifically for this.
Anti-Forensics:
On a normally-functioning and non-compromised system, one can achieve a greater level of anti-forensic / disposable computing for the entire Qubes OS system by having no Dom0 logs/state saved.
Anti-Fingerprinting:
If someone needed their Qubes OS environment to have the same fingerprint between multiple restarts, then a RAM-based Dom0 would help with achieving that.
System Security:
With a persistent boot drive, while running Dom0 from RAM, a Dom0 exploit would have to take the extra step of reaching out to compromise the drive or drive contents, beyond compromising the running Dom0 environment within RAM. So there would probably be some unaware exploits that do not assume Dom0 is even in RAM, where one’s system could be further protected against such exploits from persistent compromise beyond a restart. Not absolute security, but one more step further.
Of course, one could potentially use write-protected USB or read-only DVD boot media too. I experimented with RAM-based Dom0 and write-protected USB last year using Qubes 4.1, which I believe I wrote about some in this thread.
Layering various system attributes can potentially increase the fitness of a RAM-based Dom0 for various use cases and use conditions. A RAM-based Dom0 is another key attribute of choice that is now accessible to all who may find a use for it. Big thanks to @xuy for the technique.
My LinkedIn profile says R&D oriented CTO, and that’s pretty much why I’m in here. I talk to people who handle potentially “hot” data, or who operate in dangerous places. My big picture goal is living with Qubes daily on an HP Z420, then acquiring a stout laptop, likely a Dell Precision Xeon machine, and I expect pretty soon I’ll start getting messages that say “So … do I need to get this Qubes stuff, too?”
I’m more of an integrator than a developer, and I’d be advising people in operations.
I’ve encountered an Evil Maid in the wild and it is an ongoing issue, one that I suspect will ramp up as things in the U.S. increasingly destabilize. Ensuring the system I come home to is running the same OS I shut down when I left is important to me, and to others.
Write protected USB? Does such a thing actually exist? I recall having write protected SD adapters for microSD, this was just a simple plastic slide that put one pin out of service. I need to do some research here.
Having dom0 in ram looks like it’s good for development, testing, etc, but the added capital cost for groups of irregulars who struggle to get just A laptop that’ll run Qubes … my systems have the capability to do this, but as a rule I do not think that will be the case.
I moved at the start of the month and now I’ve got an environment where I can push off, do a half turn while rolling, and be in front of my second Z420, instead of having to run up and down stairs. I have a couple SATA SSDs, a couple small PCIe NVMe, and an external USB to SATA drive carrier. I’m working through the production scenarios - a person in the field with a single laptop, a single external drive, how do they install/run/backup? How do they handle seizure or theft of a machine?
So … I can see a dozen things I need to do prior to ever considering mucking around in dom0 internals and I’m likely the only one in my environment who would use this tmpfs solution. That could maybe change if a true read only USB device were available, but I suspect there are other keyring U2F devices that would accomplish more.
Yes. Having a securely measured and verifiable boot process, remote backup, and battery-redundant remote video/sensor physical monitoring of one’s spaces & devices would likely be important to crafting an Evil Maid solution. The Haven app was made to help with evil maid attacks, I believe.
Yes. Write protected USB devices do exist. I’ve used USB thumbdrives with a WP switch, and SATA to USB adapters with a WP switch, and USB proxies that had a WP feature. Not sure how their circuitry is implemented. Maybe usually a simple one pin disable slide, like you mentioned.
Without checking, I’ve also never heard of true RO USB (read-only), as in being like ROM (read-only memory). I’m just familiar with WP USB (write-protected). Since one probably has to be able to write/rewrite data onto the USB storage device to be generally useful, any read-only/write-protection features would probably need to be switchable or added in-line as a secondary device. A write-once/non-rewriteable optical disk (DVD, etc) via a USB connected reader is the only thing I could think of for true RO USB.
Nice. I know that feeling well. I have had to do the repetitive up and down stairs sysadmin bit many times.
If a network could be utilized without major issues (and that’s potentially a big IF in hostile scenarios), then, for local data purposes at least, I’d think it would be ideal to simply not store such data locally, and just access/load such things remotely instead.
Best of luck with your pursuits! Qubes is really a fun and powerful OS for us security-minded geeks. With how easy other monolithic OSes can be penetrated, like a hot knife through butter, I think something like Qubes is a necessity to those wanting a fighting chance at having their digital stuff remain their own.
I’ve implemented Qubes into some non-technical people’s lives too, however, it can take consistent IT assistance. For the extra effort, either be paid very well, love the people, or love the mission!
.
