Qubes in tmpfs 🤫

[qstateless]

Thanks @gonzalo-bulnes! Edit ability for the post solves, so a “wiki” works I guess.

As far as making the post its own independent topic, I’m not sure about that yet. This version of instructions are primarily meant to be a step-by-step implementation of @xuy’s how-to posts within this thread. I may decide to fundamentally upgrade and re-implement the whole Qubes Stateless system in the future, so that would certainly deserve its own topic.

For now, let’s just keep this post in-place, and I will let you know if that should change.

Thank you!

[qstateless]

Today, 2023-12-08, I have updated the Step-by-Step Instructions for Qubes Stateless.

This update includes an instruction to permanently disable Dom0 Swap.

This update also includes many re-written explanations to better explain what Qubes Stateless is and how it works. For those making use of Qubes Stateless, it may be worth re-reading.

Feel free to ask any questions or make any comments. Enjoy!

@qstateless

Edit Updates:

#6. 2023-12-08

• Added permanent disable of Dom0 Swap.

#7. 2023-12-08

• Improved several explanations throughout post.
• Minor edits.

[nealr]

I read this post yesterday and read it closer today.

I can see where this would be useful for a live USB setup, but I think it would be better as a live DVD, a sort of "take your hard drive free laptop to DEFCON’ kinda thing, where you can leave it sitting somewhere and nobody can do anything that would survive a restart.

I am not sure what value there is to having dom0 running from ram on a system that does have a disk. If something in a VM can reach out and compromise dom0, it’s already compromised Xen, and it can fiddle with VMs on disk. I guess dom0 mods for persistence, something that kinks one or more VMs each time the system started, would be a potential problem?

But the “first, let’s assume dom0 is compromised” seems like an enormous stretch. A quick Google shows there have been four ‘game over’ Xen breakouts during the life of this project. Has there every been a proof of concept demonstrated?

Isn’t there a trusted boot process that would render persistence impossible?

[qstateless]

Hi @nealr

Good thoughts…

@nealr

If something in a VM can reach out and compromise dom0, it’s already compromised Xen, and it can fiddle with VMs on disk. I guess dom0 mods for persistence, something that kinks one or more VMs each time the system started, would be a potential problem?

From a hardcore security perspective, you are generally right that a traditional boot drive is still accessible and writable, even with Dom0 running from RAM. So a breakdown of Qubes OS security via a Xen VM breakout exploit could still compromise one’s system across restarts, if a persistent drive is used.

@nealr

Isn’t there a trusted boot process that would render persistence impossible?

Detectable at least, with a TPM. I’ve read about things like UEFI Secure Boot, Heads, Anti Evil Maid, TrenchBoot (currently being integrated for Qubes AEM by 3mdeb), etc.

@nealr

I can see where this would be useful for a live USB setup, but I think it would be better as a live DVD, a sort of "take your hard drive free laptop to DEFCON’ kinda thing, where you can leave it sitting somewhere and nobody can do anything that would survive a restart.

Yes. Using write-protected USB or read-only DVD boot media would certainly increase one’s ability to eliminate the possibility of persistent exploits that could otherwise survive a system restart.

@nealr

I am not sure what value there is to having dom0 running from ram on a system that does have a disk.

I see a few general reasons for making a Qubes Dom0 boot session be disposable in RAM…

• System Administration
• Anti-Borking
• Anti-Forensics
• Anti-Fingerprinting
• System Security

System Administration:

• Having to account for and manage less state in less places can generally help ease sysadmin burdens. I personally take advantage of this benefit across a small fleet of multiple computers I administer. I’m planning to go further and use this to eliminate the use of local drives via RAM-based Qubes OS over PXE Network Boot.

Anti-Borking:

• If one wants to run scripts, tests, etc that might bork or mess up one’s Dom0, then running from RAM allows for a quick reset to clean system state. I personally have a Qubes-based development test computer setup specifically for this.

Anti-Forensics:

• On a normally-functioning and non-compromised system, one can achieve a greater level of anti-forensic / disposable computing for the entire Qubes OS system by having no Dom0 logs/state saved.

Anti-Fingerprinting:

• If someone needed their Qubes OS environment to have the same fingerprint between multiple restarts, then a RAM-based Dom0 would help with achieving that.

System Security:

• With a persistent boot drive, while running Dom0 from RAM, a Dom0 exploit would have to take the extra step of reaching out to compromise the drive or drive contents, beyond compromising the running Dom0 environment within RAM. So there would probably be some unaware exploits that do not assume Dom0 is even in RAM, where one’s system could be further protected against such exploits from persistent compromise beyond a restart. Not absolute security, but one more step further.

Of course, one could potentially use write-protected USB or read-only DVD boot media too. I experimented with RAM-based Dom0 and write-protected USB last year using Qubes 4.1, which I believe I wrote about some in this thread.

Layering various system attributes can potentially increase the fitness of a RAM-based Dom0 for various use cases and use conditions. A RAM-based Dom0 is another key attribute of choice that is now accessible to all who may find a use for it. Big thanks to @xuy for the technique.

Thoughts?

[nealr]

Yes, that clarified things for me.

My LinkedIn profile says R&D oriented CTO, and that’s pretty much why I’m in here. I talk to people who handle potentially “hot” data, or who operate in dangerous places. My big picture goal is living with Qubes daily on an HP Z420, then acquiring a stout laptop, likely a Dell Precision Xeon machine, and I expect pretty soon I’ll start getting messages that say “So … do I need to get this Qubes stuff, too?”

I’m more of an integrator than a developer, and I’d be advising people in operations.

I’ve encountered an Evil Maid in the wild and it is an ongoing issue, one that I suspect will ramp up as things in the U.S. increasingly destabilize. Ensuring the system I come home to is running the same OS I shut down when I left is important to me, and to others.

Write protected USB? Does such a thing actually exist? I recall having write protected SD adapters for microSD, this was just a simple plastic slide that put one pin out of service. I need to do some research here.

Having dom0 in ram looks like it’s good for development, testing, etc, but the added capital cost for groups of irregulars who struggle to get just A laptop that’ll run Qubes … my systems have the capability to do this, but as a rule I do not think that will be the case.

I moved at the start of the month and now I’ve got an environment where I can push off, do a half turn while rolling, and be in front of my second Z420, instead of having to run up and down stairs. I have a couple SATA SSDs, a couple small PCIe NVMe, and an external USB to SATA drive carrier. I’m working through the production scenarios - a person in the field with a single laptop, a single external drive, how do they install/run/backup? How do they handle seizure or theft of a machine?

So … I can see a dozen things I need to do prior to ever considering mucking around in dom0 internals and I’m likely the only one in my environment who would use this tmpfs solution. That could maybe change if a true read only USB device were available, but I suspect there are other keyring U2F devices that would accomplish more.

[qstateless]

@nealr

@nealr

Yes, that clarified things for me.

Glad it helped!

@nealr

I’ve encountered an Evil Maid in the wild and it is an ongoing issue, one that I suspect will ramp up as things in the U.S. increasingly destabilize. Ensuring the system I come home to is running the same OS I shut down when I left is important to me, and to others.

Yes. Having a securely measured and verifiable boot process, remote backup, and battery-redundant remote video/sensor physical monitoring of one’s spaces & devices would likely be important to crafting an Evil Maid solution. The Haven app was made to help with evil maid attacks, I believe.

@nealr

Write protected USB? Does such a thing actually exist? I recall having write protected SD adapters for microSD, this was just a simple plastic slide that put one pin out of service. I need to do some research here.

Yes. Write protected USB devices do exist. I’ve used USB thumbdrives with a WP switch, and SATA to USB adapters with a WP switch, and USB proxies that had a WP feature. Not sure how their circuitry is implemented. Maybe usually a simple one pin disable slide, like you mentioned.

@nealr

That could maybe change if a true read only USB device were available

Without checking, I’ve also never heard of true RO USB (read-only), as in being like ROM (read-only memory). I’m just familiar with WP USB (write-protected). Since one probably has to be able to write/rewrite data onto the USB storage device to be generally useful, any read-only/write-protection features would probably need to be switchable or added in-line as a secondary device. A write-once/non-rewriteable optical disk (DVD, etc) via a USB connected reader is the only thing I could think of for true RO USB.

@nealr

I moved at the start of the month and now I’ve got an environment where I can push off, do a half turn while rolling, and be in front of my second Z420, instead of having to run up and down stairs.

Nice. I know that feeling well. I have had to do the repetitive up and down stairs sysadmin bit many times.

@nealr

I talk to people who handle potentially “hot” data, or who operate in dangerous places.

@nealr

I’m working through the production scenarios - a person in the field with a single laptop, a single external drive, how do they install/run/backup? How do they handle seizure or theft of a machine?

If a network could be utilized without major issues (and that’s potentially a big IF in hostile scenarios), then, for local data purposes at least, I’d think it would be ideal to simply not store such data locally, and just access/load such things remotely instead.

@nealr

So … I can see a dozen things I need to do prior to ever considering mucking around in dom0 internals and I’m likely the only one in my environment who would use this tmpfs solution.

@nealr

My big picture goal is living with Qubes daily on an HP Z420, then acquiring a stout laptop, likely a Dell Precision Xeon machine, and I expect pretty soon I’ll start getting messages that say “So … do I need to get this Qubes stuff, too?”

Best of luck with your pursuits! Qubes is really a fun and powerful OS for us security-minded geeks. With how easy other monolithic OSes can be penetrated, like a hot knife through butter, I think something like Qubes is a necessity to those wanting a fighting chance at having their digital stuff remain their own.

I’ve implemented Qubes into some non-technical people’s lives too, however, it can take consistent IT assistance. For the extra effort, either be paid very well, love the people, or love the mission!

Cheers!

[ddevz]

@qstateless

From a hardcore security perspective, you are generally right that a traditional boot drive is still accessible and writable, even with Dom0 running from RAM. So a breakdown of Qubes OS security via a Xen VM breakout exploit could still compromise one’s system across restarts, if a persistent drive is used.

@nealr

Isn’t there a trusted boot process that would render persistence impossible?

Detectable at least, with a TPM. I’ve read about things like UEFI Secure Boot, Heads, Anti Evil Maid, TrenchBoot (currently being integrated for Qubes AEM by 3mdeb), etc.

I’ve been looking at flashable firmware on PCI devices. I wanted to mention that if you wanted a truely “persistence impossible” situation, you would need to address flashable firmware in your PCI devices, as i do not believe that gets addressed in “secure boot/measured boot/trusted boot”.

Here is a link of my attempt at addressing the situation: How to avoid flashable firmware compromises via pass-through PCI devices?

[QubesParanoia]

Good evening, this is a very good solution, I use it regularly. But there are several problems:

Why do we use zram at all in this case? As I understand it, we take a piece of all available memory, occupy it, there is now zram and there is dom0 image and other things from varlibqubes. However, since we use copying via cp with one command, it does not allow copying if the target path does not contain enough space. Simply put, we have 24 GB images that need to be loaded into RAM. We create a zram0 block of 25 GB and everything fits there, compressed and weighs 16 GB. But if we create a zram block of 23 GB, the system will not start at all with an error, since No space left on device will be in the logs when copying. As a result, unnecessary waste of processor time without any gain. Could we replace cp with find + cp or rsync? In theory, they should help to avoid such an error when copying data.

Is lzo-rle the best choice for zram0 now? I use 4.2 Qubes OS, as an experiment I changed the compression method to zstd, as a result the occupied size became 20-30% smaller, the compression time increased a little. Most tests show that the decompression time (i.e. 99% of the time zram0 is used) zstd works faster than lzo-rle

[qstateless]

@ddevz Thanks for the pointer! Bookmarked it to read.

[qstateless]

@QubesParanoia

Good day to you… Yes, having a stateless Dom0 is a great thing.

Could you post your code that you are implementing for this?

In my Step-by-Step Instructions Post, I implemented @xuy’s ZRAM code for the Dom0 RAM size savings (compared to no compression).

I see that you are saying that your Dom0 takes up ~16GB in RAM (quite large!)?

Myself and xuy have noted that we see only about ~4GB to ~6GB of RAM usage for Dom0 (~10GB to ~12GB less than you seem to be reporting!).

One possible explanation may be that you’re storing additional qubes or files in your Dom0, such as file-backed qubes in /var/lib/qubes aka the “varlibqubes” storage pool, so this contents would get copied to RAM as part of your Dom0 filesystem.

I have not played with alternative copying methods (rsync, etc) or compressions (zstd, etc), but would be genuinely interested to know if there is a meaningful improvement to be implemented here, over the ~4GB to ~6GB of Dom0 RAM usage I currently see.

Maybe seeing your code could help teach us something to further implement or iron out some issues?

[QubesParanoia]

Could you post your code that you are implementing for this?

/usr/lib/dracut/modules.d/01ramboot/tmpfs.sh

#!/usr/bin/bash
read line
case "${line:-Nn}" in
[Yy]* )
mkdir /mnt
umount /sysroot
mount /dev/mapper/qubes_dom0-root /mnt
modprobe zram 
echo "zstd" > /sys/block/zram0/comp_algorithm #30G>12GB, speed simularity lz4 (maybe little slower?) 
# echo "lz4" > /sys/block/zram0/comp_algorithm #fastest compression 30GB >16GB
echo 30G > /sys/block/zram0/disksize
/mnt/usr/sbin/mkfs.ext2 /dev/zram0
mount /dev/zram0 /sysroot
cp -a /mnt/* /sysroot
exit 0
;;
[Nn]* )
exit 0
;;
* )
exit 1
;;
esac

@qstateless

Myself and xuy have noted that we see only about ~4GB to ~6GB of RAM usage for Dom0 (~10GB to ~12GB less than you seem to be reporting!).

It’s my error. Fixed It today

I couldn’t get it to work properly with rsync or any other analogue of cp. And before copying all the data to zram, cp checks that the size of zram >= the size of the data that needs to be copied there.

[qstateless]

@QubesParanoia

Good to know about the rsync error and that cp does a size check.

I’m curious as to how much Dom0 RAM usage you are now typically seeing with this better compression enabled?

In a Dom0 terminal, I typically look at the following command…

df -m /dev/zram0

…and look at the “Used” number of RAM Megabytes taken up.

I will have to play with this soon and maybe keep this better compression algorithm as well.

I typically see Dom0 RAM usage grow to higher numbers (+1-2GB), after initial startup, with normal system use over time.

Thanks!

[qstateless]

Note the available compression algorithms for zram (in Qubes 4.2):

In a Dom0 terminal…

cat /sys/block/zram0/comp_algorithm

lzo [lzo-rle] lz4 lz4hc 842 zstd

• lzo
• lzo-rle
• lz4
• lz4hc
• 842
• zstd

I don’t know much about the pros/cons of each yet.

Will be good to do some size & speed testing in the future.

[QubesParanoia]

Standard algorithm, unless otherwise specified lzo-rle is used. It is very compatible, but on a modern OS (fedora37 dom0) it loses to other options in all respects. Lz4 is the fastest and has a minimal load on the processor during operation. Zstd (at the moment, since fedora 37 is outdated - you can’t change the compression strength, this is only available in new versions of the zram package) with a compression strength of 3. It compresses stronger than lz4. In theory, it is slower than lz4 since it is designed for stronger compression. Zstd:1-2 should in theory be faster or equal to lz4 and have a higher compression strength. But I couldn’t really notice the difference in the speed of lz4/zstd on an 8-core processor. Probably, it is very small, since it takes about a minute to send 18gb to zram (after lz4 compression it becomes 8.4).

I’ll run compression tests a bit later and send the results here and will update this post

P.s: do you know any way to track the exact time of OS launch with different algorithms? Since it’s RAM-based, there’s no point in looking at regular logs later, but maybe it can somehow write to a file or somewhere else

[qstateless]

@QubesParanoia

Thanks for the further insights into the compression algorithms.

@QubesParanoia

P.s: do you know any way to track the exact time of OS launch with different algorithms? Since it’s RAM-based, there’s no point in looking at regular logs later, but maybe it can somehow write to a file or somewhere else

I’m not sure about tracking exact time of OS launch.

OS logs may be temporarily available during a stateless session after boot, in order to manually save/copy. Not sure which to look at though.

I suppose tracking different phases could be useful. For example, the time it takes for the tmpfs.sh file to process, the boot time until sys-qubes begin autostarting, and until user login, etc.

Some of this could probably be manually timed, by hitting Esc at the disk passphrase boot step to see the scrolling text boot log and noting the seconds/minutes different phases take.

Also, one may be able to modify the tmpfs.sh file, or other files, to echo the current date time out to a file.

And maybe the compression algorithm has some speed affect upon AppVM booting & running, after Dom0 is loaded.

@QubesParanoia

I’ll run compression tests a bit later and send the results here and will update this post

Looking forward to seeing it!