Really disposable (RAM based) qubes

qubist · November 11, 2023, 1:15pm

Are there any existing GitHub issues related to (but opened before) this thread or should we probably open one?

I wonder how to approach this properly, so that it doesn’t get just quickly rejected.

Tezeria · November 11, 2023, 3:09pm

@qubist Thanks a lot!
Like @whoami , everything works perfectly!

qubist · November 11, 2023, 6:53pm

Friends,

As it seems, we have all been fooling ourselves the whole time. Our “RAM based” qubes are not what we thought they were (i.e. they are not RAM based):

github.com/QubesOS/qubes-issues

qvm-pool/volume CLI tools and files report unrealistic space usage for disposable qubes

opened 12:39PM - 11 Nov 23 UTC

emanruse

T: bug P: default

### Qubes OS release 4.1.2 ### Brief summary `qvm-*` command line t…ools report unrealistic storage usage for disposable qubes. ### Steps to reproduce Create a 1 GiB RAM based pool: ``` user@dom0:~ > mkdir -p ~/tmp/mydir user@dom0:~ > sudo mount --types tmpfs --options size=1G my_ram_pool /home/user/tmp/mydir user@dom0:~ > qvm-pool add my_ram_pool file --option revisions_to_keep=1 --option dir_path="${HOME}/tmp/mydir" ``` Create a named disposable qube: ``` user@dom0:~ > qvm-create --class DispVM AAA -P my_ram_pool --label=red user@dom0:~ > qvm-pool info my_ram_pool | grep usage usage 0 ``` Start the qube: ``` user@dom0:~ > qvm-run AAA xterm ``` and create a 500 MiB in it: ``` user@AAA:~ > dd if=/dev/urandom of=~/500mb.img bs=4k iflag=fullblock,count_bytes count=500M 128000+0 records in 128000+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 1.59195 s, 329 MB/s user@AAA:~ > ls -l 500mb.img -rw------- 1 user user 524288000 2023-11-11 14:11:37 500mb.img ``` Check space usage in dom0: ``` user@dom0:~ qvm-pool info my_ram_pool | grep usage usage 8192 user@dom0:~ > df -h ~/tmp/mydir/ Filesystem Size Used Avail Use% Mounted on my_ram_pool 1.0G 8.0K 1.0G 1% /home/user/tmp/mydir user@dom0:~ > ls -l ~/tmp/mydir/appvms/AAA/volatile.img -rw-r--r-- 1 root root 12884901888 2023-11-11 14:10:25 /home/user/tmp/mydir/appvms/AAA/volatile.img user@dom0:~ > du -hs ~/tmp/mydir/appvms/AAA/volatile.img 8.0K /home/user/tmp/mydir/appvms/AAA/volatile.img user@dom0:~ > qvm-volume info AAA:volatile | grep -E '(size|usage)' size 12884901888 usage 8192 ``` ### Expected behavior The numbers reported by the `qvm-` tools should match actuality. Then one can know how much space is left in a pool etc. ### Actual behavior For any running DispVM the reported usage is always 8 KiB and the reported volatile volume size (as well as volatile.img file size) is always 12 GiB. 12 GiB is obviously much more than the actual capacity of the pool (only 1 GiB). The actual "disk" space used by the content inside the qube itself (500 MiB) is obviously much more than the 8 KiB reported by the qvm- tools. It seems this happens only with disposables. For AppVMs it is possible to see actual increase of space usage while the qube is running.

Tezeria · November 12, 2023, 2:40pm

I admit that I’m not sure I understand everything (a bit too technical for me).

github.com/QubesOS/qubes-issues

qvm-pool/volume CLI tools and files report unrealistic space usage for disposable qubes

opened 12:39PM - 11 Nov 23 UTC

emanruse

T: bug P: default needs diagnosis C: storage affects-4.1

### Qubes OS release 4.1.2 ### Brief summary `qvm-*` command line t…ools report unrealistic storage usage for disposable qubes. ### Steps to reproduce Create a 1 GiB RAM based pool: ``` user@dom0:~ > mkdir -p ~/tmp/mydir user@dom0:~ > sudo mount --types tmpfs --options size=1G my_ram_pool /home/user/tmp/mydir user@dom0:~ > qvm-pool add my_ram_pool file --option revisions_to_keep=1 --option dir_path="${HOME}/tmp/mydir" ``` Create a named disposable qube: ``` user@dom0:~ > qvm-create --class DispVM AAA -P my_ram_pool --label=red user@dom0:~ > qvm-pool info my_ram_pool | grep usage usage 0 ``` Start the qube: ``` user@dom0:~ > qvm-run AAA xterm ``` and create a 500 MiB in it: ``` user@AAA:~ > dd if=/dev/urandom of=~/500mb.img bs=4k iflag=fullblock,count_bytes count=500M 128000+0 records in 128000+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 1.59195 s, 329 MB/s user@AAA:~ > ls -l 500mb.img -rw------- 1 user user 524288000 2023-11-11 14:11:37 500mb.img ``` Check space usage in dom0: ``` user@dom0:~ qvm-pool info my_ram_pool | grep usage usage 8192 user@dom0:~ > df -h ~/tmp/mydir/ Filesystem Size Used Avail Use% Mounted on my_ram_pool 1.0G 8.0K 1.0G 1% /home/user/tmp/mydir user@dom0:~ > ls -l ~/tmp/mydir/appvms/AAA/volatile.img -rw-r--r-- 1 root root 12884901888 2023-11-11 14:10:25 /home/user/tmp/mydir/appvms/AAA/volatile.img user@dom0:~ > du -hs ~/tmp/mydir/appvms/AAA/volatile.img 8.0K /home/user/tmp/mydir/appvms/AAA/volatile.img user@dom0:~ > qvm-volume info AAA:volatile | grep -E '(size|usage)' size 12884901888 usage 8192 ``` ### Expected behavior The numbers reported by the `qvm-` tools should match actuality. Then one can know how much space is left in a pool etc. ### Actual behavior For any running DispVM the reported usage is always 8 KiB and the reported volatile volume size (as well as volatile.img file size) is always 12 GiB. 12 GiB is obviously much more than the actual capacity of the pool (only 1 GiB). The actual "disk" space used by the content inside the qube itself (500 MiB) is obviously much more than the 8 KiB reported by the qvm- tools. It seems this happens only with disposables. For AppVMs it is possible to see actual increase of space usage while the qube is running.

What I notice on my side is that when I restart, the /ram_pool folder is completely empty (which is normal) There are just traces of.log files and files in the /run and /etc directory. If I delete the.log files and the /etc/libvirt/libxl/ram_disp.xml file right after closing the ram_disp with a bash script:

#!/bin/bash

sudo rm -rf /etc/libvirt/libxl/ram_disp.xml
sudo rm -rf /var/log/libvirt/libxl/ram_disp.log
sudo rm -rf /var/log/xen/console/guest-ram_disp.log
sudo rm -rf /var/log/qubes/qrexec.ram_disp.log
sudo rm -rf /var/log/qubes/guid.ram_disp.log
sudo rm -rf /var/log/qubes/qubesdb.ram_disp.log

All files disappear (in the /run and /etc folders as well).
There are still files in.desktop, .menu, .directory in the /home/tezeria/ folder (normal because my ram_disp is not based on a template-disposable) and the /var/lib/qubes/appvms/debian-12-ram file (which doesn’t seem to be a problem because the file is empty).
Edit: volatile.img disappear after closing ram_disp as well.

Like i said, i don’t understand everything in your link, and you know i’m not a professional lol , but i don’t see where is the trouble if there is nothing after ram_disp is closing…

Tezeria · November 12, 2023, 3:01pm

Your approach is really close to @unman’s except that you place the pool directly in the RAM (right?). Which I think is a good thing…

github.com

unman/notes/blob/master/Really_Disposable_Qubes.md

## Disposable qubes
In normal use qubes are created on, and changes written to, the disk. There is also extensive logging and signs of the qube are scattered in a number of places.
Sometimes, you want to create a qube which does not leave these traces.  
You can do this relatively simply, by creating a RAM based storage area and using it for a new storage pool.
The qube will persist until the RAM disk is deleted, or the machine is shut down.

A script like this in dom0, will create tmpfs RAM disk, create a new storage pool in that disk, and create a new qube using that pool.  
tmp_qubes.sh:  
```
#!/bin/bash
mkdir /home/user/tmp
sudo mount -t tmpfs -o size=2G new /home/user/tmp/
qvm-pool add -o revisions_to_keep=1 -o dir_path=/home/user/tmp/ newer file 
qvm-create new -P newer -t debian-11  -l purple --property netvm=tor
qvm-run new firefox-esr
```

You can remove the qube, and some of the associated artifacts by script in dom0.  
rmtmp_qube.sh:  
```

This file has been truncated. show original

qubist · November 12, 2023, 7:07pm

@Tezeria

What you don’t see is not what you don’t see (and what I did not see)!

In short:

The actual data of the DispVM (/home) is not kept in RAM but on pool00 where the DVM template resides. It is the private volume (containing /home) which we need to have in RAM but that is currently not possible (without complex tricks) due to the way Qubes OS works.

Your approach is really close to @unman’s except that you place the pool directly in the RAM (right?). Which I think is a good thing…

His script does the same. The result is the same too.

rustybird · November 13, 2023, 12:27pm

unman’s script is different because it creates an AppVM in the tmpfs pool (not a DispVM), so writes to the ‘private’ volume do end up in the tmpfs.

For DispVMs, it’s necessary to at least clone their disposable template into the tmpfs pool to achieve the same. Then set the rw property of the clone’s ‘root’ volume to False - which will be inherited by the DispVMs based on it - redirecting writes to that volume (outside the tmpfs pool) to the ‘volatile’ volume (inside the tmpfs pool). Optionally, set the ephemeral property of the clone’s ‘volatile’ volume to True.

Bob3 · November 13, 2023, 1:07pm

Thanks a lot for your dedication @qubist. I am not technical enough to guess how to use this unfortunately.

For me a few things are not clear, I will try to be the voice of less technical people:

How would I use this script? Just execute it in dom0?
How can I set the template we want to use for the dispVM?
Can I create a shortcut to click on so I can execute this script?

These functionalities are awesome, it would be great to have them in Qubes 4.2 stable release. Meanwhile if we could add to the script the creation of a shortcut icon to use this functionality easily for anyone would be very good.

qubist · November 13, 2023, 7:24pm

@rustybird

unman’s script is different because it creates an AppVM

Ha! I have somehow missed that essential detail. I guess with that trick we can have AppVMs which will be disposable due to the RAM storage. Thanks!

I am testing this right now and another question arises:

What happens if the user tries to e.g. install some software (as root) in the RAM-based AppVM or otherwise adds data to / (non-/rw)? Which volume will be used for that? When I tried, I see no increase in the usage which qvm-volume reports in neither volume (private, volatile or root).

So, where does this data go?

Could you also help with this security question:

Suppose that instead of having separate tmpfs pools for each RAM based AppVM (with separate tmpfs mounts), we have a common large one containing all img files for all such qubes. Does that approach have any anti-security consequences? The idea is to possibly create RAM-based AppVMs replacing the current disk-based sys-* disposable qubes, so that they work entirely in RAM.

qubist · November 13, 2023, 7:25pm

@Bob3

Yes.
-p template=...
Yes.

Tezeria · November 14, 2023, 4:16pm

Tanks for this explanation,
I’m going to bed less stupid tonight!

Tezeria · November 14, 2023, 4:51pm

what i do for a bash script is

Rightclicking on Desktop
Selecting ‘Create a new launcher here’
in the ‘command’s line’ bash /folder_of_the_script/your_bash_script
then i cut the file in the folder i want
After that, i could double-click on this file to execute the script.

qubist · November 14, 2023, 5:08pm

If you chmod +x <script-name>, then you can simply run it directly through /your-dir/<script-name> (no bash before it) or through <script-name> if you placed it in a dir from your PATH.

qubist · November 14, 2023, 8:15pm

@rustybird

Some more questions:

The first approach you suggested (AppVM on a tmpfs pool) works fine.

I am testing the second approach, however I am facing a strange issue after the qube is shut down in the new cleanup function, where I remove the qubes and pools:

qvm-kill "${qube_name}"
qvm-remove --force "${qube_name}" "${template_clone}"
qvm-pool remove "${pool_name}"
sudo umount "${pool_name}"

Strangely, unlike the first case, umount gives an error “target is busy”. The qubes are removed, the pool is removed, yet “target is busy”.

Trying lsof | grep <mountpoint> shows me nothing, so the only way to get out of this is to use umount --lazy (which I know may have unpredictable result and is unsafe).

I really can’t find the reason why this is happening.
Can you explain?

qubist · November 15, 2023, 4:03pm

Updated.

What’s new:

Now, the RAM based qubes are AppVMs. The script clones the DVM template into the RAM pool, sets its property template_for_dispvms to false, thus turning it into a regular AppVM, and runs the specified command. The result is that the private volume is in RAM now.

Also, the whole RAM pool is now encrypted with an ephemeral key.

Special thanks to @rustybird and @marmarek for all clarifications.

Bob3 · November 16, 2023, 6:25pm

When running the cleanup script the /var/log files of the disposable vms do not disappear. I cannot understand why.

qubist · November 16, 2023, 6:54pm

Please provide the actual steps to reproduce.

Bob3 · November 16, 2023, 7:08pm

I just put the script in /home/user/ of dom0 and name it cleanup.sh. then I run ./cleanup.sh after making the script executable and nothing happens.

qubist · November 16, 2023, 7:21pm

Strange.
What if you run it with bash -x ~/cleanup.sh? Where does it stop?

rustybird · November 16, 2023, 7:23pm

The ‘root’ volume.

The rw property of the AppVM’s ‘root’ volume should be set to False, so that writes to it are diverted to (the ‘volatile’ volume in) the pool.

You mean the pool’s ephemeral_volatile property? That’s just a default for the ephemeral property of ‘volatile’ volumes in the pool, nothing more.