I see.
So, I guess we should rather think of some snapshotting system, allowing a more efficient “template family tree” without duplication. That would not solve the problem with updates, but if the tree creation is scripted, I think it would be acceptable to e.g. run an update + recreate it once a week.
What do you think? Is there a proper way to do this with current functionality of Qubes OS?
As it happens, my tree creation IS scripted. I’d never even think of generating all of that by hand. I learned this from sven’s stuff (earlier in this thread).
It’s a combination of bash scripts and salt. (I added the salt later.) I can’t go fully salt because if I try it, templates get cloned before the software is installed on them, necessitating at least two installs of the same thing. A separate call to qubesctl must be made to create and configure each template (in fact to create each Appvm as well). I script those calls because they’re command-line monsters with five or six parameters; scripting forces them to be done in series instead of parallel.
I should probably start another thread to discuss possible options for snapshots.
On multiple templates and deduplication.
I haven’t had time to test this yet. But:
My intention on replying on this post is to restate that I understand this use case and am actually one of the users of this use case. I also use cacher project of unman over qubes 4.1, which is not supported on qubes 4.2 yet. And cacher is a bandwidth life saver. What it does is download once under cacher qube update related packages and then install those cached packages over each templates without redownloading.
If those templates could be deduplicared on block level, then the packages installed content would not be non redundant. Those templates would only occupy space for their differences: that is, differences on what is installed differently for different use cases. And there: deduplication would make sense. The only possible problem there would be a race condition over Q4.2 where multiple templates launching upgrade in parallel could attempt to download the same package at the same time at cacher level, but that should not be a problem.
So not misinterpret what I say. This approach would still take more space then having a minimal debian template shared across multiple qubes. Maintaining multiple templates still requires each templates being updated where cacher help in minimizing download repetition and speed of updating the templates which pulls from cacher, not the internet multiple times.
TLDR:
Therefore, deduplication with bees over BRTFS would be a really desired feature. But we are not there yet.
Glad to see you here!
I have read your whole thread about pool deduplication but I am not an expert on file systems and some things are not immediately clear to me. I understand the concept of snapshots and I have used them on KVM/qemu VMs by creating a basic raw origin, then building variations on top of it.
In Qubes OS, however, I have no idea how to do that. I understand your goal is to have that automated through the underlying file system on which the qube images reside. However, IIUC, that is a longer term thing, not something we can expect any time soon (at least not in a stable well-tested version), right?
So, my hope was to find some manual way of non-duplicating cloning of templates which would allow a more efficient workflow along the lines of automated creation of a template hierarchy, as in this thread.
How can this be done using the standard thin LVM of Qubes OS?
I see. Nothing of that sort exists as far as I know under QubesOS. qvm-volume revert can be abused to keep snapshots on its parents, where configuration can tell how many snapshots to keep where the parent rotate on reboot. As you might expect, a lot of changes means a lot of differences to keep track of which is something that needs a workaround somehow.
The qvm-clone comes to the rescue, but one again without deduplication, setting a new parent. And then applies the revert logic above.
My personal workaround around this today is to use wyng to store only deduplicated data of the whole backup data. But then again, when one restores a backup, there is no deduplication on what is written on the destination.
So this workaround using wyng has a limitation because the restored volume is not deduplicated. This is where I have hope toward bees on BRTFS and all the jazz I said above.
Thanks for clarifying.
I have been reading the template implementation doc, hoping to learn how snapshots in Qubes OS actually work. Unfortunately, the script which it mentions (/etc/xen/scripts/block-snapshot) does not even exist.
Then I found this, which says:
Interesting thoughts. You seem to care a lot about disk space and writes. I never did. Go for it and report back how it worked for you.
So, I have been experimenting with the decremental approach. I am facing some weird issue. Here is what happens:
apt-get purge ... ; apt-get autoremove --purgeResult:
I get errors and the process fails:
https://paste.opensuse.org/pastes/ba55cbcbb096
I have absolutely no idea why this attempts to remove qubes-core-agent.
You’re removing one of its depends (e.g. xdg-utils):
$ apt-cache show qubes-core-agent
Package: qubes-core-agent
Version: 4.2.26-1+deb12u1
Architecture: amd64
Maintainer: unman <unman@thirdeyesecurity.org>
Installed-Size: 729
Depends: apt-transport-https, dconf-cli, dmsetup, gawk, graphicsmagick, init-system-helpers, initscripts | sysvinit-utils, librsvg2-bin, locales, ncurses-term, psmisc, procps, util-linux, e2fsprogs, parted, python3-daemon, python3-qubesdb, python3-gi, python3-xdg, python3-dbus, qubes-utils (>= 3.1.3), qubes-core-qrexec, qubesdb-vm, systemd, xdg-user-dirs, xdg-utils, xen-utils-common, xen-utils-guest, xenstore-utils, python3:any, libc6 (>= 2.34), libqubes-rpc-filecopy2 (>= 4.1.7), dconf-gsettings-backend | gsettings-backend
Recommends: cups, gnome-terminal, gnome-themes-standard, haveged, libnotify-bin, locales-all, mate-notification-daemon, ntpdate, system-config-printer, qubes-core-agent-nautilus, qubes-core-agent-networking, qubes-core-agent-network-manager, x11-xserver-utils, xinit, xserver-xorg-core, xsettingsd, xterm
Conflicts: firewalld, pulseaudio-qubes (<< 4.2.0-1), qubes-core-agent-linux, qubes-core-vm-sysvinit, qubes-gui-agent (<< 4.1.6-1)
Homepage: https://www.qubes-os.org
Priority: extra
Section: admin
Filename: pool/main/q/qubes-core-agent/qubes-core-agent_4.2.26-1+deb12u1_amd64.deb
Size: 94476
SHA256: ea13bf510002a0b80f6d0acf226f90959eb20420d2bf3eefa7beb8d358f3205c
SHA1: 431c45ad5a3891260790cf798d158f4ddfab9465
MD5sum: 56b70ef04b1c4ff9e9df4fa75761c815
Description: Qubes core agent
This package includes various daemons necessary for qubes domU support,
such as qrexec services.
Description-md5: d5c21beb3c25cf7972d88fc97ad4fac4
I am removing the exact same packages which I installed previously.
The question is - why that attempts to remove more than what they installed?
No, you are not.
xdg-utils is already installed in the template. You did not install it.
It could be that you thought you installed it, but if you had checked
the logs you would have seen that this was not so.
What do you mean? If you look at the actual command line, you can see it does not contain xdg-utils. So, I am not uninstalling that explicitly.
IIUC, some of the other packages, which are on my apt-get purge command line, depends on qubes-core-agent, and for some reason it is now considered unnecessary, so apt-get tries to remove it, thus creating this mess.
Why is this happening?
OK. I found the culprit - psmisc. It somehow ended up in my command line. Removing it from there removes all errors and purging works fine. Sorry for bothering.
Go for it and report back how it worked for you.
Here is some more feedback on the decremental approach:
I have not polished it but the it works. In regards to efficiency, it is still not quite to my taste though.
First, it increases the boot time of qubes which need less packages, due to the initial package purging. To remove the packages mentioned in a previous post and clean up, it takes about 57 s (nVME storage). If I move qube’s volatile volume to tmpfs, the time is 37 s - still a significant delay. Additionally, tmpfs volatiles come at the cost of eating RAM from dom0 - the more packages are purged, the more memory is consumed by the CoW system. That might create OOM issues if running many minimized qubes (e.g. sys-*).
The main issue is obviously that GNU/Linux and its file hierarchy are simply not designed with Qubes in mind, so both approaches increase complexity and maintenance for probably marginal benefit. Maybe ZFS could reduce duplication in the incremental approach, but the additional complexity of having multiple templates remains.
If we should go for true minimalism, we need something better. No idea what exactly though. Maybe we should consider the concept of distributed operating systems.
Wait a minute. You installed things in the Template, and in the AppVM you uninstalled them?
You will have to repeat the uninstall every time you run the AppVM, since nothing done in the system area gets saved when an AppVM shuts down. (Things you say later on I didn’t quite understand might indicate you know this.) That seems like an awful lot of writes to disk, when you could do the uninstall just once in a custom template (or better yet not install in the first place).
Wait a minute. You installed things in the Template, and in the AppVM you uninstalled them?
Yes. That’s the essence of the decremental approach.
You will have to repeat the uninstall every time you run the AppVM
Sure.
That seems like an awful lot of writes to disk
There are zero writes to disk if the volatile volume is on tmpfs and the root volume has rw=false.
The only 2 problems which remain are:
(A) The additional RAM for the CoW
(B) The extra time for uninstalling packages at each AppVM boot
(A) might not be a big problem with enough physical RAM. AFAIK, the RAM for dom0 can be increased through a GRUB parameter.
(B) seems significant only if lots of packages are removed. It is possible to have 2 templates (e.g. one for more minimalist stuff, another one for office-like bloatware) but that goes against the idea of having a single template (although 2 is still better than 20).
Another method I have tried:
In the template:
apt-get install package1 package2... --download-only
In AppVM X:
apt-get install packageX && apt-get clean
That is still incremental (nothing is actually installed in the template), and it does not solve (A) and (B). It only reverts (B) - more boot-up time for AppVMs using larger software packages. Additionally, it is not quite clear to me what would happen if a package is downloaded in cache (but not installed) and a system update is run. So, additional complication.
As a possible solution to (A) and (B), here is an ugly but effective method (note the time):
root@appvm-AAA:~ # mapfile -t files < <(dpkg -L libreoffice-calc)
root@appvm-AAA:~ # time (for i in "${files[@]}"; do if [ -f "${i}" ]; then ln -sfT /dev/null "${i}"; fi ; done)
real 0m0.177s
user 0m0.096s
sys 0m0.086s
The full test, symlinking all dependencies of libreoffice-calc as well, shows:
real 0m2.691s
user 0m1.302s
sys 0m1.280s
This also solves (A) because (IIUC) symlinking does not modify actual data blocks (resulting in CoW). My test shows that the usage of the 1 GiB RAM based pool, storing the volatile volume, is 0.07% at VM boot (before symlinking) and increases to 0.34% (after symlinking). That is less than 3.5 MiB.
There is only the additional complexity of having to symlink dependencies without damaging any common ones. For example, if package1 and package2 both depend on package3, and AppVM2 needs package2 but removes package1, the automation system should not attempt to symlink package3’s files. I need to learn how to do this (tips are welcome).
Although this is somewhat hacky (and I don’t know if there may be any side effects), it should work for the purpose of reducing the attack surface (the whole point of the current multi-template exercise). No files = nothing to attack, so as long as rc.local runs early enough, we should be OK. With this approach we can have just an unitary template, where we can install whatever we plan to use, and multiple custom-minified AppVMs.
What do you think?
(I am still working on the automation system, so it is not quite polished yet. Maybe something new will come out along the way)
As far as I know, your uninstalled package won’t be updated. So basically the next time you run the VM and install the package, it’s out of date unless you also run updates on starting the VM (which would be a miserable experience). But that’s moot since I don’t even know how to do that, because as far as I know QubesUpdates works on Templates, not AppVMs. (Someone can certainly correct me if I am wrong.)
In what you quoted, I was talking about the case in which a packages is not installed in the template (and hence not uninstalled in the AppVM). It is only downloaded and stored in apt-get’s cache (in the template), then AppVMs install selectively at boot whatever they need. So, in regards to that scenario, I was wondering what would happen to the cache if the template receives an update and what control mechanisms over that exist, i.e. whether:
I am not a Debian expert, so I don’t know how that works.
As far as I know an uninstalled package (and it won’t be installed on your template) doesn’t get updated. You’ll just have to download it again…and that brings up the question of how you will know that you need to do that.
I don’t know whether it is appropriate to call it uninstalled (in the sense: installed and then removed), if it will never be installed in the template and will never be removed in the AppVM. As for the rest, it should be possible to check and compare the version on the repo and the version in local cache, then act accordingly. However, considering this whole approach does not solve (A) or (B), for the moment I am working on the symlinking one.