Automate debian-minimal based template creation

Community Guides
81

Isn’t there a system like differential/nested/inherited/chained/whatever-the-right-term-is templates?

Example:

  1. Template0: Absolutely minimal core system
  2. Service templates: each one based on Template0 + additional necessary packages. sys-* qubes will be based on those.
  3. App templates: Template0 + strictly necessary packages (e.g. everything needed for mpv). AppVMs and DispVMs will be based on those.
  4. Shared templates: will contain packages common for more than 1 template. Example: ServiceTemplate7, AppTemplate3 and AppTemplate5 using libsomething will be based on SharedTemplate1 which contains only libsomething.

Updates would work like this:

  1. Template0 updates only itself. Updates propagate automatically to all other templates, no need to duplicate data or do anything additional (except restart of qubes).
  2. Service templates: each one updates only its specific packages (restart qubes)
  3. App templates - just like 2
  4. Shared templates - just like 2.

I wonder how that can be done.

82

BTW, what do you think of this approach:

  • Start with a minimal template. All AppVMs should be based on this template.
  • Install and configure every package necessary for any VM in the template. Keep a record of every change made.
  • Create boot scripts which wipe everything from the previous step (apt-get --auto-remove purge) except what is strictly necessary for a particular VM
  • Let the relevant scripts in each AppVM’s /rw/config/rc.local
  • Configure App/DispVMs’s volatile volume’s on tmpfs, set root volume’s rw=false (prevent unnecessary SSD writes)

Then we can use and update only 1 template, taking care of customizations through the boot scripts only.

At first glance, the only disadvantage will be the slower boot of AppVMs.

What do you think?

83

Interesting thoughts. You seem to care a lot about disk space and writes. I never did. Go for it and report back how it worked for you.

The update bandwidth issue is easily solved by apt-catcher-ng.

84

I rather care about efficiency and proper hardware usage. The decremental method discussed above is not super efficient either. It is rather a workaround aimed to improve things a little. It also comes with the question whether installing a package and then removing it may have any negative security implications itself (compared to never having installed it).

What do you think?

Another (incremental) way might be to use snapshots. Example:

  • Install minimal base template. This will be T0, origin.
  • Snapshot T0 (name it T1). Install and configure package 1.
  • Snapshot T0 (name it T2). Install and configure package 2.
    …

This will still result in N templates, just without data duplication (which cloning does). Even LVM’s documentation seems to mention this approach in a Xen context.

The question is, how will updates work? If T0 needs to be modified (e.g. new systemd version), this has to propagate to other templates too, i.e. T1, T2, … TN would also need to be modified. This means that even if packageX does not need an update, the whole TemplateX would have to be recreated. Not very efficient either.

I am not very experienced with snapshots, so I may be wrong.

85

Updating is indeed the unfortunate aspect of this. Once you create a template, it’s independent of the one you cloned it from, and if the original needs an update, so does the clone.

Because, for some reason, I actually have to run a template to see that it needs updates, I actually clone my most basic template to “Update-Canary-00”. I set the memory to bare minimum on it and actually have it start up shortly after I log in. (This way I’m not running my actual base template, but I will know when it needs updating.) Similarly for other templates that occupy prominent positions in my inheritance tree (such as the base of all appvms that aren’t system appvms.

I can hear you screaming already because I end up with six qubes that run solely for the purposes of detecting a needed update. There’s no reason one cannot shut them down after half an hour (which should be enough for them to detect needed updates). Once running, after the initial check they will wait two days to check again (so sometimes I restart them). But I have 64 GB of ram so having a few qubes using 400MB each doesn’t bother me.

Anyhow, if I see that my base qube needs an update…I know I have to update them all. If the “base” qube with my browser needs an update, then all of my browser templates (about six of them) need updating.

As Sven points out, you don’t absolutely need to install one app per template, you can mix or match. You have maximum freedom there. I personally would, at a bare minimum, make sure no apps (like libreoffice, a browser, keepass, etc) are on my system qubes. (That calls for a system base template that’s barely more than the minimal template, and an app base template; System qubes would clone the system base and add whatever they need. The application base qube would have more stuff (such as, possibly, a file manager app and other stuff a user would find convenient), then you can either install all of the apps onto it, or clone it and install subsets of apps.

So one possible tree would be:

debian-12-minimal
    system-base
         networking (sys net and sys firewall are based on this)
         app-base (add user conveniences to this)
              vault-tmpl (keepass.  Really no reason to install anything else here.)
              browser-tmpl
                    work-tmpl (you'd probably want libreoffice or equivalent on this; base your work qube on this)
                   personal-tmpl (any apps you use personally on this)

Etc.

You can install whatever apps seem to make sense on your work and personal appvms on the corresponding -tmpl qubes. Here I’m assuming you’ll want a browser on both, but don’t want to have to actually install a browser twice and are willing to clone to avoid doing so. If you would rather install twice then just have work-tmpl and personal-tmpl at the same level as vault and don’t do browser-tmpl

I divide things up a lot more finely than this, and you are free to do that as well (though I suspect you would rather not). The beauty of it is the choice is yours.

86

@SteveC

You speak of beauty and freedom of choice, and I understand where your enthusiasm comes from. However, we (or at least I) still lack the choice of an approach which doesn’t involve data duplication. Although I still have no better solution to share, I suppose that if/when one exists, you may be even happier with it.

Trying to answer my own question:

[…] whether installing a package and then removing it may have any negative security implications itself (compared to never having installed it).

I suppose it actually may, especially if the installation procedure also increases fingerprint-ability in some way. So, ideally we would probably look for an incremental approach instead.

FWIW, I have looked at liteqube and in a way it seems to be decremental (through its “quarantine” procedures). However, the code seems to need significant cleanup, so personally I would not experiment in my dom0 with it.

87
88

You’re right. Essentially, I decided I was willing to duplicate things in order to reduce attack surface and and compartmentalize. By automating as much as possible, I’ve reduced the cost in my personal time a lot. But the resources are still being used (they’re cheap though). The worst aspect from my POV is that if an update to the base template needs to be done, it can take well over an hour to update all the VM templates. (I have 57 of them.) And it’s worse on my poor laptop, which struggles valiantly.

Right now we’re able to leverage off the fact that one template can support multiple AppVMs. That works because the user data and “system” areas are generally cleanly separated in Linux. If we could separate system areas into system and installed application areas, we could have another level of “template” in between the other two, and things would look like this: an OS template, an application template, and an AppVM based on the application template which in turn is based on the OS template. If you then had an OS upgrade, you’d just have to update the OS template, and everything based on it would get the updates. You might still have to update multiple templates when an application got updated though, depending on how you mix-and-match them…unless we then take the next logical step of allowing more than one application template on an AppVM. So consider an example AppVM that is running firefox and libreoffice. It could be based on a LibreOffice template AND a Firefox template…and those could be based on Debian-12. (Making sure the user doesn’t mix and match OSes in his app templates could make this even more of a nightmare to implement.) That’s about the only way I can think of where resources wouldn’t be duplicated…and I can see it would be very hard to do given the way Linux works right now. Apps get installed into /usr/bin, so you can’t function by mounting them from a different place than the OS, also in /usr/bin.

I’m not the only person who does what I do; at least one actual member of the Qubes team starts his day by booting his system, and he runs updates whether he knows he needs them or not. They then run while he cooks and eats his breakfast. It doesn’t save resources, but it does mean he doesn’t have to wait on updates.

One other concern, of course, is repeatedly writing to an SSD; there’s only so many times you can do that before it craps out on you. The good news is there are different levels of SSD (single bit cell (SLC), double bit cell (MLC), triple bit cell (TLC) and quadruple bit cell (QLC)) SLC is the longest lived but tends to be very expensive. TLC is still significantly better than QLC and you can find them readily but they will cost a bit more. The biggest difficulty here (aside from the money) is that Samsung (the biggest seller) obfuscates what kind you’re buying (they call both MLC and TLC “MLC”) but the pro model were until recently MLC not TLC. (And MLC is considerably more durable–three times the lifespan–than TLC.)

Some explanation here for anyone who needs it. Good luck figuring out what Samsung will sell you though.
Multi-Layer SSDs: What Are SLC, MLC, TLC, QLC, and PLC? (howtogeek.com)

89

@apparatus

Thanks for the link. Unfortunately, I learned how little I know about file systems and that even the experts don’t seem to agree on a practical solution which we may possibly use here.

@SteveC

57 times overhead sounds huge. Have you thought about building some package from source? In some of my VMs I use this approach, more specifically:

I do not install the existing package from distro’s repo but I build it from source (in a DispVM), using proper build options to make it run from /usr/local/bin or /opt/bin. Then I place the binaries in a bind-dir or in /usr/local/bin of a dedicated AppVM.

Another approach I use for some small Internet-related packages is to install them at qube’s boot time. In the AppVM’s (DVM’s) /rw/config/rc.local I write:

apt-get --quiet --no-install-recommends --assume-yes install <package>

After that, I configure the , so that it creates its necessary files in ~/.config. Then, my DispVMs using this AppDVM (being also RAM-based with near-zero SSD writes) simply download a few MB at boot time. In my case this is acceptable, considering the DispVM is also supposed to connect to the Internet anyway. That is not a perfect solution but the only sensible overhead is a few MB of download. Obviously, this is not suitable for an offline system, for service qubes or for those with special firewall restrictions.

Speaking of building from source, I wonder if using something like Gentoo would not be a better fit for the granular approach discussed in this thread, as it might allow additional flexibility in configuring where the binaries should be placed.

90

57 times overhead sounds huge. Have you thought about building some package from source? In some of my VMs I use this approach, more specifically:

The main effect from this (other than longer updates) is disk usage. And I’m using 34% of a 500 GB SSD. That doesn’t bother me like I imagine it would bother you. (If I were not basing things on a minimal template, things would be VERY different!)

I don’t know what building my own stuff would do…unless I do tinker with the system some beforehand as you speculated, to separate apps and system. Mucking around with such low level stuff though is outside of my comfort zone. (That said, I think there is a community gentoo option.) Besides I’m pretty sure actually being able to have two levels of templating is something the Qubes code itself won’t manage.

On the other hand…apps could be installed in /etc/local/bin on the AppVM. You could then make the AppVM a dvm template. The problem would be any data you generate can’t be saved–unless you mount some other device to /home/user/Documents (or someplace similar to that; whatever’s appropriate to the app).

The biggest customization I did to qubes was to store all data in encrypted containers on a NAS. I then created “split-veracrypt” where one VM mounts the NAS, and passes off veracrypt containers to a decryptor, which decrypts the container and passes that off to the target VM…which simply sees a mounted filesystem and has no idea it’s looking at an encrypted container on a NAS.

It lets me get by with a 500 GB drive even with 57 templates on it! Very few of my VMs are not disposable; ones that don’t connect to the internet can mount a decrypted container.

So, conceivably if I could get apps to install in /usr/local/bin I could at least free myself from having to update 57 templates just because of an update to debian-12-minimal.

91

57 templates

I wonder what threat model justifies that. If it is something you cannot share for extra security reasons, that’s OK.

I also wonder how that can be done more efficiently. Is it not more efficient to have a script-automated procedure to update the core (e.g. debian-12-minimal) template, then re-create the 57 templates (possibly using some snapshot system, I have no idea what exactly is possible), rather than start/update/shutdown each one?

Another thing which crossed my mind: Have a master template (e.g. #58) which includes all packages from the others. Then, running an update check on it, it should be possible to know which packages need an update, and based on that info, one may update only the templates which actually need it.

92

I compartmentalize a lot, and I’m connected to two different networks I don’t let talk to each other, and I also isolate USB things from the rest of the system. (For example, I don’t attach my printer to the qube I run Libre Office in; instead I print to a file (a pdf), and ship it to a VM that does nothing but talk to the printer.)

And in some cases those templates simply serve as ancestors to other templates–e.g., there’s a “net base” that has those things common to net-wifi and net-ether; there’s no appvm associated with it (it would be pointless). I could probably get rid of that one and just install the same stuff (very minor stuff) on the other two templates directly. (Since my setup files can include other files, I don’t have to worry about maintaining the same stuff in two different places–which I find to be a far more serious issue than having another template lying around. That would save one clone time too, and one start-update-shutdown cycle.

Actually it’s wrong to call it a start-update-shudown cycle; it’s really a StartTheManager-StartTheTemplate-Update-StopTheTemplate-StopTheManager cycle (and oftentimes that rodeo takes much more time than the actual update does).

Split Veracrypt also introduces a number of templates that wouldn’t otherwise be there.

I won’t claim there’s a threat model justifying this, it’s a matter of personal preference as much as anything else. But now thanks to this conversation, I am considering getting rid of some of my “this template is just an ancestor” templates (but not the one that just has firefox on it; that’s a huge package and worth not having to install six times). But that will only be about four templates.

A balance must be struck and the best balance will depend on an individual’s network speed (mine range from tolerable to two-tin-cans-and-a-taut-string): Is it cheaper to clone a qube with something installed on it, or install it twice? On a slow connection, or with a package that’s just so big that even with the cacher it takes a while to install, I’d rather clone.

93

I see.

So, I guess we should rather think of some snapshotting system, allowing a more efficient “template family tree” without duplication. That would not solve the problem with updates, but if the tree creation is scripted, I think it would be acceptable to e.g. run an update + recreate it once a week.

What do you think? Is there a proper way to do this with current functionality of Qubes OS?

94

As it happens, my tree creation IS scripted. I’d never even think of generating all of that by hand. I learned this from sven’s stuff (earlier in this thread).

It’s a combination of bash scripts and salt. (I added the salt later.) I can’t go fully salt because if I try it, templates get cloned before the software is installed on them, necessitating at least two installs of the same thing. A separate call to qubesctl must be made to create and configure each template (in fact to create each Appvm as well). I script those calls because they’re command-line monsters with five or six parameters; scripting forces them to be done in series instead of parallel.

95

I should probably start another thread to discuss possible options for snapshots.

96

On multiple templates and deduplication.

I haven’t had time to test this yet. But:

  • BRTFS as opposed to thin LVM supports deduplication
  • bees, if it was compiled and added to dom0, could either through deamon/on demand, depuplicate all those volumes (templates mostly important here).
  • Those were touched in other threads but my motivation lowered since I’m currently using thin LVM on main laptop and to test this correctly, I would need my wyng backups to be restored over BRTFS on destination laptop to bootstrap new installation as quick as possible, which is not yet supported by wyng-qubes-util restore nor wyng util directly and I do not have time to test manual instructions and possible bugs along the way.

My intention on replying on this post is to restate that I understand this use case and am actually one of the users of this use case. I also use cacher project of unman over qubes 4.1, which is not supported on qubes 4.2 yet. And cacher is a bandwidth life saver. What it does is download once under cacher qube update related packages and then install those cached packages over each templates without redownloading.

If those templates could be deduplicared on block level, then the packages installed content would not be non redundant. Those templates would only occupy space for their differences: that is, differences on what is installed differently for different use cases. And there: deduplication would make sense. The only possible problem there would be a race condition over Q4.2 where multiple templates launching upgrade in parallel could attempt to download the same package at the same time at cacher level, but that should not be a problem.

So not misinterpret what I say. This approach would still take more space then having a minimal debian template shared across multiple qubes. Maintaining multiple templates still requires each templates being updated where cacher help in minimizing download repetition and speed of updating the templates which pulls from cacher, not the internet multiple times.

TLDR:

  • if compartmentalizarion of templates is a needed feature to segregate risks of using proprietary packages and abstract forensic need of launched services by installed packages for trusted qubes and underlying templates, I’m a firm believer that multiple templates are needed. And therefore de duplication would be really helpful for that use case.
  • if a single template can be used by users not installing any risky application in a shared template, then having a single template shared across most if not all qubes is desired for both update speed and disk size consumption. But there is no one size fits all under QubesOS from my experience.

Therefore, deduplication with bees over BRTFS would be a really desired feature. But we are not there yet.

97

@Insurgo

Glad to see you here!

I have read your whole thread about pool deduplication but I am not an expert on file systems and some things are not immediately clear to me. I understand the concept of snapshots and I have used them on KVM/qemu VMs by creating a basic raw origin, then building variations on top of it.

In Qubes OS, however, I have no idea how to do that. I understand your goal is to have that automated through the underlying file system on which the qube images reside. However, IIUC, that is a longer term thing, not something we can expect any time soon (at least not in a stable well-tested version), right?

So, my hope was to find some manual way of non-duplicating cloning of templates which would allow a more efficient workflow along the lines of automated creation of a template hierarchy, as in this thread.

How can this be done using the standard thin LVM of Qubes OS?

98

I see. Nothing of that sort exists as far as I know under QubesOS. qvm-volume revert can be abused to keep snapshots on its parents, where configuration can tell how many snapshots to keep where the parent rotate on reboot. As you might expect, a lot of changes means a lot of differences to keep track of which is something that needs a workaround somehow.

The qvm-clone comes to the rescue, but one again without deduplication, setting a new parent. And then applies the revert logic above.

My personal workaround around this today is to use wyng to store only deduplicated data of the whole backup data. But then again, when one restores a backup, there is no deduplication on what is written on the destination.

So this workaround using wyng has a limitation because the restored volume is not deduplicated. This is where I have hope toward bees on BRTFS and all the jazz I said above.

99

Thanks for clarifying.

I have been reading the template implementation doc, hoping to learn how snapshots in Qubes OS actually work. Unfortunately, the script which it mentions (/etc/xen/scripts/block-snapshot) does not even exist.

Then I found this, which says:

100

@Sven

Interesting thoughts. You seem to care a lot about disk space and writes. I never did. Go for it and report back how it worked for you.

So, I have been experimenting with the decremental approach. I am facing some weird issue. Here is what happens:

  1. I clone the original minimal template and update the clone
  2. I install each and every package which any qube will use (see Note below)
  3. I shut down the template and create a test AppVM named AAA
  4. I start the test VM and in it try to remove all packages installed in 2: apt-get purge ... ; apt-get autoremove --purge

Result:

I get errors and the process fails:

https://paste.opensuse.org/pastes/ba55cbcbb096

I have absolutely no idea why this attempts to remove qubes-core-agent.