Thoughts about template management

General Discussion
1

Every now and then a new major version of the distro used as a base for our (hopefully) carefully-crafted templates is out. The “dist-upgrade my templates” approach (which we would use on a bare-metal GNU/Linux install) is cumbersome when it has to be applied to many templates, possibly may not provide all goodies of newer templates (upstream or Qubes), and globally does not seem favored around here. We seem to prefer switching a new version of the template, which also helps to minimize AppVM downtime.

But how do we make sure a newer version of a template is ready to switch an AppVM to the new version? How do we track the modifications done over the original template ? Unless you specifically take note, or back up your dpkg/rpm logs for later datamining, basically we don’t.

As I start to consider migrating my debian-10 templates to debian-11, I find myself wondering, “is the debian-10 template pristine, or did I install specific packages for a specific AppVM, and which?”.

As I look into setting up split-ssh (just an example of course, although really the one that finally triggered this post), I note that additional packages I’d need in vault are not present in the base image, and the same holds for the new qrexec definitions … and if I just install them I’m just adding to the bad trend that will make the template upgrade more painful.

How could we improve on that ?

I distinguish 3 kinds of templates (do I miss others?):

  • those installed from repository, untouched
  • those installed from repository, with modifications
  • those cloned from one installed from repository, with later modifications

The value in such templates (what would be enough to backup, aside from their qubes.xml configuration) is precisely in those (rootfs) modifications… which we don’t track.

Tracking modifications to the rootfs as they are made is hard, identifying them after the fact is costly and may not necessarily yield usable information.

But we could explore another approach: adding a new kind of templates whose rootfs cannot be modified manually (eg. they would not be possible to start the standard ways). Those would only be modifiable under control of qubes-core-admin, ie. by applying Salt repices: if the rootfs state can be reproduced programmatically, we can much more easily produce a new template using a more recent version of the base distro, and (as a bonus) we only have to backup the scripts that generate them, instead of their full contents.

As qubesctl becomes the entrypoint for all modifications to such templates, it can record the full history. Today not everyone is familiar enough with Salt to manage templates this way. As early adopters write more recipes for common tasks, and those recipes are made available as standard admin tools, it will become progressively easier to onboard new users.

Eagerly waiting for your reactions :slight_smile:

3 Likes
2

How about listing all installed packages with sudo apt list --installed, put the list in a text file and then install all of them into the new template?

3

Yes, that part is the easy one. The case of configuration files is not so readily addressed – they could be handled by installing said packages and diffing the rootfs trees, but that’s quite a heavyweight procedure even if we script it, and will not bring the advantage of lightweight backups.

With those templates installed with qvm-template (or dnf previously), my current practice is never to modify them, and clone them to a new specialized template when needed. Some hypothetical Qubes features that would support a similar workflow could be:

  • let qvm-template install the template with a user-chosen name. That would however lead to multiple downloads of the same template, and preclude any copy-on-write at LVM level – though I did not check we actually benefit from this today and anyway this benefit must surely goes away as original template and clones get updated separately
  • let installed templates never to be modified directly, only via cloning. This would ensure that any AppVM based on them would never be broken by updating the template from repository.
1 Like
4

@yann it won’t help you much now, but that’s exactly why I started to “document” any and all changes to templates in bash scripts. In other words: whenever there is a change I need to make in a template, be it configuration or installing something … I do it via a bash script for that template in dom0.

This way I can always and easily recreate a template and be confident that all changes / settings are applied.

The better solution here would be to have salt formula, but the bash script works very well for me so far.

2 Likes
5

I had started a dom0 tool for the fedora templates that ran an rpm list of both the new and old templates, did a diff between the two lists, and then finally ran an install in the new template, one by one, to get that template as close as possible to the previous template. It worked but it only really needed to do just the top level packages and I had not yet investigated how to install just those. Maybe one can just iterate over the /usr/share/applications/*.desktop list and create a dialog box with a list of check-boxes of what to move forward?