Automate debian-minimal based template creation

Community Guides
7

Recommendation:
At the end of introduction / motivation (before chapter ‘The basics’). I would add something like:

[…] the only way to truly trust code is to understand what it does.

The final result of this guide should be a bash script *.sh, actually your personal bash script with your configuration and settings, which you need to run in dom0.

Questions:
New template available

If I understood correctly, when debian-12 is out all I have to do is change the TEMPLATE value of the script:

SOURCE_TEMPLATE=debian-11-minimal to debian-12-minimal
TARGET_TEMPLATE=deb-11-min to deb-12-min

save modifications to my my-qubes-os-config.sh and run it again in dom0.

Afterwards re-assign all AppVM to new mini-templates and delete all obsolete templates. Does it make sense to also automate this steps or would it be advisable to always to this re-assigning and remove of templates manually?

2 Likes
8

Thank you @whoami for your feedback, I will incorporate it in the next draft.

when debian-12 is out all I have to do is change the TEMPLATE value of the script

Ideally. That’s what I did when upgrading from debian-10 to debian-11. But of course there is no guarantee, things might break but in case of debian they are usually well documented. Like debian-11 no longer including Qt4.

I see an upgrade just as a special case of (re-)creating your qubes. Once you tested your scripts with a particular version, they will always work with that version and chances are they will continue to work with the next.

Lol. Imagine the poor folks using Fedora and having to do this every 6 months or so. How would you even keep your sanity without automating it?

Afterwards re-assign all AppVM to new mini-templates and delete all obsolete templates. Does it make sense to also automate this steps or would it be advisable to always to this re-assigning and remove of templates manually?

Here is what I do:

qvm-ls --field=name, template > tmp
chmod +x tmp
nano tmp

In nano I then

  1. remove the first line, the dom0 line, all the template lines and all the lines of qubes based on a dvm template. As a result I now have a list of all my qubes and their templates.
  2. add qvm-prefs at the beginning of each line
  3. search and replace deb-10 by template deb-11
  4. safe & exit nano

Then qvm-shutdown --all --wait && ./tmp … and done.

2 Likes
9

Well, I’m using fedora-35 from the very beginning of my daily-driver Qubes using (last November) and fedora-36 is just out a week ago?
Regardless the fact that I started to use debian-11-minimal(s) in parallel, thanks to you, of course.

1 Like
10

Hi @enmus, I should have looked it up before posting:

Fedora maintains its releases for “approximately 13 months” and there is a new release “approximately every six months”. So one doesn’t have to upgrade every 6 months, but one could.

Regarding debian-minimal, I am uncomfortable with the “thanks to you”. The debian template maintainer is @unman. He does all the work and almost everything I try to (re-)contribute here is more or less a result of him answering my questions (see the archives). He and the other core team members are the ones making all this even possible.

1 Like
11

I am not. If there wasn’t your topic about minimal(s) I’d probably never use debian.

1 Like
12

If you wanted to automate this process and put it in a script: (it assumes all your dvm-based vms contain dvm in their name)

qvm-ls --field=name,template,class | grep AppVM | grep -v dvm | awk '{print "qvm-prefs " $1 " template " $2}' | sed 's%deb-10%deb-11%g' > tmp
2 Likes
13

This is awesome!

Going to implement this this week. For the add-feature helper scripts, do those just need to be separate .sh files in the same directory to be called properly?

I think it makes sense to not publish the bash scripts for all the reasons you enumerate, but I wonder if the features and templates tables would make sense as a repo in Qubes-Community/contents/configuration, to enable community engagement?

Am I correct that a package won’t list dependencies that are assumed, due to them being in the base Debian image, but that are not present in Debian minimal? The missing piece that I find intimidating is knowing when a package will break, which is solved by this template/feature combination!

For example, I’d benefit from a signal-desktop or element-desktop ‘template table’ that enumerates the ‘features’ these applications rely on (presumably at least a few libraries etc).

2 Likes
14

Just a personal note: I found qubes-core-agent-passwordless-root not to be required for Signal.

Update: I was able to install element on the same signal-template so no further requirements should be necessary.

Update 2: since we’re talking about messaging apps, I took the liberty to try out session with the same installed packages, and… it works just fine: repo info: https://deb.oxen.io/

4 Likes
15

Yes. I just use ~/bin.

Once the note is complete I will offer it to the Qubes-Community project to adopt and extend. For now it’s a first draft (what I could do on one Sunday afternoon) and I intent to work a lot more on it before I’ll consider it finished.

I published it unfinished because that was better then to keep promising something and also to get all the feedback I can.

Interesting question. Unfortunately I don’t know. However I have a hard time imagining that there are “assumed” packages. That doesn’t sound like a good idea to me. I think it’s rather that some packages just have incomplete dependency information … and other dependencies are kind of bugs. Like signal hanging when there is no notification daemon. It should check before calling it.

Coming up, but I’ve also repeatedly posted this particular example to the forum. A little searching should help you find it.

2 Likes
16

I’m definitely a newbie, but I hate “cruft” (unused stuff), so I’ve been working through this.

I had the worst time with the firewalls. I’d create a template and try to turn it into a dvm template. And that wouldn’t work. So I’d leave it as a template and I couldn’t get the firewall vm to use it as a template.

Finally, tonight, the light dawned.

First you create a template. Install network stuff, the dom0 update thing, and so on, onto that template.

Then you create an ordinary VM from that template…and turn it into a DVM template. That’s what

    qvm-create --template OrigTemplate --label puce DVMTempl
    qvm-prefs DVMTempl template_for_dispvms True
    qvm-features DVMTeml appmenus-dispvm 1

are for and you do not run them on the (first) template!!!)

THAT–a different entity from the template you installed software onto–can now be pointed to by sys-firewall as its template.

Maybe I’m just really dense. Or maybe I’m not really dense and it should be spelled out this blatantly in the directions because most people might fail to realize this. I’ll leave that up to @Sven 's judgment.

By the way, thank you; I’ve learned a lot whilst doing this. I also have about a zillion templates right now! (Life made interesting by the fact that I am compartmentalizing wifi and ethernet, so I end up with separate sys-net and sys-firewall stuff…and all the templates they depend on.

(In fact, this is going through my minimal wifi firewall and wifi net qubes.)

1 Like
17

For dvm-templates used for sys vms, you could consider qvm-features DVMTeml appmenus-dispvm '' instead.
If you’re not going to generate disp vms on-demand, this will prevent the dvm showing at the top of the Q app-menu.

18

Good point…I probably do not want that for firewall templates.

On the other hand, for my disposable VM generator which I just created…I do want that.

Thanks!

19

Okay, so that template I did want to keep in the menu? It runs firefox, so I can browse and blow it away (as someone put it, a very private session).

I got tired of firefox always opening their welcome to firefox and privacy tabs on first start, and so I decided I wanted a “just barely set up” firefox on the DVM template.

No way to just run the DVM Template itself, that I could see; it invariably opens things disposable.

So, I pointed its networking to “none”, unchecked the box where it is a DVM Template, then fired up the browser. I made my changes, got rid of those two stupid tabs, closed it, and went back and made the App VM back into a DVM template.

Now, no matter what I do with that appmenus-dispvm flag, I can’t get the DVM template to show up in the menu at the top.

Is it possible that once I’ve run the template as a regular system, it is forever contaminated somehow and won’t show in the menu? (It does, if I scroll all the way to the bottom of the menu, let me start firefox from there and my settings are preserved…but I don’t want to have to do all that scrolling, it should be at the top where it’s convenient.) If so, how do I get disposable firefox to not show me those doggone tab pages? Should I run it in the AppVMTemplate just to set it up? (I know someone will scream at that thought.)

20
1 Like
21

Scroll down the Q appmenu until you reach the Template (disp) section. That’ll open the dvm template. However it is not recommended to open the browser in the template because that may lead to profile fingerprinting. If you’ve already done so and wish to remove existing profiles run the following in the dvm-template:

[user@deb-11-firefox-dvm ~] rm -r /home/user/.mozilla

You should instead consider using Firefox’s autoconfig (as mentioned in the guide above: [Guide] Automatically install extensions and configure new (dispvm) hardened Firefox profiles with arkenfox user.js and policies) and/or Firefox policies. I wrote a brief explanation here:
How to install addons's or arkenfox profile in template vm so it is there in disposable vm? - #14 by BEBF738VD

update: the linked guide now contains a full explanations of policies as well.

For this issue specifically, add the following to the policies.json file mentioned in the guide:

    "FirefoxHome": {
      "Search": true,
      "TopSites": false,
      "SponsoredTopSites": false,
      "Highlights": false,
      "Pocket": false,
      "SponsoredPocket": false,
      "Snippets": false,
      "Locked": false
    },
    "Homepage": {
      "URL": "about:home",
      "Locked": false,
      "StartPage": "homepage"
    },

This will disable everything unnecessary and will leave you with a plain homepage with a search-bar.

1 Like
22

Yep, this solved it.

I don’t know what was the deal with my machine not showing up in the upper menu even though I set that feature flag. But when I renamed it, it did show up. So now I had a perfectly good disposable template with firefox set up the way I wanted…but with a profile to track. There may be a bug here. Anyhow, I simply re-ran the script to create the thing (thanks to sven for the inspiration for that), then went through the procedure referenced above, and now I have a good disposable template without the profile.

23

Just creating a set of minimal templates… I am facing one issue with this:


#!/bin/bash

# WARNING: Never run any code that you don’t trust. And the only way to
# truly trust code is to understand what it does.
# Therefore, read carefully the following documents before you run this script:
# source: https://svensemmler.org/notes/deb-min-templates
# source: https://micahflee.com/2021/11/introducing-qube-apps
# official: https://www.qubes-os.org/doc/templates/minimal
# 
# To run this script command the following in a dom0 Terminal:
# Prerequisite: SOURCE_TEMPLATE=deb-11-m (debian-xx-minimal)
# 1. copy *.sh to dom0: qvm-run --pass-io SOURCE_APPVM 'cat ...' > DOM0_TARGET
# 2. set execute permission: chmod +x ./tpl-deb-m-flatpak.sh
# 3. run script: ./tpl-deb-m-flatpak.sh

# define and clone templates
SOURCE_TEMPLATE=deb-11-m
FLATPAK_TEMPLATE=deb-11-m-flatpak
qvm-clone $SOURCE_TEMPLATE $FLATPAK_TEMPLATE
# AppVM attributes and AppVM creation (optionally)
APPVM_NAME=flatpak
APPVM_LABEL=blue
read -p "Do you want to create an AppVM afterwards? (y/n) :  " appvm_prompt

# install all but only necessary packages
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install qubes-core-agent-networking -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install qubes-core-agent-network-manager -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install pulseaudio-qubes -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install qubes-usb-proxy -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install qubes-core-agent-nautilus -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install nautilus -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install zenity -y"
# required for flatpak app
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-setuptools -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-stdeb -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install dh-python -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install flatpak -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-pyside2.qtcore -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-pyside2.qtgui -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-pyside2.qtwidgets -y"
# required only for the build process
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install git -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install fakeroot -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install build-essential -y"


# create a temporary AppVM to download git repository and build script
qvm-create temporary-appvm -t $FLATPAK_TEMPLATE --label red

# clone micahflee's git repository and run the script
qvm-run --pass-io -u root temporary-appvm "git clone https://github.com/micahflee/qube-apps.git"
qvm-run --pass-io -u root temporary-appvm "cd qube-apps && ./build_deb.sh"

# move deb package to flatpak template and purge setup
qvm-run --pass-io -u root temporary-appvm "qvm-move-to-vm $FLATPAK_TEMPLATE deb_dist/*.deb"
qvm-shutdown --wait temporary-appvm
qvm-remove temporary-appvm -f
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt remove git -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt remove fakeroot -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt remove build-essential -y"

qvm-run --pass-io -u root $FLATPAK_TEMPLATE "dpkg -i QubesIncoming/temporary-appvm/*.deb -y && poweroff"

echo -e "\nYou have installed the Qube-Apps Flatpak Installer & Updater by @micahflee, please read Micah Lee’s blog post about ‘Qube-Apps’.\n\nSOURCE: https://micahflee.com/2021/11/introducing-qube-apps \nIMPORTANT: You should increase the private storage on your Flatpak-AppVM.\nREMARK: This is special use-case of a minimal template and it is not and can never be tested with all possible Flatpaks. Consequently, there could be special configurations and additional packages required to run a flatpak. \n"

if [ $appvm_prompt = 'y' ]; then
	qvm-create $APPVM_NAME -t $FLATPAK_TEMPLATE --label $APPVM_LABEL
	echo "INFO: AppVM < $APPVM_NAME > has been created."
fi

I see one error in the build process:

dpkg-builpackage: erro: gain-root-command 'fakeroot' not found

I am not sure if it is an issue related to the fakeroot or something within micahflee’s script?

24

OK, so I tried to automate all this profile and settings stuff. Fortunately, I changed the name of the qubes I was generating.

The new disposable qube simply shut down immediately when I started firefox in it.

I removed the new stuff, figuring I’d made some bonehead error copying text.

This qube now contains nothing but a straight network-and-firefox-esr installation, and still when I start it, it simply shuts down immediately without displaying firefox esr. (Xterm works fine).

Journal CTL displays no error message. The qube starts, it shuts down, as if that’s what I had somehow configured it to do.

So I must be making some bonehead mistake somewhere. What could it be?

25

In dom0 qvm-run QUBE xterm
When the terminal opens, run firefox-esr and check for error messages.

You can also check for relevant information in /var/log/qubes/guid.<name>.log (change <name> with the qube’s name).

26

OK (and thank you for a fast response!)

I didn’t think to run it from within Xterm. OK, to clarify, the old disposable VM, which doesn’t do this, is named MidInternet, its template is deb11a-internet-dvm. This is the one which works (almost) perfectly [It has one issue I haven’t mentioned yet, and won’t mention now to keep the thread from branching ridiculously]. I had manually added all of the arkenfox stuff to it.

The new one is MidInternetX, and its template is deb11a-internet-dvmX. This one if I run firefox, terminates immediately and I never see firefox open. This started out as my attempt to automate the Arkenfox stuff from a script; when it behaved like this I stripped out the Arkenfox stuff and (in desperation) even other apps I was installing.

When I run an Xterm on MidInternetX, and run firefox, firefox actually starts, with of course the two tabs I wanted to get rid of. That’s more than happens from the menu. The only outputs are:

  1. The invariable complaints about my “locale” not being supported. Aside from generating annoying error messages, this is never an actual damn issue.

  2. “[GFX1-]: glxtest: libpci missing.”

However, I tried this with MidInternet as well (the one that works), and got exactly the same output.

I also tried scrolling to the bottom of the menu and starting Firefox in MidInternetX there (as I understand it from earlier, this opens it in the DVM template without making it disposable), and it works fine there (two extra tabs, becuase the Arkenfox stuff is not done in MidInternetX’s template).

So there’s somehow something bodged up with the disposable-ness of the thing, but I’ll be dipped if I can figure out what that is. Like I said this is probably a stupid, forehead-smacker of an error but it’s been driving me nuts for two days.