Automate debian-minimal based template creation

Sven · May 25, 2022, 5:05am

Yes. I just use ~/bin.

Once the note is complete I will offer it to the Qubes-Community project to adopt and extend. For now it’s a first draft (what I could do on one Sunday afternoon) and I intent to work a lot more on it before I’ll consider it finished.

I published it unfinished because that was better then to keep promising something and also to get all the feedback I can.

Interesting question. Unfortunately I don’t know. However I have a hard time imagining that there are “assumed” packages. That doesn’t sound like a good idea to me. I think it’s rather that some packages just have incomplete dependency information … and other dependencies are kind of bugs. Like signal hanging when there is no notification daemon. It should check before calling it.

Coming up, but I’ve also repeatedly posted this particular example to the forum. A little searching should help you find it.

SteveC · June 11, 2022, 6:09am

I’m definitely a newbie, but I hate “cruft” (unused stuff), so I’ve been working through this.

I had the worst time with the firewalls. I’d create a template and try to turn it into a dvm template. And that wouldn’t work. So I’d leave it as a template and I couldn’t get the firewall vm to use it as a template.

Finally, tonight, the light dawned.

First you create a template. Install network stuff, the dom0 update thing, and so on, onto that template.

Then you create an ordinary VM from that template…and turn it into a DVM template. That’s what

    qvm-create --template OrigTemplate --label puce DVMTempl
    qvm-prefs DVMTempl template_for_dispvms True
    qvm-features DVMTeml appmenus-dispvm 1

are for and you do not run them on the (first) template!!!)

THAT–a different entity from the template you installed software onto–can now be pointed to by sys-firewall as its template.

Maybe I’m just really dense. Or maybe I’m not really dense and it should be spelled out this blatantly in the directions because most people might fail to realize this. I’ll leave that up to @Sven 's judgment.

By the way, thank you; I’ve learned a lot whilst doing this. I also have about a zillion templates right now! (Life made interesting by the fact that I am compartmentalizing wifi and ethernet, so I end up with separate sys-net and sys-firewall stuff…and all the templates they depend on.

(In fact, this is going through my minimal wifi firewall and wifi net qubes.)

BEBF738VD · June 11, 2022, 8:26am

For dvm-templates used for sys vms, you could consider qvm-features DVMTeml appmenus-dispvm '' instead.
If you’re not going to generate disp vms on-demand, this will prevent the dvm showing at the top of the Q app-menu.

SteveC · June 11, 2022, 8:54am

Good point…I probably do not want that for firewall templates.

On the other hand, for my disposable VM generator which I just created…I do want that.

Thanks!

SteveC · June 11, 2022, 9:21pm

Okay, so that template I did want to keep in the menu? It runs firefox, so I can browse and blow it away (as someone put it, a very private session).

I got tired of firefox always opening their welcome to firefox and privacy tabs on first start, and so I decided I wanted a “just barely set up” firefox on the DVM template.

No way to just run the DVM Template itself, that I could see; it invariably opens things disposable.

So, I pointed its networking to “none”, unchecked the box where it is a DVM Template, then fired up the browser. I made my changes, got rid of those two stupid tabs, closed it, and went back and made the App VM back into a DVM template.

Now, no matter what I do with that appmenus-dispvm flag, I can’t get the DVM template to show up in the menu at the top.

Is it possible that once I’ve run the template as a regular system, it is forever contaminated somehow and won’t show in the menu? (It does, if I scroll all the way to the bottom of the menu, let me start firefox from there and my settings are preserved…but I don’t want to have to do all that scrolling, it should be at the top where it’s convenient.) If so, how do I get disposable firefox to not show me those doggone tab pages? Should I run it in the AppVMTemplate just to set it up? (I know someone will scream at that thought.)

tzwcfq · June 12, 2022, 5:21am

BEBF738VD · June 12, 2022, 7:36am

Scroll down the Q appmenu until you reach the Template (disp) section. That’ll open the dvm template. However it is not recommended to open the browser in the template because that may lead to profile fingerprinting. If you’ve already done so and wish to remove existing profiles run the following in the dvm-template:

[user@deb-11-firefox-dvm ~] rm -r /home/user/.mozilla

You should instead consider using Firefox’s autoconfig (as mentioned in the guide above: [Guide] Automatically install extensions and configure new (dispvm) Firefox profiles with arkenfox user.js and policies) and/or Firefox policies. I wrote a brief explanation here:
How to install addons's or arkenfox profile in template vm so it is there in disposable vm? - #14 by BEBF738VD

update: the linked guide now contains a full explanations of policies as well.

For this issue specifically, add the following to the policies.json file mentioned in the guide:

    "FirefoxHome": {
      "Search": true,
      "TopSites": false,
      "SponsoredTopSites": false,
      "Highlights": false,
      "Pocket": false,
      "SponsoredPocket": false,
      "Snippets": false,
      "Locked": false
    },
    "Homepage": {
      "URL": "about:home",
      "Locked": false,
      "StartPage": "homepage"
    },

This will disable everything unnecessary and will leave you with a plain homepage with a search-bar.

SteveC · June 13, 2022, 5:01am

Yep, this solved it.

I don’t know what was the deal with my machine not showing up in the upper menu even though I set that feature flag. But when I renamed it, it did show up. So now I had a perfectly good disposable template with firefox set up the way I wanted…but with a profile to track. There may be a bug here. Anyhow, I simply re-ran the script to create the thing (thanks to sven for the inspiration for that), then went through the procedure referenced above, and now I have a good disposable template without the profile.

whoami · June 13, 2022, 11:41am

Just creating a set of minimal templates… I am facing one issue with this:


#!/bin/bash

# WARNING: Never run any code that you don’t trust. And the only way to
# truly trust code is to understand what it does.
# Therefore, read carefully the following documents before you run this script:
# source: https://svensemmler.org/notes/deb-min-templates
# source: https://micahflee.com/2021/11/introducing-qube-apps
# official: https://www.qubes-os.org/doc/templates/minimal
# 
# To run this script command the following in a dom0 Terminal:
# Prerequisite: SOURCE_TEMPLATE=deb-11-m (debian-xx-minimal)
# 1. copy *.sh to dom0: qvm-run --pass-io SOURCE_APPVM 'cat ...' > DOM0_TARGET
# 2. set execute permission: chmod +x ./tpl-deb-m-flatpak.sh
# 3. run script: ./tpl-deb-m-flatpak.sh

# define and clone templates
SOURCE_TEMPLATE=deb-11-m
FLATPAK_TEMPLATE=deb-11-m-flatpak
qvm-clone $SOURCE_TEMPLATE $FLATPAK_TEMPLATE
# AppVM attributes and AppVM creation (optionally)
APPVM_NAME=flatpak
APPVM_LABEL=blue
read -p "Do you want to create an AppVM afterwards? (y/n) :  " appvm_prompt

# install all but only necessary packages
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install qubes-core-agent-networking -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install qubes-core-agent-network-manager -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install pulseaudio-qubes -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install qubes-usb-proxy -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install qubes-core-agent-nautilus -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install nautilus -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install zenity -y"
# required for flatpak app
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-setuptools -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-stdeb -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install dh-python -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install flatpak -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-pyside2.qtcore -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-pyside2.qtgui -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install python3-pyside2.qtwidgets -y"
# required only for the build process
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install git -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install fakeroot -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt install build-essential -y"


# create a temporary AppVM to download git repository and build script
qvm-create temporary-appvm -t $FLATPAK_TEMPLATE --label red

# clone micahflee's git repository and run the script
qvm-run --pass-io -u root temporary-appvm "git clone https://github.com/micahflee/qube-apps.git"
qvm-run --pass-io -u root temporary-appvm "cd qube-apps && ./build_deb.sh"

# move deb package to flatpak template and purge setup
qvm-run --pass-io -u root temporary-appvm "qvm-move-to-vm $FLATPAK_TEMPLATE deb_dist/*.deb"
qvm-shutdown --wait temporary-appvm
qvm-remove temporary-appvm -f
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt remove git -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt remove fakeroot -y"
qvm-run --pass-io -u root $FLATPAK_TEMPLATE "apt remove build-essential -y"

qvm-run --pass-io -u root $FLATPAK_TEMPLATE "dpkg -i QubesIncoming/temporary-appvm/*.deb -y && poweroff"

echo -e "\nYou have installed the Qube-Apps Flatpak Installer & Updater by @micahflee, please read Micah Lee’s blog post about ‘Qube-Apps’.\n\nSOURCE: https://micahflee.com/2021/11/introducing-qube-apps \nIMPORTANT: You should increase the private storage on your Flatpak-AppVM.\nREMARK: This is special use-case of a minimal template and it is not and can never be tested with all possible Flatpaks. Consequently, there could be special configurations and additional packages required to run a flatpak. \n"

if [ $appvm_prompt = 'y' ]; then
	qvm-create $APPVM_NAME -t $FLATPAK_TEMPLATE --label $APPVM_LABEL
	echo "INFO: AppVM < $APPVM_NAME > has been created."
fi

I see one error in the build process:

dpkg-builpackage: erro: gain-root-command 'fakeroot' not found

I am not sure if it is an issue related to the fakeroot or something within micahflee’s script?

SteveC · June 15, 2022, 10:54am

OK, so I tried to automate all this profile and settings stuff. Fortunately, I changed the name of the qubes I was generating.

The new disposable qube simply shut down immediately when I started firefox in it.

I removed the new stuff, figuring I’d made some bonehead error copying text.

This qube now contains nothing but a straight network-and-firefox-esr installation, and still when I start it, it simply shuts down immediately without displaying firefox esr. (Xterm works fine).

Journal CTL displays no error message. The qube starts, it shuts down, as if that’s what I had somehow configured it to do.

So I must be making some bonehead mistake somewhere. What could it be?

BEBF738VD · June 15, 2022, 11:05am

In dom0 qvm-run QUBE xterm
When the terminal opens, run firefox-esr and check for error messages.

You can also check for relevant information in /var/log/qubes/guid.<name>.log (change <name> with the qube’s name).

SteveC · June 15, 2022, 11:26am

OK (and thank you for a fast response!)

I didn’t think to run it from within Xterm. OK, to clarify, the old disposable VM, which doesn’t do this, is named MidInternet, its template is deb11a-internet-dvm. This is the one which works (almost) perfectly [It has one issue I haven’t mentioned yet, and won’t mention now to keep the thread from branching ridiculously]. I had manually added all of the arkenfox stuff to it.

The new one is MidInternetX, and its template is deb11a-internet-dvmX. This one if I run firefox, terminates immediately and I never see firefox open. This started out as my attempt to automate the Arkenfox stuff from a script; when it behaved like this I stripped out the Arkenfox stuff and (in desperation) even other apps I was installing.

When I run an Xterm on MidInternetX, and run firefox, firefox actually starts, with of course the two tabs I wanted to get rid of. That’s more than happens from the menu. The only outputs are:

The invariable complaints about my “locale” not being supported. Aside from generating annoying error messages, this is never an actual damn issue.
“[GFX1-]: glxtest: libpci missing.”

However, I tried this with MidInternet as well (the one that works), and got exactly the same output.

I also tried scrolling to the bottom of the menu and starting Firefox in MidInternetX there (as I understand it from earlier, this opens it in the DVM template without making it disposable), and it works fine there (two extra tabs, becuase the Arkenfox stuff is not done in MidInternetX’s template).

So there’s somehow something bodged up with the disposable-ness of the thing, but I’ll be dipped if I can figure out what that is. Like I said this is probably a stupid, forehead-smacker of an error but it’s been driving me nuts for two days.

SteveC · June 15, 2022, 11:34am

Yep.

SMDH.

Apparently it does this if the templateVM is running.

And of course I was leaving it running so it would be more convenient to check that it’s moving the files around.

BEBF738VD · June 15, 2022, 11:36am

The errors are not too bad, you should be able to fix the second one by installing libpci3 and libpciaccess0 in the template.

Strongly discouraged for the reasons I mentioned before, you should remove the .mozilla folder.

You’re supposed to install the policies.json and autoconfig.js files in the template, not in the dvm. So it shouldn’t matter what dvm you’re using (as it does NOT require any kind of extra configuration), as long as they’re using the same template.

I suggest that you create a new dvm-template and do not touch that at all, and delete the old one.

SteveC · June 15, 2022, 11:39am

I do that as part of the script.

I ONLY did that this time to see whether it would do something different. That one, in fact, is already gone.

To clarify, all installations (including arkenfox stuff, when enabled) happen ONLY to deb11a-internet-dvmX. At the very end of that script, I create MidInternetX from deb11a-internet-dvmX

BEBF738VD · June 15, 2022, 11:39am

Fair.

Sven · June 16, 2022, 2:53am

In it’s current form the ‘note’ is about template creation, not qube creation (disposable templates). That’s of course something I also automate, especially for the disposable sys-qubes.

Thank you for the feedback, I’ll add a clarification and later a second ‘note’ on how to automate the creation of some common qubes.

Sven · June 16, 2022, 2:59am

@whoami as a rule of thumb: do it first manually in the actual cloned template, step-by-step and find/solve all issues this way. Then when you have a sequence that works, write the dom0-based shell script that reflects it.

If there is an issue with this specific debian-minimal based flatpack install thingy… make a separate thread for it please.

whoami · June 17, 2022, 7:57am

Yea, I did it like this, step-by-step. Wasn’t easy for me as ‘low-code-developer’

Anyways, I have made a template for the templates …

With this I am building up my scripts based on your explanations. I will publish it on Gitea and share the link in the coming days…

tanky0u · June 17, 2022, 9:07am

Looking forward to seeing that.