Dom0' UpdateVM' templates requirements to install templates with qvm-template from dom0?

For general user deployment this is not true.
That is, for Debian and Fedora based qubes not currently need to be
installed.

For minimal templates (advanced users only) - issue?documentation required?
For Whonix? What is the real cause of problem?

identification of the actual issue would be useful,

I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.

If can help please tell how. sys-whonix updatevm here no cacher. Test for Insurgo because he treasure. unman also extraordinary precious wizard so happy if can help. Now have qubes-repo-templates install in whonix-gw-16. Should revert? What tests useful?

I have not extensively tested this. I have just witnessed that qubes-repo-templates package is not installed under debian/fedora/whonix, as referred in currently opened issue, with mea culpa telling my setup was the cause of my issue (cacher with repositories all modified to be compatible with apt-cacher-ng. Therefore, newly deployed repositories by qubes-repo-templates package in templates were not apt-cacher-ng compatible and the requests from those templates simply silently failed, until I deactivated updates-proxy-setup service, with qubes-repo-templates package installed to infirm my prior hypothesis. I configured my qubes to check and report to dom0 for available updates, and to do so, I need those templates to be apt-cacher-ng compatible, and use apt-cacher-ng update proxy through enabling updates-proxy-setup on those qubes. By doing so, not modifying /etc/qubes/repo-templates/qubes-templates.repo, deployed by qubes-repo-templates package results in dom0 qvm-template failing. Other configuration will oviously have different result, while qubes-repo-templates package still needs to be deployed and Qubes documentation updated accordingly and where shaker/cacher currently is not affected by the bad behavior, simply because cacher use case relies on users manually updating templates and not relaying on qubes reporting available updates (qubes-update-check service on qubes) nor having qubes rely on apt-cacher-ng to report such updates nor use apt-cacher-ng to fetch repository packages (updates-proxy-setup is nly enabled on Templates through shaker/cacher. App qubes, such as sys-firewall if defined as qubes dom0 updatevm, will be used through normal https connection and won’t fail qvm-template calls from dom0 if cacher is not used. @unman : makes sense?


Conclusion is confirming that qvm-template is lacking proper error reporting in case qubes-repo-templates is not deployed on dom0’s updatevm Template, current Qubes OS documentation not referring to qvm-template at all, while requirements of a additioanl qubes-repo-templates package deployment under templates is nowhere to be found, nor automatically installed.

@Confused : can you validate that
[user@dom0 ~]$ qvm-prefs sys-firewall template
What is template?

If template Fedora/Debian without qubes-repo-templates package installed in that template:
[user@dom0 ~]$ qubes-prefs updatevm sys-firewall
[user@dom0 ~]$ qvm-template install fedora-37

And report back?

[user@dom0 ~]$ qvm-prefs sys-firewall template
sys-firewall-dvm
[user@dom0 ~]$ qvm-prefs sys-firewall-dvm template
ks-16-min-fw

Note: ks-16-min-fw is debian-11-minimal distro-morph to kicksecure-qubes-cli without qubes-repo-templates

[user@dom0 ~]$ qubes-prefs updatevm
sys-whonix
[user@dom0 ~]$ qubes-prefs updatevm sys-firewall
[user@dom0 ~]$ qvm-template install fedora-37
[Qrexec] /etc/qubes-rpc/qubes.TemplateSearch: line 3: /usr/lib/qubes/qvm-template-repo-query: No such file or directory
ERROR: qrexec call 'qubes.TemplateSearch' failed.
[user@dom0 ~]$ qubes-prefs updatevm sys-whonix
[user@dom0 ~]$ qvm-template install fedora-37
usage: qvm-template [--verbose] [--quiet] [--help] [--repo-files REPO_FILES] [--keyring KEYRING] [--updatevm UPDATEVM] [--enablerepo REPOID] [--disablerepo REPOID]
                    [--repoid REPOS] [--releasever RELEASEVER] [--refresh] [--cachedir CACHEDIR] [--keep-cache] [--yes]
                    {install,reinstall,downgrade,upgrade,download,list,info,search,remove,purge,clean,repolist,migrate-from-rpmdb} ...
qvm-template: error: Template 'fedora-37' not found.
[user@dom0 ~]$ qubes-prefs updatevm
sys-whonix
[user@dom0 ~]$ qvm-template list
Installed Templates
debian-11-minimal  0:4.0.6-202108191622  qubes-templates-itl
fedora-32          0:4.0.6-202010192324  qubes-templates-itl
fedora-32-minimal  0:4.0.6-202010191916  qubes-templates-itl
fedora-36          0:4.0.6-202205270243  qubes-templates-itl
fedora-36-minimal  0:4.0.6-202205270243  qubes-templates-itl
whonix-gw-16       0:4.0.6-202206270941  qubes-templates-community-testing
whonix-ws-16       0:4.0.6-202206270840  qubes-templates-community-testing
Available Templates
debian-10-minimal  0:4.0.6-202010131933  qubes-templates-itl
debian-10          0:4.0.6-202009131420  qubes-templates-itl
debian-11-minimal  0:4.0.6-202108191622  qubes-templates-itl
debian-11          0:4.0.6-202201171556  qubes-templates-itl
fedora-32-minimal  0:4.0.6-202010191916  qubes-templates-itl
fedora-32-xfce     0:4.0.6-202010191916  qubes-templates-itl
fedora-32          0:4.0.6-202010192324  qubes-templates-itl
fedora-33-minimal  0:4.0.6-202102131637  qubes-templates-itl
fedora-33-xfce     0:4.0.6-202102140100  qubes-templates-itl
fedora-33          0:4.0.6-202102131637  qubes-templates-itl
fedora-34-minimal  0:4.0.6-202110020922  qubes-templates-itl
fedora-34-xfce     0:4.0.6-202110020922  qubes-templates-itl
fedora-34          0:4.0.6-202201160401  qubes-templates-itl
fedora-35-minimal  0:4.0.6-202205081759  qubes-templates-itl
fedora-35-xfce     0:4.0.6-202205081759  qubes-templates-itl
fedora-35          0:4.0.6-202205081759  qubes-templates-itl
fedora-36-minimal  0:4.0.6-202205270243  qubes-templates-itl
fedora-36-xfce     0:4.0.6-202205270243  qubes-templates-itl
fedora-36          0:4.0.6-202205270243  qubes-templates-itl

:confused: @Insurgo should I enable qubes-templates-itl-testing or did output help?

Well, in the above, debian-11-minimal is not expected to have qubes-repo-templates package deployed.

Here you confirm that sys-whonix, based on whonix-16-gw, was dom0 updatevm

You switched it to sys-firewall, based on

Interestingly, here, qvm-template’s Qrexec output hints about qvm-template-repo-query not having proper

And fails. I guess this is linked with qvm-template fails with unhelpful error if qubes-repo-templates not installed in UpdateVM · Issue #7605 · QubesOS/qubes-issues · GitHub directly, since you are using a minimal template and qubes-repo-templates package is not deployed there.

Then you did:

Which didn’t give any output, which is what I confirmed being bogus since whonix-gw-16, Fedora-36 and Debian-11 standard templates should have qubes-repo-templates package installed by default. Not sure what @unman meant there:

Since from your sole current testing, you confirmed that whonix-gw-16 is not deploying qubes-repo-templates package, where I confirmed that it was not deployed under fedora-36.

A quick check on debian-11 template:

user@debian-11:~$ sudo apt install qubes-repo-templates
[...]
After this operation, 19.5 kB of additional disk space will be used.
Get:1 http://HTTPS///deb.qubes-os.org/r4.1/vm bullseye/main amd64 qubes-repo-templates amd64 4.1.2-1+deb11u1 [5,028 B]
Fetched 5,028 B in 0s (24.4 kB/s)                
Selecting previously unselected package qubes-repo-templates.
(Reading database ... 167991 files and directories currently installed.)
Preparing to unpack .../qubes-repo-templates_4.1.2-1+deb11u1_amd64.deb ...
Unpacking qubes-repo-templates (4.1.2-1+deb11u1) ...
Setting up qubes-repo-templates (4.1.2-1+deb11u1) ...

Confirms that qubes-repo-templates packages should be installed in all templates, otherwise its a bug.

My conclusion is that qvm-template being used under dom0 is currently missing an expected package dependency all standard deployed templates (whonix-gw-16+, debian-11+ and fedora-36+), that is, qubes-repo-templates deployed as default.

Or the repository file should come with what provides /usr/lib/qubes/qvm-template-repo-query script: qubes-core-agent-dom0-updates, or qubes-core-agent-dom0-updates having a dependency on qubes-repo-templates packages for all distributions.

Edit: Crosslinked this post from qvm-template fails with unhelpful error if qubes-repo-templates not installed in UpdateVM · Issue #7605 · QubesOS/qubes-issues · GitHub and documented issue there for devels and upstream documentation to be corrected. The way I understand it, qvm-template doesn’t work as of today for anybody but the ones having qubes-repo-templates deployed manually. This sure is a bug.

1 Like

What worries me is that you are going to mislead users, based on your
own individual (broken?) set-up, and lack of systematic testing.

Test with a vanilla install.

  1. Set up a new minimal template and qube:
qvm-clone debian-11-minimal template-updater
qvm-create updater -t template-updater -l purple
  1. Test update proxy.
    In template-updater, install qubes-core-agent-networking.
    Enable qubes-updates-proxy service in updater.
    Set updater as default target for qubes.UpdateProxy in
    /etc/qubes/policy/30-user.policy

Update a template? OK

  1. Test dom0 updates.
    In template-updater, install qubes-core-agent-dom0-updates

qubes-prefs updatevm updater

Restart updater. Confirm that qubes-repo-templates is not installed , and
there are no definitions of dom0 template repositories under /etc/qubes

In dom0, run qvm-template list. OK
qvm-template-gui, install a template. OK

  1. Test modified repo definitions.
    Modify repositories in templates-updater to be compatible with
    apt-cacher-ng.
    Shutdown template-updater.
    restart updater.

Update a template? OK
In dom0, run qvm-template list. OK
qvm-template-gui, install a template. OK

  1. Conclusion

The only package required to act as updatevm, beside networking, is
qubes-core-agent-dom0-updates.

Both debian-11 and fedora-36 templates can be used as base for updatevm
qube, and work without adding qubes-repo-templates package.

I am not saying there is no issue. I am saying you have not identified
the issue, and have generalised to a faulty conclusion.
You should change the title and edit the first entry to make the position
clear.

Hi,
I’m suffering of similar troubles after installing cacher and I can confirm that in my cacher qube isn’t installed qubes-core-agent-dom0-updates.
Do You think installing it may solve my troubles?
Can I do any other check You think may be useful for clarify the problem?
Thanks a lot,
cheers,
M.

@unman : i will retest again next week.
I clarified my setup, where other users missing qubes-repo-templates package (the repository file for templates repositories is provided by quebes-repo-templates under dom0’s updatevm’s template)

I agree this lacks systemic testing.
Was not yet able to install new template following your extra-space additional thin LVM, since additional paths under dom0 are used by qvm-template as opposed to qubes-dom0-update (dom0’s ~/.cache is used to store piped rpm to dom0, as well as /var/tmp paths for extraction). As of now, my wyng metadata cache under dom0 is consuming a lot of space which prevents me to test qvm-template download systematically from clean setup).

Hi Mila

Simply installing cacher should not (and in most cases does not) cause
problem.

What exactly is your set-up?
What is your updatevm - qubes-prefs or Global Settings GUI.
What is template for updatevm?
Have you made any other changes?

What are your “similar troubles”?

@Insurgo - I’ll go back and see if I understand your setup,and try to
create it.

“my wyng metadata cache under dom0 is consuming a lot of space” - good
grief. Can’t it be stored off-qube?

Hi unman,
sorry, I’ve mispelled it.
When talking about troubles, I’m talking about templates not cached by cacher qube (as oel 8 and similiar).
I’ve tried putting proxy in yum.conf, but it still cannot reach external internet.

$ qubes-prefs updatevm
sys-whonix

but I’ve yet tried to put it to somewhat else, without succeding.
Thanks for Your help!
Cheers,
M.

I’d like to add (custom) content of my 30-user.policy:

qubes.UpdatesProxy * oel-8-template @defafult allow target=sys-whonix

(before last line (with target=cacher))
Thanks again.
M.

I can wipe it off and ask for it to be synced back prior of next backup through arch-init from.

Will test again next week, but as of now I’m really puzzled by you saying quebes-repo-templates not being needed. What that package provides is repo definition for dom0 templates + gpg signatures, nothing more or less.

@mila this is off topic. Other threads are related to cacher, this thread mixed it in only because my setup had that mixed in.

Please post your issue in cacher/apt-cacher-ng threads where it is fit.

Here we try to isolate qvm-template dom0’ updatevm’ template package installation requirements and discuss about them only.

Why puzzled?
If you use a Debian updatevm, do you install the fedora 32 dom0 repository
definitions?
If you add a new repository in dom0, do you add that definition in the updatevm?

Puzzled exactly because of that.

Because qubes-dom0-update and qvm-template are not calling the same codepaths, and because qubes-repo-templates package is explicitly dom0 templates repo definitions to be added to dom0’s updatevm’s Template and exist for debian, fedora, and because rpc calls depend on the /etc/qubes/templates-repo/repo to be defined under dom0 updatevm template, and are fedora-32 related because of dom0. Have you checked what qubes-repo-template provides on debian/fedora templates?

I will check again like said but now there is something I do not understand for sure with qvm-template requirements and will read code linked to dom0 rpc calls linked to qvm-template. I checked quickly with and without qubes-repo-templates, thought i understood but now I definitely doubt with you saying that package is not required.

You understand that the updateVM does not need to have dom0
repositories defined, so you must understand that dom0 repo definitions
are passed threw to the updateVM.
Why would you think that qvm-template would not follow the same pattern?

This is a puzzle to me, but a matter of psychology.
Let’s look at code.

I have told you that the updateVM does not need to have the template
repo definitions installed, and I have shown you how to prove this for
yourself.

The process is documented here.

In dom0, look at /usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template.py
qrexec_payload builds the payload for the qrexec call to the updateVM.
You can see the payload write ---, and then pulls in the
repository definitions.

payload += '---\n'
for path in args.repo_files:
    with open(path, 'r', encoding='utf-8') as fd:
        payload += fd.read() + '\n'

This is then used for the qrexec call to the updateVM

On the updateVM, the qrexec function is /etc/qubes-rpc/qubes.TemplateDownload,
which calls /usr/lib/qubes/qvm-template-repo-query
There you will see:

while IFS= read -r line; do
    if [ "$line" = "---" ]; then
        break
    fi
    case "$line" in
        --enablerepo=*|\
        --disablerepo=*|\
        --repoid=*|\
        --releasever=*|\
        --refresh)
            OPTS+=("$line")
            ;;
        *)
            SPEC="$line"
            ;;
    esac
done

repodir=$(mktemp -d)
trap 'rm -r "$repodir"' EXIT
cat > "$repodir/template.repo"

The incoming payload is parsed until the --- signal, then what follows
is written out to a temporary template.repo, used in the dnf call.

This is how the updateVM can use the template repository definitions from
dom0 without them being “installed”.

Again, this process works under latest updates in 4,1 for Debian and
Fedora based updateVM qubes.
Apparently, it does not work for you.
I hope this helps you to troubleshoot your problem and identify the root
of your issue.

As an aside, why are there Debian and Fedora packages
qubes-repo-template, containing repository definitions for Qubes
templates?
You can see that it is a dependency of qubes-core-admin-client, a
package that provides “Tools to manage Qubes system using Admin API”
If you use an adminVM, you have to have the template repository
definitions there.
I think this is not your situation.

I never presume to speak for the Qubes team.


When I comment in the Forum or in the mailing lists I speak for myself.

2 Likes

@unman: you are totally right. I forgot to check on which depends qubes-repo-templates, which would have answered my question. Your analysis of code went deeper than my own and shows that qubes-dom0-updater and qvm-templates-gui are reusing a similar but different logic, that is, to copy the repository definitions from dom0 to its updatevm, but uses those repo list differently: one through the update proxy (qvm-template) the other without (qubes-dom0-update).

My setup, as written elsewhere, encompasses whonix support as of now by having apt-cacher-ng proxy report (lying) that apt-cacher-ng is torrified at all times (that is, by modifying its offered web page which is checked by whonix templates prior of permitting the proxy to be accessed).

My setup activates qubes update notifications to dom0 as per normal Qubes deployment (qvm-service sys-whonix qubes-update-check on). To be able to report updates to dom0, those qubes also need to be able to download packages information (qvm-service sys-whonix updates-proxy-setup on) through the update proxy, and to do so, needs to have proper modified and compatible apt-cacher-ng repository definitions. And for those qubes to be able to talk to cacher, dom0 needs to have @anyvm policy under 30-user.policy file:
qubes.UpdatesProxy * @anyvm @default allow target=cacher.

You covered this elsewhere and for that i’m sorry:

Basically, as of now, it is not possible to have cacher reporting qubes’s Templates available updates to dom0 unless a downstream everyday used qube has both qubes-update-check and
updates-proxy-setup services enabled. But by doing so, it is impossible to use qvm-template or qvm-template-gui with a dom0 updatevm configured to use cacher.

So long story short, since my dom0 is configured to have sys-whonix has its updatevm, or any other template also configured to use cacher, it is impossible to have both downstream qubes reporting for its parent template available updates AND be used to download templates through qvm-template/qvm-template-gui.

So, the fix on my setup was to deactivate updates-proxy-setup service on dom0’s defined updatevm:

[user@dom0 ~]$ qvm-service sys-whonix
qubes-update-check   on
updates-proxy-setup off

Doing so, running qvm-template-gui/qvm-template succeeds. But of course, sys-whonix doesn’t report to dom0 availabe updates unless whonix-gw-16 Template is started by itself. It’s also impossible to download temporarily (session-based) packages under sys-whonix, since sys-whonix doesn’t have standard repository definitions, since whonix-gw-16 still has its repositories modified to use apt-cacher-ng (most common with debian-11).

As I explained in different posts, I really appreciate Qubes to be able to install software momentarily in app qubes. This permits to install software that won’t persist across reboots, for example in sys-net to do some forensic analysis. Same for my app dvms where I do not see advantage in having those in parent Templates. We had that discussion for which your conclusion was totally valid: users might be confused into why software they installed vanished on next reboot of that app qube. This is valid, but also permitted in normal Qubes installation and really useful.

As your current shaker/cacher implementation, this is not an issue for anybody then myself.
My current possible mitigations, following my modified cacher implementation also caching updates for whonix-gw-16 and whonix-wks-16, is to either:

  • deactivate sys-whonix’s updates-proxy-setup service so that dom0 can continue to use sys-whonix as its updatevm (and template downloader), while sys-whonix cannot report for whonix-gw-16 available updates anymore nor install packages temporarily following its whonix-gw-16 repository modifications to comply with apt-cacher-ng proxy.
  • Revert to your upstream cacher configuration and have whonix not having its repositories and packages cached by apt-cacher-ng (sad since whonix OS updates are common as of now across debian-11 and whonix)
  • Modify dom0’s /etc/qubes/repo-templates/qubes-templates.repo so that its URLs are apt-cacher-ng compliant.

The reason I got really confused (puzzled) between qubes-dom0-updates and qvm-template behavior differences can be grasped with the following:

On my current setup:

[user@dom0 ~]$ qubes-prefs updatevm
sys-whonix
[user@dom0 ~]$ qvm-service sys-whonix
qubes-update-check   on
updates-proxy-setup  on

Succeeds:

user@host:~$ sudo apt install htop
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  lm-sensors
The following NEW packages will be installed:
  htop
0 upgraded, 1 newly installed, 0 to remove and 14 not upgraded.
Need to get 127 kB of archives.
After this operation, 328 kB of additional disk space will be used.
Get:1 tor+http://HTTPS///deb.debian.org/debian bullseye/main amd64 htop amd64 3.0.5-7 [127 kB]
Fetched 127 kB in 0s (325 kB/s)
Selecting previously unselected package htop.
(Reading database ... 51973 files and directories currently installed.)
Preparing to unpack .../htop_3.0.5-7_amd64.deb ...
Unpacking htop (3.0.5-7) ...
Setting up htop (3.0.5-7) ...
Processing triggers for qubes-core-agent (4.1.37-1+deb11u1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for man-db (2.9.4-2) ...
user@host:~$
[user@dom0 ~]$ sudo qubes-dom0-update
Using sys-whonix as UpdateVM to download updates for Dom0; this may take some time...
No updates available

Fails:

[user@dom0 ~]$ qvm-template list
Installed Templates
debian-11          0:4.0.6-202201171556  @commandline
debian-11-minimal  0:4.0.6-202108191622  qubes-templates-itl
whonix-gw-16       0:4.0.6-202201160402  @commandline
whonix-ws-16       0:4.0.6-202201160402  @commandline

From there, of course, if I disable updates-proxy-setup:

[user@dom0 ~]$ qvm-shutdown sys-whonix --force --wait
[user@dom0 ~]$ qvm-service sys-whonix updates-proxy-setup off
[user@dom0 ~]$ qvm-start sys-whonix

qvm-template-gui works:

qubes-dom0-updates also works:

But of course, installing anything in sys-whonix won’t, and whonix-gw-16 available updates won’t be reported to dom0:

So I am still puzzled onto why qubes-dom0-update would work while qvm-template wouldn’t. The mechanisms in play between qvm-template and qubes-dom0-updates is different, the latter requiring me to change my setup, which otherwise works fulfilling all my current needs, also caching whonix updates, permitting me to have ephemeral package installation into all my app qubes and also report available Templates updates to dom0 until I tried to install new templates into dom0 through any of my qubes, which relies into the same qvm-template mechanisms, where dom0 does something different than when qubes-dom0-update is called.

@unman : Makes sense?

@unman

Basically, the question can be shortened down to:
Why qvm-template relies on dom0’s updatevm’s updates-proxy-setup (and dom0 @anyvm policy) where qubes-dom0-updates doesn’t?

qubes-dom0-update doesn’t care about appvm’s update proxy and succeeds getting dom0 available updates and reports them correctly to dom0 (talks directly to the internet through its updatevm, using dom0’s template repo definitions passed to it).

But qvm-template doesn’t work and relies on dom0’s updatevm updates-proxy-setup and dom0 updatevm policy, and would require dom0’s /etc/qubes/repo-templates/qubes-templates.repo being apt-cacher-ng compliant to be useable?

1 Like

There’s a lot here - I’ll try to reply in the morning.