3isec-qubes-cacher-1.10-1 breaks Debian/Fedora TemplateVM updates

User Support
, ,
1

I installed the cacher package according to the steps outlined in https://qubes.3isec.org/tasks.html on my Qubes setup running 4.1.2

The installation of 3isec-qubes-cacher-1.10-1.fc32.x86_64.rpm via dom0 went well, but afterward, none of my Debian or Fedora TemplateVMs could update via apt/dnf update.

Debian 12

E: Failed to fetch http://HTTPS///deb.debian.org/debian/dists/bookworm/InRelease Connection failed [IP: 127.0.0.1 8082]
E: Failed to fetch http://HTTPS///deb.debian.org/debian-security/dists/bookworm-security/InRelease Connection failed [IP: 127.0.0.1 8082]
E: Failed to fetch http://HTTPS///deb.qubes-os.org/r4.1/vm/dists/bookworm/InRelease Connection failed [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.

Fedora 38

Errors during downloading metadata for repository ‘fedora’:
Curl error (52): Server returned nothing (no headers, no data) for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-38&arch=x86_64&protocol=http [Empty reply from server]
Error: Failed to download metadata for repo ‘fedora’: Cannot prepare internal mirrorlist: Curl error (52): Server returned nothing (no headers, no data) for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-38&arch=x86_64&protocol=http [Empty reply from server]

My cacher AppVM is connected to the same netVM which is successfully providing network to other AppVMs. However, when I try to ping out from the cacher AppVM I get

Ping: connect: Network is unreachable

ip link indicates eth0 is down. I brought it up via ip link set eth0 up but still can’t ping out from the cacher AppVM.

What am I missing here?

2

Usually when I see messages like that in a VM, I find that I have the VM set up to try to use the proxy, but the UpdateProxy policy is pointing to sys-firewall not to sys-cacher. (or vice versa, if it seems to be having trouble finding https://mirrors… without HTTPS in it.)

In other words, you may need to fix your policy to match what your VM is trying to do (that’s easier than fixing the VM to conform to the policy).

I’ll assume the former case here since it would be consistent with what you report.

It works a bit differently in 4.2 versus 4.1. In 4.2 you want to look at the file /etc/qubes/policy.d/50-config-updates.policy and there should be a line at the end that contains @type:TemplateVM and ends with target=sys-cacher. If it says something like target=sys-firewall, you may be able to fix your issue by editing that line to read sys-cacher.

In 4.1 the line will appear in a different file (grep for UpdatesProxy); I’m not sure which one because my 4.1 install has been tinkered with a lot.

On the other hand your sys-cacher doesn’t seem to have network connectivity, so this probably isn’t the problem, or at least it’s not the ONLY problem.

3

On 4.1, I haven’t encountered this problem.
It seems to me that you need to address the networking problem first.
Why is the cacher offline when you have a netvm set? This looks
seriously wrong.
Check that you have qubes networking set up with apt list installed qubes-core-agent-networking

systemctl status apt-cacher-ng will tell you if the service is running,
but until you resolve the network problem that’s no help.

4

Thanks @unman.

It looks like the template-cacher TemplateVM was created based on my Global default template (fedora-38-minimal) and therefore did not install apt-cacher-ng (obviously) or qubes-core-agent-networking (less obvious).

Even after manually installing those packages, my Fedora TemplateVMs error out when updating over clearnet (not TOR or VPN) connections.

Errors during downloading metadata for repository ‘updates’:
Status code: 403 for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f38&arch=x86_64&protocol=http (IP: 127.0.0.1)
Error: Failed to download metadata for repo ‘updates’: Cannot prepare internal mirrorlist: Status code: 403 for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f38&arch=x86_64&protocol=http (IP: 127.0.0.1)

I then tried to uninstall 3isec-qubes-cacher-1.10-1 from dom0 via sudo dnf remove 3isec-qubes-cacher-1.10-1.fc32.x86_64 to try to reinstall it with debian-11 set as the Global default template. The PRERUN scriptlet appeared to run, but also fail:

error: %preun(3isec-qubes-cacher-1.10-1.fc32.x86_64) scriptlet failed, exit status 20
Error in PREUN scriptlet in rpm package 3isec-qubes-cacher
Verifying : 3isec-qubes-cacher-1.10-1.fc32.x86_64 1/1
Failed:
3isec-qubes-cacher-1.10-1.fc32.x86_64

Is the 403 error still a known issue? And if so, how do we “clear the cacher before trying another update”?

How should cacher be uninstalled? I found this unanswered forum post while trying to troubleshoot my unsuccessful uninstall and reinstall of cacher.

5

Update: I was able to uninstall 3isec-qubes-cacher from dom0 by running:

sudo dnf remove 3isec-qubes-cacher-1.10-1
sudo rpm --noscripts -e 3isec-qubes-cacher

And then reinstalled it after setting my Global template to debian-12 and running the following from dom0:

sudo dnf install ./3isec-qubes-cacher-1.10-1.fc32.x86_64.rpm -y

The reinstallation threw these errors:

whonix-ws-16: ERROR (exit code 20, details in /var/log/qubes/mgmt-whonix-ws-16.log)
whonix-gw-16: ERROR (exit code 20, details in /var/log/qubes/mgmt-whonix-gw-16.log)
warning: %post(3isec-qubes-cacher-1.10-1.fc32.x86_64) scriptlet failed, exit status 20
Error in POSTIN scriptlet in rpm package 3isec-qubes-cacher
Verifying : 3isec-qubes-cacher-1.10-1.fc32.x86_64 1/1
Installed:
3isec-qubes-cacher-1.10-1.fc32.x86_64

I noticed that the template-cacher TemplateVM’s sudo systemctl status apt-cacher-ng indicates apt-cacher-ng is installed as a masked service (expected), but qubes-core-agent and qubes-core-agent-networking were not installed (unexpected).

I noticed that the cacher AppVM doesn’t seem to have the /etc/apt-cacher-ng/ directory or the apt-cacher-ng service as per systemctl status apt-cacher-ng’s output:

Unit apt-cacher-ng.service could not be found.

I verified cacher has network connectivity via ping. However, now neither debian nor fedora TemplateVMs can update through cacher.

debian-12

E: Failed to fetch http://HTTPS///deb.debian.org/debian/dists/bookworm/InRelease Connection failed [IP: 127.0.0.1 8082]
E: Failed to fetch http://HTTPS///deb.debian.org/debian-security/dists/bookworm-security/InRelease Connection failed [IP: 127.0.0.1 8082]
E: Failed to fetch http://HTTPS///deb.qubes-os.org/r4.1/vm/dists/bookworm/InRelease Connection failed [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.

fedora-38

Fedora 38 - x86_64 0.0 B/s | 0 B 00:01
Errors during downloading metadata for repository ‘fedora’:
Curl error (1): Unsupported protocol for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-38&arch=x86_64&protocol=http [Received HTTP/0.9 when not allowed]
Error: Failed to download metadata for repo ‘fedora’: Cannot prepare internal mirrorlist: Curl error (1): Unsupported protocol for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-38&arch=x86_64&protocol=http [Received HTTP/0.9 when not allowed]

It is still not clear to me how these cacher VMs are being built (in general, and why they are missing some of these packages in particular) or how this package should be uninstalled and reinstalled.

6

I dont see how this could happen - the clone.sls should ensure that a
debian template is installed, and that the template used by cacher is
cloned from debian-X-minimal.
But apparently it has happened. (Feel the need for a baffled emoji here)

How do you remove the package? Usually I would just remove the package
with dnf remove 3isec-qubes-cacher, but as your instal is completely
borked do it manually.

First roll back any changes made to the templates - there’s a salt state at
/srv/salt/cacher/restore_templates.sls that should do this:
sudo qubesctl --skip-dom0 --show-output --targets=templates state.apply cacher.restore_templates

Then edit /etc/qubes/policy.d/30-user.policy and remove any reference
to qubes.UpdatesProxy .

Finally:

sudo dnf remove 3isec-qubes-cacher`
sudo rpm --noscripts -e 3isec-qubes-cacher

should clean everything up on the rpm side.

Check that /srv/salt/cacher is deleted.

7

I attempted to remove the package from dom0 via

sudo dnf remove 3isec-qubes-cacher-1.10-1

But it errored out toward the end:

error: %preun(3isec-qubes-cacher-1.10-1.fc32.x86_64) scriptlet failed, exit status 20
Error in PREUN scriptlet in rpm package 3isec-qubes-cacher
Verifying : 3isec-qubes-cacher-1.10-1.fc32.x86_64 1/1
Failed:
3isec-qubes-cacher-1.10-1.fc32.x86_64
Error: Transaction failed

That’s why/how I came across having to also run the following from dom0:

sudo rpm --noscripts -e 3isec-qubes-cacher

I reverted the changes to dom0’s /etc/qubes/policy.d/30-user.policy and deleted /srv/salt/cacher via

sudo rm -r /srv/salt/cacher

This command did not return any output (I did not run this command before):

sudo qubesctl --skip-dom0 --show-output --targets=templates state.apply cacher.restore_templates

Re-installing cacher via dom0 (sudo dnf install ./3isec-qubes-cacher-1.10-1.fc32.x86_64.rpm) results in this error (I don’t recall any errors the first time I installed this):

warning: %post(3isec-qubes-cacher-1.10-1.fc32.x86_64) scriptlet failed, exit status 20
Error in POSTIN scriptlet in rpm package 3isec-qubes-cacher
Verifying : 3isec-qubes-cacher-1.10-1.fc32.x86_64 1/1
Installed:
3isec-qubes-cacher-1.10-1.fc32.x86_64

Interestingly, this time I was able to update debian-12, but fedora-38 continues to error out with the 403 error as before:

Errors during downloading metadata for repository ‘updates’:
Status code: 403 for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f38&arch=x86_64&protocol=http (IP: 127.0.0.1)
Error: Failed to download metadata for repo ‘updates’: Cannot prepare internal mirrorlist: Status code: 403 for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f38&arch=x86_64&protocol=http (IP: 127.0.0.1)

Also, I noticed this time the cacher AppVM shows the apt-cacher-ng service and has more files in its /etc/apt-cacher-ng/ directory. Not sure if that qubesctl command did something different.