Issues with apt-cacher-ng

unman · September 3, 2022, 1:16pm

Forum logistics

Thank you @Sven

Sven · September 3, 2022, 3:27pm

Forum logistics

Done

unman · September 3, 2022, 4:16pm

Forum logistics

Sorry I was bamboozled by the fork.

I wanted the General Discussion to be “Issues with apt-cacher-ng”
and the User Support thread to be “Caching updates for non-templates using
apt-cacher-ng”

Sven · September 4, 2022, 3:18am

Forum logistics

It shall be so.

enmus · October 15, 2022, 9:32pm

@Insurgo, @unman,

In /etc/apt/apt.conf.d/90whonix I found this:

Summary

## apt caching proxy:
## apt-cacher-ng
##
## If you want to use apt-cacher-ng,
## you have to disable the apt-get uwt wrapper:
##     create a file /etc/uwt.d/50_user.conf with
##     uwtwrapper["/usr/bin/apt-get"]="0"
##

So I did it.

In the same file I have found this

Summary

## Please do not edit 90whonix file! This is because with next
## Whonix update, this file may get replaced with an improved version.
## That is why you created your own file /etc/apt/apt.conf.d/50user instead.

So I created it with these options enabled

Summary

## Working.
Acquire::http { Proxy "http://127.0.0.1:8082"; };
##
## Untested.
Acquire::http::Proxy "http://127.0.0.1:8082/";
Acquire::https::Proxy "http://127.0.0.1:8082/";
Acquire::ftp::Proxy "ftp://127.0.0.1:8082/";
Acquire::tor::Proxy "http://127.0.0.1:8082/";

According to /etc/uwt.d/30_uwt_default.conf I created 50_uwt_user.conf and specifically disabled

Summary

uwtwrapper["/usr/bin/apt"]="0"
uwtwrapper["/usr/bin/apt-get"]="0"

I have changed all sources lists in Whonix to point to cacher adding http://HTTPS/// to the each link. Onion addresses aren’t working because I’m point to cacher anyway and from there to tor via sys-whonix, so I commented them in.

In dom0, in my 30_user.policy I have changed whonix vm to point to cacher:

# Upgrade Whonix TemplateVMs through sys-whonix.
qubes.UpdatesProxy      *   @tag:whonix-updatevm    @default    allow target=cacher
# Deny Whonix TemplateVMs using UpdatesProxy of any other VM.
qubes.UpdatesProxy      *   @tag:whonix-updatevm    @anyvm      deny
qubes.UpdatesProxy	* vault @anyvm deny
...
...

Started update in whonix template

Summary

user@host:~$ sudo apt update
Hit:1 tor+http://HTTPS///deb.debian.org/debian bullseye InRelease
Hit:2 http://HTTPS///contrib.qubes-os.org/deb/r4.1/vm bullseye InRelease
Get:3 tor+http://HTTPS///deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Hit:4 tor+http://HTTPS///deb.debian.org/debian-security bullseye-security InRelease
Hit:5 http://HTTPS///contrib.qubes-os.org/deb/r4.1/vm bullseye-testing InRelease
Get:6 tor+http://HTTPS///deb.debian.org/debian bullseye-backports InRelease [49.0 kB]
Hit:7 http://HTTPS///deb.qubes-os.org/r4.1/vm bullseye-testing InRelease
Hit:8 tor+http://HTTPS///fasttrack.debian.net/debian bullseye-fasttrack InRelease
Hit:9 http://HTTPS///deb.qubes-os.org/r4.1/vm bullseye-securitytesting InRelease
Hit:10 tor+http://HTTPS///deb.whonix.org bullseye InRelease                    
Get:11 tor+http://HTTPS///deb.debian.org/debian bullseye-updates/main amd64 Packages.diff/Index [12.8 kB]
Get:12 tor+http://HTTPS///deb.debian.org/debian bullseye-backports/main amd64 Packages.diff/Index [63.3 kB]
Get:13 tor+http://HTTPS///deb.debian.org/debian bullseye-updates/main amd64 Packages T-2022-10-15-2035.13-F-2022-10-15-2035.13.pdiff [286 B]
Get:13 tor+http://HTTPS///deb.debian.org/debian bullseye-updates/main amd64 Packages T-2022-10-15-2035.13-F-2022-10-15-2035.13.pdiff [286 B]
Get:14 tor+http://HTTPS///deb.debian.org/debian bullseye-backports/main amd64 Packages T-2022-10-15-2035.13-F-2022-10-15-2035.13.pdiff [387 B]
Get:14 tor+http://HTTPS///deb.debian.org/debian bullseye-backports/main amd64 Packages T-2022-10-15-2035.13-F-2022-10-15-2035.13.pdiff [387 B]
Fetched 170 kB in 1min 10s (2,420 B/s)                                         
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
user@host:~$

Can someone please confirm this as well as if there’s some security unwise actions in this?

Insurgo · October 16, 2022, 4:53pm

Hmm.

I guess this is under whonix templates.

My understanding is that it is another way of disabling whonix check for torrified update proxy.

For the moment I have no opinion on this, which implementation discussion is happening under shaker at

github.com/unman/shaker

Cacher incompatible with sys-whonix (updates fail for whonix-ws and whonix-gw)

opened 02:56PM - 19 Aug 22 UTC

tlaurion

EDIT: - Working, undesired implementation cacher proxy, advertising itself as g…uaranteeing tor proxy even if false, is at: https://github.com/unman/shaker/issues/10#issuecomment-1221116802 - The only way whonix template updates could be cached as all other templates if repo defs modigied to compoy with apt-cacher-ng requirements is if a cacher-whonix version was created, deactivating whonix-gw's tinyproxy and replacing it with apt-cacher-ng so that sys-whonix would be the update+cache proxy. ---- When cacher is activated, whonix-gw and whonix-ws templates cannot be updated anymore, since both whonix-gw and whonix-ws templates implement a check through systemd qubes-whonix-torified-updates-proxy-check at boot. ~Also, cacher overrides whonix recipes applied at salt install from qubes installation, deployed when the user specifies that he wants all updates to be downloaded through sys-whonix.~ ~The standard place where qubes defines, and applies policies on what to use as update proxies to be used is under `/etc/qubes-rpc/policy/qubes.UpdatesProxy` which contains on standard install:~ Whonix still has policies deployed at 4.0 standard place: ``` $type:TemplateVM $default allow,target=sys-whonix $tag:whonix-updatevm $anyvm deny ``` And where cacher write those at standard q4.1 place https://github.com/unman/shaker/blob/3f59aacbad4e0b10e69bbbcb488c9111e4a784bc/cacher/use.sls#L7-L9 ~First thing first, I think both cacher and whonix should agree on where UpdatesProxy settings should be prepended/modified, which I think historically (and per Qubes documentation as well) it should be under `/etc/qubes-rpc/policy/qubes.UpdatesProxy` for clarity and not adding confusion.~ Whonix policies needs to be applied per q4.1 standard under Qubes. Not subject of this issue. @unman @adrelanos @fepitre --------- The following applies proper tor+cacher settings: https://github.com/unman/shaker/blob/3f59aacbad4e0b10e69bbbcb488c9111e4a784bc/cacher/change_templates.sls#L5-L13 Unfortunately, whonix templates implement a sys-whonix usage check which prevents templates to use cacher. This is documented over https://www.whonix.org/wiki/Qubes/UpdatesProxy, and is the result of `qubes-whonix-torified-updates-proxy-check` systemd service started at boot. Source code of the script can be found at https://github.com/Whonix/qubes-whonix/blob/98d80c75b02c877b556a864f253437a5d57c422c/usr/lib/qubes-whonix/init/torified-updates-proxy-check Hacking around current internals of both project, one can temporarily disable cacher to have torified-updates-proxy-check check succeed and put its success flag that subsists for the life of that booted Templatevm. We can then reactivate cacher's added UpdatesProxy bypass and restart qubesd, and validate cacher is able to deal with tor+http->cacher->tor+https on Whonix TemplatesVMs: 1- deactivate cacher override of qubes.UpdateProxy policy: ``` [user@dom0 ~]$ cat /etc/qubes/policy.d/30-user.policy #qubes.UpdatesProxy * @type:TemplateVM @default allow target=cacher ``` 2- restart qubesd ``` [user@dom0 ~]$ sudo systemctl restart qubesd [user@dom0 ~]$ ``` 3- Manually restart whonix template's torified-updates-proxy-check (here whonix-gw-16) `user@host:~$ sudo systemctl restart qubes-whonix-torified-updates-proxy-check` We see that whonix applied his state at: https://github.com/Whonix/qubes-whonix/blob/98d80c75b02c877b556a864f253437a5d57c422c/usr/lib/qubes-whonix/init/torified-updates-proxy-check#L46 ``` user@host:~$ ls /run/updatesproxycheck/whonix-secure-proxy-check-done /run/updatesproxycheck/whonix-secure-proxy-check-done ``` 4- Manually change cacher override and restart qubesd ``` [user@dom0 ~]$ cat /etc/qubes/policy.d/30-user.policy qubes.UpdatesProxy * @type:TemplateVM @default allow target=cacher [user@dom0 ~]$ sudo systemctl restart qubesd ``` 5- check functionality of downloading tor+https over cacher from whonix template: ``` user@host:~$ sudo apt update Hit:1 http://HTTPS///deb.qubes-os.org/r4.1/vm bullseye InRelease Hit:2 tor+http://HTTPS///deb.debian.org/debian bullseye InRelease Hit:3 tor+http://HTTPS///deb.debian.org/debian bullseye-updates InRelease Hit:4 tor+http://HTTPS///deb.debian.org/debian-security bullseye-security InRelease Hit:5 tor+http://HTTPS///deb.debian.org/debian bullseye-backports InRelease Get:6 tor+http://HTTPS///fasttrack.debian.net/debian bullseye-fasttrack InRelease [12.9 kB] Hit:7 tor+http://HTTPS///deb.whonix.org bullseye InRelease Fetched 12.9 kB in 7s (1,938 B/s) Reading package lists... Done Building dependency tree... Done Reading state information... Done ``` Problem with this is that qubes update processes will start templates and try to apply updates unattended, and this obviously won't work unattended. The question is then how to have whonix templates do a functional test for it to see that torrified updates **are possible** instead of whonix believing he is the only one providing the service? The code seems to implement curl check, but also doesn't work even if cacher is exposed on as a tinyproxy replacement, listening on 127.0.0.1:8082. Still digging, but at the end, we need to apply mitigation ( disable Whonix check) or proper functional testing from Whonix, which should see that the torrified repositories are actually accessible. ----- How to fix this? Some hints: 1- cacher and whonix should modify the policy at the same place to ease troubleshooting and understanding of what is modified on the host system, even more when dom0 is concerned. I think cacher should prepend `/etc/qubes-rpc/policy/qubes.UpdatesProxy` 2- Whonix seems to have thought of a proxy check override: https://github.com/Whonix/qubes-whonix/blob/685898472356930308268c1be59782fbbb7efbc3/etc/uwt.d/40_qubes.conf#L15-L21 @adrenalos: not sure this is the best option, and I haven't found where to trigger that override so that the check is bypassed? 3- At Qubes OS install, torrified updates and torrifying all network traffic (setting sys-whonix as default gateway) is two different things, the later not being enforced by default. Salt recipes are available to force updates through sys-whonix when selected at install, which dom0 still uses after cacher deployment: ![29-Q41_options_selection_done](https://user-images.githubusercontent.com/827570/156628021-d3ec34af-e620-48b8-aa39-45d84c968769.jpeg) So my setup picked up sys-whonix as the default gateway for cacher since I configured my setup to use sys-whonix proxy as default, which permits tor+http/HTTPS to go through after applying manual mitigations. But that would not necessarily be the case for default deployments but would need to verify, sys-firewall being the default unless changed. @unman: on that, I think the configure script should handle that corner case and make sure sys-whonix is the default gateway for cacher if whonix is deployed. Or your solution wants to work independently of Whonix altogether (#6) but there would be discrepancy between Qubes installation options, what most users use and what is available out of the box after installing cacher from rpm: https://github.com/unman/shaker/blob/3f59aacbad4e0b10e69bbbcb488c9111e4a784bc/cacher.spec#L50-L60 4- That is, cacher cannot be used as of now for dom0 updates either. Assigning dom0 updates to cacher gives the following error from cacher: `sh: /usr/lib/qubes/qubes-download-dom0-updates.sh: not found` So when using cacher + sys-whonix, sys-whonix would still be used by dom0 (where caching would not necessarily makes sense since dom0 doesn't share same fedora version then templates, but I understand that this is desired to change in the near future. Point being here: sys-whonix would still offer its tinyproxy service, sys-whonix would still be needed to torrify whonix templates updates and dom0 would still depend on sys-firewall, not cacher, on a default install (without whonix being installed). Maybe a little bit more thoughts needs to be given to the long term approach of pushing this amazing caching proxy forward to not break things on willing testers :) @fepitre: adding you to see if you have additional recommendations, feel free to tag Marek if you feel like so later on, but this caching proxy is a really long awaited feature (https://github.com/QubesOS/qubes-issues/issues/1957) which would be a life changer if salt recipes, cloning from debian-11-minimal and sepecializing templates for different use cases, and bringing a salt store being the desired outcome from all of this. Thank you both. Looking forward for a cacher that can be deployed as "rpm as a service" on default installations. We are close to that, but as of now, this doesn't work, still, out of the box and seems to need a bit of collaboration from you both. Thanks guys!

Insurgo · October 17, 2022, 6:09pm

@unman : I am still confused on some of the internals.

Wanting to reply on a forum thread, and not having installed anything under my disp sys-net for a while, I wanted to install wireshark there in current session to install additional tools I needed just for that session…

And fell into another rabbit hole.

First, as I normally am able to do with current setup on other qubes:

[user@sys-net ~]$ sudo dnf update
Fedora 36 - x86_64                                                                                                                       11  B/s | 547  B     00:48    
Errors during downloading metadata for repository 'fedora':
  - Status code: 500 for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64&protocol=http&protocol=http (IP: 127.0.0.1)
Error: Failed to download metadata for repo 'fedora': Cannot prepare internal mirrorlist: Status code: 500 for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64&protocol=http&protocol=http (IP: 127.0.0.1)

Hmm?

[user@dom0 ~]$ qvm-service sys-net 
clocksync            on
qubes-update-check   on
updates-proxy-setup  on

Ok…
Let’s compare with sys-firewall, which works:

[user@sys-firewall ~]$ sudo dnf update
Fedora 36 openh264 (From Cisco) - x86_64                                                                                                0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'fedora-cisco-openh264':
  - Curl error (56): Failure when receiving data from the peer for https://codecs.fedoraproject.org/openh264/36/x86_64/os/repodata/repomd.xml [Received HTTP code 403 from proxy after CONNECT]
Error: Failed to download metadata for repo 'fedora-cisco-openh264': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Fedora 36 - x86_64 - Updates                                                                                                             15 MB/s |  28 MB     00:01    
^C^C^C^C^C^C^C^C^C^CKeyboardInterrupt: Terminated.

[user@dom0 ~]$ qvm-service sys-firewall
qubes-update-check   on
updates-proxy-setup  on

Any explanation on this?

[user@dom0 ~]$ sudo cat /etc/qubes/policy.d/30-user.policy 
qubes.UpdatesProxy  *  @anyvm  @default  allow target=cacher
#qubes.UpdatesProxy  *  @tag:whonix-updatevm    @default    allow target=sys-whonix
#qubes.UpdatesProxy  *  @type:AppVM  @default  allow target=cacher
#qubes.UpdatesProxy  *  @type:TemplateVM  @default  allow target=cacher

[user@sys-net ~]$ wget 127.0.0.1:8082
--2022-10-17 14:03:14--  http://127.0.0.1:8082/
Connecting to 127.0.0.1:8082... connected.
HTTP request sent, awaiting response... 403 Filtered
2022-10-17 14:03:14 ERROR 403: Filtered.

[user@sys-firewall ~]$ wget 127.0.0.1:8082
--2022-10-17 14:03:42--  http://127.0.0.1:8082/
Connecting to 127.0.0.1:8082... connected.
HTTP request sent, awaiting response... 406 Usage Information
2022-10-17 14:03:42 ERROR 406: Usage Information.

I would have expected to have the same behavior between sys-firewall and sys-net.

Another question: how to change the default of a qvm-service? I recently created a new qube, and that qube doesn’t have updates-proxy-setup by default.
As we discussed before, I do not really see how a qube having qubes-update-check service on, depending on cacher to be able to provide package list to notify dom0 could be able to do so without having updates-proxy-setup also enabled.

Consequently, for the usage I do of cacher, permitting me to install softwares I can sporadically need to have into sys-net or other disposable, I want my qubes to be able to report for templates available updates when I use them, as well as being able to have any qube I use be able to install software in them, since I am well aware, from my use case, that those won’t survive reboot as well. And for that, I would love to have updates-proxy-setup enabled by default unless I deactivate those manually.

How to accomplish this?

Insurgo · October 17, 2022, 6:25pm

From https://dev.qubes-os.org/projects/core-admin-client/en/latest/manpages/qvm-service.html:

`--default` `` `, ` `-D` `` `, ` `--delete` `` `, ` `--unset` ``
Reset service to its default state (remove from the list). Default state means “lets VM choose” and can depend on VM type (NetVM, AppVM etc).

@unman : I understand that updates-proxy-setup would need to be enabled by default in the templates/dispvm templates?

enmus · October 18, 2022, 6:07pm

I’m getting this here

user@cacher:~$ wget 127.0.0.1:8082
--2022-10-18 14:06:33--  http://127.0.0.1:8082/
Connecting to 127.0.0.1:8082... connected.
HTTP request sent, awaiting response... 406 Usage Information
2022-10-18 14:06:33 ERROR 406: Usage Information.

Insurgo · October 18, 2022, 6:48pm

Your here is linked to something unsupported, from what i quickly surveyed on that post. I would suggest using shaker/cacher so that project can improve from its deployed settings instead of @unman or the community trying to figure out errors that would not occur by following known to work recipes.

I won’t comment on that own thread content, whike this off topic comment referring to that thread of inppace upgrade gone wrong will be misleading in this thread.

enmus:

user@cacher:~$ wget 127.0.0.1:8082
--2022-10-18 14:06:33--  http://127.0.0.1:8082/
Connecting to 127.0.0.1:8082... connected.
HTTP request sent, awaiting response... 406 Usage Information
2022-10-18 14:06:33 ERROR 406: Usage Information.

In other circumstances, receiving usage information in a working setup would be a good news. This means the web service is ready to receive direct connection and permits the user to deal with the cache settings from a web browser. Under correct circumstances, if cacher is setuped to be netvm of a qube, that qube couod connect directly to cacher’s ip address’ port or if proxy-enable-setup service is activated for a qube, to talk to 127.0.0.1:8082 to access cache configuration settihgs directly.

enmus · October 19, 2022, 7:43pm

Offtopic

The fix was trivial, or rather quick. I linked to the other topic not knowing if it’s something about apt-caher-ng started not to work, regardless of the template, especially getting 403’s. I hope I clarified my reasoning now.