Apt-cacher: non-deterministic 503 connection refused

monoxide059 · April 19, 2025, 8:57pm

When updating qubes with qubes updater or installing packages with saltstack i pretty often get 503 Connection refused errors.

I have not found out some reasons for this; it isnt important if they are onion repos or not, no matter if the sys-cacher qube was already started or not - i still get the error. If I try run the salt state again immediately after the failure then it works.

unman · April 19, 2025, 11:40pm

https://apt-cache.privex.io/acng-doc/html/troublefaq.html#prob

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

monoxide059 · May 8, 2025, 2:43pm

This issue still persits in two different ways:

update really fails

Updating template-sys-mullvad
Refreshing package info
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:3 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:3 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:3 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Err:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:3 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Reading package lists...
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/dists/bookworm-security/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm-backports/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch http://HTTPS///repository.mullvad.net/deb/beta/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
Ign:1 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:3 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:4 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:1 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:4 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:3 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:1 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:3 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:4 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Err:1 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:3 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:4 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Reading package lists...
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/dists/bookworm-security/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm-backports/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch http://HTTPS///repository.mullvad.net/deb/beta/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2

Updating fails… not? (Qubes Update GUI shows that the update failed)

Updating template-torrent
Refreshing package info
Ign:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:2 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:2 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:2 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Err:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:2 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Reading package lists...
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/dists/bookworm-security/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm-backports/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.
Hit:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Hit:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Hit:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Hit:4 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I used your notes to setup sys-apt-cacher-ng and this post.
These are my saltstack states (at least the important ones):

{% if grains['id'] == 'template-sys-apt-cacher-ng' %}

template-sys-apt-cacher-ng--mask:
  service.masked:
    - name: apt-cacher-ng

template-sys-apt-cacher-ng--mask-privoxy:
  service.masked:
    - name: privoxy

template-sys-apt-cacher-ng--configure:
  file.managed:
    - name: /etc/apt-cacher-ng/acng.conf
    - source: salt://sys-apt-cacher-ng/files/acng.conf
    - user: root
    - group: root
    - makedirs: True

template-sys-apt-cacher-ng-privoxy:
  file.append:
    - name: /etc/privoxy/config
    - text: forward-socks5t / 10.152.152.10:9153 .

'chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng':
  cmd.run

'chmod -R a+rX,g+rw,u+rw /var/cache/apt-cacher-ng':
  cmd.run

{% endif %}

and

[workstation user ~/QubesIncoming/dom0]% cat configure.sls 
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

{% if grains['id'] == 'sys-apt-cacher-ng' %}

cacher-enable-rc-local:
  file.managed:
    - name: /rw/config/rc.local
    - user: root
    - group: root
    - mode: 755
  
cacher_/rw/config/rc.local:
  file.append:
    - name: /rw/config/rc.local
    - text: |
        systemctl unmask apt-cacher-ng
        systemctl start apt-cacher-ng
        /usr/sbin/nft insert rule qubes custom-input tcp dport 8082 accept
        systemctl unmask privoxy
        systemctl start privoxy

cacher_/rw/config/qubes-firewall-user-script:
  file.append:
    - name: /rw/config/qubes-firewall-user-script
    - text: /usr/sbin/nft insert rule qubes custom-input tcp dport 8082 accept

/rw/config/qubes-bind-dirs.d/50_user.conf:
  file.managed:
    - source:
      - salt://sys-apt-cacher-ng/files/50_user.conf
    - user: root
    - group: root
    - makedirs: True

{% endif %}

/var/log/apt-cacher.err is empty.
/var/log/apt-cacher.log shows no special error, unless that a 503 http error occured.

Edit:
I think privoxy sometimes dies on startup? in rc.local it gets unmasked and started, but:

root@sys-apt-cacher-ng:~# systemctl status privoxy
privoxy.service - Privacy enhancing HTTP Proxy
     Loaded: loaded (/lib/systemd/system/privoxy.service; enabled; preset: 
enabled)
     Active: inactive (dead)
       Docs: man:privoxy(8)
             https://www.privoxy.org/use

But journalctl -u privoxy -b -n 50 shows nothing and if i run systemctl start privoxy it works

Edit 2:
hm, privoxy gets started 2 minutes after apt-cacher-ng started? I am not sure what the reason for this behavior is? However, it explains why i get these 503 errors. sys-apt-cacher-ng has shutdown-idle enabled, so it practically always needs to be restarted when installing things / when updating. Now that i know the fact that it starts 2 min after apt-cacher-ng i started sys-apt-cacher-ng, waited a few minutes and installed things with saltstack & updated my system - it works withour errors.

unman · May 8, 2025, 3:25pm

@monoxide059 I’m too tired just now to read the detail of your post.
I’ll return to it in the morning.

You can increase the logging level by setting Debug option in acng.conf
and restarting service.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

monoxide059 · May 8, 2025, 5:22pm

I improved logging both in apt-cacher-ng and in privoxy, apt-cacher.err was still empty, and I didn’t saw anything interesting in privoxy logs.
Please let me know if the logs are still interesting after my second edit in the post were i gave more information about the error, then i will post it - they are pretty big