Apt-cacher: non-deterministic 503 connection refused

This issue still persits in two different ways:

  1. update really fails
Updating template-sys-mullvad
Refreshing package info
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:3 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:3 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:3 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Err:1 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:3 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Reading package lists...
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/dists/bookworm-security/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm-backports/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch http://HTTPS///repository.mullvad.net/deb/beta/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
Ign:1 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:3 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:4 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:1 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:4 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:3 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Ign:1 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:3 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:4 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
Err:1 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:3 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:5 http://HTTPS///repository.mullvad.net/deb/beta bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:4 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Reading package lists...
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/dists/bookworm-security/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm-backports/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch http://HTTPS///repository.mullvad.net/deb/beta/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
W: Target Translations (main/i18n/Translation-en) is configured multiple times in /etc/apt/sources.list.d/mullvad.list:1 and /etc/apt/sources.list.d/mullvad.list:2
  1. Updating fails… not? (Qubes Update GUI shows that the update failed)
Updating template-torrent
Refreshing package info
Ign:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:2 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:2 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Ign:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Ign:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Ign:2 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Ign:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Err:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:2 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Err:4 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
  503  Connection refused [IP: 127.0.0.1 8082]
Reading package lists...
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion/dists/bookworm-security/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian/dists/bookworm-backports/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Failed to fetch tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm/dists/bookworm/InRelease  503  Connection refused [IP: 127.0.0.1 8082]
E: Some index files failed to download. They have been ignored, or old ones used instead.
Hit:1 tor+http://deb.qubesosfasa4zl44o4tws22di6kepyzfeqv3tg4e3ztknltfxqrymdad.onion/r4.2/vm bookworm InRelease
Hit:2 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm InRelease
Hit:3 tor+http://2s4yqjx5ul6okpp3f2gaunr2syex5jgbfpfvhxxbbjwnrsvbk5v3qbid.onion/debian bookworm-backports InRelease
Hit:4 tor+http://5ajw6aqf3ep7sijnscdzw77t7xq4xjpsy335yb2wiwgouo7yfxtjlmid.onion bookworm-security InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

I used your notes to setup sys-apt-cacher-ng and this post.
These are my saltstack states (at least the important ones):

{% if grains['id'] == 'template-sys-apt-cacher-ng' %}

template-sys-apt-cacher-ng--mask:
  service.masked:
    - name: apt-cacher-ng

template-sys-apt-cacher-ng--mask-privoxy:
  service.masked:
    - name: privoxy

template-sys-apt-cacher-ng--configure:
  file.managed:
    - name: /etc/apt-cacher-ng/acng.conf
    - source: salt://sys-apt-cacher-ng/files/acng.conf
    - user: root
    - group: root
    - makedirs: True

template-sys-apt-cacher-ng-privoxy:
  file.append:
    - name: /etc/privoxy/config
    - text: forward-socks5t / 10.152.152.10:9153 .

'chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng':
  cmd.run

'chmod -R a+rX,g+rw,u+rw /var/cache/apt-cacher-ng':
  cmd.run

{% endif %}

and

[workstation user ~/QubesIncoming/dom0]% cat configure.sls 
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

{% if grains['id'] == 'sys-apt-cacher-ng' %}

cacher-enable-rc-local:
  file.managed:
    - name: /rw/config/rc.local
    - user: root
    - group: root
    - mode: 755
  
cacher_/rw/config/rc.local:
  file.append:
    - name: /rw/config/rc.local
    - text: |
        systemctl unmask apt-cacher-ng
        systemctl start apt-cacher-ng
        /usr/sbin/nft insert rule qubes custom-input tcp dport 8082 accept
        systemctl unmask privoxy
        systemctl start privoxy

cacher_/rw/config/qubes-firewall-user-script:
  file.append:
    - name: /rw/config/qubes-firewall-user-script
    - text: /usr/sbin/nft insert rule qubes custom-input tcp dport 8082 accept

/rw/config/qubes-bind-dirs.d/50_user.conf:
  file.managed:
    - source:
      - salt://sys-apt-cacher-ng/files/50_user.conf
    - user: root
    - group: root
    - makedirs: True

{% endif %}

/var/log/apt-cacher.err is empty.
/var/log/apt-cacher.log shows no special error, unless that a 503 http error occured.

Edit:
I think privoxy sometimes dies on startup? in rc.local it gets unmasked and started, but:

root@sys-apt-cacher-ng:~# systemctl status privoxy
privoxy.service - Privacy enhancing HTTP Proxy
     Loaded: loaded (/lib/systemd/system/privoxy.service; enabled; preset: 
enabled)
     Active: inactive (dead)
       Docs: man:privoxy(8)
             https://www.privoxy.org/use

But journalctl -u privoxy -b -n 50 shows nothing and if i run systemctl start privoxy it works

Edit 2:
hm, privoxy gets started 2 minutes after apt-cacher-ng started? I am not sure what the reason for this behavior is? However, it explains why i get these 503 errors. sys-apt-cacher-ng has shutdown-idle enabled, so it practically always needs to be restarted when installing things / when updating. Now that i know the fact that it starts 2 min after apt-cacher-ng i started sys-apt-cacher-ng, waited a few minutes and installed things with saltstack & updated my system - it works withour errors.