Cacher (Apt-cacher-ng): issues with fedora updates

User Support
,
61

You can try this.

These issues happen particularly with “updates” repositories. I think this is
because you are hitting issues with poor mirroring.
They are reported by users not using apt-cacher-ng.

I find in many cases that simply retrying at a later time will work - my
guess is that this is because the mirrors have synced and no other
updates have been pushed in the mean time.
Let me know how you get on.

62

E.g., Fedora-35-minimal and Fedora-36 updates failed this morning.
Tried again now, and all good.

63

Ok, thanks. I’ll try on different intervals.

On another note, would it just be easier to run a squid proxy at my firewall? Then I could leverage that for updating other non-qubes systems I have.

64

Also, just a few minutes later, some started working. Some still fail. So I’ll keep trying the others.

65

I see that worked for you.
I’m sure this is because of mirror syncing in fedora.
You could get round this by pointing all the Fedora repo calls to a
single server - you do this in the Remap definition. I don’t recommend
that.

Of course, in the associated GitHub discussion,
there are many options, including squid.
I find squid over baked for the purpose, and apt-cacher-ng works well
with minimal overhead. YMMV

I never presume to speak for the Qubes team. When I comment in the Forum or in the mailing lists I speak for myself.
66

Thanks for the github discussion link!

I’ll check it out, as I’ve got one qubes system so far (laptop), but am going to repurpose a server to be my new workstation. Plus I’ve got multiple VMs that would benefit from the rpm/deb caching.

I used to run local repos, but that ended up being even more traffic as I was caching packages I’m not using.

67

I had to make this change to get Fedora 38 templates to update. Without the change, I always had to manually boot the template to run dnf clean all.

I’m not using shaker but I got to the point where I did copy over the shaker acng.conf and I manually replicated the
update_template salt steps.

1 Like
68

For me, it randomly succeeds, regardless of False is set to zchunk or not.
So I totally gave up of not to clean all first…

69

I’m having issues with the fedora-cisco-openh264 repository in particular and have read through everything I could find related to that repo, mirrors, apt-cacher-ng, 403 errors, curl 18 errors, etc. and this thread. I thought I’d try to summarize everything to try to centralize troubleshooting steps.

  1. Make sure the Fedora TemplateVM can update outside of cacher/apt-cacher-ng before complicating things.
    • From Dom0, inspect
      • The Fedora TemplateVM’s tags via qvm-tags <TemplateVM name> to ensure it doesn’t have any tags that might interfere with Updates.Proxy routing
      • Dom0’s /etc/qubes/policy.d/50-config-updates.policy and /etc/qubes/policy.d/30-user.policy (if it exists) to ensure the Fedora TemplateVM is set to update through a known-working Update qube and not cacher (yet).
    • From the Fedora TemplateVM,
      • Make sure the repos start with https:// and not the cacher proxy prefix of http://HTTPS/// via sed -i 's^http://HTTPS///^https://^' /etc/yum.repos.d/*
      • Attempt to update via dnf update to verify functionality. If it doesn’t, troubleshoot and resolve those issues before proceeding.
  2. Make sure the template-cacher TemplateVM is healthy
    • Complete the steps above for the template-cacher TemplateVM and monitor for and resolve any issues during apt update like missing or damaged packages.
    • Shutdown template-cacher; also shutdown cacher, if it was running, to ensure any changes propagate down to it.
  3. Configure the Fedora TemplateVM’s repos for cacher’s proxy prefix via sed -i 's|https://|http://HTTPS///|' /etc/yum.repos.d/*
  4. Prepare cacher for Fedora updates and troubleshooting
    • Clear the logs via rm -r /var/log/apt-cacher-ng/*
    • Clear the cache via rm -r /var/cache/apt-cacher-ng/*
    • Modify the following properties in /etc/apt-cacher-ng/acng.conf
      • VfilePatternEx: .*fedora.*updateinfo.*xml.zck$|^/\?release=[0-9]+&arch=.*|.*/RPM-GPG-KEY.*|.*\?repo=fedora|.*pkg.tar.zst.sig
      • DontCache: .*fedora.*updates.*updateinfo.xml.zck .*fedora.*repomd.xml
    • Restart the apt-cacher-ng service via systemctl restart apt-cacher-ng
  5. Attempt to update the Fedora TemplateVM via dnf update to verify functionality. If it doesn’t, troubleshoot and resolve those issues.

I’m not very clear on:

  • When to modify /etc/apt-cacher-ng/fedora_mirrors_extra and the expected syntax
  • The appropriate syntax for metalink or baseurl in the /etc/yum.repos.d/ repo files.

Hope I got this right and helps move us all toward speedy resolutions!

1 Like
70

Thanks, but this works for me, at least at the moment and trying only manual update from within the template while using dnf clean all first:

metalink=http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-$releasever&arch=$basearch&protocol=http

etalink=http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch&protocol=http

metalink=http://HTTPS///mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch&protocol=http

But no guarantees it’ll last… so I usually manually edit problematic repo links during update when it doesn’t work, while bored to listen it actually works and it’s something on my side.

71

Fedora mirrors can be in a shocking state. (Worse than Qubes.)

The openh264 request returns:
https://codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml,
which isnt particularly helpful, and that returns 403,404 500 errors at different times.

You could try commenting out the metalink and inserting:
baseurl=http://HTTPS///codecs.fedoraproject.org/openh264/40/x86_64/os/
in the repo definition.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

72

Now that’s relieving to hear, cause it’s so annoying…

73

Thank you for the suggested baseurl. I tried different fedora-cisco-openh264 baseURLs, but wasn’t quite sure what I was doing.

Unfortunately, this one returns the same curl error as my other experimental fedora-cisco-openh264 baseurls as well as the stock RPM Fusion for Fedora 40 - Free, RPM Fusion for Fedora 40 - Free - Updates, RPM Fusion for Fedora 40 - Nonfree, and RPM Fusion for Fedora 40 - Nonfree - Updates metalinks with the http://HTTPS/// prefix:

Errors during downloading metadata for repository ‘fedora-cisco-openh264’:
- Curl error (18): Transferred a partial file for http://HTTPS///codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml [transfer closed with 303 bytes remaining to read]

The following repos seem to operate just fine: Fedora 40 - x86_64, Fedora 40 - x86_64 - Updates, and Qubes OS Repository for VM (updates). Initially, I think I only had problems with fedora-cisco-openh264, but think the other repos started having problems. I don’t *think* I modified the other repos, but I might have. That’s why I wanted to firm up my troubleshooting steps in my post above.

74

I gave you a working URL - why did you try “different” baseURLS?

Open the file /etc/yum.repos.d/fedora-cisco-openh264.repo in an editor
Comment the line beginning “metalink” by prepending #
Insert the line given:
baseurl=http://HTTPS///codecs.fedoraproject.org/openh264/40/x86_64/os/

It’s best not to make changes if you are unsure what you are doing. That
baseURL entry works reliably for other users.

Open a root terminal in your caching qube.
cd /var/cache/apt-cacher-ng
rm -rf codecs.fedoraproject.org
Edit /etc/apt-cacher-ng/acng.conf to include value: Debug: 7
systemctl restart apt-cacher-ng

attempt update with the baseURL given
examine logs in /var/log/apt-cacher-ng

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

75

I only meant to communicate that I had tried different baseurls in my troubleshooting before posting here and that I was appreciative that you gave me a working baseurl that I had not yet tried in my experimenting. The errors I posted above were from the baseurl you gave me with the metalink commented out.

Interestingly, there was no codecs.fedoraproject.org entry in this directory. Neither now nor in my previous troubleshooting.

This is a good step–I’ll add it to my troubleshooting post above. I had not done this before (UPDATE: I guess there’s a limit to updating previous posts). As such, I’m not sure if this is a new, insightful, or expected log output:

1723380008|M|Download of codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml started
1723380008|I|364|127.0.0.1|codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml [HTTP error, code: 400]
1723380008|E|364|127.0.0.1|codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml [HTTP error, code: 400]
1723380009|M|Download of codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml started
1723380009|I|364|127.0.0.1|codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml [HTTP error, code: 400]
1723380009|E|364|127.0.0.1|codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml [HTTP error, code: 400]
1723380009|M|Download of codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml started
1723380009|I|364|127.0.0.1|codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml [HTTP error, code: 400]
1723380009|E|364|127.0.0.1|codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml [HTTP error, code: 400]
1723380009|M|Download of codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml started
1723380009|I|364|127.0.0.1|codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml [HTTP error, code: 400]
1723380009|E|364|127.0.0.1|codecs.fedoraproject.org/openh264/40/x86_64/os/repodata/repomd.xml [HTTP error, code: 400]

76

Hi again.

I wanted to revisit this to see if anything might’ve changed in the past few months.

I made sure the fedora TemplateVM had the appropriate qvm-tag to use my apt-cacher-ng VM for updates and that my fedora repo URLs had the http://HTTPS/// prefix.

While dnf update -y received the occasional and expected 503 errors, they eventually went away and were replaced with several consistent curl error 56s and 18s (sample of each below):

Curl error (56): Failure when receiving data from the peer for https://fedora.ip-connect.vn.ua/linux/releases/41/Everything/x86_64/os/repodata/repomd.xml [CONNECT tunnel failed, response 403] - https://fedora.ip-connect.vn.ua/linux/releases/41/Everything/x86_64/os/repodata/repomd.xml - repomd.xml
Curl error (18): Transferred a partial file for http://fedora.ip-connect.vn.ua/linux/releases/41/Everything/x86_64/os/repodata/repomd.xml [end of response with 150 bytes missing] - http://fedora.ip-connect.vn.ua/linux/releases/41/Everything/x86_64/os/repodata/repomd.xml - repomd.xml

I verified these URLs were valid in my browser.

I double-checked that the cacher repo mod command didn’t miss anything in the /etc/yum.repos.d/ directory

grep -Flr "https://" /etc/yum.repos.d/ 2>&1

The failing URLs are coming from somewhere else–maybe the .xml files?

I tried to troubleshoot the curl errors by creating a /root/.curlrc file with

proxy=http://127.0.0.1:8082

Deleted this when it had no affect.

I also attempted various permutations of http/s proxies (to no avail):

export http_proxy=http://127.0.0.1:8082
export https_proxy=https://127.0.0.1:8082
export https_proxy=http://127.0.0.1:8082

Found Setting up a cache server for apt packages which mentioned configuring the PassThroughPattern attribute in template-cacher’s /etc/apt-cacher-ng/acng.conf and so I added a wide-open config here

PassThroughPattern: .*

This also had no affect.

I saw @unman mentioned enabling “extended fedora repository lists” in
Fedora 39 Errors during downloading metadata for repository 'fedora': - Status code: 500 - #5 by unman, but I’m not sure if that would affect these curl errors I’ve been trying to research and troubleshoot.

Any thoughts?

77

Nothing has changed. It should work. You should be able to use an apt-cacher-ng qube. I just confirmed that I am able to perform dnf upgrade successfully with my Fedora 40 template. No problems with the openh264 or base fedora repositories. Although, I only tested from my Fedora 40 templates, as I haven’t installed/upgraded to Fedora 41, yet.

Your more recent errors are different from earlier in three ways:

  • Fedora version: 41, not 40
  • Repo: Fedora repo, not the Cisco openh264 repo
  • HTTP error: 403, not 400

The HTTP 403 errors are different from your earlier HTTP 400 errors. HTTP 403 indicates that your connection is forbidden/blocked. Are you using a VPN, Tor, or some other kind of proxied connection for your system updates?

78

All good points. Thank you for your response, @sm95

I just tried setting my apt-cacher-ng qube to use a non-TOR/VPN NetVM (sys-net), but the errors persisted for both my fedora 40 and 41 TemplateVMs (see below). My PassThroughPattern is still set to .*

Refreshing package info
Errors during downloading metadata for repository ‘fedora-cisco-openh264’:

79

If you use that PassThroughPattern you might as well not use a caching
proxy at all. What it means is that all traffic matching that
pattern is not cached.

Response 403 means that access on the server to the requested resource
is forbidden. Without far more detail about your set up it’s impossible
to help. I’d suggest changing the update proxy to tinyproxy (nothing
lost since you are not caching the data) and checking that updates work
correctly.

I never presume to speak for the Qubes team. When I comment in the Forum I speak for myself.
80

Hi @unman ,
Thank you as always for your work and the time you spend responding here in the forum.

The only reason I had temporarily set my PassThroughPattern to .* was to try to reduce variables that could be causing problems with my Fedora updates through cacher. My Fedora TemplateVMs update just fine through sys-net when the Dom0 policy routes updates through that VM and the Fedora repositories are set to `https://…", so I’m not sure what it is about having cacher broker the updates that is causing issues.

I apologize for not providing enough detail here–what additional information can I provide to be of better assistance?