How to update non-template qubes over qrexec while using apt-cahcer-ng as netVM & everything in policies talk about cacher

I have set fully functional apt-cacher-ng qube and am updating all the templates and dom0 via it.

Now, in general I always try to install packages first in disposables to check whatever, but for obvious reasons I cannot do that. So, what I do, I have a backup of non-apt-cacher-ng yum.repos.d and sources.list 9and all other for different distros), so when I want to install something, I overwrite those in disposables with originals, then being able to download.

I tried to find persistent way for both disposable templates and app qubes via bind dirs, hopefully to bind qubes.UpdatesProxy and then to customize it to common https:// instead of http://HTTPS///, but I couldn’t find even on dev.qubes-os what rule to include in it that would treat app qubes.

Even better, is it possible to create such a rule in a dom0’s qubes.UpdatesProxy, something like

$type:AppVM @default allow,target=apt-cacher-ng-cube

Thanks in advance for any pointers.

If you want to use the qubes.UpdatesProxy service you will have to
install it and enable it in the qube.
You could start the service from rc.local

Or you could put the caching proxy upstream of the qubes, and then have
a file in /etc/apt/apt.conf.d -
10Proxy:

Acquire::http::Proxy "CACHE_IP_ADDRESS:PORT";

Either use bind-dirs for that file, or store it in /rw/config and write
it in place from rc.local

Thanks, I have managed to set it but only if cacher is set as netVM for the qube.

If any other qube is set as netVM, the error is thrown

Error

Err:1 http://HTTPS///deb.debian.org/debian bullseye InRelease
Connection failed [IP: 10.x.x.x 8082]

Is it possible sys-whonix for example to be set as netVM for the qube, yet it to updates via apt-cacher-ng?

Or, I am missing something in configuration?

Yes, you are missing something.
If sys-whonix is set as netvm, then all traffic is send through Tor.
This means that the traffic aimed at 10.x.x.x will be routed through Tor

  • it never has a chance to hit the caching proxy because it is encrypted
    by the time it hits that proxy

Thanks unman. Indeed I missed the fact and i was just intended to find out how I could bypass tor for local addresses in Qubes, but then, it turned out that this won’t work even when clearnetVM is set for AppVM.
I have specifically enabled updates-proxy-setup service and disabled qubes-updates-proxy service as per How to install software | Qubes OS, and also bound 10Proxy with the content

#Use Qubes Update Proxy
Acquire::http::Proxy “http://10.x.x.x:8082/”;
Acquire::tor::proxy “http://10.x.x.x:8082/”;

which content I actually already found in /etc/apt/apt/conf.d/01qubes-proxy in a template.
It would work only if apt-cacher-ng would set as netVM.
What am I missing?

Is there a chance i can expect some concrete advise how to set AppVm, or should I continue to investigate myself?

Is there a chance you will provide enough concrete information to allow
anyone to help you?

I have told you that you can configure the proxy by setting a file under
/etc/apt/apt.conf.d - this is the canonical way under Debian.
Provided that you do not have a netvm set which routes traffic over Tor
or a VPN, or nowhere, this will work wherever the caching proxy is set
upstream.

Saying this “wont work” isn’t enough, since this works for me. If not, I
wouldn’t have advised it.

You have to provide enough detail for me to understand what issue you
have, how your system is configured, and so on.
Here’s where you could start, using all Debian-11 based qubes.
Disable `updates-proxy-setup``
Start off with just the 10Proxy file.
Confirm that this setup works when the proxy is the netvm.
Create a new qube that provide network and set it between the test qube
and the proxy.
???
Feedback.

I never presume to speak for the Qubes team.


When I comment in the Forum or in the mailing lists I speak for myself.

You are probably right that I’m missing to give some info that would help. Thanks for your response, anyway. Indeed appreciated.

I’ll try it again:

  • I’m using debian-11 based app qube, named debian-11-test
  • I have specifically disabled updates-proxy-setup service, by adding it to the Service tab, and then deselected it. I did this because you wrote to disable it.
  • I have set a clearnet sys-firewall as a netVM to the qube.
  • I have created 10Proxy file with the content:

#Use Qubes Update Proxy
Acquire::http::Proxy “http://10.y.y.y:8082/”;
Acquire::tor::proxy “http://10.y.y.y:8082/”;

where 10.y.y.y is a local address of an apt-cacher-ng service qube. I have got this address from Qube Manager.

  • I have put the file from within debian-11-test qube into maually created tree /rw/etc/apt/apt.conf.d/.
  • I have created 50_user.conf file with the content

binds+=( ‘/usr/lib/python3/dist-packages/qubesidle/idleness_monitor.py’ )
binds+=( ‘/etc/apt/apt.conf.d/10Proxy’ )

and put it into
/rw/config/qubes-bind-dirs.d/

  • Restarted debian-11-test app qube.
  • Checked if 50_user.conf and 10Proxy are in place. They are.
  • Intentionally shutdown apt-cacher-ng qube
  • Ran sudo apt update in debian-11-test.
  • Starting apt-cacher-ng qube isn’t invoked by this command
  • In debian-11-test qube, expectantly the error is thrown:

Err:1 http://HTTPS///deb.debian.org/debian bullseye InRelease
Could not connect to 10.y.y.y:8082 (10.y.y.y), connection timed out
Err:2 http://HTTPS///deb.debian.org/debian-security bullseye-security InRelease
Unable to connect to 10.y.y.y:8082:
E: Failed to fetch http://HTTPS///deb.debian.org/debian/dists/bullseye/InRelease Could not connect to 10.y.y.y:8082 (10.y.y.y), connection timed out

  • Manually started apt-cacher-ng qube.
  • Retried to update app qube.
  • Same error again.
  • Restarted app qube and retried to update it, while all the time service qube is up.
  • Same error again.
  • Noticed that if 10Proxy is in debian-11 template’s /etc/apt/apt.conf.d/, template won’t update with the error,

Err:9 http://HTTPS///deb.debian.org/debian bullseye Release
Cannot initiate the connection to 10.y.y.y:8082 (10.y.y.y). - connect (101: Network is unreachable)

so moved it to /etc/apt. Update worked.

  • Changed everything in debian-11-test app qube, so to 10Proxy is actually in (/rw)/etc/apt/.
  • Restarted debian-11-test qube and tried to update it.
  • Now the error is thrown

E: Failed to fetch http://HTTPS///deb.qubes-os.org/r4.1/vm/dists/bullseye-testing/InRelease Could not resolve ‘HTTPS’

  • Back 10Proxy to apt.conf.d and specifically enabled updates-proxy-setup service

  • Restarted and tried to update. Again same timed out/unable to connect error.

  • Removed updates-proxy-setup service from the list, leaving it that way to its default state whatever that was.

  • Restarted app qube and tried again. Same timed out / unable to connect error.

  • Set apt-cacher-ng qube as netVM. Without restart, update works.

  • Set netVM to none - “Network unreachable” error.

  • Back to specifically disabling updates-proxy-setup

  • Created disposable sys-cacher netVm based on debian-11-minimal-firewall-dvm-template, set ‘apt-cacher-ng’ as its netVM, and set sys-cacher as debian-11-test app qube netVm.

  • Prior to that, set debian-11-minimal-firewall-dvm-template 10Proxy in /rw/etc/apt/apt.conf.d, with correspondent 50_user.conf in /rw/config/qubes-bind-dirs.d/.

  • Updating sys-cacher via apt-cacher-ng qube expectantly works, while inherited 10Proxy from its template.

  • Setting sys-cacher as debian-11-test qube’s netVM and tried to update, while 10Proxy left in app qube as well.
    Update finally works.
    Browsing works as well

As a noob, I can only conclude that some package or (networking) service is installed/created during creating netVM which doesn’t happen when creating app qube.

Is there a way not to clutter Qubes (and RAM and CPU) with yet another qube?

Thanks for the idea to set netVM in between. Probably wouldn’t occur to me to try it.

Deleted

No, it’s not this, most probably. This inter-netVM can be updated only if its netVM is apt-cacher-ng too, so the same problem.
I now don’t see the point to put netVM between app qube and proxy, when app qube will be updated anyway when attached to proxy.

I think you are confused between the processes for using the caching
proxy over qrexec, (as for templates), and using it over normal
networking.

Looking at your extensive notes, I see that you have adopted a
scattergun approach, trying many different things and changing your
targets and configuration. It’s much better to identify a specific aim,
and focus on solving that specific problem. To do this, break down your
major task in to small steps and solve each step at a time.
Don’t mix in bind-dirs at this stage - don’t switch the target - take
small steps.

Access using normal networking.

  1. To access the proxy using normal Qubes networking, you specify the
    actual IP address of the proxy in the 10_Proxy file.
    To be processed by apt processes, this file should be in
    /etc/apt/apt.conf.d
    Write that file (the contents you show look right to me.)

  2. Set the caching proxy as the netvm of your debian-11-test qube.

  3. sudo apt update

  4. This should work - if it doesn’t, check the IP address you specified
    is correct.

  5. If apt update still does not work, see if you can access
    http://10.y.y.y:8082 in a browser. You should see a configuration
    window. If you don’t, it suggests that either apt-cacher-ng is not running,
    access is blocked via nftables, the IP address is incorrect, or you have
    not set the proxy as netvm.

  6. Solve that problem: the sensible place to start would be to work in the proxy
    qube. You can check the status using systemctl status apt-cacher-ng,
    and check availability by pointing a browser at the same address.
    If the service is running, make sure that you have a rule in the INPUT
    chain giving access to tcp port 8082, by specifying ACCEPT tcp dpt:8082
    You can check this by running sudo iptables -L -nv.
    If you do not have such a rule, add one:
    sudo iptables -I INPUT -p tcp --dport 8082 -j ACCEPT

  7. Test again from the proxy

  8. Test again from debian-11-test

Access using normal networking, where the proxy is not direct netvm.

  1. Once you have the first case working, create a new qube with
    provides_network set, and the proxy as netvm.
  2. Set this qube as the netvm for debian-11-test.
  3. Confirm you can access the proxy from this new qube. (Steps as
    above.)
  4. Confirm you can access the proxy from debian-11-test.

N.B

This configuration will work provided that the intermediate qube is not
encrypting the traffic or routing it elsewhere - a VPN or Tor proxy
will do this, so don’t put them in the netvm chain.

You can use apt-cacher-ng as a drop in replacement for tinyproxy,
so you can run it on e.g. sys-firewall.
If you want to do this, make sure that you have disabled the tinyproxy
service, and that the caching proxy service is able to start and run.

Access using qrexec

This is the method used by templates, but you can apply it to any qube.
updates-proxy-setup creates a listening service at 127.0.0.1:8082,
which forwards traffic over qrexec to the target.
The target is specified in dom0, in 4.0 using /etc/qubes-rpc/policy/qubes.UpdatesProxy,
and in 4.1 by a policy file in /etc/qubes/policy.d

To make this work in a qube:

  1. Begin with a clean qube.
  2. Enable the updates-proxy-setup service
  3. Start the qube.
  4. Confirm that the service is running:
    systemctl status qubes-updates-proxy-forwarder.socket
  5. The service should show “active(listening)” - if it does not, solve
    that problem. Attempt to manually start the service, and check the
    status again.
  6. Confirm that the file has been created at /etc/apt/apt.conf.d/01qubes-proxy specifying the proxy at http://127.0.0.1:8082/
  7. Make sure that you have an entry in the relevant policy file for this
    new qube, pointing to the caching proxy. (In 4.1 it’s recommended that
    you create a new policy file in the same location with a lower number _
    e.g 30_user.policy - to hold custom changes to the policy.)
    The syntax is slightly different between 4.0 and 4.1, so make sure you
    sue the right form.
    In 4.1 it would look like TEST @default allow target=CACHING_PROXY -
    change the uppercase to match you setup.
  8. sudo apt update

Don’t have TWO proxy specifications in /etc/apt/apt.conf.d/ - that’s
confusing for you and for your qubes.

Summary

Break down your target in to small steps.
Keep the specification of the target and of each step as minimal as
possible.
Focus on solving each small step in order.
Once you have solved one, move on to the next.

When you have reached the target, you may want to add more steps - e.g.
in this case adding bind-dirs. That is not necessary to start with.

It takes some time to learn how to do this, particularly in Qubes,
where there may be many confounding factors.
It also takes time to learn what steps are needed
to reach a certain target: this can lead to XY problems.
That’s fine - everyone starts somewhere.
If you need help ask - it’s helpful for people to say how the system is
configured, what they want to achieve, what steps they have taken so
far, and where they are stuck.

I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.
3 Likes

First of all,

Thank you for such a detailed and nicely organized response. I guess it would be of immense help to other users too.

I just want to be short now. From point to point, I have already achieved that, but obviously my terminology in my previous posts was too noobish to emphasize the fact. What I have learned here in addition is that I could access configuration from the browser if apt-cacher-ng service was properly working. Tried that too and it also worked in all three qubes (test, inter-vm, and caching-proxy).

Everything is working, after setting 10_Proxy as per your initial tip.

Probably, but more probably was that I just didn’t express myself properly. From the very beginning that was my goal, keeping in mind that in my initial post I was trying to express the idea:

and finally, I got the answer from you I was looking for:

with a slight modification, as per Qubes Architecture Next Steps: The New Qrexec Policy System | Qubes OS, so it actually looks like

qubes.UpdatesProxy * debian-11-test @default allow target=apt-cacher-ng-qube

And, following your

updates now works, whatever netVM is set, because of course this time it is done via qrexec, but I’m still worried about that * because I was warned I missed proper number of columns.

One thing is left, at least for me, to conclude this topic.
I’d like to allow all qubes except vault to be updated over qrexec. Would the policy look like this in that case:

qubes.UpdatesProxy * vault @default deny target=apt-cacher-ng-qube
qubes.UpdatesProxy * * @default allow target=apt-cacher-ng-qube

I am afraid of the second * in the second rule above: it probably ssumes templates as well, and I wouldn’t want to mess something with that, especially when whonix is in the matter.

Thanks again for a great example how to write guides, on tips how to approach to problems and, above all, on your time.

qubes.UpdatesProxy * vault @anyvm deny
qubes.UpdatesProxy * @anyvm @default allow target=cacher

It looks, this would be the proper way so far. Still have to learn about RPC policies, I admit that while the syntax is pretty much clear, content isn’t yet (for example what +Argument is and could be, is there @type:AppVM since it is mentioned but it seem it doesn’t work, etc…).

It looks, this would be the proper way so far. Still have to learn about RPC policies, I admit that while the syntax is pretty much clear, content isn't yet (for example what +[Argument](https://www.qubes-os.org/news/2020/06/22/new-qrexec-policy-system/#new-column-syntax) is and could be, is there @type:AppVM since it is [mentioned](https://github.com/QubesOS/qubes-core-qrexec/blob/master/doc/multifile-policy.markdown) but it seem it doesn't work, etc...). The syntax change I was trying to highlight in 4.1 is the absence of a comma. (You had quoted a rule *with* a comma - `allow,target=cacher`)

Here’s a case where an argument is useful -
In my split-ssh, I have a sys-agent qube which holds multiple ssh-agents.
A client qube can access qubes.SshAgent to access the “work” agent, but
not the “personal” agent.
The policy could be:

Qubes.SshAgent  work      client  @default  allow target=sys-agent
Qubes.SshAgent  personal  client  @anyvm    deny

I haven’t used @type:AppVM, but I’ll try it when I’m back in 4,1

1 Like

Thanks for the feedback in advance.

But, I noticed something rather unexplained to me.

  1. Namely, I tried to install some app in disposable qube, via cacher qube of course. I then checked it there, if it worked, how it worked, if there is some strange running it, and so on (what I am capable of, to be more precise). Everything went fine, so I decided to install it on a template.
    Started template (all debian), started procedure to import keys, repositories, etc, and finally to install it.

To my surprise, downloading of the very same app started again, and not from the cache in a cacher qube?! How do I assume this? Because I waited for more than 5 minutes app to be downloaded in a template, since cacher qube is chained to sys-whonix, and the speed is as we all know around several hundreds of KB/s.

Did you ever tried this? If you did, did you experience the same and there is some logical explanation, and if you didn’t try can you try it please and let me know so I could perform my sanity check. I don’t find it crucial, so absolutely understand if not replying on this.

The only reason this might happen is that it wouldn’t be good to cash the package if it turns out to be bad in disposable, which is good but in a way defies the purpose of cacher qube. Anyway somehow I doubt that there is such a mechanism in Qubes. It isn’t related to apt-cacher-ng as I see it, since apt-cacher-ng itself isn’t aware of templates and disposables, so it would have to do something with Qubes.

  1. My fedora-dvm-template updates via qrexec and cacher qube smoothly. Yet, sys-usb disp qube which is based on it, doesn’t. I have checked for possible differences between dvm-template and sys-usb disp, and couldn’t find differences.
    01qubes-proxy is on the same place in both, in both updates-proxy-setup service is enabled (to the latter I had to manually type and add it to the Service tab), and I rather get an error:

Error: Error downloading packages:
Status code: 500 for http://HTTPS///mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 (IP: 127.0.0.1)

and in sudo journalctl -f I found

Apr 20 09:23:00 sys-usb tinyproxy[2399]: opensock: Could not retrieve info for HTTPS

So, obviously port 8082 is missing above, if it’s missing, so I specifically added and disabled tinyproxy service and restarted sys-usb. No luck. Specifically enabled service and restarted. No luck again.
These are mi knowledge limits at the moment, so if you have some idea what and where to research, I’d be grateful. always willing to learn.

Indeed confusing, and I don’t have an idea what further steps to tke in order to diganose it.

  1. In addition, in my /etc/yum.conf.d/qubes-proxy.conf

it is already

proxy=http://127.0.0.1:8082/

in sys-usb, so stiil have no idea what it might be…

Any news on this?

@unman From my tests, to have all qubes being able to install applications (since there repo definitions changed per cacher deployment)

It is required to

  • have the AppVM policy statement to be able use RPC calls to cacher
[user@dom0 ~]$ cat /etc/qubes/policy.d/30-user.policy 
#qubes.UpdatesProxy  *  @tag:whonix-updatevm    @default    allow target=sys-whonix
qubes.UpdatesProxy  *  @type:AppVM  @default  allow target=cacher
qubes.UpdatesProxy  *  @type:TemplateVM  @default  allow target=cacher

(Note that in above case, I am able to use cacher for tor+https use cacher since I have modifed userinfo.html content in template-cacher so that it exposes its proxy as enforcing tor. That is ok for my use case, since I have cacher use sys-whonix as netvm. This is discussed under https://github.com/unman/shaker/issues/10 and more concisely for implementation details under https://github.com/unman/shaker/issues/12).

  • have all qubes wanting to install software until reboot to have the qubes-proxy-setup service activated, otherwise repository definitions containing http://HTTPS/// are simply failing from app-qubes (normal afterall).

@enmus does that answer your question?

@unman : I will not open a cacher issue as of now, but this is an issue. I think all app qubes should use the RPC call as the templates do for software install. After all, users might install a software inside of a qube/dispvm first to see if it behaves correctly, and the dispvm template will be derived from the templatevm which is configured to use cacher as well. I do not see any case where not having qubes-proxy-setup feature activated.

Thoughts?

I’ve already said why I think this is a bad idea.
That marker makes sense where the updateVM is guaranteed to go through
Tor, as is supposedly the case with sys-whonix.
It doesn’t make sense in a case where it’s trivial for the user to change
things so that the updateVM does not go through Tor. For the cacher,
the only setting is that the netvm or netvm of some upstream qube is set
to a Tor proxy. But that is quite fragile, and easily changed in error.
If such a change were made then Whonix would allow updates that dont go
through Tor, and I assume that that wood be unwelcome in Whonix.
(Actually it’s this sort of check that is one reason why I dont use
Whonix, but that’s a separate issue.)

This is a major intervention, and I am not clear it is the right think
to do.
The solution I have promoted so far is to use a proxy setting in the
qube and set netvm to cacher; no one has had any problem with that.
I think that I prefer that option - not least because it requires users
to think about whether they should be installing software in the qube
(doesn’t work without user action), or the template (works).
We still see posts about this - “I installed foo and it disappeared
when I restarted the qube”.

I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.

18 posts were split to a new topic: Issues with apt-cacher-ng