Curl-proxy / wget-proxy scripts in Templates so users can add GPG distro keys linked to added external repositories

I was curious to trying out the various methods mentioned above so I made a deb-signal template only to find out none seem to work:

wget:

$ export https_proxy=http://127.0.0.1:8082
$ wget -O- https://updates.signal.org/desktop/apt/keys.asc
Connecting to 127.0.0.1:8082... connected.
Proxy tunneling failed: CONNECT denied (ask the admin to allow HTTPS tunnels)Unable to establish SSL connection.

curl:

$ curl --proxy http://127.0.0.1:8082/ --tlsv1.2 --proto =https --max-time 180 "https://updates.signal.org/desktop/apt/keys.asc"
curl: (56) Received HTTP code 403 from proxy after CONNECT

I then noticed that when launching the command, my apt-cacher qube was started and I found the following command to be working:

curl --proxy http://127.0.0.1:8082/ --tlsv1.2 --max-time 180 "http://HTTPS///updates.signal.org/desktop/apt/keys.asc"

So I had to remove --proto =https and change https:// to http://HTTPS///.

Is this behavior to be expected when not dealing with repositories but an apt-cacher qube is enabled?

It’s offtopic asking that here.

@marmarek :

@adw @marmarek @sven @Demi @fsflover

If you prefer not interrupting user, but warning him that he is about to download stuff on the internet from a non-networked Template, that section could be changed to simply call the -proxy (here curl-proxy as an exemple) instead of exiting and asking the user to change curl to curl-proxy manually…:

if [ "$(qubesdb-read /type)" == "TemplateVM" ]; then 
	if ! $(qubesdb-read /qubes-gateway > /dev/null 2>&1); then 
		echo "You attempted to download a file with curl in a Template without direct internet access. (no network assigned)." >&2
		echo "" >&2
		echo "The attempted downloading command was: curl $@." > &2 
		echo "" >&2
		echo "Automatically translating to: curl-proxy $@" >&2
		echo "" >&2
		echo "Continuing in 3 seconds... Type CTRL+C to prevent the download now!" >&2
		sleep 3
		/usr/bin/curl-proxy "$@"
	else
		echo "You attempted to download a file with curl in a Template WITH DIRECT INTERNET ACCESS. (network assigned)." >&2
		echo "" >&2
		echo "The attempted downloading command was: curl $@" >&2
		echo "" >&2
		echo "Continuing in 3 seconds... Type CTRL+C to prevent the download now!" >&2
		sleep 3
		/usr/bin/curl "$@"
	fi
else
	echo "curl download attempt not in TemplateVM. Continuing..." >&2
	/usr/bin/curl "$@"
fi

The resulting behavior would become similar to a Template with network access configured externally in Qubes, while using the proxy through curl-proxy translated call automatically (snippet):

The attempted downloading command was: curl -so /etc/apt/trusted.gpg.d/oxen.gpg https://deb.oxen.io/pub.gpg

Automatically translating to: curl-proxy -so /etc/apt/trusted.gpg.d/oxen.gpg https://deb.oxen.io/pub.gpg

Continuing in 3 seconds... Type CTRL+C to prevent the download now!

@deeplow @marmarek :
This seems like the only way to stop/warn/make aware/prevent users from running a whole snippet of code borrowed from the internet which would include curl/wget commands from actually doing harm and render otherwise trusted template into untrusted template, effectively limiting what would be downloaded before it happens.

The code snippet copy pasted in shell is going to download 3 files? The user receives 3 warnings! Other more obscure code snippets using something else then curl/wget will just not get access to the internet, which I think is the best tradeoff, not requiring to widen proxy access to the whole template nor giving global network access externally to the template.

Otherwise enticing the user to open the door wide open by either telling the whole template to use proxy / inviting the user to give internet globally to the Template per Qubes core documentation.

Yes, apt-cacher-ng is replacing the tinyproxy and listening in it’s place at 8082. You had to actually configure your apt-cacher in it’s config file to use that port. :wink:

1 Like

He already has.

@adw For some reason, I feel you are the person to be convinced of the proposed approach.

@adw to me this is a balance movement, which maybe went too far one way and then too far the other. @marmarek proposed to implement exactly the 4th step of your full circle picture you posted earlier: open up the non-networked Template. By an alias, or by documentation. This approach gives the exact same result as assigning network to a Template externally, and wil lbe as confusing for end users. Please take a minute and thing more broadly. We can fix this.

Please take 1 minute and look at the output of the PoC.

If you do not like the fact that non-networked Templates (default) were requiring the user to manually replace wget/curl with wget-proxy/curl-proxy on the command line, I suggested a non-blocking version translating curl/wget to curl-proxy/wget-proxy calls automatically, pausing for 3 seconds prior of complying with user request so the user can interrupt the request. That would be the safer default approach, permitting to educate/raise awareness/not block standard calls/follow upstream instructions without customization.

The idea here again being to give safer by defaults options, teaching users what are Qubes differences (non-obstructively), while not having binary allow all/block everything approach by default.

Please let me know your constructive thoughts, remembering that users that know what they are doing will go as @sven. While people not knowing what they are doing will open up everything. We need middle ground, and permitting wget/curl (which most software installation methods talk about) are covered by PoC.

Exporting a variable enables network access just in this terminal (and apps started from there), and only until it’s closed. That’s a major advantage over connecting netvm, while simpler than wrappers. But see below.

Wrappers that educate the user are indeed great idea. I think this could be the main reason to use wrappers instead of just env variable.

But also, I think outside of TemplateVM, those should not be set at all (not setting aliases, not printing anything extra etc) - the risk of breaking normal use cases is too great.

2 Likes

@marmarek:

What is not covered ?

The normal qubes use case uses the same wrappers.

The if [ "$(qubesdb-read /type)" == "TemplateVM" ]; then should be when setting aliases, instead of within the wrappers.

Not sure I follow, while just realizing that links to special anchors related to post headers are not taken into consideration so all my past links pointing to different parts of my PoC are all linking to the main “# header” not “## subheaders”. Sigh.

That wrapper would be available in Template and then in qubes. The wrapper is consequently called the same way per aliases defined. And the same code is being used, changing behavior if in TemplateVM, and then if in a networked TemplateVM specifically, or otherwise in a qube (not being a TemplateVM). Is there something I haven’t tested correctly @marmarek?

Quoting again, to be sure you saw output:

Then:

And finally, qubes:

While I proposed an alternative non-blocking solution for non-networked Template to use directly curl-proxy/wget-proxy, after pausing for 3 seconds (so the user can interrupt calls done in random ran scripts actually trying to download stuff on the internet with curl/wget):

Which outputs:

Let me know what you would improve @marmarek. Wrapper would improve UX/educate users/prevent most probable disasters, indeed.

With most of the users it always is about “I know what I’m doing”, “It won’t happen to me”, “I have nothing to hide anyway”.

So, the first thing next that will happen would be users asking how to workaround this “annoyance”, as a following of a routine not to read what is echoed after the first couple of successful and carefully implemented attempts/imports?

The last proposition is making curl and wget call wget-proxy and curl-proxy automatically, warning the user adding a delay of 3 seconds so they can cancel an unwanted download, the script showing the URL that will be downloaded otherwise. I don’t have anything better to propose to address the problem space (warn, educate, not open the Template wide giving it complete internet access through global proxy access/giving template internet access). Otherwise, if not dealt in code, it will need to be dealt by documentation. I always prefer the former.

A knowledgeable user could call wget-proxy/curl-proxy directly. The wrappers are the ones warning and adding a reaction time delay. If a user is able to replace those calls, he understands those calls and can change those calls. That was the base logic behind my PoC. With the added delay that security shoukd be based on trust but inspect principles, which tinyproxy is lacking right now, not keeping logs to inspect after the fact.

At this point I think i made my point. I made a PoC. I pass the ball to other users which complained many times in GitHub issues and other threads, following their own annoyances and push this forward and modify/propose alternatives and resolve this total annoyance for new comers.

As usual the best is always to have a solution being implemented upstream. In Qubes. To address the problem in a way that fits its user needs. I advocated for the most vulnerable and not teksavvy.

This is now your turn (users) to challenge and propose better alternatives in the goal of making Qubes better for all.

Hopefully doing so by still thinking of the most vulnerable out there which are probably less knowledgeable then yourself.

Allow me to add my point of view to this topic.

I am already uncomfortable with the fact that any program running in a template can get network access to any destination by simply using the proxy at port 8082. I understand that I can address this by setting a different policy and/or setting firewall rules for the qube that hosts the UpdateProxy. However, we are talking about the default here.

The issue discussed in this thread is of course real, but it ought to be addressed by educating users, better documentation, community support and hopefully one day a “salt store” offering trusted recipes to install software and configure Qubes OS (@unman works on this I think). In the meantime maybe by adding some popular application repositories and keys to the default install as @Demi proposed. Maybe.

What the wrappers proposed here aim at is to allow users to follow instructions they don’t understand to install who knows what software on a system they choose for heightened security. And the justification that they might otherwise do even worse things is none at all. There is only so much Qubes OS can do to protect users. Doing less to prevent them from becoming frustrated and sabotaging themselves can’t really be an argument – can it?

1 Like

And maybe even more by the fact of a most user friendly action imagined - choosing netVM in Qube Settings for the template. Setting proxy is somewhat advanced, but the former is so easy and teasing.

My intention wasn’t to discourage @Insurgo to continue brainstorming, but only to bring in user’s point of view.

I just see this as a safety feature as any other in Qubes, which the one have to learn how to workaround if (s)he wants to use Qubes. I’m at the forum too short, but I haven’t seen a safety feature that wasn’t challenged or asked how to be circumvented. Shared folders/storages, VMs communicating to each other directly, bypassing safety dialog boxes, remote access to dom0, installing whatever in templates, installing whatever in dom0, copying from/to dom0 more friendly, escaping using CTRL+SHIFT+c/v, etc, etc… And probably all of them, I’ll repeat myself here, they all know what they’re doing, or nah, it won’t happen to them, or they don’t have to hide anything, anyway.
Well, there are users too that will not challenge any of these, that adopted the logic and want to see Qubes even more tighter in terms of security.
There is simply not much space for conformism with the astronaut’s suit while in space :smiley:

@Sven

  1. If the proxy was logging downloaded content, the principles of “trust but inspect” could be applied after the fact, if using such proxy. Giving total network access, without passing by the proxy, is the other option. So here, as of now, we have an unmonitored proxy, that users bypass per instructions to install whatever they want. I still think that enforcing a proxy is a pretty good way to do things. Yes, anything knowing where the proxy is (apt/dnf) can use it. But the problem here is not the safety mechanism, but the lack of current auditing possibilities of what went through it.
  2. The approach of exposing the proxy system wide is the counter-proposition given by @marmarek here. If implemented, the user could call enable-network-proxy which would give system wide internet access through the proxy. Once again, I would not personally have a problem here (proxy is used, no direct access per Template assignation of a gateway), IF tinyproxy was logging url accessed.
  1. My latest proposition (I feel people do not read) is to pause (3 seconds) each wget/curl commands typed in Template terminal to tell the user what is wget/curl, and that if not intercepted, xyz file will be downloaded per past command (in script or command line). This is a non-blocking proposal. A script calling wget/curl would be intercepted by the wrapper, pausing for 3 seconds at each wget/curl call.
  2. By implementing 3 different behaviors (TemplateVM without network assigned, TemplateVM with network assigned, qubes), everything calling wget/wget is intercepted by the wrappers, where only TemplateVMs have different paths of operation: Non-Networked Template informs the user that curl-proxy/wget-proxy will be called with the URL in 3 seconds, giving the user possibility to CTRL-C, while networked TemplateVM gives a warning, pause for 3 seconds with the wget/curl URL that will be downloaded next. Application Qubes just receives a message saying that they are going to download something and doesn’t pause. The idea here is for the user to have a chance to stop each individual downloads that would happen in a script ran under a TemplateVM while not blocking the user from downloading stuff with curl/wget (since most upstream instructions are using those 2 common download tools), permitting scripts to still work. The blocking/annoying approach would be to ask user for confirmation, which would break scripts.
  3. @Sven We could go more advanced then this, but that would be more complicated to explain to users, which would otherwise just open wide internet access. For example, scripts are called in terminal in a different way. We could block that, so the user has to type commands one by one interactively on the terminal, instead of pausing for 3 seconds each time curl/wget is called. But the user would get even more frustrated, and would assign network to template even more easily.

Maybe its time to just ask basics and get global consent on basic principles.

  1. @sven: Do you agree that having tinyproxy logs would permit core documentation to give guidelines to be able to inspect in retrospect what was downloaded in a Template, since an external proxy is used? Do we agree that since that proxy is external, it is more complicate to obfuscate network accesses and would give the possibility to confirm/infirm trusted state of said Template?
  2. We agree that Templates can access the internet if commands explicitly express how to use defined proxy in their wget/curl commands.
  3. Compromise of such Templates through internet downloaded stuff would require past actions to have defined system wide bypasses, explicitely defining how to use such proxy, and commands ran in TemplateVM outside of simply downloading required stuff, not running there anything else

@Sven The proposed solution, pausing for 3 seconds, warning users of downloads that will happen in TemplateVMs (with proxy if no network, without proxy if network is given) was my best idea to deal with all use cases. A user knowing what he is doing will most probably give direct internet access to cloned Template, and the wrapper would pause for 3 seconds prior of calling wget/curl. No big deal. Both TemplateVMs use case can still call wget-proxy and curl-proxy, which is not a regression following your comment on security, since bypasses measures for commands to directly use proxy could still be used by code snippets trying to circumvent TemplateVM to download stuff. But the person writing installation scripts could now simplify them by calling wget-proxy and curl-proxy directly, bypassing the 3 seconds warning pause.

Makes sense? Otherwise:

Would be helpful if GPG+Repositories are deployed. But then again, for having done that in the past, I received a lot of complaints, which some in this thread already shown same disagreement: Why would we do that choice for the user of deploying Signal repository? (Why Qubes would be choosing to deploy Signal?)
Deploying “persona” related packages (salt recipes, repositories, GPG keys, even personalized TemplateVMs including the precedent predeployed) is an old dream going back to 2019 if I recall well.
But meanwhile, each time a user installs/reinstalls (Q4.1) with new templates (time to get back clean and install repositories and software again) is a big burden for users, which didn’t thought they would have to do it again, and forgot the complications they had to deal with years ago. Some will have made some scripts to ease the process (not a lot know salt), but in most cases following upstream software installation documentation is ok. The problem arises when the user needs to find security measures circumvention.

I hope we agree here that users should be able to just follow Signal/Session guides. Having a 3 second warning prior of having expected result is, I think, a pretty correct result. On that, I do not understand this thread reluctance.

This is why I advocate for curl-proxy and wget-proxy (which basically are better defaults then calling curl and wget with obscure and unsafe options), while wrappers are teaching them how to script this properly, and do simple substitutions when following upstream instructions, which would nullify the need from Qubes to package GPG keys and repositories (maintainership of those added) and continue to give Qubes OS freedom, while warning them about what they are going to do prior of actually impacting their templates.

@enmus It is hard to implement security without impacting convenience. We are talking about safer default (proxy is there now, which is a good thing, while not permitting proper audit to serve its full purpose, which is the problem here).

In short

  1. We need tinyproxy logs so core documentation can tell users to inspect logs prior of trusting a new created/modified Template. This proxy could become a better ally, not something most users want to circumvent because its in the way. We should wrap around it better
  2. We should not push users to give wide internet access to Templates but in last resort
  3. We should guide users in their learning steps, reducing friction without compromising security principles

Note that for HTTPS connections (and you should use HTTPS) you can only get target host, not full URL.

1 Like

@marmarek Agreed (apt-cache-ng being another subject altogether I do not want to mix here). We could output timestamps in the wrappers to better match what would be in tinyproxy entries. Point here, if there is 4 entrees in tinyproxy where only 3 prompts were given in Template, some questions should arise from the user and inspection of code snippet ran should occur. But yet again, nothing will be perfect. Just improved.

@marmarek @Demi @Sven @fsflover @enmus @deeplow

Please help me improve this PoC. (tar.gz archive, instructions below)

  • This is the blocking version of what was posted above, but archived for convenience of testing. (Non-Networked Template will prevent automatic download and tell the user to replace wget/curl to wget-proxy/curl-proxy manually from command line).

  • The non-blocking curl-wrapper example (translating curl to curl-proxy calls automatically on non-networked TemplateVM) is here.

Was created by:

user@debian-11:/$ sudo tar zcvf wget-curl-wrappers-config-changes-debian.tar.gz /usr/bin/curl-proxy /usr/bin/curl-wrapper /usr/bin/wget-wrapper /usr/bin/wget-proxy /etc/profile.d/download-wrappers-aliases.sh /etc/bash.bashrc 
tar: Removing leading `/' from member names
/usr/bin/curl-proxy
tar: Removing leading `/' from hard link targets
/usr/bin/curl-wrapper
/usr/bin/wget-wrapper
/usr/bin/wget-proxy
/etc/profile.d/download-wrappers-aliases.sh
/etc/bash.bashrc

So you can inspect code by downloading/extracting it locally in any qube directory.
You can test it by cloning debian-11 to debian-11-proxytest or whatever, and then copying the downloaded archive to your cloned debian-11-proxytest TemplateVM (qvm-copy from command line or righ-clicking in Files application from the qube that downloaded the archive):

cd /
sha256sum ~/QubesIncoming/NameOfOriginQube/wget-curl-wrappers-config-changes-debian.tar.gz
397c6e3e8e792a4fcb48c7401b575dab30e51afb58eea4f7ee350a62a7aa2fb4  ~/QubesIncoming/NameOfOriginQube/wget-curl-wrappers-config-changes-debian.tar.gz
sudo tar zxvf ~/QubesIncoming/NameOfOriginQube/wget-curl-wrappers-config-changes-debian.tar.gz

Reboot Template (or just open a new terminal.)
/etc/bash.bashrc will reload /etc/profie.d/*.sh and the aliases should be there already working so that curl/wget are replaced by curl-wrapper/wget-wrapper, which you can confirm by typing alias in the terminal.

Then try calling curl and wget to download stuff:

  • TemplateVM without assigned network (default)
  • TemplateVM with assigned network (core instructions)
  • from created qubes depending on cloned template.

Please report experience/frustrations/desired improvements: what you would like to be the Qubes default TemplateVM behavior dealing with command line downloaders and why.