Curl-proxy / wget-proxy scripts in Templates so users can add GPG distro keys linked to added external repositories

@Insurgo I am not trying to be dense and I admit I haven’t read the entire thread – so ignore me if that’s over the line.

But now I have the impression what we are really talking about is to create Qubes OS specific scripts called ‘curl-proxy’ and ‘wget-proxy’ and deploy them to all templates so that we can then document somewhere:

If you see ‘curl’ or ‘wget’ in an install script, replace them with ‘curl-proxy’ and ‘wget-proxy’

… so you, I and other can in future just point to that documentation page and say “do this”.

But then, we could simply document:

In case of ‘curl’ add the ‘–proxy http://127.0.0.1:8082/’ parameter and in case of ‘wget’ add ‘-e use_proxy=yes -e http_proxy=127.0.0.1:8082’

Same effect, no changes in Qubes OS required. Alternatively we could educate users about ~/.wgetrc and
~/.curlrc and provide examples that already include the required proxy config. Yes?

In the case of Signal, Element, and other widely used applications, I plan to include the public keys in a package that is installed by default. This means that the “download the public key” step is not only unnecessary, it’s actively harmful: the copy in the OS is trustworthy, while the one downloaded might not be (if someone fooled a CA into giving them a rogue certificate, say).

More generally, I think @adw is much better than I am explaining these things than I am :slight_smile:. This post of his is right on the money.

I see your point @demi. However, I don’t think in this particular case it will help the situation. Users won’t look for how to specifically install software X in Qubes. Rather they will look for upstream instructions (e.g. from signal’s website). So they will inevitable try to run the wget command to obtain the gpg. They’ll do all sort of things in a desperate attempt to get it to work.

In your solution this requires prior knowledge of what a GPG key is and that it’s available in the keyrings. But most new users will not know that. However, for more advanced users equiped with that knowledge it will be a nice to have.

On the other hand in @Insurgo’s solution the user will type the commands and see an error indicating a clear solution (replacing wget with wget-proxy) – something which hands the solution to the user instead of waiting for them to go look for it.

1 Like

Here comes the PoC.

Config changes

/etc/profile.d/download-wrapper-aliases.sh

alias curl="curl-wrapper"
alias wget="wget-wrapper"
# sudo will not use aliases unless https://linuxhandbook.com/run-alias-as-sudo/
alias sudo="sudo "

Append the following under /etc/bash.bashrc

#/etc/profile.d scripts are not loaded in interactive non-login shells (https://linuxhandbook.com/run-alias-as-sudo/)
if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i
    fi
  done
  unset i
fi

Scripts

/usr/bin/wget-wrapper

#!/bin/env bash
# wget-wrapper script around wget
#  Goal is to give a chance to the user of knowing what is going to happen next, educate and explain what is different in Templates
#
# This wrapper script detects if the wget command was ran in a TemplateVM (Which by default is prohibited)
#   In case the Template is trusted (No Network assigned), it warns the user to call wget-proxy instead and exits
#   In case the Template has network assigned, it warns the user the template should not be trusted anymore and proceeds in 3 seconds
#   In case we are not in a template, just add a warning and proceed with original wget call
# 
#Depends on : 
# this file being deployed in OS searchable path (eg: /usr/bin)
# alias being previously defined on the system prior of running wget. 
#   To test this:
#     alias wget="wget-wrapper"
#   To permanently define the alias for a user, add the alias in ~/.bashrc


if [ "$(qubesdb-read /type)" == "TemplateVM" ]; then 
	if ! $(qubesdb-read /qubes-gateway > /dev/null 2>&1); then 
		echo "You attempted to download a file with wget in a Template without direct internet access. (no network assigned)." >&2
		echo "" >&2
		echo "The attempted downloading command was: wget $@" >&2
		echo "" >&2
		echo "You either:" >&2 
		echo " 1- need to replace wget by wget-proxy in previous attempted shell command line (Recommended)." >&2
		echo " 2- assign a Networking qube to this Template which should consequently be less trusted. (Untrusted)." >&2
		exit 1
	else
		echo "You attempted to download a file with wget in a Template WITH DIRECT INTERNET ACCESS. (network assigned)." >&2
		echo "" >&2
		echo "The attempted downloading command was: wget $@" >&2
		echo "" >&2
		echo "Continuing in 3 seconds... Type CTRL+C to prevent the download now!" >&2
		sleep 3
		/usr/bin/wget "$@"
	fi
else
	echo "wget download attempt not in TemplateVM. Continuing..." >&2
	/usr/bin/wget "$@"
fi

/usr/bin/curl-wrapper

#!/bin/env bash
# curl-wrapper script around curl
#  Goal is to give a chance to the user of knowing what is going to happen next, educate and explain what is different in Templates
#
# This wrapper script detects if the curl command was ran in a TemplateVM (Which by default is prohibited)
#   In case the Template is trusted (No Network assigned), it warns the user to call curl-proxy instead and exits
#   In case the Template has network assigned, it warns the user the template should not be trusted anymore and proceeds in 3 seconds
#   In case we are not in a template, just add a warning and proceed with original curl call
# 
#Depends on : 
# this file being deployed in OS searchable path (eg: /usr/bin)
# alias being previously defined on the system prior of running curl. 
#   To test this:
#     alias curl="curl-wrapper"
#   To permanently define the alias for a user, add the alias in ~/.bashrc


if [ "$(qubesdb-read /type)" == "TemplateVM" ]; then 
	if ! $(qubesdb-read /qubes-gateway > /dev/null 2>&1); then 
		echo "You attempted to download a file with curl in a Template without direct internet access. (no network assigned)." >&2
		echo "" >&2
		echo "The attempted downloading command was: curl $@" >&2
		echo "" >&2
		echo "You either:" >&2 
		echo " 1- need to replace curl by curl-proxy in previous attempted shell command line (Recommended)." >&2
		echo " 2- assign a Networking qube to this Template which should consequently be less trusted. (Untrusted)." >&2
		exit 1
	else
		echo "You attempted to download a file with curl in a Template WITH DIRECT INTERNET ACCESS. (network assigned)." >&2
		echo "" >&2
		echo "The attempted downloading command was: curl $@" >&2
		echo "" >&2
		echo "Continuing in 3 seconds... Type CTRL+C to prevent the download now!" >&2
		sleep 3
		/usr/bin/curl "$@"
	fi
else
	echo "curl download attempt not in TemplateVM. Continuing..." >&2
	/usr/bin/curl "$@"
fi

/usr/bin/curl-proxy

#!/bin/env bash
curl --proxy http://127.0.0.1:8082/ --tlsv1.2 --proto =https --max-time 180 $@

/usr/bin/wget-proxy

#!/bin/env bash
https_proxy=http://127.0.0.1:8082/ http_proxy=http://127.0.0.1:8082/ wget --secure-protocol=TLSv1_2 --timeout=180 "$@"

Testing instructions

  1. Clone Template
  2. Deploy scripts
  3. Apply system wide changes above related to aliases (Should probably file a bug report @marmarek ?)
  4. Reproduce results
  5. Assign network to Template. Reproduce results
  6. Create qube based on previous cloned template. Reproduce results.

No network assigned to cloned Template use case:

user@debian-11:~$ alias
alias curl='curl-wrapper'
alias ls='ls --color=auto'
alias sudo='sudo '
alias wget='wget-wrapper'

Non-Networked Template: gpg key download of signal from user

user@debian-11:~$ wget -O- https://updates.signal.org/desktop/apt/keys.asc 
You attempted to download a file with wget in a Template without direct internet access. (no network assigned).

The attempted downloading command was: wget -O- https://updates.signal.org/desktop/apt/keys.asc

You either:
 1- need to replace wget by wget-proxy in previous attempted shell command line (Recommended).
 2- assign a Networking qube to this Template which should consequently be less trusted. (Untrusted).

user@debian-11:~$ wget-proxy -O- https://updates.signal.org/desktop/apt/keys.asc | gpg --dearmor > signal-desktop-keyring.gpg
--2022-04-21 10:52:38--  https://updates.signal.org/desktop/apt/keys.asc
Connecting to 127.0.0.1:8082... connected.
Proxy request sent, awaiting response... 200 OK
Length: 3090 (3.0K) [application/pgp-signature]
Saving to: ‘STDOUT’

-                                     100%[=======================================================================>]   3.02K  18.1KB/s    in 0.2s    

2022-04-21 10:52:41 (18.1 KB/s) - written to stdout [3090/3090]

Non-Networked Template: gpg key download of session (sudo curl call per instructions) use case:

user@debian-11:~$ alias
alias curl='curl-wrapper'
alias ls='ls --color=auto'
alias sudo='sudo '
alias wget='wget-wrapper'

user@debian-11:~$ sudo curl -so /etc/apt/trusted.gpg.d/oxen.gpg https://deb.oxen.io/pub.gpg
You attempted to download a file with curl in a Template without direct internet access. (no network assigned).

The attempted downloading command was: curl /etc/apt/trusted.gpg.d/oxen.gpg https://deb.oxen.io/pub.gpg

You either:
 1- need to replace curl by curl-proxy in previous attempted shell command line (Recommended).
 2- assign a Networking qube to this Template which should consequently be less trusted. (Untrusted).

user@debian-11:~$ sudo curl-proxy -so /etc/apt/trusted.gpg.d/oxen.gpg https://deb.oxen.io/pub.gpg

user@debian-11:~$ ls -al /etc/apt/trusted.gpg.d/oxen.gpg
-rw-r--r-- 1 root root 2213 Apr 21 11:58 /etc/apt/trusted.gpg.d/oxen.gpg

Assigned network to Template use case:

Networked Template: Signal wget use case (normal user)

user@debian-11-networked:~$ wget -O- https://updates.signal.org/desktop/apt/keys.asc | gpg --dearmor > signal-desktop-keyring.gpg
You attempted to download a file with wget in a Template WITH DIRECT INTERNET ACCESS. (network assigned).

The attempted downloading command was: wget -O- https://updates.signal.org/desktop/apt/keys.asc

Continuing in 3 seconds... Type CTRL+C to prevent the download now!
--2022-04-21 12:01:23--  https://updates.signal.org/desktop/apt/keys.asc
Resolving updates.signal.org (updates.signal.org)... 172.64.155.131, 104.18.32.125, 2606:4700:4400::6812:207d, ...
Connecting to updates.signal.org (updates.signal.org)|172.64.155.131|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3090 (3.0K) [application/pgp-signature]
Saving to: ‘STDOUT’

-                   100%[===================>]   3.02K  --.-KB/s    in 0s      

2022-04-21 12:01:23 (13.0 MB/s) - written to stdout [3090/3090]

Networked Template: Session sudo curl use case (runs as root)

user@debian-11-networked:~$ sudo curl -so /etc/apt/trusted.gpg.d/oxen.gpg https://deb.oxen.io/pub.gpg
You attempted to download a file with curl in a Template WITH DIRECT INTERNET ACCESS. (network assigned).

The attempted downloading command was: curl -so /etc/apt/trusted.gpg.d/oxen.gpg https://deb.oxen.io/pub.gpg

Continuing in 3 seconds... Type CTRL+C to prevent the download now!

user@debian-11-networked:~$ ls -al /etc/apt/trusted.gpg.d/oxen.gpg
-rw-r--r-- 1 root root 2213 Apr 21 12:02 /etc/apt/trusted.gpg.d/oxen.gpg

curl and wget direct calls in Networked Templates

user@debian-11-networked:~$ wget
You attempted to download a file with wget in a Template WITH DIRECT INTERNET ACCESS. (network assigned).

The attempted downloading command was: wget 

Continuing in 3 seconds... Type CTRL+C to prevent the download now!
wget: missing URL
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.

user@debian-11-networked:~$ curl
You attempted to download a file with curl in a Template WITH DIRECT INTERNET ACCESS. (network assigned).

The attempted downloading command was: curl 

Continuing in 3 seconds... Type CTRL+C to prevent the download now!
curl: try 'curl --help' or 'curl --manual' for more information

Normal Qube depending on that cloned template use case

user@wrapper-test:~$ wget
wget download attempt not in TemplateVM. Continuing...
wget: missing URL
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.

user@wrapper-test:~$ curl
curl download attempt not in TemplateVM. Continuing...
curl: try 'curl --help' or 'curl --manual' for more information

Notes

@marmarek @adw @Sven @fsflover @Demi @deeplow : Am I making my point?
@Demi @marmarek: Fell upon two interesting bugs

  1. sudo cannot invoque aliases unless alias sudo="sudo " per How to Run an Alias With Sudo in Linux
  2. /etc/profile.d/* are ignored when using graphical shells (interactive, non-login shells: Why are scripts in /etc/profile.d/ being ignored (system-wide bash aliases)? )

This is why /etc/bash.bashrc was modified to also source /etc/profile.d/*.sh.
@marmarek @demi @fepitre : Qubes Bug/Feature?

Edit: Sorry for all the edits. I’m done now.
Edit2. I’m done for real, sorry. This is why writing documentation is so hard.

2 Likes

As you can see in my PoC

  1. Some upstream instructions (See Session example) asks the user to call curl with sudo. (wait what? sudo? curl? gpg? I just want to use session in 5 minutes, damnit. I’m already late.)
    1- sudo calls would require also the user to apply some changes on that user as well, otherwise commands will fail. And no, I do not believe having the use pass additional command line options is a good way to deal with the problem (adds more problems.)
  2. As @deeplow said, when things do not work as expected, non-teksavvy users will do extended stupid shit. True story. Sometimes to the point of having to reinstall (doing things in dom0 is the next stupid step a lot of users will do, taking snippets found on the internet…)
  3. If you add that to stress (I’m not even sure my camera will work in Signal… Damnit, already 5 minutes late to the meeting…) users even do more extended stupid shit. This is not a good time to neither ask them to find best practices or ask them to even memorize/ask them to type the proxy extensions to their wget and curl calls.
  4. This is a time to tell them what to safely do. Now. And remind them, at the same time, that if they are in a cloned Template (good, at least!) with wide internet access (meh, but better then in the main trusted Template @adw), that that Template should now be untrusted to use widely in other qubes depending on it (as the core documentation now reflects. Thanks!)

I hope that with the console output given in previous PoC post, I was able to prove my point and the pertinence of this massive UX improvement for Qubes OS (@marmata?). Of course, take it, modify it, improve it. But please consider the need and effectiveness of such solution.

Otherwise, as @deeplow said and I cannot agree more:

Missing piece:

  • tinyproxy logs should at least be kept in memory and a return on past actions should be possible to at least audit what urls were accessed by that update proxy for TemplateVMs using it.

An alternative idea to the alias: set https_proxy=http://127.0.0.1:8082 env variable. This will make curl/wget work automatically, in this terminal only. No need to change anything in upstream installation instruction then. The steps are then:

  1. Open terminal in the template
  2. Enter export https_proxy=http://127.0.0.1:8082
  3. Follow upstream instructions

The command is easy to enter wrong, but we could have an alias like enable-network-proxy. I think that’s significantly simpler than wrapping curl, wget and whatnot. And also, should allow only https traffic, but not plain http (at least in theory).

5 Likes

But, if users have to enter this, how will they know they need to enter this?

2 Likes

I was curious to trying out the various methods mentioned above so I made a deb-signal template only to find out none seem to work:

wget:

$ export https_proxy=http://127.0.0.1:8082
$ wget -O- https://updates.signal.org/desktop/apt/keys.asc
Connecting to 127.0.0.1:8082... connected.
Proxy tunneling failed: CONNECT denied (ask the admin to allow HTTPS tunnels)Unable to establish SSL connection.

curl:

$ curl --proxy http://127.0.0.1:8082/ --tlsv1.2 --proto =https --max-time 180 "https://updates.signal.org/desktop/apt/keys.asc"
curl: (56) Received HTTP code 403 from proxy after CONNECT

I then noticed that when launching the command, my apt-cacher qube was started and I found the following command to be working:

curl --proxy http://127.0.0.1:8082/ --tlsv1.2 --max-time 180 "http://HTTPS///updates.signal.org/desktop/apt/keys.asc"

So I had to remove --proto =https and change https:// to http://HTTPS///.

Is this behavior to be expected when not dealing with repositories but an apt-cacher qube is enabled?

It’s offtopic asking that here.

@marmarek :

@adw @marmarek @sven @Demi @fsflover

If you prefer not interrupting user, but warning him that he is about to download stuff on the internet from a non-networked Template, that section could be changed to simply call the -proxy (here curl-proxy as an exemple) instead of exiting and asking the user to change curl to curl-proxy manually…:

if [ "$(qubesdb-read /type)" == "TemplateVM" ]; then 
	if ! $(qubesdb-read /qubes-gateway > /dev/null 2>&1); then 
		echo "You attempted to download a file with curl in a Template without direct internet access. (no network assigned)." >&2
		echo "" >&2
		echo "The attempted downloading command was: curl $@." > &2 
		echo "" >&2
		echo "Automatically translating to: curl-proxy $@" >&2
		echo "" >&2
		echo "Continuing in 3 seconds... Type CTRL+C to prevent the download now!" >&2
		sleep 3
		/usr/bin/curl-proxy "$@"
	else
		echo "You attempted to download a file with curl in a Template WITH DIRECT INTERNET ACCESS. (network assigned)." >&2
		echo "" >&2
		echo "The attempted downloading command was: curl $@" >&2
		echo "" >&2
		echo "Continuing in 3 seconds... Type CTRL+C to prevent the download now!" >&2
		sleep 3
		/usr/bin/curl "$@"
	fi
else
	echo "curl download attempt not in TemplateVM. Continuing..." >&2
	/usr/bin/curl "$@"
fi

The resulting behavior would become similar to a Template with network access configured externally in Qubes, while using the proxy through curl-proxy translated call automatically (snippet):

The attempted downloading command was: curl -so /etc/apt/trusted.gpg.d/oxen.gpg https://deb.oxen.io/pub.gpg

Automatically translating to: curl-proxy -so /etc/apt/trusted.gpg.d/oxen.gpg https://deb.oxen.io/pub.gpg

Continuing in 3 seconds... Type CTRL+C to prevent the download now!

@deeplow @marmarek :
This seems like the only way to stop/warn/make aware/prevent users from running a whole snippet of code borrowed from the internet which would include curl/wget commands from actually doing harm and render otherwise trusted template into untrusted template, effectively limiting what would be downloaded before it happens.

The code snippet copy pasted in shell is going to download 3 files? The user receives 3 warnings! Other more obscure code snippets using something else then curl/wget will just not get access to the internet, which I think is the best tradeoff, not requiring to widen proxy access to the whole template nor giving global network access externally to the template.

Otherwise enticing the user to open the door wide open by either telling the whole template to use proxy / inviting the user to give internet globally to the Template per Qubes core documentation.

Yes, apt-cacher-ng is replacing the tinyproxy and listening in it’s place at 8082. You had to actually configure your apt-cacher in it’s config file to use that port. :wink:

1 Like

He already has.

@adw For some reason, I feel you are the person to be convinced of the proposed approach.

@adw to me this is a balance movement, which maybe went too far one way and then too far the other. @marmarek proposed to implement exactly the 4th step of your full circle picture you posted earlier: open up the non-networked Template. By an alias, or by documentation. This approach gives the exact same result as assigning network to a Template externally, and wil lbe as confusing for end users. Please take a minute and thing more broadly. We can fix this.

Please take 1 minute and look at the output of the PoC.

If you do not like the fact that non-networked Templates (default) were requiring the user to manually replace wget/curl with wget-proxy/curl-proxy on the command line, I suggested a non-blocking version translating curl/wget to curl-proxy/wget-proxy calls automatically, pausing for 3 seconds prior of complying with user request so the user can interrupt the request. That would be the safer default approach, permitting to educate/raise awareness/not block standard calls/follow upstream instructions without customization.

The idea here again being to give safer by defaults options, teaching users what are Qubes differences (non-obstructively), while not having binary allow all/block everything approach by default.

Please let me know your constructive thoughts, remembering that users that know what they are doing will go as @sven. While people not knowing what they are doing will open up everything. We need middle ground, and permitting wget/curl (which most software installation methods talk about) are covered by PoC.

Exporting a variable enables network access just in this terminal (and apps started from there), and only until it’s closed. That’s a major advantage over connecting netvm, while simpler than wrappers. But see below.

Wrappers that educate the user are indeed great idea. I think this could be the main reason to use wrappers instead of just env variable.

But also, I think outside of TemplateVM, those should not be set at all (not setting aliases, not printing anything extra etc) - the risk of breaking normal use cases is too great.

2 Likes

@marmarek:

What is not covered ?

The normal qubes use case uses the same wrappers.

The if [ "$(qubesdb-read /type)" == "TemplateVM" ]; then should be when setting aliases, instead of within the wrappers.

Not sure I follow, while just realizing that links to special anchors related to post headers are not taken into consideration so all my past links pointing to different parts of my PoC are all linking to the main “# header” not “## subheaders”. Sigh.

That wrapper would be available in Template and then in qubes. The wrapper is consequently called the same way per aliases defined. And the same code is being used, changing behavior if in TemplateVM, and then if in a networked TemplateVM specifically, or otherwise in a qube (not being a TemplateVM). Is there something I haven’t tested correctly @marmarek?

Quoting again, to be sure you saw output:

Then:

And finally, qubes:

While I proposed an alternative non-blocking solution for non-networked Template to use directly curl-proxy/wget-proxy, after pausing for 3 seconds (so the user can interrupt calls done in random ran scripts actually trying to download stuff on the internet with curl/wget):

Which outputs:

Let me know what you would improve @marmarek. Wrapper would improve UX/educate users/prevent most probable disasters, indeed.

With most of the users it always is about “I know what I’m doing”, “It won’t happen to me”, “I have nothing to hide anyway”.

So, the first thing next that will happen would be users asking how to workaround this “annoyance”, as a following of a routine not to read what is echoed after the first couple of successful and carefully implemented attempts/imports?