The firewall documentation says:
“Qubes does not support running any networking services (e.g. VPN, local DNS server, IPS, …) directly in a qube that is used to run the Qubes firewall service (usually sys-firewall) for good reasons. In particular, if you want to ensure proper functioning of the Qubes firewall, you should not tinker with iptables or nftables rules in such qubes.”
which sounds confusing because:
sys-firewall is not a service, it is a VM- the “good reasons” have not been explicitly clarified
- it seems impossible “not to tinker” with iptables or nftables rules if one wants to configure a qube running a DNS because those rules are necessary for proper packet routing
IOW, even if we deploy the network infrastructure proposed in the doc:
sys-net <--> sys-firewall-1 <--> network service qube <--> sys-firewall-2 <--> [client qubes]
we still need firewall rules in the network service qube (which will run the dnscrypt-proxy service). Does this remove the need for sys-firewall-2 which only direct DNS traffic to the network service qube? The doc says:
"The sys-firewall-2 proxy ensures that:
- Firewall changes done in the network service qube cannot render the Qubes firewall ineffective.
- Changes to the Qubes firewall by the Qubes maintainers cannot lead to unwanted information leakage in combination with user rules deployed in the network service qube.
- A compromise of the network service qube does not compromise the Qubes firewall."
Re. 1: Even without sys-firewall-2, the Qubes firewall (sys-firewall-1) is still separate from the network service qube. So, it is not clear how exactly sys-firewall-2 ensures anything in regards to that.
Re. 2: That highly depends on the actual changes and the actual user rules. Example: Suppose the developers (deliberately or by mistake) switch the default policy of the FORWARD chain from DROP to ACCEPT. Then maybe the network service qube (not having any firewall rules, as both advised and impossible) can forward traffic through sys-firewall-1 and sys-firewall-2 can do nothing about it.
Re. 3: Just like in 1, sys-firewall-2 has nothing to do with that.
Regardless of the above confusion in documentation, one can assume that “good reasons” means improved security through additional isolation of firewall stuff from DNS. Having a second firewall between the network service qube and the client cubes can reduce the possibility of leakage from client qubes to the Internet. It also creates another possibility (more on that below).
The other confusion is that Qubes OS still uses the legacy iptables, making us dependent on it through the package qubes-core-agent-networking. To make things even more complicated, Qubes OS mixes that with nftables, making the whole thing very difficult to understand and manage. In my trials, I found it sufficient to use only iptables rules, as explained below.
So, the goal is to deploy the following network infrastructure:
[network uplink]
└── sys-net
└── sys-firewall
├── sys-dns
│ └── sys-wall
│ ├── qube-1
│ ├── qube-2
│ ├── [...]
│ └── qube-n
└── sys-whonix
└── [anonymized-qubes]
Preparation
Install fedora-37-minimal template and update it.
In dom0:
sudo qubes-dom0-update qubes-template-fedora-37-minimal
sudo qubesctl --show-output --skip-dom0 --targets fedora-37-minimal state.sls update.qubes-vm
Create a minimal disposable sys-dns qube:
Preserve the original template as an untouched starting point for other qubes.
In dom0:
qvm-shutdown fedora-37-minimal
qvm-clone fedora-37-minimal f37-m-net
Install software in the cloned template
In dom0:
qvm-run -u root f37-m-net xterm
As per docs, qubes-core-agent-networking seems necessary for anything network related:
In f37-m-net:
dnf install qubes-core-agent-networking dnscrypt-proxy vim-minimal
systemctl disable dnscrypt-proxy
Customize files in /etc/dnscrypt-proxy.
Create user and group, so the service does not run as root:
groupadd --system dnscrypt
useradd --system --home /run/dnscrypt-proxy --shell /bin/false --gid dnscrypt dnscrypt
usermod --lock dnscrypt
This directory should be the same as the one used in subsections of section [sources] in /etc/dnscrypt-proxy/dnscrypt-proxy.toml. I am using a subdir of /run in order to hopefully have cache in RAM (/run is a tmpfs mount):
mkdir -p /run/dnscrypt-proxy
Set proper ownership and permissions:
chown dnscrypt:dnscrypt /run/dnscrypt-proxy
chmod go-rwx /run/dnscrypt-proxy
chown -R dnscrypt:dnscrypt /etc/dnscrypt-proxy
chmod -R go-rwx /etc/dnscrypt-proxy
Use the same user in /etc/dnscrypt-proxy/dnscrypt-proxy.toml:
user_name = 'dnscrypt'
Create a disposable DNS template:
In dom0:
qvm-shutdown f37-m-net
qvm-create -C AppVM --template f37-m-net --label red f37-m-dns-dvm
qvm-prefs f37-m-dns-dvm template_for_dispvms True
qvm-create -C DispVM --template f37-m-dns-dvm --label orange sys-dns
qvm-run -u root f37-m-dns-dvm xterm
In f37-m-dns-dvm move the config files to /rw dir to make it specific to the disposable qube only:
mv /etc/dnscrypt-proxy /rw/
In /rw/config/rc.local:
#!/bin/sh
# This script will be executed at every VM startup, you can place your own
# custom commands here. This includes overriding some configuration in /etc,
# starting services etc.
ipt='/usr/sbin/iptables'
# allow redirects to localhost
/usr/sbin/sysctl -w net.ipv4.conf.all.route_localnet=1
"${ipt}" -I INPUT -i vif+ -p tcp --dport 53 -d 127.0.0.1 -j ACCEPT
"${ipt}" -I INPUT -i vif+ -p udp --dport 53 -d 127.0.0.1 -j ACCEPT
# block connections to other DNS servers
"${ipt}" -I FORWARD -i vif+ -p tcp --dport 53 ! -d 127.0.0.1 -j DROP
"${ipt}" -I FORWARD -i vif+ -p udp --dport 53 ! -d 127.0.0.1 -j DROP
"${ipt}" -t nat -F PR-QBS
"${ipt}" -t nat -A PR-QBS -p udp --dport 53 -j DNAT --to-destination 127.0.0.1
"${ipt}" -t nat -A PR-QBS -p tcp --dport 53 -j DNAT --to-destination 127.0.0.1
echo 'nameserver 127.0.0.1' > /etc/resolv.conf
# https://github.com/DNSCrypt/dnscrypt-proxy/wiki/Installation-linux
# https://wiki.archlinux.org/title/Dnscrypt-proxy#Enable_EDNS0
echo 'options edns0' >> /etc/resolv.conf
ln -s /rw/dnscrypt-proxy /etc/dnscrypt-proxy
/usr/bin/systemctl start dnscrypt-proxy.service
In dom0:
qvm-run -u root f37-m-net xterm
In f37-m-net:
rm -rf /etc/dnscrypt-proxy
Test if the service works:
In dom0:
qvm-shutdown f37-m-dns-dvm f37-m-net
qvm-run -u root sys-dns xterm
In sys-dns:
systemctl status dnscrypt-proxy
The above should show that the service is active and running.
Optional (for more details):
systemctl restart dnscrypt-proxy; journalctl --output=short-monotonic -f -u dnscrypt-proxy
after completion of the start process the journal shows something like:
[...]
[ 5611.615623] sys-dns dnscrypt-proxy[2951]: [2023-01-22 21:50:02] [NOTICE] - 519ms dnswarden-uncensor-dc-swiss
[ 5611.615696] sys-dns dnscrypt-proxy[2951]: [2023-01-22 21:50:02] [NOTICE] - 584ms pryv8boi
[ 5611.615780] sys-dns dnscrypt-proxy[2951]: [2023-01-22 21:50:02] [NOTICE] - 760ms altername
[ 5611.615845] sys-dns dnscrypt-proxy[2951]: [2023-01-22 21:50:02] [NOTICE] Server with the lowest initial latency: scaleway-ams (rtt: 89ms)
[ 5611.615944] sys-dns dnscrypt-proxy[2951]: [2023-01-22 21:50:02] [NOTICE] dnscrypt-proxy is ready - live servers: 33
List processes running as user dnscrypt:
In sys-dns:
ps -U dnscrypt -u dnscrypt u
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
dnscrypt 2951 0.2 2.5 1301200 100776 ? Ssl 21:49 0:01 /usr/bin/dnscrypt-proxy -config /etc/dnscrypt-proxy/dnscrypt-proxy.toml -child
Create and configure the new minimal disposable firewall sys-wall:
We have already installed qubes-core-agent-networking in f37-m-net, so we simply use that template.
qvm-create -C AppVM --template f37-m-net --label red f37-m-firewall-dvm
qvm-prefs f37-m-firewall-dvm template_for_dispvms True
qvm-create -C DispVM --template f37-m-firewall-dvm --label green sys-wall
Configure the network structure based on the initial diagram.
In dom0:
qvm-prefs sys-dns netvm sys-firewall
qvm-prefs sys-dns autostart true
qvm-prefs sys-dns provides_network true
qvm-prefs sys-wall netvm sys-dns
qvm-prefs sys-wall provides_network true
Configure the firewall rules:
In dom0:
qvm-run -u root f37-m-firewall-dvm xterm
In f37-m-firewall-dvm edit /rw/config/rc.local
#!/bin/sh
# This script will be executed at every VM startup, you can place your own
# custom commands here. This includes overriding some configuration in /etc,
# starting services etc.
ipt='/usr/sbin/iptables'
# redirect all dns-requests to sys-dns
"${ipt}" -t nat -F PR-QBS
"${ipt}" -t nat -A PR-QBS -d 10.139.1.1/32 -p udp --dport 53 -j DNAT --to-destination 10.138.26.87
"${ipt}" -t nat -A PR-QBS -d 10.139.1.1/32 -p tcp --dport 53 -j DNAT --to-destination 10.138.26.87
"${ipt}" -t nat -A PR-QBS -d 10.139.1.2/32 -p udp --dport 53 -j DNAT --to-destination 10.138.26.87
"${ipt}" -t nat -A PR-QBS -d 10.139.1.2/32 -p tcp --dport 53 -j DNAT --to-destination 10.138.26.87
# block connections to other DNS servers
"${ipt}" -t nat -A PR-QBS -p udp --dport 53 -j DNAT --to-destination 0.0.0.0
"${ipt}" -t nat -A PR-QBS -p tcp --dport 53 -j DNAT --to-destination 0.0.0.0
Set sys-wall as NetVM for a qube and test in a terminal in that qube:
[user@disp1463 ~]$ host gnu.org
gnu.org has address 209.51.188.116
gnu.org has IPv6 address 2001:470:142:5::116
gnu.org mail is handled by 10 eggs.gnu.org.
[user@disp1463 ~]$ host google-analytics.com
google-analytics.com host information "This query has been locally blocked" "by dnscrypt-proxy"
google-analytics.com host information "This query has been locally blocked" "by dnscrypt-proxy"
google-analytics.com host information "This query has been locally blocked" "by dnscrypt-proxy"
That’s because my blocked_names_file has a line blocking the last host. This confirms the configuration is working.
Test if it is possible to circumvent the blockage and use another DNS server (8.8.8.8):
[user@disp1463 ~]$ host google-analytics.com 8.8.8.8
;; connection timed out; no servers could be reached
It seems sys-wall woks as expected too.
Finally, set sys-wall as NetVM for all qubes which should use DNScrypt.
TODO:
-
DNScrypt-proxy allows name (un)blocking. Using an dedicated combination of sys-dns and sys-wall for a qube it is possible to block everything and allow only one domain (or a set of them), e.g. *.mybank.com. With proper scripting and UI, this may be a possible solution to a problem discussed long time ago.
-
Another thing is getting rid of iptables and using nftables. This seems something developers need to do in relation to qubes-core-agent-networking. Has it been reported and/or considered?
-
Ideally, dnscrypt-proxy should be integrated in Qubes OS out of the box.
Comments, suggestions and corrections are very welcome. I don’t pretend to be an expert, just sharing what worked for me.
P.S. Sorry for the huge delay. I had some troubles and this whole thing took longer than expected. Still, better late than never, I hope.