Firewall VM with Custom Rules behind a VPN = No Internet

Hi Everybody,

I am experiencing an issue where an AppVM won’t connect to the internet when its Firewall VM is behing a VPN AND contains enabled custom firewall rules. Here is my setup:

Qubes 4.0.3
Qubes-vpn-support
OpenVPN as VPN provider
WIFI VM > VPN VM > Firewall VM > AppVM
All these VMs are based on Fedora32 minimal templates.
Rules use domain names instead of IPs

Two observations:

  1. AppVM connects without issues and applies Firewall VM rules when bypassing the VPN VM to connect: WIFI VM > Firewall VM > AppVM.

  2. WIFI VM > VPN VM > Firewall VM > AppVM connects without issues if the Firewall VM rules are disabled or deleted.

I’ve tested this behavior with
smtp, pop3, pop3s, imap, imaps, http, https with UDP, TCP and Any
I’ve used there rules for specific websites and for all traffic under the given port/protocol. Behavior is the same in all scenarios.

I know this probably has to do with the iptables and the firewall lock down state the VPN package sets up. However, I have no Idea on how to relate iptables to actual firewall rules.

Any assistance would be much appreciated. Thank you.

I also have this issue, the same Qubes 4.0.3.

But why do you replace sys-net with wifi VM ?

How about VPN VM between firewall and app VM ?

why do you put VPN VM between wifi and firewall VM ?

Hi tava

This is because my system doesn’t have an Ethernet adapter

I tested with that setup and obtained the same results

I presume you mean as opposed to putting it between the Firewall and App VMs? If so, this is because it allows for having different AppVMs connected to different firewalls while allowing all of them to be behind one VPN VM

Looking forward to learn whether I am missing something obvious or dig deeper into what could be the problem. Thanks again

Hi again everyone,

I have made some more testing after updates and I made some disturbing findings

First, observation 2 from my post above is no longer true. Here are the observations after performing maintenance template updates:

  1. WIFI VM > VPN VM > Firewall VM (rules enabled)) > AppVM = no internet (?)
  2. WIFI VM > VPN VM > Firewall VM (rules disabled) > AppVM = no internet (?)
  3. WIFI VM > Firewall VM (rules disabled) > VPN VM > AppVM = Internet to any site
  4. WIFI VM > Firewall VM (rules ENABLED) > VPN VM > AppVM = Internet to ANY site
  5. Removing VPN VM from all of the above produces expected behavior in all cases.

As you can see in none of the cases above FW rules seem to be being applied at all, except for case 5.

More alarming however, is case 4. Why are FW rules being completely ignored in this case?

I have read all posts resulting from a “VPN” search in this forum and the qubes-users google group as well as the VPN-support and vpn tunnel instructions and the Qubes’ documentation Firewall and VPN pages and I can still not figure out the reason for this behavior. I cannot find an explanation of why swapping the order between VPN and FW VMs is affecting internet connectivity in this way.

To someone like me, with only basic networking knowledge, Firewall VM and VPN VM combination seem to produce erratic connectivity behavior at best and cannot be trusted to isolate and control traffic properly.

At the moment it seems to me that the effective use of a VPN and Firewall VM combination, if there is such a thing, is restricted to advanced-networking due to the prohibiting complexity involved in setting it up.

Therefore, I kindly ask if someone could assist me with understanding if this is indeed erratic behavior or if I am lacking knowledge and context to understand it.

Thanks in advanced,
SU

I could be completely off-base here, but wouldn’t having the VPN VM before the firewall VM mean that all the traffic is already encapsulated/encrypted (if that’s the right term) by the VPN and thus unable to be processed by the firewall?

So to me, examples 3 & 4 are acting as expected. I would have expected 1 & 2 to work… but obviously they aren’t. No very helpful I know :frowning:.

I could be completely off-base here, but wouldn’t having the VPN VM before the firewall VM mean that all the traffic is already encapsulated/encrypted (if that’s the right term) by the VPN and thus unable to be processed by the firewall?

Yes - the traffic at the firewall is encrypted and heading for the VPN
servers, so “firewall rules” relevant to the appVM are not triggered. In
fact the firewall sees NO traffic from the appVM

So to me, examples 3 & 4 are acting as expected. I would have expected 1 & 2 to work… but obviously they aren’t. No very helpful I know :frowning:.

need more detail to comment on 1&2 - what are "custom firewall rules,
how implemented? Please post detail AND detail on your openvpn config.

Hi Rooftop and unman,

Thanks for the reply. @unman just a follow up question before addressing yours.

Wouldn’t in these cases (3 & 4) the FirewallVM act as a firewall for the VPN VM? Meaning that: if the rules in FirewallVM do not include provisions for the VPN servers, then traffic out of the VPN VM should be blocked when the Firewall VM rules are enabled. Is this assumption not correct?

Now, to your questions. I’ll try to explain it as clearly as I can.

Custom firewall rules:
qvm-firewall <vmname> list
0 accept www.website.com tcp 443 - - - -
1 accept - - - dns - - -
2 accept - icmp - - - - -
3 drop - - - - - - -

Rule 0 I entered via the FirewallVM GUI. Rules 1, 2 and 3 were not entered by me and are not visible in the FirewallVM’s GUI.

Open VPN configuration details:
I followed steps 1 - 4 from https://github.com/tasket/Qubes-vpn-support to install; without any deviations, as I also utilized openvpn as my provider. I performed this procedure in a DisposableVM Template so that the actual VPN VM is a Static (named) DisposableVM.

I tested the link in the DisposableVM Template as instructed between steps 2 & 3 from the Qubes-vpn-support link above. I also tested the stop/restart of qubes-vpn-handler in the VPN VM after install to ensure internet gets cut off when the tunnel goes down. Both tests were successful. The only result that I noticed being different from the expected in the above guide is that I do not get a notification ‘LINK IS UP’ when the VPN VM starts. I however used websites such as “whatsmyip” to determine whether the detected location matched that of my VPN server’s -not sure if this is enough to also determine whether the traffic is being encrypted-. The configuration files themselves I downloaded from the VPN provider’s website and placed them in the DisposableVM Template where instructed without any modification.

Other configurations which may or may not be relevant:

  1. I use “Randomize all Ethernet an Wifi connections” and “Randomize your hostname” configurations from https://github.com/Qubes-Community/Contents/blob/master/docs/privacy/anonymizing-your-mac-address.md in the WIFI VM.
  2. All VMs are Static (named) DisposableVMs, except for the AppVM. The AppVM is an non-static unnamed DisposableVM.

I appreciate your time and knowledge so let me know if there is any other information I can provide

Was www.website.com your VPN provider? Because that would make sense and match some of the results you have been getting. If not, you can ignore all the following conjecture :slight_smile:

If the VPN is operating over port 443 to www.website.com then by opening that port in the firewall you will allow VPN traffic to any site on the internet (same as if the was no firewall - your cases 3 & 4).

Put the firewall before the VPN and you would have no access to any site except www.website.com because it was being blocked by the firewall before it even gets to the VPN (case 1).

No idea about case 2, unless you didn’t fully disable the firewall (maybe you deleted the rule but forgot to select Allow all outgoing internet connections?

That’s my best guess anyway…

Hi Rooftop and unman,

Thanks for the reply. @unman just a follow up question before addressing yours.

Wouldn’t in these cases (3 & 4) the FirewallVM act as a firewall for the VPN VM? Meaning that: if the rules in FirewallVM do not include provisions for the VPN servers, then traffic out of the VPN VM should be blocked when the Firewall VM rules are enabled. Is this assumption not correct?

Yes, in those cases the FirewallVM acts as a firewall for the VPN.
The results would be dependent on how you have configured the firewall.

Now, to your questions. I’ll try to explain it as clearly as I can.

Custom firewall rules:
qvm-firewall <vmname> list
0 accept www.website.com tcp 443 - - - -
1 accept - - - dns - - -
2 accept - icmp - - - - -
3 drop - - - - - - -

Rule 0 I entered via the FirewallVM GUI. Rules 1, 2 and 3 were not entered by me and are not visible in the FirewallVM’s GUI.

Rules 1 and 2 are default Qubes configuration, as explained at the
bottom of the “Firewall rules” tab.
Rule 3 is because you have selected “Limit outgoing connections to…”

So, assuming that “www.website.com”, port 443 is your VPN provider, this
rule will allow VPN traffic. You aren’t restricting traffic from the
client at all.
That means that any traffic passing through the VPN will be allowed.

If you remove rule 0, then the VPN will be closed, and no traffic
will be allowed at all from vpn clients.
But if you disable the firewall (I’m not sure I understand what you
mean by this), so that there are no restrictions on the traffic then
the VPN will work AND the clients will be able to connect to any
host/port.

On the cases in 1 and 2, then this will depend entirely on what rules
you have set for the firewall.
If you want to pursue this, then perhaps you can give some detail on
what rules you have enabled, and what you mean by “rules disabled” in
case 2.

@Rooftop and @unman, thanks again for your continued support.

My meaning of rules enabled / disabled is as follows:

Rules enabled = Selecting radio button option to “Limit outgoing connections to …”
Rules disabled = Selecting radio button option to “Allow all outgoing internet connections”

This was also my understanding and working assumption. Thanks for confirming

However, www.website.com is not the VPN provider’s website. Therefore, I as well would have expected case 4 to follow your logic above and result in no traffic; as none of rules 0-3 present include the VPN provider’s website. This is why I found case 4 so alarming.

The entire extent of my configuration to FirewallVM rules involved:

  1. Adding Rule 0 via the FirewallVM GUI, and
  2. Alternating between the above-described enabled and disabled states in cases 1-5.

The FirewallVM was the same actual qube and Rule 0 remained unmodified by me throughout all of these cases.

With this, I call out what I identify to be the 3 problems:

  1. Case 1: www.website.com is not reachable when “Limit outgoing connections to …” is selected and rule 0 exists.
  2. Case 2: Neither www.website.com nor any other websites are reachable when “Allow all outgoing internet connections” is selected.
  3. Case 4: www.website.com (which is not the VPN server) and all other websites are reachable when “Limit outgoing connections to …” is selected and rule 0 exists.

I suspect cases 1 & 2 share the same root-cause; however I lack knowledge to understand / troubleshoot what that root-causes may be and whether it is related to that from problems observed in case 4. Therefore, I am hoping you could provide some more guidance.

Finally, if it aids in providing more context, my goal is to have the setup described in case 1.

Once this issue is resolved, I would plan to modify the approach and have different AppVMs each connect to their respective FirewallVM and have all these FirewallVMs go through a common VPN VM. That VPN VM then would connect to its own FirewallVM so that all outgoing its traffic is tunneled and encrypted before entering the WIFI VM and allowed only to the VPN Provider’s servers. Example below:

WIFI VM > VPN FirewallVM (traffic to VPN server only) > VPN VM > FirewallVM-A (traffic to website list A only) > AppVM-A

WIFI VM > VPN FirewallVM (traffic to VPN server only) > VPN VM > FirewallVM-B (traffic to website list B only) > AppVM-B

Best regards,
SU

Your eventual approach seems like it should work.

But I can’t understand what is happening in your tests.

I would suggest that you go back to WIFI VM > VPN VM > APP-VM. Get that working (which I believe you have in the past).

Then add in the VPN Firewall VM (traffic to VPN server only). Get that working, or failing that, report back with the firewall rule(s) you used. It might be easier to work this out one step at a time.

Yes, there’s something wrong with your results and I cant duplicate
them.
Particularly odd is that your test 2 result changed.

Alright thanks.

Correct. This has always worked well.

I set this up and tested the firewall by adding one vpn server at the time. Tested access only through that sever successfully. This is working fine.

I agree. I created brand new Firewall VMs for the VPN (VPN-FW-VM) and for the AppVM (FirewallVM).

Both from the same minimal template.

Before adding FirewalVM to the chain, I added the rules for the vpn server to the newly created VPN-FW-VM and connected as suggested by @Rooftop:

WIFI VM > VPN-FW-VM > VPN VM > APP-VM. I set the radio button to “Limit outgoing internet connections to …” And used qvm-firewall command in dom0 to add the rules instead of the GUI. As aforementioned, this produced expected results: I was able to confirm that websites could only be accessed from the AppVM when connected to the vpn servers for which “accept” rules existed in the firewall.

I then added the newly created FirewallVM without any rules yet set and “Allow all outgoing internet connections” activated. Internet access remained unchanged as expected.

I then proceeded to enable “Limit outgoing internet connections to …” And add one rule for a specific websiteA.com using the same qvm-firewall process in dom0. This was the exact same procedure used to to add the vpn server rules in the VPN-FW-VM.

However, after doing this there was no access to websiteA.com (unexpected). Neither was there access to any site (expected).

I deleted the rule and added ine for websiteB.com amd repeated the test. I did this procedure with several different websites so that only a rule for one website at the time existed at all times in FirewallVM.

I noticed erratic behavior where some websites would work with the rules and most wouldn’t.

After all this I’m of the impression that it may be website related (Perhaps DNS or load balancing?)

In any case, I guess I would just have to do wothout a firewall, which is obviously not acceptable. I’ll go back to using qubes once I figure out a solution to this. Thanks for the support.

SU

Ok. If you are trying to restrict access to websites that may be behind CDN’s or load balancing, then the firewall isn’t going to work because it only works on IP addresses (the website address is converted into an IP address when the rule is created).

For this situation you need a web proxy (something like squid maybe). So your setup would then be:

WIFI VM > VPN-FW VM > VPN VM > Proxy VM

APP VM (No network - see below)

The proxy server on the Proxy VM can be used to restrict access to the sites you require (since it will see the site names, not just the IP addresses).

As your APP VM has no networking, you will need to enable networking between the APPl VM and the Proxy VM. See the Qubs Firewall documentation (especially he section Opening a single TCP port to other network-isolated qube. Since you will only be opening a single port (the proxy server port) to the Proxy VM, I don’t think you need another firewall between APP VM and Proxy VM. Obviously you will need to setup the browser in App VM to use the proxy.

Also note that your APP VM will have no networking other that allowed via the proxy server (which sounds like what you want).

One last note - I haven’t tried any of this - this is speculation based on reading the docs :slight_smile:

If none of the above makes sense, I might try to set up a test system in the new year and see if I can get it working.

If you examine the qubes-firewall table in the fw below the proxy,
you’ll see that a single domain name can generate more than 1 IP
address in the rule.

To add to the complexity, many websites load data from many domains, so
simply whitelisting one may not allow the website to load (properly).

That said, I have exactly the setup that SomeUser has , and I cant reproduce their
failing.

What is www.websiteA.com?
What are the other sites that SomeUser tried?
What is the content of nft list table qubes-firewall
If you load the vpn client manually with --verb 6 (for openvpn), do you
see significant traffic when attempting to open the web site?

Hi All,

Apologies for the delay in seeing your responses. Please see my comments below.

The tested websites are:
www.google.com
www.qubes-os.org

Correct. I remember reading this, and @Rooftop’s comments, in the Qubes Firewall documentation. That would make perfect sense if not for the following fact:

The sites in question are still accessible via Firewall VM when the VPN-VM is not present in the chain.
This suggests to me that the access is crippled when introducing VPN-VM into the chain and not merely by fact that URLs are converted to IPs when rules are applied.

Right. I have noticed this with some streaming websites.

Running the above in the VPN VM returns the following:

table ip qubes-firewall {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
		ip saddr <Firewall.VM.ip.address> jump qbs-<Firewall-VM-ip-address>
	}

	chain qbs-<Firewall-VM-ip-address> {
		reject with icmp type admin-prohibited
	}
}

Note that this does not reflect the output of the qvm-firewall command in the FIrewallVM:
qvm-firewall <Firewall VM> list
0 accept www.google.com - - - - - -
1 accept www.qubes-os.org - - - - - -
2 accept - - - dns - - -
3 accept - icmp - - - - -
4 drop - - - - - - -

If I remove VPN-VM from the chain and connect Firewall VM directly to WIFI VM, I get this when running the nft command in the WIFI VM:

table ip qubes-firewall {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state established,related accept
		iifname != "vif*" accept
		ip saddr Firewall.VM.ip.address> jump qbs-<Firewall-VM-ip-address>
	}

	chain qbs-<Firewall-VM-ip-address> {
		ip daddr { 172.217.23.4 } accept
		ip daddr { 185.199.108.153, 185.199.109.153, 185.199.110.153, 185.199.111.153 } accept
		ip daddr { Default.Route/DNS.IP } tcp dport 53 accept
		ip daddr { Default.Route/DNS.IP } udp dport 53 accept
		ip protocol icmp accept
		reject with icmp type admin-prohibited
		reject with icmp type admin-prohibited
	}
}

This matches the output of the qvm-firewall command in the FIrewallVM listed above

First test:
I performed this test and noticed an ongoing “ping” every 8s when looking at the journalctl qubes-vpn-handler log.
I then proceeded to configure the Firewall VM to allow all outgoing traffic.
I loaded the websites above on a web browser and was able to observe alterations in the traffic pattern log indicating that data was being exchanged. The websites loaded.

Second test:
I then proceeded to configure the Firewall VM as shown by qvm-firewall <Firewall VM> list above.
I again looked at the log and noticed the ongoing “ping” every 8s.
Finally, I attempted to load the websites. I noticed no change in the traffic pattern while the web browser attempted to load the websites. The browser eventually returned the page not found message. The traffic pattern remained unaltered throughout this process.

Thanks fort the suggestion, though this sounds like adding yet another network service provider to the list. I would much rather figure out the issue at hand and avoid that, if possible, since I already have my reservations about trusting a VPN provider in the first place. Thanks again

SU

I had hoped that this had been put to bed, because I cant
reproduce the issue you are having. And it seems to me that the
“problem” you are reporting has changed over the course of this thread.

I’m not sure why you are surprised at the result of nft in the vpn -
isn’t this just what you want? You want traffic to be diverted to the
VPN and you don’t want it forwarded ‘as is’ to sys-net, or whatever is
upstream of the vpn qube.

The thing that bothered you most originally was this case:
4. WIFI VM > Firewall VM (rules ENABLED) > VPN VM > AppVM = Internet to ANY site
But we have discussed this in detail and I hope you understand why there
is not a problem here.

The current problem you report is not one you originally had:

  1. WIFI VM > VPN VM > Firewall VM (rules enabled)) > AppVM = no internet (?)
  2. WIFI VM > VPN VM > Firewall VM (rules disabled) > AppVM = no internet (?)
    The fact that you didn’t have this problem originally, combined with the
    fact that I cant reproduce your issue, makes me think that you have
    simply configured the VPN/Firewall combination incorrectly during the
    course of your testing.

I suggest that you start from scratch - set up the VPN fresh, test it
with cases 1 and 2 with a new FirewallVM.
I’ll do the same.

I started afresh with a new vpn qube, new vpn-fw and tester qube.
Apart from minor hacks to the vpn setup because I dont use IPv6, it was
plain install.
Confirmed that vpn - test worked: tester access via VPN
vpn - vpn-fw - tester (no rules) - tester had full access via VPN.
vpn - vpn-fw - tester (custom rules) - tester had access via VPN
restricted as per the rules.

I set custom rules using domain names, and IP addresses. No problem with
either.
I used both Debian and Fedora based vpn-fw: no problem with either.

Again, I am unable to reproduce the problem as reported.

Hi all,

I see. I have started the whole setup process from scratch with the same results: VPN fresh works, adding Firewall VM breaks it. I followed the same process I have described in previous posts. I do not see where I should/could change or adjust the setup steps as they are as per the available documentation without any variations. Continuing to repeat this would be insanity at this point.

Your results obviously prove my goal is possible and the fault lies somewhere in my setup. Since I lack the ability to further diagnose and resolve, I will desist.

Thank you all for your time and support
SU

This thread has been difficult to follow, and I don’t have any experience using a VPN with Qubes, so there’s not much I can add to what @unman has already said.

However, I’ll add some foundational knowledge that I’ve learned when it comes to using firewall rules:

  1. Firewall rules for “someqube” are enforced on the netvm/proxyvm that “someqube” is connected to.
  2. That netvm/proxyvm qube must have the ability to resolve hostnames, or it will not allow traffic from “someqube” to flow.

If your goal is to only allow www.google.com from AppVM, there should be a FirewallVM in-between VPN VM and AppVM,

Wifi VM < VPN VM < Firewall VM < App VM

The firewall rules should then be configured in AppVM’s Qube Settings / Firewall.

Then, to make sure point two is not a factor, look at the logs on Firewall VM, specifically journalctl -u qubes-firewall.

If there exists a message such as “Failed to parse rules, blocking traffic”, this condition must be rectified. How can it happen?

  • VPN VM did not connect to the remote server quick enough such that a DNS request resolves successfully
  • VPN VM has issues responding to DNS for some other reason.
  • VPN VM DNS server just didn’t respond

If it is a timing condition, open a terminal in FirewallVM and do sudo systemctl restart qubes-firewall while observing the journal to force the firewall to try again.

2 Likes