Debian-10-minimal firewall issues

User Support
1

Cross-posting here for visibility:

Original thread: https://groups.google.com/g/qubes-users/c/6e8oyr5GdXk

On Friday, 21 August 2020 at 16:58:48 UTC+8 54th Parallel wrote:

I’m having the same issue with disposable firewalls built on debian-10-minimal, with the minimum amount of packages, on brand-spanking-new installations (plural) being unreliable firewalls. They sometimes function but not all the time–and this is what’s scary, because there’s no way of knowing without manually checking all the time. The warning prompts when editing firewall rules aren’t useful indicators since they always appear regardless of whether filtering is happening.

I ran systemctl in both and found that qubes-firewall.service is not running in either, despite having manually activated them. I’m not a technical person, but this seems like a pretty critical issue to me (unreliable firewall with no indicator)–a warning about using minimal debian as templates for firewalls should be put up somewhere highly visible.

This unreliability has been bugging me for a while and I’ve been testing and testing (to the best of my abilities) before realizing that this is almost certainly not a user issue, so Sven, the OP, probably either ran into the issue again, didn’t know about his deactivated firewalls, or didn’t report the issue.

After some more probing around, I think I’ve found the issue, and that what I wrote earlier contains inaccuracies. The unreliable firewall might not be a debian-10-minimal issue, though the warning prompt that appears whenever editing firewall rules in a connected VM is.

My setup has two firewalls–one behind sys-net and another behind a VPN VM. Though the two firewalls are clones of one another, the sys-net firewall works (responds to rules set in appVMs) and the proxyVM firewall doesn’t. This is what caused me to think that deb-10-min firewalls in general are unreliable–some things are connected to the net-firewall (sometimes) and most are connected to the VPN-firewall. This makes it look like the firewalls work sometimes.

I have two laptops running Qubes with the same setup. Of the four firewalls, all with qubes-firewall explicitly enabled, only one actually has the qubes-firewall.service show up after typing ‘systemctl | grep firewall’ [so unlike what I said earlier, not even the working net-firewall is consistent, so this might not be a positioning issue]. Each of these firewalls were created in fresh but updated installations of 4.0.3 with the absolute minimum amount of packages (qubes-core-agent-passwordless-root (so I can configure sudo prompt), qubes-core-agent-networking, apparmor*) and the typical settings, along with qubes-vm-hardening (vm-boot-protect enabled).

Any insight into this would be greatly appreciated since this is a massive headache for me.

Update: Starting qubes-firewall.service manually in a VPN-firewall via sudo systemctl start qubes-firewall leads to qubes-firewall.service loaded failed failed being displayed in systemctl. On the other laptop, both firewall had loaded active active after doing the same.

Update 2: The VPN-firewalll actually starts up qubes-firewall that quickly turns from loaded active active to loaded failed failed (less than 1 min).

2

So now we have an unholy mix between top and bottom-posting, which
suggests that cross-posting may not be such a good idea.
On the other hand, perhaps this all looks fine in the web interface.

To deal with the question here, I use debian minimal templates
extensively, (NOT with qubes-vm-hardening), and have never seen this
issue - neither the unreliable firewall nor the warning prompts issue.
I’ve asked fellow users who also use debian-minimal and they do no not
recognise the issue.
So either it’s qubes-vm-hardening, or something about your system - so
more detailed information would be helpful.

3

Hi Unman,

Sorry for cross-posting–it’s only a thing for this urgent post while I transition to Discourse. Since I have the luxury of choosing where to continue the thread, I’ll gladly pick this place over Google Groups.

The steps below should cover everything and is taken from a manual I wrote and follow of how I should safely install and setup a fresh Qubes OS:

 

Steps to replicate my firewall (assume setting is default if unspecified):

Hardware

  • Dell Inspiron with i7-1065G7; NVidia GPU
  • Dell Vostro with i5-10210U; iGPU

Start with a clean copy of debian-10-minimal

  • Install apparmor apparmor-utils apparmor-profiles apparmor-profiles-extra qubes-core-agent-passwordless-root qubes-core-agent-networking
  • Change kernelopts to nopat apparmor=1 security=apparmor'
  • Install qubes-vm-hardening and configure dom0 sudo prompt

Create DispVM template based on above template

  • Disable services cups, qubes-updates-proxy, and qubes-update-check
  • Disable ipv6 qvm-features debian-10-minimal ipv6 ''
  • Enable service qubes-firewall
  • Enable service vm-boot-protect-root
  • Start above once for initialization

Create DispVMs based on above DispVM template

  • Changes above carry over
  • Disable vm-boot-protect-root
  • Enable vm-boot-protect

Insert firewall DispVMs into network as follows (only relevant parts shown):

sys-net (disp) — sys-firewall-1 (disp) — sys-vpn (disp) — sys-firewall-2 (disp) — appVMs

 

It’s basically linear, with sys-vpn sandwiched between two firewalls. This is why there’s a ‘sys-net firewall’ and a ‘VPN-firewall’. (If my system has Whonix, it’d have sys-firewall-1 as its netVM, and I’m also considering a ‘Tor-firewall’) I chose to separate sys-vpn and sys-firewall-2 despite knowing that sys-vpn can also act as a firewall. This is because I want to have another layer of isolation. In short, firewall-1 takes orders from sys-vpn and only allows VPN traffic in or out of the system, while firewall-2 takes orders from the appVMs and filters the contents of the VPN connection.

Since the two firewalls in a system are clones of each other, and the two systems were setup following the same manual, the only difference is in positioning within the network, but then even the networks are the same. Why only one sys-net firewall works reliably (i.e. qubes-firewall.service starts and stays up), I do not know. I would have probably solved this if I knew.

Another Google Groups/mailing list user thinks the setup doesn’t make sense but our understanding of how Qubes firewalls work seem to differ, so my understanding might be completely off-base.

 

As for your debian-minimal friends not recognizing the issue, it seems I am not the only person who has it, as the Google Groups thread I replied to was started by someone with the same issue (emphasis mine):

On Tuesday, 4 February 2020 at 08:29:47 UTC+8 sv...@svensemmler.org wrote:
Hi,

I created a sys-firewall based on debian-10-minimal:

  • qvm-clone debian-10-minimal deb-10-sys-firewall
  • qvm-create --template deb-10-sys-firewall --label blue dvm-sys-firewall
  • qvm-prefs dvm-sys-firewall template_for_dispvms True
  • qvm-create --class DispVM --template dvm-sys-firewall --lable blue sys-firewall
  • qvm-prefs sys-firewall provides_network True
  • qvm-prefs sys-firewall netvm sys-net

Then in deb-10-sys-firewall (template):

  • sudo apt-get install qubes-core-agent-networking qubes-core-agent-dom0-updates
  • attempting to install iproute tells me that this package no longer exists and I shall try iproute2
  • iproute2 does exist and was already installed

Then in dvm-sys-firewall (template for disposable):

  • added “iptables -I FORWARD 2 -s 10.137.0.21 -d 10.137.0.25 -j ACCEPT” to /rw/config/qubes-firewall-user-script

Then shut everything down and started sys-firewall.

Result:

  • network connectivity working
  • the above mentioned iptables rule is working (.21 can connect to .25)
  • qubes-qube-manager gives me this error when I try to edit the firewall rules of any qube connected to sys-firewall: “Networking qube does not support ‘qubes-firewall’ - firewall restrictions will not be applied.”
  • however it does not give me this error when I try to edit other qubes connected to sys-whonix

Any ideas?

/Sven

–
public key: https://www.svensemmler.org/0x8F541FB6.asc
fingerprint: D7CA F2DB 658D 89BC 08D6 A7AA DA6E 167B 8F54 1FB6

https://groups.google.com/g/qubes-users/c/6e8oyr5GdXk/m/ZFOpE_1gEAAJ

The problem I have looks the same, with the added issue of qubes-firewall.service either not starting or not staying up (see updates in original post). Because the warning “'networking qube does not support ‘qubes-firewall’” is always up, even when the firewall works, it doesn’t function as an indicator. My gut is telling me the issue might actually have to do with it being a disposable VM, but this is the gut of a non-technical person.

Anyways, thanks for wading through this wall of text even if you don’t have a solution. Might be a good impetus to learn how to configure Mirage firewalls (I have a feeling that’ll be harder than learning Qubes, given my skillset).

5

Did you actually say which of the machines works and which one doesnt? I
may have missed that.
You can troubleshoot this in a number of ways.
You could swap the firewall positions - this should allow you to
determine if the placement in your Qubes configuration is the issue - I
doubt that it is.
Then start with a basic minimal template configuration and confirm that
it works as expected. Once you are happy there, start adding in the
additional packages/configuration steps, to find at what stage things
break.
I see Sven’s problem, but I’m not convinced it is the same as yours.

The thing to do in this sort of situation is to start with what works
and make small incremental steps until you find where it breaks. It
could be that it is some combination if things that leads to breakage,
but fortunately Qubes allows you to make these sots of changes easily,
so trouble shooting is much easier than on a normal system.

6

Also, if you read that thread on the mailing list you will see that
Sven reworked his setup and everything worked as expected, which
confirms my view that the Debian-10-minimal firewall works fine.
You should confirm this and then take small steps to find where your
setup breaks.
Don’t forget to post your findings back to the mailing list once you have
identified the problem.

7

I’ve done some experimenting with my firewalls based on my hunches:

Hunch: Starting proxyVMs one after another confuses some part of the system and leads to the errors

Finding: Manually spacing out the booting of the VMs (i.e. not starting them together as a stack) does nothing.

Hunch: Since the VPN VM sandwiched between firewalls itself a proxyVM, QubesOS starts the firewall service in it by default, leading to three firewalls running in a chain. This might confuse the system.

Finding: Disabling the firewall service in the VPN VM does nothing; systemctl still shows qubes-firewall.service as not starting on boot. Manaually starting qubes-firewall often leads to persistent ‘loaded failed failed’ status.

Serendipitous finding?: Starting the firewall VMs with the ethernet cable plugged in seems to work, but I haven’t experimented with this enough to make any statements, especially since everything feels so whimsical and inconsistent.

Observation and conjecture: The only consistent point is that qubes-firewall service fails to start on boot. This might have to do with the disposable nature of the VMs and when it was cloned (assuming I understand dispVMs).

Right now my temporary solution is to configure the firewall VMs to run sudo systemctl start qubes-firewall on boot, then manually spam check with systemctl | grep firewall. A lot of the times the firewall service starts off fine with loaded active running, but then fails with loaded failed failed after a few seconds or a few minutes. If the service survives this perilous period, then it’s usually home free and works as advertised. Sometimes the service works from the start.

 

Yes, you’re right in saying that my issues don’t match those of Sven’s. His main issue is the warning prompt. He also mentions multiple times that everything works as it does. This is explicitly different from mine, and I misspoke when I said that “the problem I have looks the same” as my issue is that the firewalls aren’t working. Though I admit my fault, I want to argue that our issues are connected in an important way–with the warning prompt no longer corresponding to whether the firewall is actually working or not, there’s no way that I know of to have a quick and easy way of seeing whether the firewalls are working.

 

Since both machines have the same configuration and both firewalls in both machines seem to have the same issue (after more extensive observation and testing). I previously said “Of the four firewalls, all with qubes-firewall explicitly enabled, only one actually has the qubes-firewall.service show up”, which gives the impression of this particular firewall on this particular machine consistently working, but after more observation I can say that this is not the case–it behaves similarly to all other firewalls.

 

The testing above was done before reading your replies. Your suggestions are scientific and methodical, but would take a lot of time and effort. Since I have a stop-gap solution, I want to take a break from this, then later put resources into learning how to configure Mirage (I can’t seem to find a clear and understandable explanation on how to do so).

I apologize for any confusion caused by unclear or inaccurate reporting, but want to note that I’m doing what I can as a non-technical person probing what is possibly a highly complex technical issue.

8

I was under the mistaken impression that editing Mirage firewall rules involved editing the .ml files in OCaml, then recompiling the mirage kernel every time you wanted to change rules. This is why I was using Debian firewalls because that hurdle was like climbing a cliff for me.

I got that impression from a few forum posts and testing it way back when I first started playing around with this OS before understanding how firewalls work here (I directly edited the firewall rules of Mirage VMs, then sulked when that didn’t work).

Now that I’ve tested it without having a half-baked conception of how Qubes firewalls work, I realized that its configured just like regular VM firewalls. Massive thumbs ups to the Mirage team.

As much as I want to get to the bottom of this, I can only report my shallow probings (as in the previous post) as a technical breakdown is beyond my current abilities. My solution is to use Mirage, now that I’ve figured it out, as (AFAIK) unikernels are much more stable and reliable and I get the additional upside of having more free resources and a much smaller attack surface.

Thanks for putting up with me.

1 Like
9

Ok, so today it happened again (I am documenting my template/qube creation to prepare for bullseye / kicksecure 16 and eventual salting)

As I mentioned before, I had satisfied myself once again that the firewall was actually doing it’s job, but the Qube Manager complained anyway.

So I looked at the code (why I didn’t choose this approach earlier is beyond me)…

if netvm is not None and
not netvm.features.check_with_template(
‘qubes-firewall’,
False):
QtGui.QMessageBox.warning(
None,
self.tr(“Qube configuration problem!”),
self.tr("The ‘{vm}’ AppVM is network connected to "
“’{netvm}’, which does not support firewall!
”
"You may edit the ‘{vm}’ VM firewall rules, but these "
"will not take any effect until you connect it to "
“a working Firewall VM.”).format(
vm=self.vm.name, netvm=netvm.name))

… they key here would be the “.check_with_template”. So while

qvm-features sys-firewall returned “qubes-firewall 1” all along, the template sys-firewall was based on did only return “qubes-firewall” (without the one).

So a quick qvm-features call later to set the 1 to the qubes-firewall feature of the template … and now the Qubes Manager doesn’t complain anymore.

2 Likes
10

There is a related qubes-issue now: debian-10-minimal requires 'gcr' package for sys-firewall GUI · Issue #6381 · QubesOS/qubes-issues · GitHub

@fiftyfourthparallel: has my solution worked for you? If so, could you mark my post as the solution please?

1 Like
11

Hi Sven,

I must have glanced at it, hearted it, and then forgotten to come back to it since I was distracted.

This solves the mystery of why the GUI keeps telling us the d10m firewall doesn’t support firewall.

The other part of my issue is that qubes-firewall.service itself wasn’t starting in the firewallVM, according to systemctl, and this has major security ramifications. I’m not technical so maybe there’s a link between the two that’s not apparent to me. Is it because the app/dispVM checks with the template and when it doesn’t return the qubes-firewall 1 it doesn’t start the service?

I’ve been using qubes-mirage-firewall exclusively since learning about how to use it, so I can’t really say whether changes and updates have fixed the issue, but it’s good to know that the GUI mystery has been solved–thank you!

Edit: I went and read the Github link you provided and realized that the GUI and the systemctl services issue are linked, and that your post should solve the underlying issue, so I’ve marked your it as a solution.