Need a little help with firewall / iptables setup

tomacto · November 13, 2020, 10:19pm

Hi there.

I am trying to set up a qube for housing differential backups (taken with restic). To do so I need to run an sftp server in this backup qube.

None of the qubes I wish to backup from are usually connected to the internet so I have cloned sys-firewall as internal-firewall and set its netvm to none. The qubes to be backed up and the backup qube itself have this internal-firewall as their netvm.

In internal-firewall’s /rw/config/qubes-firewall-user-script I have:

iptables -I FORWARD 2 -s 10.137.0.0/24 -d 10.137.0.bb -j ACCEPT

allowing all local machines which are assigned it as it’s netvm to access the backup qube (10.137.0.bb).

And in the backup qubes /etc/config/rc.local I have:

iptables -I INPUT -s 10.137.0.xx -j ACCEPT

where 10.137.0.xx is the machine to be backed up that I am currently testing from.

The backup qubes /etc/ssh/sshd_config looks reasonable. Though I don’t think these last two points are essential information right now since I think whatever wall I’m hitting, I’m hitting it in the internal-firewall VM.

I can ping without issue from xx to bb and can see the activity with iptables -t filter -vnL, with packets arriving on both machines, on the FORWARD chain of internall-firewall and the INPUT chain of the backup qube at the rules shown above, as well as packets on the OUTPUT chain of the backup qube, and the FORWARD chain of the internal-firewall at the rule which accepts packets with cstate RELATED,ESTABLISHED.

But when I try ssh 10.137.0.bb from xx I receive no route to host and only see one packet on the OUTPUT chain of internal-firewall, nothing more.

(There are no other modifications to the firewall rules on either machine but for these here.)

I expect (and am hoping) that this is a simple failing with my understanding of how iptables rules are stated, as I have had only limited experience with it until now, but hopefully somebody here has better knowledge and some insight into this and can point me in the right direction.

Thanks for the time taken to read this and for any help you can give. T.

unman · November 14, 2020, 12:08am

Hi there,
instead of doing that, it’s much easier to use a qrexec service between
the qubes, which allows you to keep them offline, and control the
interaction using standard qubes-rpc policy files.
I’ve something at https://github.com/unman/qubes-sync which should help you to
set this up. Happy to give more detail if you find my notes hard to
follow.

If you must pursue the internal-firewall route, (and why not?), then
check that your /rw/config/qubes-firewall-user-script has been
triggered.
Otherwise everything looks fine to me.
What is happening on the INPUT side of the ssh server? Put in a logging
rule in first position.

tomacto · November 14, 2020, 1:11am

Thanks for your reply.

I had thought bout a qrexec solution before setting down this path actually but thought that it wasn’t possible as restic does not run as a server itself. I’m sure I do not have a grip on qrexec’s full potential though and perhaps I am misguided, so will take a closer look at it and your solution soon.

However, I would really like to get this going through the internal-firewall if at all possible. I would like to use it for various services, testing web apps, etc. in future so it would not be wasted, and as I say this firewall is not itself connected to sys-net so the machines all are still kept offline.

Anyway so, I stuck in the LOG rule as you suggested on the INPUT chain of the backup qube, but as suspected there was nothing coming through.

Perhaps more interestingly though, I went ahead and also added one to the FORWARD chain of the firewall and there is nothing there either.

unman · November 14, 2020, 2:00am

qrexec is quite capable, and also very easy to use.
It’s trivial to set up an rsync or ssh daemon and then have a listener
on some port redirecting to that service. Then on the client side you
set a listener service (for the other side of the qrexec tunnel).
Then on the client connect to localhost:port and qrexec magic connects
this to the daemon on the server.
This is the mechanism used for proxy updates, and it can be used/abused
for quite anything.
I use this mechanism for rsync and sshfs, and for separating the mail
functions to different qubes: I can read and compose mail “offline”, and
have the online qubes as disposable qubes.

Turning to your problem, what template are you using for
internal-firewall?
If you insert the LOG rule in 1st position, what traffic do you see?
With ping and then with ssh connection?

tomacto · November 14, 2020, 4:11am

Cool. Yeah, I used it the other week to attach my phone to a VM by forwarding adb to its daemons port running directly inside a usb qube, to prevent the device disconnecting when android switched modes. It was quite impressive, I’ll defintely play about with it some more.

The template of the firewall is fedora-32, it’s a straight clone of the standard (and unaltered) sys-firewall, and the only changes I’ve made are the rules discussed here. The backup qube is debian-10.

I had the LOG rule at the top before too, so there is nothing, but I’ve now added a LOG rule as the first rule of every chain in every table that was lighting up when watching ìptables -vnL on both firewall and target. Here’re the results.

ping "bb" (from “xx”)

___________________
The firewall logged:
---
**raw-PREROUTING**
  IN=vif3.0 OUT= MAC=<mac1> SRC=10.137.0.xx DST=10.137.0.bb <infoOne...>

**mangle-PREROUTING**
  IN=vif3.0 OUT= MAC=<mac1> SRC=10.137.0.xx DST=10.137.0.bb <infoOne...>

**nat-PREROUTING**
  IN=vif3.0 OUT= MAC=<mac1> SRC=10.137.0.xx DST=10.137.0.bb <infoOne...>

**mangle-FORWARD**
  IN=vif3.0 OUT=vif10.0 MAC=<mac1> SRC=10.137.0.xx DST=10.137.0.bb <infoOne...>

**filter-FORWARD**
  IN=vif3.0 OUT=vif10.0 MAC=<mac1> SRC=10.137.0.xx DST=10.137.0.bb <infoOne...>

**mangle-POSTROUTING**
  IN= OUT=vif10.0 SRC=10.137.0.xx DST=10.137.0.bb <infoOne...>

**nat-POSTROUTING**
  IN= OUT=vif10.0 SRC=10.137.0.xx DST=10.137.0.bb <infoOne...>

**raw-PREROUTING**
  IN=vif10.0 OUT= MAC=<mac1> SRC=10.137.0.bb DST=10.137.0.xx <infoTwo...>

**mangle-PREROUTING**
  IN=vif10.0 OUT= MAC=<mac1> SRC=10.137.0.bb DST=10.137.0.xx <infoTwo...>

**mangle-FORWARD**
  IN=vif10.0 OUT=vif3.0 MAC=<mac1> SRC=10.137.0.bb DST=10.137.0.xx <infoTwo...>

**filter-FORWARD**
  IN=vif10.0 OUT=vif3.0 MAC=<mac1> SRC=10.137.0.bb DST=10.137.0.xx <infoTwo...>

**mangle-POSTROUTING**
  IN= OUT=vif3.0 SRC=10.137.0.bb DST=10.137.0.xx <infoTwo...>

______________________
The backup qube logged:
---
**filter-INPUT**
  IN=eth0 OUT= MAC=<mac2> SRC=10.137.0.xx DST=10.137.0.bb <infoThree...>

**filter-OUTPUT**
  IN= OUT=eth0 SRC=10.137.0.bb DST=10.137.0.xx <infoFour...>

ssh "bb" (from “xx”)

___________________
The firewall logged:
---
**raw-PREROUTING**
  IN=vif3.0 OUT= MAC=<mac1> SRC=10.137.0.xx DST=10.137.0.bb <infoFive...>

**mangle-PREROUTING**
  IN=vif3.0 OUT= MAC=<mac1> SRC=10.137.0.xx DST=10.137.0.bb <infoFive...>

**nat-PREROUTING**
  IN=vif3.0 OUT= MAC=<mac1> SRC=10.137.0.xx DST=10.137.0.bb <infoFive...>

**mangle-FORWARD**
  IN=vif3.0 OUT=vif10.0 MAC=<mac1> SRC=10.137.0.xx DST=10.137.0.bb <infoFive...>

**raw-OUTPUT**
  IN= OUT=vif3.0 SRC=10.137.0.fw DST=10.137.0.xx <infoSix...> [SRC=10.137.0.xx DST=10.137.0.bb <infoFive...> ]

**mangle-OUTPUT**
  IN= OUT=vif3.0 SRC=10.137.0.fw DST=10.137.0.xx <infoSix...> [SRC=10.137.0.xx DST=10.137.0.bb <infoFive...> ]

**filter-OUTPUT**
  IN= OUT=vif3.0 SRC=10.137.0.fw DST=10.137.0.xx <infoSix...> [SRC=10.137.0.xx DST=10.137.0.bb <infoFive...> ]

**mangle-POSTROUTING**
  IN= OUT=vif3.0 SRC=10.137.0.fw DST=10.137.0.xx <infoSix...> [SRC=10.137.0.xx DST=10.137.0.bb <infoFive...> ]

______________________
The backup qube logged:
---
<exactly nothing>

Oh, and since it’s not showing up at filter-FORWARD, I also set the default on that chain briefly to ACCEPT to see if that made any difference. It did not!

icequbes1 · November 14, 2020, 5:35am

If you’re getting a ‘no route to host’, I suspect:

You are hitting a REJECT iptables rule somewhere
The backup qube isn’t connected to the firewall qube, or somehow lost connectivity?
- Edit: based on your last reply, I think this is where the issue is
You’re modifying routing rules on some qubes and undoing the 32-bit subnetting?
Typo somewhere? Wrong IP address somewhere maybe? Make sure the IPs haven’t changed on you, though that should only happen if you created a new qube.

From the info you’ve posted, what you have should do it. You’ve got:

Forwarding enabled at the netvm so that qubes can talk to each other
The backup qube wide open to the client on the INPUT side

I also agree with unman, doing this with qrexec will remove the iptables complexity and the need for your firewall qube.

To understand the qrexec/RPC method, think of it like this: If you have a server qube and client qube, in the client qube you are doing something similar to local port forwarding for SSH, except the transport isn’t SSH; it’s qrexec. qrexec is the Qubes implementation that uses Xen capabilities like vchans for inter-VM communication details.

If you wanted to open up all ports on a server qube to a client qube (which by all means is naturally dangerous, this is only to demonstrate capability), you would do:

Edit (dom0) /etc/qubes-rpc/policy/qubes.ConnectTCP
Add a line clientqubename serverqubename ask,default_target=serverqubename, this will prompt you to confirm allowing making the TCP connection

Any port in serverqubename is now accessible to clientqubename; pretend it’s like clientqubename having localhost access to serverqubename (no annoying firewall rules to worry about, very dangerous!)

To ssh into serverqube from clientqube (such that localhost:8022 redirects to serverqube:22):
```
qvm-connect-tcp 8022:serverqubename:22
ssh -p 8022 localhost
```

But don’t do that; that’s just to show how simple it is. Create new named Qubes RPC services for each service you want to expose from serverqube.