I swear I can’t figure this crap out. All of the tutorials are different, most of them I get stuck halfway through because of a missing command… it’s infuriating. There just isn’t anything Qubes specific. Even Debian specific stuff doesn’t work right in a Debian qube…
I had bigger plans, but right now I would be immensely happy if I got just basic routing working. I just want to give a specific qube some physical ethernet ports, and whatever connects to those ethernet ports (with crossed cables of course) should connect to the internet through whatever netvm I assigned to the qube. That’s the simplest iteration of what I want… and I can’t even get that working.
Everything seemed to go well until it asks me to ping 192.168.1.1. Up until that point I had been mapping IP’s in my head (from his format to qubes format, eg. 192.168.10.x to 10.137.0.x) but then he mentions 192.168.1.1 and 192.168.0.1, neither of which were mentioned before in the tutorial… not sure if that’s a typo or if he’s talking about his router or what.
Please tell me I’m not the only Qubes user that’s ever needed a router… I just can’t be, I refuse to believe it… Not with the unmatched potential of being able to chain and tunnel different networks any way you want from one single easy to use GUI… Name another OS that lets you do that… come on I know you’re out there…
edit: Yes I set it to HVM mode with no memory balancing and did add the nic’s, which are named ens7, ens8, and ens9 from the qube’s point of view
I was a little confused about what you were trying to do at first, is this right?
External non-qubes device → physical NIC connected to a StandaloneVM → sys-firewall → sys-net → local gateway → internet
The tutorial is assuming that 192.168.0.1 is your local gateway to the internet. 192.168.1.1 was used once in error (these are the most common default subnets for home routers). To test that the external device and the StandaloneVM are routing correctly, ping the 10.137.0.x address of your VM’s eth0. To test the full path through Qubes you will want to ping the IP of your local router to the internet. It will be the default gateway of sys-net.
When testing and troubleshooting, “ip route show” can be used on your external linux device, the StandaloneVM, sys-firewall and sys-net to see where the next hop of traffic will go. The line beginning with “default via…” should be your path to the internet.
192.168.10.x addresses in the tutorial refer to the network between your external device and your StandaloneVM. You probably do not need to change that and it can cause issues if the subnet you use here overlaps with the internal qubes address spaces of 10.13x.0.x or the subnet of your local network.
This is where things can get difficult. If you are using the Qubes Debian 11 template there are quite a few things already in place. Qubes networking is set up by a couple scripts, /etc/xen/scripts/vif-route-qubes and /usr/lib/qubes/qubes_setup_dnat_to_ns. The new iptables rules from the tutorial might need to be reordered to work.
For testing and learning purposes, it may be easier to use install media to install Debian into a HVM rather than starting from the Qubes template. You might need to manually configure eth0 and DNS in that case, but it would match what tutorials are expecting better.
Alright so after a lot of trial and error I finally got SOME form of routing working. Not exactly what I wanted but at least I got a win after struggling so long, and at least now I know my crossover cables actually work.
For some reason Debian’s base install doesn’t come with ifconfig so I couldn’t set up eth0, however I managed to pull it off with another Debian-based distro. The furthest I got with the terminal and the tutorial was getting a connection from the HVM to sys-firewall; I never got any of the other NICs working.
What did end up working however was NetworkManager applet. First I set eth0 to ‘Manual’ and copied IPv4 settings from the Qubes Manager to get an internet connection, and then set all the other NICs to ‘Shared to other computers’. At that point I got all three NICs (eth1/2/3) to route to the outside internet through eth0, as well as directly to the HVM.
I haven’t tried to replicate this in a regular qube, but even if it works the same as it does in the HVM, I still don’t know how to control and restrict the traffic between each NIC connection and other devices at the different levels of my network. I need the following types of connections to be possible:
NIC <-> services local to the HVM
NIC <-> HVM <-> fellow ethernet devices eg. 10.42.x.x
NIC <-> HVM <-> sys-firewall <-> other Qubes eg. 10.137.0.x
NIC <-> HVM <-> sys-firewall <-> sys-net <-> other devices on my network eg. 192.168.0.x
NIC <-> HVM <-> sys-firewall <-> sys-net <-> the outside internet
Bridges first, walls second.
From that list, the only ones that work right now are #1 and #5, and that’s with NetworkManager. I was expecting #2 to work out of the box too but it doesn’t. I tried sudo iptables -I FORWARD 2 -s $source_ip -d $destination_ip -j ACCEPT and vice versa on the HVM (which normally works for getting two qubes to talk to each other in a netvm) but it didn’t work. I suspect it has something to do with the fact that each NIC IP only has two digit groups in common instead of three (eg. 10.42.0.72 and 10.42.1.25). As for #3 and #4… well that’s what I’ll be working on in the meanwhile lol.
So I guess my next question is: should I even keep bothering with NetworkManager? If so, is there any way to configure it other than the applet (eg. for telling it to put all of the NICs in the same network so they can see each other and to toggle outside internet access independently of firewall rules)? And if not, is there a way to “translate” or transpose my current network configuration into something ifconfig can digest?
ifconfig is deprecated, and has been since stretch - it’s removed from the base install in favour
of ip
The reason is that it was part of net-tools which was unmaintained and
lacked ipv6 support: Debian isn’t alone in this change.
Similarly iptables has been replaced by nftables - when you use the iptables command in recent Debian you are actually using nft - at some stage
that wrapper will be removed so you should start becoming familiar with
nft. As a start, I’ll show the nft equivalents for iptables commands.
I cant help you with Network Manager - if I tried I would just be
reviewing the same documents that are available to you.
As far as the connections you want, they are quite simple.
First you need to ensure that the “router” (HVM) has routing enabled -
Since you say that #5 works I’m assuming that you have done this and
have a working configuration.
#1. This is simple - you need to identify the ports used by those
services and open them: e.g for an ssh server on port 22, to be
available from eth1: iptables -I INPUT -p tcp -m tcp -i eth1 --dport 22 -j ACCEPT nft insert rule ip filter INPUT iifname "eth1" tcp dport 22 accept
#2. This might be simple if I understood what you want. What are “fellow
ethernet devices”? What is the significance of the class B example?
#3 - This is simple and covered in the documentation
You need rule(s) on sys-firewall which allow traffic between attached
qubes. How complicated a rule is will depend on which qubes you
want to connect and how you want them to e connected. Do you want all connections to be available to every qube? Limited
connections to specific qubes?
Remember, as far as sys-firewall is concerned all the traffic from HVM including traffic that has come in via the NIC comes from the HVM IP
address. This means that you have to decide whether you want HVM to be
able to access those qubes too.
By default every qube allows all outgoing traffic and allows all
RELATED and ESTABLISHED traffic. If you don’t want this you can block any
outgoing traffic to other qubes; iptables -I OUTPUT -d 10.137.0.1 -j DROP nft insert rule ip filter OUTPUT ip daddr 10.137.0.1 drop
#4 - This should be straightforward, and if you have #5 working should
work.
Again, as far as those network providing qubes are considered, the
traffic comes from the IP address of the attached qube, regardless of
where it originated. sys-net does not do any filtering based on target
IP address, and contains a correctly set route to the local network.
You can confirm this by ip route and iptables -L -nv, or nft list table filter
So if this is not working, you need to look elsewhere.
This being Qubes, there are some specific things to look at - you can
put configuration files in /rw/config, as explained here
An advantage of nft is that you can make atomic changes to the ruleset -
this means that all changes are made immediately in the transition from
one state to another.
Once you have the nft ruleset as you want it you can dump the contents
to a file , and xen load that file - you can do this from scripts in /rw/config/qubes-firewall.d. (This replaces iptables-save and
iptables-restore.)
To dump the rules: echo "nft flush ruleset" > backup.nft nft list ruleset >> backup.nft
To load it: nft -f backup.nft
If you store the dump in /rw/config, then it will be accessible for per
qube settings.
I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.
