I did not manage to get sys-net openbsd working. Can someone help me saying what’s wrong here?
@omgsolucky I cant see your images.
It would be helpful if you could summarise what they show.
I assume that you have the OpenBSD system online, and can access the
internet from it.
How have you set forwarding on that sys-net? What do your firewall rules
look like on that system?
How did you configure the netvm of that system? What is the content of
the forward and custom-forward chains?
Yes i can access the internet @unman
My /etc/examples/pf.conf its exactly the one of the guide. I only use re0 instead of em0. My etc/resolv.conf cfg: nameserver 192.168.18.1 # resolvd: re0. Mirage-firewall netvm is none, sys-net-openbsd netvm is mirage-firewall & personal netvm is mirage-firewall. Advanced configurations of mirage-firewall: 32mb initial & 32mb max, 1 vcpu, kernel mirage-firewall, pvh mode & kernel opts --ipv4=10.137.0.31 --ipv4-gw=10.137.0.2. My mirage-firewall IP was just the first one. I was able to see this ip from mirage-firewall settings. However, this ip just dissapeared for now and don’t have informations of ip, dns, etc, of the mirage-firewall. Just blank space. And my OpenBSD IP is the same as ipv4-gw, so everything fine here, i guess.
When i try to open some website on my personal vm, which is my vm for test for this setup, the website enter in a loop loading and after a few minutos it give a error: Possible security risk looking up this domain. Firefox can't protect your request for this site's address through our secure DNS provider. Firefox wasn't able to connect to mozilla.cloudflare-dns.com. When i try to ping 4.4.4.4 in console of personal, seeing tcpdump -ni xnf0 logs inside OpenBSD, it shows 10.137.0.31 > 4.4.4.4: icmp: echo request. When i try to ping qubes-os.org, it shows 10.137.0.31.59172 > 10.139.1.1.53: 63771+ A? qubes-os.org.(30). And so much spam trying to connect to firefox’s dns. 10.137.0.31.7604 > 10.139.1.2.53: 49471+ AAAA? mozilla.cloudflare.com.(44)
Edit: I will manually copy all my configuration of /etc/examples/pf.conf so it will be better to you.
set skip on lo
match in proto { udp tcp } from xnf0:network to any port domain rdr-to 9.9.9.9 port domain
pass in proto { udp tcp } from xnf0:network to any port domain rdr-to 9.9.9.9 port domain
match out on re0 inet from xnf0:network to any nat-to re0
pass out on re0 inet from xnf0:network to any nat-to (re0)
block return
pass
block return in on ! lo0 proto tcp to port 6000:6010
block return out log proto {tcp udp} user _pbuild
As there are packets arriving in openbsd (and the src address is that of qubes-mirage-fw), everything seems to be fine in all the other qubes.
Can you check whether packets are coming out of your openbsd vm (e.g., tcpdump on your re0 interface should show the icmp requests packets to 4.4.4.4, as well as the replies and also your DNS requests to 9.9.9.9).
Ok, so I use sys-net (openbsd) + mirage-firewall for all networking qubes and it works.
But problem is that when my windows-11 hvm is using mirage firewall, if I shut down it, mirage-firewall always gets shut down.
When Windows is networked through this all other VM fails to start.
My system has 64 GB RAM out of tat windows use 8GB, mirage using 64MB & sys-net of openbsd uses 400MB.
Yes I also managed to reproduce that, thank you for reporting. The issue rises when the windows VM disconnects its virtual interface. I currently have no workaround and I’ll try to fix that for next release.
These are my logs from tcpdump -ni re0. They help in something? I cannot understand nothing from this dump
I struggle with them too because the packets that are going out of your BSD VM are not related to your ping (i.e., no ICMP from 192.168.18.109 to 4.4.4.4 if 192…109 is your computer IP address).
To me the issue is probably in the NAT configuration inside BSD but I don’t see it.
Exactly. I don’t have idea of what’s happening. But maybe i can say that i skipped the step “Enter the DNS nameservers listed in the ‘Qube Manager.”, since the bsd install didn’t asked me to put dns nameservers.
Maybe the problem is here. It’s the only thing that i didn’t followed in the guide.
@dalih: If you want to try out, I’ve proposed a PR for handling HVM AppVM (Fix HVM clients AppVMs by palainp · Pull Request #219 · mirage/qubes-mirage-firewall · GitHub), you should be able to build the unikernel by yourself in any AppVM by installing podman, cloning qubes-mirage-firewall repository and switch to my palainp:fix-dm-appvm branch and using ./build-with.sh podman, the hashsum should match.
@omgsolucky: I’ve spotted a typo in your /etc/pf.conf file, the line pass out on em0 inet still refers to em0. If that was fixed, and as there is no evidence of internal traffic going out of sys-net-openbsd, can you check net.inet.ip.forwarding=1 is set in /etc/sysctl.conf?
1 Like
Hello! I’m @omgsolucky . I lost the account so i will continue the discussion from this account.
Yes i’ve noticed that when i was uploading the screenshots. I changed and fixed but nothing changed. That’s why i upload the image with that typo, without sending another with the configuration fixed.
Yes it is setted.
I think that my mistake is from the step
I don’t understand that part. I have to put ipv4=X.X.X.X to my ip address of mirage-firewall? Because i was putting the X.X.X.X the ip address of my mirage-firewall when it was connected with sys-net, which don’t make sense. When i was setting up the network, i setted like that:
But instead of sys-firewall, i used mirage-firewall. Is that right? What i have to put on X.X.X.X? My mirage-firewall doesn’t have any ipaddress, since it is not connected to any net qube.
The guide seems a bit confuse to me. Sorry, my english is not very good.
X.X.X.X should be the address given by qvm-prefs (you can also see it with qubes-manager). Y.Y.Y.Y should be the address of openbsd. As your first tcpdump log on xnf0 shows icmp packets I think the arguments are ok and qubes-mirage-firewall correctly forwards the packets. To me the issue lives in openbsd.
But the question is: When i put the exactly network setup the guide asked to put, the mirage-firewall stays with no net qube, so mirage-firewall will don’t have ip address, obviously, since it don’t have a net qube. So i was putting in the kernelopts the ip address for mirage-firewall when it was with sys-net as net qube. This is right?
I understand. I think that the error is that i setup something wrong. Some steps i cannot reproduce them exactly as the guide suggests. For example, the part of putting DNS Nameservers and gateway from qube manager to my attached network adapter. I could not do this. I selected autoconf for ipv4, skip ipv6 and the installer didn’t give me any options to setup the dns nameservers and gateway.
Well autoconf should be fine, and youve confirmed that you can access
the internet from the OpenBSD machine, so that’s all good.
It would be useful if you could dump the contents of your
pf configuration in the bsd qube.
You can do this from /etc/pf.conf (or wherever you store them).
I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.
My /etc/pf.conf is that:
# $OpenBSD: pf.conf, v 1.5.5 2017/12/03 20:40:04 sthen Exp $
#
# See pf.conf (5) and /etc/examples/pf.conf
set skip on lo
match in proto { udp tcp } from xnf0:network to any port domain rdr-to 9.9.9.9 port domain
pass in proto { udp tcp } from xnf0:network to any port domain rdr-to 9.9.9.9 port domain
match out on re0 inet from xnf0:network to any nat-to re0
pass out on re0 inet from xnf0:network to any nat-to (re0)
block return # block stateless traffic
pass # establish keep-state
# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010
# Port build user does not need network
block return out log proto {tcp udp} user _pbuild
it’s the exactly one from the guide. I only replaced em0 to re0.
My /etc/resolv.conf:
nameserver 192.168.18.1 # resolvd: re0
lookup file bind
To me:
- everything seems fine up to openbsd (because you can dump packets with tcpdump on xnf0 with relevant traffic regarding your ping test, e.g., inside openbsd, A? and AAAA? requests from 10.137.0.31 (your mirage-firewall IP that has NATed requests from client VM) to 19.139.1.1 and .2),
- everything after openbsd as well (because you can access the internet from openbsd, and you can dump traffic from your local network, e.g., multicast traffic from 192.168.18.1, port scanning from 192.168.18.253, and you laptop seems to have 192.168.18.109)
Therefore I suspect the NAT inside openbsd. The DNS redirection is done with the two first lines (match in proto { udp tcp } from xnf0:network to any port domain rdr-to 9.9.9.9 port domain and pass in proto { udp tcp } from xnf0:network to any port domain rdr-to 9.9.9.9 port domain) and you don’t have the requests on the re0 dump (and btw your /etc/resolv.conf configuration is not used as we redirect to 9.9.9.9).
Then the traffic is NATed with the two next lines (match out on re0 inet from xnf0:network to any nat-to re0 and pass out on re0 inet from xnf0:network to any nat-to (re0)).
And finally the ip_forwarding property in /etc/sysctl.conf should be set to permit openbsd to forward a packet from one interface to another.
I don’t have openbsd here, and I’m unsure how you can check if the pf rules are correctly parsed and running, and if ip_forwarding is correctly set. But to me the issue is somewhere between the two interfaces.
This must be right.
@connor.blane
I always find it useful to start with the most basic ruleset, confirm
that is working, and then add complexity.
In this case, the mirage-firewall performs NAT itself - so you can
recast the rules using its address: (say 10.137.0.100)
# $OpenBSD: pf.conf, v 1.5.5 2017/12/03 20:40:04 sthen Exp $
#
# See pf.conf (5) and /etc/examples/pf.conf
match in proto { udp tcp } from 10.137.0.100 to any port domain rdr-to 9.9.9.9 port domain
pass in proto { udp tcp } from 10.137.0.100 to any port domain rdr-to 9.9.9.9 port domain
match out on em0 inet from 10.137.0.100 to any nat-to em0
pass out on em0 inet from 10.137.0.100 to any
set skip on lo
block return # block stateless traffic
pass # establish keep-state
# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010
You could omit rules 1 and 2, by setting /etc/resolv.conf to 9.9.9.9 in
client qubes, but that’s trivial working config.
@qEawma5f Hi, Is there a way to setup it with dnscrypt like options with rethinkdns or nextdns.
Hello,
At first try, Qubes said sd0 not available to install. After restarting Qubes and resuming this morning, I no longer have this error.
After config xnf0, it asks me if I want to config vlan0 … if I do, then the next line asks me if I want to config vlan1 (which didn’t exist in previous
DNS nameserver “listed in Q.Manager” … I don’T see any such in Q.mgr, only IP (I set it to qube’IP)
EDIT: Found it (dumb me) it’s in the qube settings, not in the qube manager
Error: firmware.openbsd.org : no address associated with name => relinking to make a unique kernel
Exit to shall, halt or reboot ? (I guess reboot ?)
From the sys-firwall ? from the sys-net ? form the newly created openbsd ?
=> In openbsd, in /etc/, I don’t have “examples”
NextDNS is not supported on OpenBSD.
You can fairly easily install dnscrpt on OpenBSD - there are guides
available. Then you can redirect to whatever DNS provider you will.
The pf rules here capture DNS traffic and redirect it to a chosen
resolver. You would need to change this to redirect to the internal
service, and let that handle the outgoing DNS traffic.
I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.