Combination of OpenBSD sys-net + MirageFW

Has someone here ever managed on get these 2 NetVMs to work together? Interested in to do so to have a more hardened system.

I tried (following Use OpenBSD as NetVM · Issue #5294 · QubesOS/qubes-issues · GitHub) without luck, openBSD failed to use my network interface :frowning:

BTW if you succeed to run openBSD and use your network card, OpenBSD still lacks netback drivers (Use OpenBSD as NetVM · Issue #5294 · QubesOS/qubes-issues · GitHub). So you’ll also have to compile a specific mirage version that “invert” the way netvm is used (manage a dynamic uplink by palainp · Pull Request #178 · mirage/qubes-mirage-firewall · GitHub barely tested as I currently don’t have an openBSD sys-net but will be happy to work on again).

I’m also interested in something similar to this setup so I gave a try. I used HardenedBSD instead of OpenBSD for sys-net.

My NIC is compatible with BSD and I was able to connect to the internet in sys-net during this test.

I wasn’t able to reach the internet in the VM’s that had mirage firewall has the netvm :frowning:

@palainp I compiled that specific version of mirage-firewall and set kernelopts to ‘–ipv4=$MIRAGE’S_QUBES_IP --ipv4-gw=$BSD’S_QUBES_IP’

sha256sum: 32272239ebe24c23ed0e888395400aeaf09a95e854b6e5ab35af25263b94dcd8

This is my setup (sys-net has mirage-firewall has the netvm and mirage-firewall has no netvm):

defaultrouter=“$ROUTER_IP”
ifconfig_$NIC=“inet $LOCAL_IP netmask 255.255.255.0 mtu 1500”
ifconfig_xn0=“inet $SYS-NET_QUBES_IP netmask 255.0.0.0 mtu 1500”

I’m able to ping sys-net qube’s IP in a VM that has mirage-firewall has netvm but nothing else.

If I try to ping 1.1.1.1 in a VM that has mirage-firewall has the netvm I receive the requests in sys-net in the xn0 interface but sys-net doesn’t reply.

I’m not able to ping mirage-firewall from sys-net.
I’m not able to ping VM’s that have mirage-firewall has netvm.

How is this supposed to work, is the traffic from my NIC meant to be forwarded to xn0? does multicast relate to any of this?

I’m sorry for the confusion.

Thank you for the test bed! :slight_smile:
I think there should be some NAT somewhere between NIC an xn0 (edit: and a default route via NIC)? This way the 1.1.1.1 ping packet will go outside and back to the AppVM via sys-net and fw.

Enabling forwarding with (sysctl net.inet.ip.forwarding=1) makes it so sys-net also receives request packets in the NIC interface.

cat /etc/pf.conf
nat pass on $NIC_IF from {xn0:network} to any → ($NIC_IF)

Makes sys-net receive request and reply packets in both interfaces but the VM doesn’t get the response, I don’t know where to go from here :frowning:

This when trying to ping 1.1.1.1 because if I try to ping a domain it gets stuck at the xn0 interface.

I don’t get how to add a default route via NIC :confused:

@unman mentions a firewall script on his OpenBSD as netvm notes, I wish I could see it.

Any thoughts?

I’ll figure it out this weekend and share the solution

I was hopeful at first, now my hope is dead.

@palainp You mentioned you tried mirage firewall with Linux instead of BSD in this same scenario, how did you implement NAT?

This was my final setup:

/etc/rc.conf:
pf_enable="YES"
gateway_enable="YES"

defaultrouter="$ROUTER_GATEWAY"

ifconfig_$NIC_IF="inet $LOCAL_IP netmask 255.255.255.0 mtu 1500"
ifconfig_xn0="inet $BSD_QUBES_IP netmask 255.0.0.0 mtu 1500"

# Not Needed
static_arp_pairs="router firewall dns1 dns2"
static_arp_router="$ROUTER_GATEWAY $ROUTER_MAC_ADDRESS"
static_arp_firewall="$MIRAGE_QUBES_IP fe:ff:ff:ff:ff:ff"
static_arp_dns1="10.139.1.1 fe:ff:ff:ff:ff:ff"
static_arp_dns2="10.139.1.2 fe:ff:ff:ff:ff:ff"
/etc/pf.conf:
nat on $NIC_IF inet from xn0:network to any -> $NIC_IF

The final result is the same as before, the rest of what I tried wasn’t progress.

Ping 1.1.1.1 from VM that has mirage-fw as NETVM; $MIRAGE_IF gets request and replies but nothing happens in the VM that requested it.

DNS is probably easy to get the same result as the ping request, I just didn’t bother to do it yet since it would take me a while and I knew it wouldn’t contribute to the wanted result.

My concern is that it isn’t a skill issue on my end, and this mirage-fw branch isn’t working as intended?

I also tried to use OpenBSD instead of HardenedBSD the result was the same.

first, can you ping sys-net from sys-firewall when it’s using hardenedbsd?

if it’s not working, what does a tcpdump in sys-net show when you try to ping from sys-firewall?

Thank for you test, and I’ve not deeply tested that branch, so I may have done somethin wrong. So far I understood your setup is: AppVM → mirage-fw ← BSD (which control a NIC connected to LAN)
and with ping 1.1.1.1 from AppVM you got packets in BSD but nothing else. If my understanding is correct having nat between the 2 BSD interfaces and a default route via NIC should resolve the ping request (maybe it worth having tcpdump or wireshark here?)
I failed to have a linux sys-net with a NIC connected to LAN (I lack some Qubes knowledge here) so I only tried to the situation:
AppVM → mirage-fw ← linux and where a ping from AppVM is forwarded to the linux VM.

EDIT: Warning my OpenBSD knowledge is inexistant. According to OpenBSD PF: Network Address Translation do you have something like: pass out on $NIC_IF inet from xn0:network to any nat-to $NIC_IP (that’s a very similar syntax as your /etc/pf.conf so I’m not sure what are the differences)?

@solene: as it’s a mirage unikernel we can’t launch commands from the firewall :frowning:

That is the message that leads me thinking nat should be sufficient.

EDIT: and btw I have the same hashsum as you with ./build-with.sh podman.

@nalea if you have enough hope, would you mind to try out a new branch where I added some logs:

The hashsum is b2703fd8f4dd7c18af80681aad30f39f22e549571b9777048215b7180be344c7

In my test setup I have:
AppVM (10.137.0.x) → mirage-fw (10.137.0.y) ← linux (10.137.0.z)
And with `ping 10.137.0.z) in AppVM I have in the logs:

INF [dispatcher] packet from client 10.137.0.x -> 10.137.0.z
INF [dispatcher] transmit 10.137.0.y -> 10.137.0.z
INF [dispatcher] packet from client 10.137.0.z -> 10.137.0.y
INF [dispatcher] transmit 10.137.0.z -> 10.137.0.x

So the firewall correctly nat from one client to the netvm gateway in both directions.

As I don’t have an outgoing interface ping 1.1.1.1 doesn’t resolve but is received in 10.137.0.z.

What I have in mind is that the code for forwarding packets from the netvm client may be buggy and these traces would help :slight_smile:

@palainp Of course, I really want this setup to work.

AppVM (10.137.0.X) → mirage-fw (10.137.0.Y) ← HardenedBSD (10.137.0.Z)

ping 1.1.1.1 in AppVM (10.137.0.X) - (nat on $NIC_IF inet from xn0:network to any → $NIC_IF)

INF [dispatcher] packet from client 10.137.0.X -> 1.1.1.1
INF [dispatcher] transmit 10.137.0.Y -> 1.1.1.1
WRN [dispatcher] Incorrect source IP 1.1.1.1 in IP packet from 10.137.0.Z (dropping)

Result: ping requests and replies from 10.137.0.Y to one.one.one.one inside xn0

ping 10.137.0.Z (HardeneBSD) in AppVM (10.137.0.X) - (nat on $NIC_IF inet from xn0:network to any → $NIC_IF)

INF [dispatcher] packet from client 10.137.0.X -> 10.137.0.Z
INF [dispatcher] transmit 10.137.0.Y -> 10.137.0.Z
INF [dispatcher] packet from client 10.137.0.Z -> 10.137.0.Y
INF [dispatcher] transmit 10.137.0.Z -> 10.137.0.X

Result: ping requests and replies from 10.137.0.Y to 10.137.0.Z inside xn0

AppVM (10.137.0.X) → mirage-fw (10.137.0.Y) ← OpenBSD (10.137.0.Z)

ping 1.1.1.1 in AppVM (10.137.0.X) - (pass out on $NIC_IF inet from xnf0:network to any nat-to $NIC_IF)

INF [dispatcher] packet from client 10.137.0.X -> 1.1.1.1
INF [dispatcher] transmit 10.137.0.Y -> 1.1.1.1
WRN [dispatcher] Incorrect source IP 1.1.1.1 in IP packet from 10.137.0.X (dropping)

Result: ping requests and replies from 10.137.0.Y to 1.1.1.1 inside xnf0

ping 10.137.0.Z (OpenBSD) in AppVM (10.137.0.X) - (pass out on $NIC_IF inet from xnf0:network to any nat-to $NIC_IF)

INF [dispatcher] packet from client 10.137.0.X -> 10.137.0.Z
INF [dispatcher] transmit 10.137.0.Y -> 10.137.0.Z
INF [dispatcher] packet from client 10.137.0.Z -> 10.137.0.Y
INF [dispatcher] transmit 10.137.0.Z -> 10.137.0.X

Result: ping requests and replies from 10.137.0.Y to 10.137.0.Z inside xnf0

I’m also not able to ping AppVM (10.137.0.X) from OpenBSD/HardenedBSD (10.137.0.Z)
INF [client_eth:dom$XID:10.137.0.Z] who-has 10.137.0.X unknown address; not responding

:frowning:

is it a typo or did you type xn instead of xnf?

the output suggests that sys-net is relaying the data from the appvm to 1.1.1.1 without NAT.

@solene Not a typo, the interface is named xn0 in HardenedBSD and xnf0 in OpenBSD, how can I be sure NAT is being used?

Neat!
I suppose the line from your OpenBSD test:

WRN [dispatcher] Incorrect source IP 1.1.1.1 in IP packet from 10.137.0.X (dropping)

is:

WRN [dispatcher] Incorrect source IP 1.1.1.1 in IP packet from 10.137.0.Z (dropping)

And if that is true I think you moved forward for fixing that issue :slight_smile:

The fw normally doesn’t allow packets with src IP different from the client VM ip, which is not our case anymore in that setup (the sys-net can relay packets from outside, which have by definition another src IP (here 1.1.1.1 and not 10.137.0.Z).
Therefore I added a new commit on the same branch which allow to forward a paquet with a different src IP if it comes from our netvm, would you mind to git pull, rebuild and test again?

EDIT: regarding the ping from sys-net to appVM, it can be a firewall issue from your AppVM which might not respond to ping? (I’ve not done to much with nft, I stil have old vm with iptables for testing and have to add something about that).

1 Like

Just to be sure to understand, does that test ok?
Based on the logs from mirage-fw it should as the message INF [dispatcher] transmit 10.137.0.Z -> 10.137.0.X indicates that a packet is emitted to 10.137.0.X (so the correct side and client) and the src address is 10.137.0.Z (this is the target address for the ping).

Regarding OpenBSD sorry that’s right I indeed made a typo

ping 10.137.0.Z (HardeneBSD) in AppVM (10.137.0.X)
And yes I do receive the packets in AppVM
I’m rebuilding with the new commit and trying again

1 Like

@palainp I’m happy to announce that you just made my week. I’m able to ping 1.1.1.1 from AppVM with HardenedBSD and OpenBSD.
Now I just need to figure out how to NAT the DNS :slight_smile:

1 Like

Nice!
I’ll back port to the PR on github without the log lines (if you have a github nickname I’ll put it in the commit comment). I’m unsure if I’ve done all correctly about the DNS requests, can you check if they are forwarded to your sys-net?