Combination of OpenBSD sys-net + MirageFW

User Support
1

Has someone here ever managed on get these 2 NetVMs to work together? Interested in to do so to have a more hardened system.

2

I tried (following Use OpenBSD as NetVM Ā· Issue #5294 Ā· QubesOS/qubes-issues Ā· GitHub) without luck, openBSD failed to use my network interface :frowning:

BTW if you succeed to run openBSD and use your network card, OpenBSD still lacks netback drivers (Use OpenBSD as NetVM Ā· Issue #5294 Ā· QubesOS/qubes-issues Ā· GitHub). So youā€™ll also have to compile a specific mirage version that ā€œinvertā€ the way netvm is used (manage a dynamic uplink by palainp Ā· Pull Request #178 Ā· mirage/qubes-mirage-firewall Ā· GitHub barely tested as I currently donā€™t have an openBSD sys-net but will be happy to work on again).

3

Iā€™m also interested in something similar to this setup so I gave a try. I used HardenedBSD instead of OpenBSD for sys-net.

My NIC is compatible with BSD and I was able to connect to the internet in sys-net during this test.

I wasnā€™t able to reach the internet in the VMā€™s that had mirage firewall has the netvm :frowning:

@palainp I compiled that specific version of mirage-firewall and set kernelopts to ā€˜ā€“ipv4=$MIRAGEā€™S_QUBES_IP --ipv4-gw=$BSDā€™S_QUBES_IPā€™

sha256sum: 32272239ebe24c23ed0e888395400aeaf09a95e854b6e5ab35af25263b94dcd8

This is my setup (sys-net has mirage-firewall has the netvm and mirage-firewall has no netvm):

defaultrouter=ā€œ$ROUTER_IPā€
ifconfig_$NIC=ā€œinet $LOCAL_IP netmask 255.255.255.0 mtu 1500ā€
ifconfig_xn0=ā€œinet $SYS-NET_QUBES_IP netmask 255.0.0.0 mtu 1500ā€

Iā€™m able to ping sys-net qubeā€™s IP in a VM that has mirage-firewall has netvm but nothing else.

If I try to ping 1.1.1.1 in a VM that has mirage-firewall has the netvm I receive the requests in sys-net in the xn0 interface but sys-net doesnā€™t reply.

Iā€™m not able to ping mirage-firewall from sys-net.
Iā€™m not able to ping VMā€™s that have mirage-firewall has netvm.

How is this supposed to work, is the traffic from my NIC meant to be forwarded to xn0? does multicast relate to any of this?

Iā€™m sorry for the confusion.

4

Thank you for the test bed! :slight_smile:
I think there should be some NAT somewhere between NIC an xn0 (edit: and a default route via NIC)? This way the 1.1.1.1 ping packet will go outside and back to the AppVM via sys-net and fw.

5

Enabling forwarding with (sysctl net.inet.ip.forwarding=1) makes it so sys-net also receives request packets in the NIC interface.

cat /etc/pf.conf
nat pass on $NIC_IF from {xn0:network} to any ā†’ ($NIC_IF)

Makes sys-net receive request and reply packets in both interfaces but the VM doesnā€™t get the response, I donā€™t know where to go from here :frowning:

This when trying to ping 1.1.1.1 because if I try to ping a domain it gets stuck at the xn0 interface.

I donā€™t get how to add a default route via NIC :confused:

@unman mentions a firewall script on his OpenBSD as netvm notes, I wish I could see it.

Any thoughts?

6

Iā€™ll figure it out this weekend and share the solution

7

I was hopeful at first, now my hope is dead.

@palainp You mentioned you tried mirage firewall with Linux instead of BSD in this same scenario, how did you implement NAT?

This was my final setup:

/etc/rc.conf:
pf_enable="YES"
gateway_enable="YES"

defaultrouter="$ROUTER_GATEWAY"

ifconfig_$NIC_IF="inet $LOCAL_IP netmask 255.255.255.0 mtu 1500"
ifconfig_xn0="inet $BSD_QUBES_IP netmask 255.0.0.0 mtu 1500"

# Not Needed
static_arp_pairs="router firewall dns1 dns2"
static_arp_router="$ROUTER_GATEWAY $ROUTER_MAC_ADDRESS"
static_arp_firewall="$MIRAGE_QUBES_IP fe:ff:ff:ff:ff:ff"
static_arp_dns1="10.139.1.1 fe:ff:ff:ff:ff:ff"
static_arp_dns2="10.139.1.2 fe:ff:ff:ff:ff:ff"

/etc/pf.conf:
nat on $NIC_IF inet from xn0:network to any -> $NIC_IF

The final result is the same as before, the rest of what I tried wasnā€™t progress.

Ping 1.1.1.1 from VM that has mirage-fw as NETVM; $MIRAGE_IF gets request and replies but nothing happens in the VM that requested it.

DNS is probably easy to get the same result as the ping request, I just didnā€™t bother to do it yet since it would take me a while and I knew it wouldnā€™t contribute to the wanted result.

My concern is that it isnā€™t a skill issue on my end, and this mirage-fw branch isnā€™t working as intended?

I also tried to use OpenBSD instead of HardenedBSD the result was the same.

8

first, can you ping sys-net from sys-firewall when itā€™s using hardenedbsd?

if itā€™s not working, what does a tcpdump in sys-net show when you try to ping from sys-firewall?

9

Thank for you test, and Iā€™ve not deeply tested that branch, so I may have done somethin wrong. So far I understood your setup is: AppVM ā†’ mirage-fw ā† BSD (which control a NIC connected to LAN)
and with ping 1.1.1.1 from AppVM you got packets in BSD but nothing else. If my understanding is correct having nat between the 2 BSD interfaces and a default route via NIC should resolve the ping request (maybe it worth having tcpdump or wireshark here?)
I failed to have a linux sys-net with a NIC connected to LAN (I lack some Qubes knowledge here) so I only tried to the situation:
AppVM ā†’ mirage-fw ā† linux and where a ping from AppVM is forwarded to the linux VM.

EDIT: Warning my OpenBSD knowledge is inexistant. According to OpenBSD PF: Network Address Translation do you have something like: pass out on $NIC_IF inet from xn0:network to any nat-to $NIC_IP (thatā€™s a very similar syntax as your /etc/pf.conf so Iā€™m not sure what are the differences)?

@solene: as itā€™s a mirage unikernel we canā€™t launch commands from the firewall :frowning:

10

That is the message that leads me thinking nat should be sufficient.

EDIT: and btw I have the same hashsum as you with ./build-with.sh podman.

11

@nalea if you have enough hope, would you mind to try out a new branch where I added some logs:

The hashsum is b2703fd8f4dd7c18af80681aad30f39f22e549571b9777048215b7180be344c7

In my test setup I have:
AppVM (10.137.0.x) ā†’ mirage-fw (10.137.0.y) ā† linux (10.137.0.z)
And with `ping 10.137.0.z) in AppVM I have in the logs:

INF [dispatcher] packet from client 10.137.0.x -> 10.137.0.z
INF [dispatcher] transmit 10.137.0.y -> 10.137.0.z
INF [dispatcher] packet from client 10.137.0.z -> 10.137.0.y
INF [dispatcher] transmit 10.137.0.z -> 10.137.0.x

So the firewall correctly nat from one client to the netvm gateway in both directions.

As I donā€™t have an outgoing interface ping 1.1.1.1 doesnā€™t resolve but is received in 10.137.0.z.

What I have in mind is that the code for forwarding packets from the netvm client may be buggy and these traces would help :slight_smile:

12

@palainp Of course, I really want this setup to work.

AppVM (10.137.0.X) ā†’ mirage-fw (10.137.0.Y) ā† HardenedBSD (10.137.0.Z)

ping 1.1.1.1 in AppVM (10.137.0.X) - (nat on $NIC_IF inet from xn0:network to any ā†’ $NIC_IF)

INF [dispatcher] packet from client 10.137.0.X -> 1.1.1.1
INF [dispatcher] transmit 10.137.0.Y -> 1.1.1.1
WRN [dispatcher] Incorrect source IP 1.1.1.1 in IP packet from 10.137.0.Z (dropping)

Result: ping requests and replies from 10.137.0.Y to one.one.one.one inside xn0

ping 10.137.0.Z (HardeneBSD) in AppVM (10.137.0.X) - (nat on $NIC_IF inet from xn0:network to any ā†’ $NIC_IF)

INF [dispatcher] packet from client 10.137.0.X -> 10.137.0.Z
INF [dispatcher] transmit 10.137.0.Y -> 10.137.0.Z
INF [dispatcher] packet from client 10.137.0.Z -> 10.137.0.Y
INF [dispatcher] transmit 10.137.0.Z -> 10.137.0.X

Result: ping requests and replies from 10.137.0.Y to 10.137.0.Z inside xn0

AppVM (10.137.0.X) ā†’ mirage-fw (10.137.0.Y) ā† OpenBSD (10.137.0.Z)

ping 1.1.1.1 in AppVM (10.137.0.X) - (pass out on $NIC_IF inet from xnf0:network to any nat-to $NIC_IF)

INF [dispatcher] packet from client 10.137.0.X -> 1.1.1.1
INF [dispatcher] transmit 10.137.0.Y -> 1.1.1.1
WRN [dispatcher] Incorrect source IP 1.1.1.1 in IP packet from 10.137.0.X (dropping)

Result: ping requests and replies from 10.137.0.Y to 1.1.1.1 inside xnf0

ping 10.137.0.Z (OpenBSD) in AppVM (10.137.0.X) - (pass out on $NIC_IF inet from xnf0:network to any nat-to $NIC_IF)

INF [dispatcher] packet from client 10.137.0.X -> 10.137.0.Z
INF [dispatcher] transmit 10.137.0.Y -> 10.137.0.Z
INF [dispatcher] packet from client 10.137.0.Z -> 10.137.0.Y
INF [dispatcher] transmit 10.137.0.Z -> 10.137.0.X

Result: ping requests and replies from 10.137.0.Y to 10.137.0.Z inside xnf0

Iā€™m also not able to ping AppVM (10.137.0.X) from OpenBSD/HardenedBSD (10.137.0.Z)
INF [client_eth:dom$XID:10.137.0.Z] who-has 10.137.0.X unknown address; not responding

:frowning:

13

is it a typo or did you type xn instead of xnf?

the output suggests that sys-net is relaying the data from the appvm to 1.1.1.1 without NAT.

14

@solene Not a typo, the interface is named xn0 in HardenedBSD and xnf0 in OpenBSD, how can I be sure NAT is being used?

15

Neat!
I suppose the line from your OpenBSD test:

WRN [dispatcher] Incorrect source IP 1.1.1.1 in IP packet from 10.137.0.X (dropping)

is:

WRN [dispatcher] Incorrect source IP 1.1.1.1 in IP packet from 10.137.0.Z (dropping)

And if that is true I think you moved forward for fixing that issue :slight_smile:

The fw normally doesnā€™t allow packets with src IP different from the client VM ip, which is not our case anymore in that setup (the sys-net can relay packets from outside, which have by definition another src IP (here 1.1.1.1 and not 10.137.0.Z).
Therefore I added a new commit on the same branch which allow to forward a paquet with a different src IP if it comes from our netvm, would you mind to git pull, rebuild and test again?

EDIT: regarding the ping from sys-net to appVM, it can be a firewall issue from your AppVM which might not respond to ping? (Iā€™ve not done to much with nft, I stil have old vm with iptables for testing and have to add something about that).

1 Like
16

Just to be sure to understand, does that test ok?
Based on the logs from mirage-fw it should as the message INF [dispatcher] transmit 10.137.0.Z -> 10.137.0.X indicates that a packet is emitted to 10.137.0.X (so the correct side and client) and the src address is 10.137.0.Z (this is the target address for the ping).

17

Regarding OpenBSD sorry thatā€™s right I indeed made a typo

18

ping 10.137.0.Z (HardeneBSD) in AppVM (10.137.0.X)
And yes I do receive the packets in AppVM
Iā€™m rebuilding with the new commit and trying again

1 Like
19

@palainp Iā€™m happy to announce that you just made my week. Iā€™m able to ping 1.1.1.1 from AppVM with HardenedBSD and OpenBSD.
Now I just need to figure out how to NAT the DNS :slight_smile:

1 Like
20

Nice!
Iā€™ll back port to the PR on github without the log lines (if you have a github nickname Iā€™ll put it in the commit comment). Iā€™m unsure if Iā€™ve done all correctly about the DNS requests, can you check if they are forwarded to your sys-net?