Fortifying sys-net: A Shift to OpenBSD

Intro

Hello guys, this guide will explain how you can replace your existing sys-net with a much more hardened sys-net by using OpenBSD.

OpenBSD provides high-quality driver code, strong security features, good hardware support, low resource use, and a smaller attack surface compared to Linux. Using OpenBSD for sys-net adds important diversity to your network setup. For example, if there’s a major exploit in the Linux distribution, using OpenBSD alongside Linux means that an attacker would need to find two separate vulnerabilities to compromise the system.

Additionally, a standard Fedora template includes many packages that you might not need. While some might suggest using a minimal template, it’s worth asking: why not go further and use a more secure operating system? Choosing OpenBSD not only reduces potential risks but also strengthens the overall security of your network. This way, you can take advantage of a secure OS while avoiding unnecessary vulnerabilities.

Why OpenBSD?

In theory, you could use other BSD distributions like HardenedBSD (which is a fork of FreeBSD which doesn’t do well on security) or NetBSD, but OpenBSD is a better choice because it is security focused. It has well-reviewed driver code, high-quality code, smart ways to prevent attacks, great hardware support, uses fewer resources, and has a smaller attack surface compared to Linux.

Before we begin

:warning: I highly recommend that you familiarize yourself with basic configuration in OpenBSD. You’ll need to set up certain features, such as doas (an alternative to sudo). This will help you understand how it works, as it differs significantly from Linux.

There are some (outdated) guides available on the forum and GitHub, but they are unclear and miss important details and updates.

I’d like to give a special thanks to @palainp and @solene. They have been very helpful in setting this up, they supported me through many issues during the setup of this.

@palainp is one of the maintainers of the Qubes OS Mirage firewall, and I will create a separate guide on how to use it with for general use. He has helped me with setting up the OpenBSD sys-net and Mirage Firewall.

@solene is one of the document maintainers of openbsd and has helped me stricting the rulesets for the openbsd firewall.

This is one of my first guides, so please don’t hold back on your feedback. I’m still in the learning process of both creating guides and setting up something this advanced.

Setup the Qube

I used the following command to create the OpenBSD Qube as a standalone VM. While it’s possible to set it up as a template VM and create AppVMs from it, we will focus on the standalone VM for now.

qvm-create \
<name it as you want i'm using sys-net-openbsd) \
--standalone \
--property virt_mode=HVM \
--property kernel='' \
--label (give it a color as you want) \

After you have created the Qube go to the ‘Qube Manager’ and edit the following settings:

  • Give it 30 gb of system storage
  • Disable ‘Include in memory balancing’
  • Set default disposable template to ‘(none)’
  • Enable ‘Provides network’
  • Set the Net qube to sys-firewall

:warning: Please do not delete your existing sys-net just yet.

Installation of OpenBSD

Launch a disposable VM and download your OpenBSD ISO file from this link:
https://www.openbsd.org/faq/faq4.html#Download

Install the install76.iso for amd64.

Once you’ve downloaded the ISO, be sure to verify it using the SHA256 file, which is also available from the same link.

Booting up

It’s important to mention that OpenBSD does not include certain network firmware in the installation ISO due to copyright and licensing restrictions. As a result, there are two ways to set up OpenBSD, and I will cover both methods.

First, we need to check if you need to install any firmware manually. To do this, attach your Ethernet or network controller to your standalone OpenBSD using the ‘Qube Manager’ in the ‘Devices’ tab. Then, return to the ‘Advanced’ tab and click on ‘Boot qube from CD-ROM.’ After that, choose ‘from file in qube,’ locate your disposable VM, and click on the ISO file.

:information_source: Before you boot, make sure to shut down your original sys-net. You cannot boot OpenBSD if the controllers are attached while sys-net is still running.

Quote from OpenBSD:

For licensing reasons, some firmware cannot be directly distributed with OpenBSD. The fw_update tool will automatically download and install any missing firmware, but this requires a working internet connection.

Setting up OpenBSD

Once you’ve successfully booted up, take a close look at the boot logs. If you see something like what’s in the screenshot, you likely need a firmware. The error will show up after everything has fully loaded and you reach the setup screen. Please follow Setup Guide 2.
image

If you don’t see the same error or if you’re using an Ethernet cable for your internet connection, please follow Setup Guide 1.

Setup guide 1

Follow the setup guide after you have succesfully booted up.

  • Press I.
  • Select your keyboard layout.
  • Choose your system hostname.
  • Use the xnf0 network interface.
  • Enter the IPv4 address provided by QubesOS, for example, 10.137.x.x (you can also find this in the ‘Qube Manager’).
  • Set the netmask to 255.0.0.0.
  • Skip IPv6.
  • Configure additional interface with DHCP from your attach network controller for example em0
  • Enter the gateway listed in the ‘Qube Manager.’
  • Enter the DNS nameservers listed in the ‘Qube Manager.’
  • Setup root password
  • Don’t start sshd by default
  • Go with the defaults for the next questions about X Window System
  • Setup a user
  • Go with the disk sd0
  • Don’t encrypt the disk
  • Go with MBR
  • auto layout for the disk setup
  • Follow the instructions to install the sets it’s on cd0
  • You don’t need game76.tgz and x.tgz packages deselect them with -game* and -x*
  • You can proceed without verification, but if you want, you can add the sha256.sig file to the ISO using Isomaster after downloading it from the OpenBSD site. However, we won’t cover that in this guide, so just choose ‘yes’ for now.
  • Press done.
Setup guide 2

You need to install your firmware.

Access the shell by pressing S and enter halt -p. Once OpenBSD has shut down, return to the Qube Manager and go to the Devices tab. Remove the attached controller and ensure that your sys-firewall is still set as the net qube for OpenBSD. After that, boot back up (refer to the ‘booting up’ section if you need help).

Follow the setup guide after you have succesfully booted up.

  • Press I.
  • Select your keyboard layout.
  • Choose your system hostname.
  • Use the xnf0 network interface.
  • Enter the IPv4 address provided by QubesOS, for example, 10.137.x.x (you can also find this in the ‘Qube Manager’).
  • Set the netmask to 255.0.0.0.
  • Skip IPv6.
  • There’s no need to configure additional network interfaces; just enter ‘done.’
  • Enter the gateway listed in the ‘Qube Manager.’
  • Enter the DNS nameservers listed in the ‘Qube Manager.’
  • Setup root password
  • Don’t start sshd by default
  • Go with the defaults for the next questions about X Window System
  • Setup a user
  • Go with the disk sd0
  • Don’t encrypt the disk
  • Go with MBR
  • auto layout for the disk setup
  • Follow the instructions to install the sets it’s on cd0
  • You don’t need game76.tgz and x.tgz packages deselect them with -game* and -x*
  • You can proceed without verification, but if you want, you can add the sha256.sig file to the ISO using Isomaster after downloading it from the OpenBSD site. However, we won’t cover that in this guide, so just choose ‘yes’ for now.
  • Press done.

After rebooting and logging in, look up how to install the firmware for your network controller it should be straightforward. You can use the fw_update command to install the firmware for your network controller. Once you’ve done that, reattach the controller to your standalone OpenBSD and boot up again. Then, set up your controller. Some guides which should help.

OpenBSD FAQ: Installation Guide
OpenBSD FAQ: Networking

To check your network interfaces, run the command ifconfig. You should see several interfaces listed. The ones that we are interest in are xnf0, which is the virtual interface attached from sys-firewall that you configured during installation, and your network controller, such as iwn0. Make sure both interfaces are functioning properly by pinging 1.1.1.1 for xnf0 If the output is too long, you can redirect it to a file for easier viewing by using the command ifconfig > test This will allow you to scroll through the output more conveniently.

Network setup

We need to configure the network.

Set your net Qube to use sys-firewall as followed:
Set sys-firewall as n/a (no net qube)
Set sys-net-openbsd as sys-firewall
Set the APPvm as sys-firewall

sys-net-openbsd ← sys-firewall → appvm

Next, copy the sysctl.conf file from the /etc/examples directory:

cp /etc/examples/sysctl.conf /etc/

Then, edit the sysctl.conf file using vi and uncomment the following line:

net.inet.ip.forwarding=1

After making these changes, reboot your OpenBSD system.

Next, you will need to set up the firewall within OpenBSD using Packet Filter (PF). Open the /etc/pf.conf file with vi and make sure it looks like this:

Replace em0 with your network controller (e.g., iwm0) and change 9.9.9.9 to your preferred DNS nameserver.

The firewall rules are a work in progress and are not yet fully strict.

# $OpenBSD: pf.conf, v 1.5.5 2017/12/03 20:40:04 sthen Exp $
#
# See pf.conf (5) and /etc/examples/pf.conf

set skip on lo

match in proto { udp tcp } from xnf0:network to any port domain rdr-to 9.9.9.9 port domain
pass in proto { udp tcp } from xnf0:network to any port domain rdr-to 9.9.9.9 port domain
match out on em0 inet from xnf0:network to any nat-to em0
pass out on em0 inet from xnf0:network to any nat-to (em0)

block return  # block stateless traffic
pass # establish keep-state

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild

Apply the changes with pfctl -f /etc/pf.conf

Firewall Configuration

The sys-firewall requires specific configuration to effectively route traffic between Qubes. Unfortunately, I did not test this with sys-firewall. Instead, I went with qubes-mirage-firewall, which is relatively easy to set up. If anyone has successfully configured sys-firewall, please share your method so we can consider adding it to the guide.

Setup instructions from qubes-mirage-firewall on their GitHub

Configure firewall with OpenBSD-like netvm

OpenBSD is currently unable to be used as netvm, so if you want to use a BSD as your sys-net VM, you’ll need to set its netvm to qubes-mirage-firewall (see #146 for more information). That means you’ll have AppVMs -> qubes-mirage-firewall <- OpenBSD with the arrow standing for the netvm property setting.

In that case you’ll have to tell qubes-mirage-firewall which AppVM client should be used as uplink:

qvm-prefs --set mirage-firewall -- kernelopts '--ipv4=X.X.X.X --ipv4-gw=Y.Y.Y.Y'

with X.X.X.X the IP address for mirage-firewall and Y.Y.Y.Y the IP address of your OpenBSD HVM.

Here are some resources for setting up sys-firewall and routing traffic successfully. However, please note that these resources may be outdated and likely require some rework:

Firewall | Qubes OS (not outdated)
GitHub · Where software is built (outdated)

Your hardenend sys-net should now be functional! Test it by configuring sys-firewall as your net Qube and try pinging or browsing the internet.

8 Likes

Thank you for the guide, I’ll be able to setup an OpenBSD to enhance the firewall rules as we discussed earlier.

However, I have some nitpicks about the intro.

Can you explain? This statement is wrong to me.

I disagree, except if we talk about inbound connections which does not really make sense on Qubes OS.

This can be fixed.

this is far stretched

no

5 Likes

To me, the value of using OpenBSD as sys-net is to provide some diversity in the network chain. For instance, if there is a huge exploit in the Linux distro used for both sys-net and sys-firewall, the same exploit will work twice.

If you put a different product with different vulnerabilities, like OpenBSD + linux, you need to have two very bad vulnerabilities simultaneously.

An alternative could be to use OpenBSD as sys-firewall, but this would require another linux sys-firewall below to handle the firewall rules of the qubes (which are applied on their netvm), which is not a security issue because diversity was introduced above in the chain. An advantage of this setup is you don’t need to have your network interfaces compatible with OpenBSD.

8 Likes

I agree that OS diversity is the key point here, and that’s also why having a mirage unikernel in the chain would help :slight_smile:

Regarding the implementation of OpenBSD as a sys-firewall, in addition to the firewall rules, it seems that the BSD family does not have the Xen driver to provide network (but I don’t have pointers to show that, that’s what I’ve read some times ago), and so OpenBSD is used as a sys-net and its netvm points to the firewall.

1 Like

It’s just the Xen integration is lacking, there is no vchan support or paravirtualized kernels (NetBSD might have it, it has been a first class Xen dom0/domU 10 years ago). OpenBSD uses xnf driver for virtual interfaces, xnf stands for “Xen NetFront”, xnf(4) - OpenBSD manual pages

if you use OpenBSD as sys-net, it can’t have a netvm, I don’t understand the sentence

You’re right, sorry for my imprecision. I was thinking of backend drivers, not frontend drivers :slight_smile:

The current configuration with OpenBSD as sys-net, is to act as a client, so with the netvm property set (xnf connected to firewall, wired/wifi interface connected outside). This is due to the lack of backdrivers to provide the network. Therefore, without these drivers, OpenBSD cannot be inside the VM chain.

Thank you for your detailed feedback. I appreciate you taking the time to point out these issues in my guide.

You’re right, and I see now that some of my statements were inaccurate or exaggerated. I’ll revise the guide to address these points. I want to add that I’ve gathered what other people had said, and have not investigated thoroughly about what benefits OpenBSD provides in this scenario. Your comment about the value of using OpenBSD as sys-net to provide diversity in the network chain is actually the key point, and I should’ve not put claims without researching well. I made the guide in 2 hours, which should say a lot about its depth and accuracy.

My intention was to highlight potential areas for improvement, but I see now that some claims were misleading. I’ll update the guide to provide more accurate information. Thank you again for helping me improve the accuracy of this information.

I don’t understand, it should be able to deal with 2 xnf interfaces when using as sys-firewall?

I think so, but I don’t know how to create the second interface (I never tried, I always used the automagic creation with netvm). And the other two sides of these interfaces need to be linux/mirage guests. But I might be totally wrong here, I’m far from expert.

If Qubes OS provides a virtual network interface when the qube is set as a netvm in sys-net, and provides a network interface when not used as a netvm, it should provide 2 network interfaces when used as a netvm but having a netvm. It should work, I see no reasons it should not.

Using OpenBSD as a direct sys-firewall replacement might not work because sys-firewall has a lot of vif interfaces (all logical interfaces within the qube) created by Qubes OS tooling. So, keeping a linux sys-firewall is still mandatory due to this, but like this [Linux sys-net] -> [OpenBSD firewall] -> [sys-firewall]

I’ve been using OpenBSD as sys-net for some time and have posted about
it in the past.
I’ve always found use as a separate firewall painful and unnecessary,
but ymmv.
What’s missing from the original guide are instructions on passing
traffic between qubes and the BSD sys-net - documented here and also
it should make clear that when properly implemented the Qubes firewall
functions as expected.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

2 Likes

Is there a guide for proper implementation by any chance?

Updated the guide

  • Rewrote the intro with correct facts
  • Added a section about Firewall Configuration
1 Like

This seems to be very feasable. Do you think that using this guideline I can try OPNsense as well? I know that OPNsense is based on freeBSD, but I dunno how different openBSD is from FreeBSD…

OpenBSD hardware compatibility is better for wifi interfaces, hardening works out of the box, there is nothing to tweak.

FreeBSD can be tweaked in many ways but it’s a full time job to figure how, and wifi support is really bad.

There is a “fork” named HardenedBSD that tries to keep in sync with FreeBSD but with security features, it used to be the base system for OPNsense but the people doing OPNsense decided out of the blue to switch to FreeBSD a while ago.

When I use OpenBSD as sys-net replacement with the physical network interfaces attached and “provides network” checked, it has no xnf0 interface, any idea?

To me that’s because OpenBSD lacks the backend drivers and can’t provides network. This is also why the netvm property is set to point to the firewall and is considered a client by the firewall, which is responsible for forwarding packets between clients and one special of them, the default gateway.

1 Like

Some notes:

  • I recommend to set up OpenBSD without the network interfaces to download ALL firmwares with this (as root):
cd /root/
mkdir firmware
cd firmware
pkg_add wget
wget --continue --accept "*.tgz" --no-directories --no-parent --recursive http://firmware.openbsd.org/firmware/$(uname -r)/
pkg_delete wget
pkg_delete -a
reboot
  • you can find your network interfaces names like this
ifconfig | grep mtu

xnf0 is the virtual interface of qubes os, lo0 is loopback and pflog0 is for logging packets. The other are the physical interfaces.

  • to install the download firmwares, once you booted with the physical interfaces

As root, go to /root/firmware and run fw_update $file where $file is a firmware file in the directory that starts like a physical interface. For instance, I have iwx0 and there is a firmware file iwx-firmware-20240513p0.tgz.

  • if you have a wireless and ethernet interfaces, you can aggregate them in a trunk failover to automatically switch from one to the other on the same network, this allows unplugging your computer and use the wifi with no interruption.

https://dataswamp.org/~solene/2018-08-30-openbsd-trunk.html

1 Like

Dont do this. The OpenBSD sys-net is not providing network in Qubes
terms. It is attached to sys-firewall, and sys-firewall provides network
to the BSD sys-net and all other qubes by passing traffic between them.
All of these are downstream of sys-firewall in Qubes networking terms.

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

5 Likes

I came across @unman’s guide half a decade ago (phrased as such to highlight just how quickly time flies) but I’ve never been able to muster the courage to try it since I had my hands full with just Linux and the guide just isn’t layman-friendly, to put it mildly. I feared I would end up setting myself and/or my house on fire.

Your is far more accessible and actually gives me a fighting chance at doing this without getting firemen involved. Thank you!

1 Like