So I was following these great instructions to the letter (my goal was to first set this up with the normal sys-firewall, then go and test a particular build of mirage-firewall that’s supposed to work in such a setup):
opened 03:53PM - 05 Oct 21 UTC
Qubes OS's implementation of sys-net is limiting as an internet facing router. S… ince Qubes OS (Qubes) is a virtualization platform, it is reasonable to implement a more featured OS on the external interface instead.
This guide will get you an instance of OpenBSD that services the hardware Ethernet interface. You can then choose what functionality you want to install, such as a hardened router, a firewall, IDPS, etc.
Credit to notes **[1]** and comments **[2]** from @unman.
I highly recommend following these steps in the order presented. (See discussion at bottom for details.):
1. Preparations:
a. Shutdown all VMs.
b. Backup dom0, sys-net, and sys-firewall.
c. All steps can be performed in the GUI except:
-- i. Step 5, creating the openbsd-21 HVM TemplateVM, and
-- ii. Step 10, running the script in sys-firewall.
2. In the sys-firewall VM change these settings:
a. Confirm "Provides network" is set to "True".
b. Set NetVM, aka "Networking", to none.
c. Add in "Services" the following:
-- i. "clocksync" daemon.
-- ii. "qubes-update-check".
-- iii. "qubes-updates-proxy".
-- iv. "qubes-yum-proxy".
3. In the sys-net VM settings:
a. Remove all hardware devices.
b. Confirm "Provides network" is "True".
c. Set NetVM, aka "Networking", to "sys-firewall".
d. Change virtualization mode to "PVH".
e. Change the following "Services":
-- i. Remove "clock-sync".
-- ii. Add "qubes-yum-proxy".
-- iii. Add "qubes-updates-proxy".
-- iv. Add "qubes-update-check".
4. Start the sys-firewall VM.
5. Create a HVM as a class of TemplateVM **[3]** **[4]** with these settings:
a. Keeping the naming convention, my example uses the name "openbsd-21".
b. dom0 command line (note that the kernel property is set to two single tick marks, aka, "none"):
`qvm-create openbsd-21 --class TemplateVM --property virt_mode=HVM --property kernel='' --label purple`
c. Set NetVM, aka "Networking", to "sys-firewall".
d. Set "Start qube automatically on boot" to "False".
e. Add network hardware "Devices". (For me it's 00:19.0 Ethernet controller: Intel Corp...)
f. Confirm the kernel is set to "none".
6. Create a new HVM with class of AppVM using the openbsd-21 HVM TemplateVM in Step 5 above with these settings:
a. Name "openbsd-21-sysnet".
b. Set NetVM, aka "Networking", to "sys-firewall".
c. Set "Start qube automatically on boot" to "False".
d. Add network hardware "Devices". (For me it's 00:19.0 Ethernet controller: Intel Corp...)
7. Note the static IP assignment to "openbsd-21-sysnet" by Qubes.
a. From Qubes Manager find the IP address assigned to the "openbsd-21-sysnet" VM.
-- i. For me, it is 10.137.0.16
b. If you have any trouble with this step then try this:
-- i. Reboot Qubes OS.
-- ii. Start "openbsd-21-sysnet" VM.
-- iii. Let it run until it at least identifies the Xen virtual ethernet interface "xn0".
-- iv. Stop "openbsd-21-sysnet" VM.
8. Install OpenBSD in the "openbsd-21" HVM TemplateVM.
a. Assign interfaces. Your LAN interface to sys-firewall is going to be xn? For me, LAN is xn0 and WAN is em0.
b. Assign the static IP address from Step 7 above. For me it is 10.137.0.16 to the "xn0" interface.
c. Change root password.
d. Shutdown the "openbsd-21" TemplateVM.
9. Now start "openbsd-21-sysnet" and confirm the above settings copied from the parent "openbsd-21" TemplateVM:
a. Interface assignments. (Again, my LAN is xn0 and WAN is em0).
b. Interfaces' IP addresses.
-- i. For me, WAN is set by DHCP and the LAN static is 10.137.0.16/24.
-- ii. Note that this LAN static IP address was set by Qubes when openbsd-21-sysnet was instantiated in Step 7.
c. For some reason I had to repeat the OpenBSD configuration in "openbsd-21" TemplateVM twice to make the root password change.
10. In "sys-firewall", create a script to route traffic to "openbsd-21-sysnet":
a. The script needs to set the gateway, allow other VMs access, and identify the DNS server. For me it is like this:
```
#!/usr/bin/sh
#
# Logging Function
sysidps_log () { echo -e "$(date "+%F %T")" "$1" | tee -a >&1 "$log_file"; }
log_msg_fail="Timed out waiting on gateway."
log_msg_chk="Checking gateway for ICMP echo reply..."
log_msg_up="Gateway replying to ping requests!"
log_msg_rte="Gateway set as the default route."
log_msg_gw="Virtual ifaces routing through gateway."
log_msg_ns="Name servers added to routing table."
log_file="/var/log/qubes/sys-idps"
#
# Hard code static IP addresses. You'll need at least one name server.
gw_ip=10.137.0.14
#ns1_ip=10.137.252.1
ns2_ip=10.137.1.1
#
# Wait for gateway to come up.
# Maximum wait time is 8 hrs before exits with failure.
# Your Gateway Must Reply To (Does Not Block) ICMP Echo Requests!
up_reply="0% packet loss,"
ping_chk="false"
declare -i time_out=0
time_base=$(date "+%s")
while [ "$ping_chk" != "$up_reply" ]; do
if [ $time_out -lt 300 ] ; then sleep 1 ;
elif [ $time_out -lt 600 ] ; then sleep 3 ;
elif [ $time_out -lt 1200 ] ; then sleep 5 ;
elif [ $time_out -lt 1800 ] ; then sleep 8 ;
elif [ $time_out -lt 3600 ] ; then sleep 15 ;
elif [ $time_out -lt 14400 ] ; then sleep 30 ;
elif [ $time_out -lt 28800 ] ; then sleep 60 ;
else sysidps_log "$log_msg_fail" && exit 1 ;
fi;
sysidps_log "$log_msg_chk";
ping_chk=$(ping -c 1 -r $gw_ip | grep "0% packet loss" | cut -d ' ' -f 6-8);
time_out=$(date "+%s")-$time_base;
done
sysidps_log "$log_msg_up"
#
# Set gateway as default route.
# Running "ip route add" first makes discovering gw_vif easy.
ip route add default via $gw_ip
gw_vif=$(ip route show default | cut -d ' ' -f 5)
sysidps_log "$log_msg_rte"
#
# Add all VMs vif routing to/from gateway.
iptables --insert FORWARD --in-interface vif+ --out-interface $gw_vif --jump ACCEPT
iptables -t raw --insert PREROUTING --in-interface $gw_vif --jump ACCEPT
sysidps_log "$log_msg_gw"
#
# Add routing to name server resolution. Call Qubes name server script.
rm -f /etc/resolv.conf
if [ "X$ns1_ip" != "X" ] ; then echo "nameserver $ns1_ip" >> /etc/resolv.conf ; fi
if [ "X$ns2_ip" != "X" ] ; then echo "nameserver $ns2_ip" >> /etc/resolv.conf ; fi
/usr/lib/qubes/qubes-setup-dnat-to-ns
sysidps_log "$log_msg_ns"
#
exit 0
```
11. You could make this a background process in /rw/config/rc.local. If so, then set openbsd-21-sysnet to "Start qube automatically at boot". I sometimes tail /var/log/qubes/sys-idps.
a. For me, it looks like this:
```
# Poll for and then set up system gateway:
/rw/config/qubes-openbsd-sysnet-script &
```
12. Set the "fedora-32-dvm" "NetVM" to "sys-net".
a. This works around Qubes' 10.138.0.0/24 networking on disposable VMs.
b. Repeat for all your DisposableVMs.
c. Note: In Step 3 the "sys-net" Networking was set to "sys-firewall".
13. Thoughts and concerns:
a. You have to manually manage a few of things:
-- i. Match the LAN static IP address for "openbsd-21-sysnet" to what Qubes auto-assigns; conceivably, your actions could inadvertently change it.
-- ii. Setting your static IP values used in the script.
b. This requires editing the "openbsd-21" TemplateVM to make changes that persist in "openbsd-21-sysnet".
-- i. My OpenBSD instance in the "openbsd-21" didn't save my root account changes until the third try. Just triple check that all your changes were saved if your "openbsd-21-sysnet" AppVM has issues.
c. Several steps given above are only cautionary, or good practice.
d. A couple of things that Qubes hard codes:
-- i. TemplateVM updates to proxy via "sys-net". So I had to enable some services on "sys-net".
-- ii. Disposable templates route through "sys-net". So, "sys-net" here forwards to "sys-firewall".
e. During initial testing occasionally the hypervisor backend wouldn't release the PCI ethernet controller. Hence, Step 7.b. I haven't has any trouble since.
f. This is a working integration guide and shell script. Edits are welcomed.
[1] https://github.com/unman/notes/blob/master/openBSD_as_netvm
[2] https://groups.google.com/g/qubes-users/c/MpXLhz5COvM/m/smHA6a0VBgAJ
[3] https://www.qubes-os.org/doc/standalones-and-hvms/
[4] https://dev.qubes-os.org/projects/core-admin-client/en/latest/manpages/qvm-create.html
For me the problem happens when I’m installing OpenBSD at these steps:
Install OpenBSD in the “openbsd-21” HVM TemplateVM.
I can set the static IP address shown in the Qubes Manager to xnf0, but setting up em0 ethernet (or leaving it by default) doesn’t seem to work. I get this error that keeps popping throughout the install and after when I set it to autoconfig
em0: watchdog: head 4 tail 0 TDH 4 TDT 4
Would appreciate any help in resolving this issue.
1 Like
Maybe try following the instructions for OpenBSD Use OpenBSD as NetVM · Issue #5294 · QubesOS/qubes-issues · GitHub
Hardened BSD is a FreeBSD distribution, unrelated to OpenBSD.
The instructions in Use OpenBSD as NetVM · Issue #5294 · QubesOS/qubes-issues · GitHub are exactly the same as the ones I linked above. And I used OpenBSD, not Hardened BSD.
Oh ok, the issue title is wrong then.
1 Like