[4.2.3] sys-vpn: rc.local not called anymore?

[juju42]

I migrated from R4.1 to 4.2.3 recently.
I reviewed my firewall configuration and converted to nftables (adapted from the new nftables one. see below).
The only issue left seems that /rw/config/rc.local is not executed at start and when I reboot dom0, I have to re-executed it manually.
rc.local does not match anything in /etc/
and `/rw/config/rc.local` doesn't seem to be working seems to go in same direction

service is present but dead as depending on missing file

[user@VPN ~]$ systemctl status rc-local.service
○ rc-local.service - /etc/rc.d/rc.local Compatibility
     Loaded: loaded (/usr/lib/systemd/system/rc-local.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
       Docs: man:systemd-rc-local-generator(8)
[user@VPN ~]$ grep ExecStart /usr/lib/systemd/system/rc-local.service
ExecStart=/etc/rc.d/rc.local start
[user@VPN ~]$ ls -l /etc/rc.d/rc.local
ls: cannot access '/etc/rc.d/rc.local': No such file or directory

but still plenty of references in Code search results · GitHub

Nftables vpn

(without any masquerading)

# Qubes R4.2, VPN - WIP
# https://forum.qubes-os.org/t/wireguard-vpn-setup/19141
# https://forum.qubes-os.org/t/configuring-a-proxyvm-vpn-gateway/19061
# https://github.com/QubesOS/qubes-core-agent-linux/blob/main/network/qubes-ipv4.nft

#flush ruleset
#flush table tbl_x

# removing masquerading
flush chain qubes postrouting
# review general filter rules
flush table ip filter

table ip qubes {

	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		oifgroup 2 accept
		oif "lo" accept
		# no masquerading for vpn
		# masquerade
	}

	# Prevent the qube to forward traffic outside of the VPN
	# https://forum.qubes-os.org/t/wireguard-vpn-setup/19141#p-86722-killswitch-12
	chain custom-forward {
		oifname eth0 counter drop
	}
}
table ip6 qubes {

	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		oifgroup 2 accept
		oif "lo" accept
		# masquerade
	}

	# Prevent the qube to forward traffic outside of the VPN
	chain custom-forward {
		oifname eth0 counter drop
	}
}

table ip filter {
	chain FORWARD {
		type filter hook forward priority filter; policy accept;
		ct state related,established counter packets 0 bytes 0 accept
		ip daddr 10.252.116.5 oifname "tun0" udp dport 53 counter packets 0 bytes 0 accept
		ip daddr 10.252.116.5 oifname "tun0" tcp dport 53 counter packets 0 bytes 0 accept
		iifname "tun0" counter packets 0 bytes 0 accept
		oifname "tun0" counter packets 0 bytes 0 accept
		iifname "eth0" counter packets 0 bytes 0 drop
		oifname "eth0" counter packets 0 bytes 0 drop
	}

	chain OUTPUT {
		# type filter hook output priority filter; policy drop;
		type filter hook output priority filter; policy accept;
		ip daddr 158.69.212.202 udp dport 1194 counter packets 0 bytes 0 accept
		oifname "eth0" skgid 979 counter packets 0 bytes 0 accept
		# allow VPN all, not just icmp
		oifname "tun0" ip protocol icmp counter packets 0 bytes 0 accept
		# oifname "tun0" ip protocol tcp dport 22 counter packets 0 bytes 0 accept  # NOK
		oifname "tun0" ip protocol tcp counter packets 0 bytes 0 accept
		oifname "tun0" ip protocol udp counter packets 0 bytes 0 accept
		iifname "tun0" ip protocol icmp counter packets 0 bytes 0 accept
		iifname "tun0" ip protocol tcp counter packets 0 bytes 0 accept
		iifname "tun0" ip protocol udp counter packets 0 bytes 0 accept
		oifname "lo" counter packets 0 bytes 0 accept
		counter packets 0 bytes 0
	}

	chain INPUT {
		type filter hook input priority filter; policy accept;
		ip saddr 10.8.0.1 iifname "tun0" ip protocol icmp counter packets 0 bytes 0 accept
		ip saddr 10.8.0.0/24 iifname "tun0" ip protocol icmp counter packets 0 bytes 0 accept
		ip saddr 10.137.0.0/24 oifname "tun0" ip protocol icmp counter packets 0 bytes 0 accept
		ip saddr 10.137.0.0/24 oifname "tun0" ip protocol tcp counter packets 0 bytes 0 accept
		ip saddr 10.137.0.0/24 oifname "tun0" ip protocol udp counter packets 0 bytes 0 accept
	}
}

[DVM]

/rw/config/rc.local is run by the qubes-misc-post systemd service.
You can try to check the status of the service and see if it shows any errors with sudo journalctl -u qubes-misc-post.

For firewall rules, you should try to use /rw/config/qubes-firewall-user-script instead.

[juju42]

@DVM

sudo journalctl -u qubes-misc-post.

Ah. Thanks.
This point to a permission denied for an openvpn file “openvpn[1142]: Options error: --ca fails with ‘ca.crt’: Permission denied (errno=13)”.
Strangely no issue with same config in interactive. and file is in 0644… directories are ok too. can cat the file as user nobody. Systemd unit does not list any restrictions or specific user. Don’t know from where does this come from.

[DVM]

Are you by any chance using fedora as your base system for your vpn qube? It could be a selinux permission issue.

[juju42]

it is a fedora. always selinux but not sure why change from past fedora38 template to 40.

from ls -Z /rw/config/vpn/, I get two different permission set system_u:object_r:initrc_exec_t:s0 or unconfined_u:object_r:etc_t:s0. No idea where the initrc_exec comes from…
matchpathcon -V DIR seems to say initrc_exec should be used.
restorecon -v FILE puts everything under initrc_exec except ovpn config file which is unconfined_u:object_r:initrc_exec_t. still not working.

if forcing sudo chcon -u system_u -t initrc_exec_t FILE.ovpn… still get error
“VPN misc-post.sh[1202]: Options error: In [CMD-LINE]:1: Error opening configuration file: FILE.ovpn”

if sudo chcon -u system_u -t etc_t FILE.ovpn (type that I see for other files in same directory), same

if sudo chcon -u unconfined_u -t etc_t FILE.{ovpn,key,crt,ca}, fails on “openvpn[1153]: WARNING: Failed running command (–up/–down): could not execute external program”

[user@VPN ~]$ sudo journalctl -xe -l --no-pager  | grep -i denied
Dec 01 15:24:38 VPN audit[1222]: AVC avc:  denied  { getattr } for  pid=1222 comm="qubes-vpn-handl" path="/usr/sbin/nft" dev="xvda3" ino=285476 scontext=system_u:system_r:openvpn_t:s0 tcontext=unconfined_u:object_r:iptables_exec_t:s0 tclass=file permissive=0
[user@VPN ~]$ sudo journalctl -xe -l --no-pager  | grep -i denied | audit2allow
#============= openvpn_t ==============
allow openvpn_t iptables_exec_t:file getattr;
[user@VPN ~]$ sudo grep -rin openvpn_t /etc/selinux/
/usr/bin/grep: /etc/selinux/targeted/policy/policy.33: binary file matches

from sudo semanage fcontext -l | grep -e vpn, script should be system_u:object_r:openvpn_unconfined_script_exec_t:s0 and config system_u:object_r:openvpn_etc_t:s0 but getting same issue than above with nft.

any default policy doing both openvpn and iptables/nft?

[DVM]

I saw “qubes-vpn-handler” in your message which should come from GitHub - tasket/Qubes-vpn-support: VPN configuration in Qubes OS so I tried it (from this pull request for nftables support) with a fresh fedora 40 template and had some problems with selinux as expected.
This is what I did to get my openvpn connection to work (example using mullvad openvpn config):

# Convert configs to system_u:openvpn_etc_t (inside /rw/config/vpn/)
sudo chcon -u system_u -t openvpn_etc_t mullvad_ca.crt mullvad_userpass.txt update-resolv-conf vpn-client.conf

# Convert qubes-vpn-support scripts to system_u:bin_t (inside /rw/config/)
sudo chcon -u system_u -t bin_t qubes-vpn-ns qubes-vpn-openvpn-script qubes-vpn-setup

# Convert qubes-vpn-support systemd service to system_u:systemd_unit_file_t (inside /rw/config/)
sudo chcon -R -u system_u -t systemd_unit_file_t qubes-vpn-handler.service qubes-vpn-handler.service.d

After rebooting the qube, my openvpn configuration was up and my connected qubes could access the Internet using the VPN.

[juju42]

Thanks a lot @DVM !
Not using Qubes-vpn-support. Setup likely predates those.

sudo chcon -u system_u -t openvpn_etc_t /rw/config/vpn/{CONFIG.{ovpn,key,crt},ca.crt}
sudo chcon -u system_u -t bin_t  /rw/config/qubes-vpn-handler.sh /rw/config/vpn/qubes-vpn-handler.sh

(no systemd unit files)
=> reboot, still not starting

$ sudo journalctl -xe -l --no-pager  | grep -i denied 
Dec 01 20:31:25 VPN audit[1227]: AVC avc:  denied  { getattr } for  pid=1227 comm="qubes-vpn-handl" path="/usr/sbin/nft" dev="xvda3" ino=285476 scontext=system_u:system_r:openvpn_t:s0 tcontext=unconfined_u:object_r:iptables_exec_t:s0 tclass=file permissive=0
Dec 01 20:31:25 VPN audit[1227]: AVC avc:  denied  { getattr } for  pid=1227 comm="qubes-vpn-handl" path="/usr/sbin/nft" dev="xvda3" ino=285476 scontext=system_u:system_r:openvpn_t:s0 tcontext=unconfined_u:object_r:iptables_exec_t:s0 tclass=file permissive=0

also tried

$ sudo chcon -u system_u -t iptables_exec_t  /rw/config/qubes-vpn-handler.sh /rw/config/vpn/qubes-vpn-handler.sh
[reboot, nok]
$ sudo journalctl -xe -l --no-pager  | grep -i denied 
Dec 01 20:36:19 VPN audit[1210]: AVC avc:  denied  { execute } for  pid=1210 comm="openvpn" name="qubes-vpn-handler.sh" dev="xvdb" ino=275 scontext=system_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:iptables_exec_t:s0 tclass=file permissive=0
Dec 01 20:36:19 VPN misc-post.sh[1210]: Options error: --up script fails with 'qubes-vpn-handler.sh': Permission denied (errno=13)
$ sudo journalctl -xe -l --no-pager  | grep -i denied | audit2allow


#============= openvpn_t ==============
allow openvpn_t iptables_exec_t:file execute;

Also, try with selinux custom module but it does not persist after reboot

audit2allow -M myopenvpn 
sudo semodule -i myopenvpn.pp

[DVM]

@juju42

Also, try with selinux custom module but it does not persist after reboot

You can try 2 things:

• Move and install the file in the template
• Move the file to /home or /rw to make the file persistent within the vpn qube and put the install command in rc.local before calling the vpn/firewall commands.

[juju42]

Thanks again for the follow-up.
I did multiple try and update to get to following selinux policy

$ cat myopenvpn.te 
# pp build
# checkmodule -M -m -o myopenvpn.mod myopenvpn.te
# semodule_package -o myopenvpn.pp -m myopenvpn.mod

module myopenvpn 1.0;

require {
	type iptables_exec_t;
	type openvpn_t;
	class file { getattr open execute read execute_no_trans map };
	class netlink_netfilter_socket { create getopt setopt };
}

#============= openvpn_t ==============
allow openvpn_t iptables_exec_t:file { getattr open execute read execute_no_trans map };
allow openvpn_t self:netlink_netfilter_socket { create getopt setopt };

and adding semodule -i /rw/config/path/myopenvpn.pp in /rw/config/rc.local

Still openvpn does not start at boot even if no more avc denied logs. Openvpn log returns:

[...]
2024-12-04 21:17:35 TUN/TAP device tun0 opened
2024-12-04 21:17:35 net_iface_mtu_set: mtu 1500 for tun0
2024-12-04 21:17:35 net_iface_up: set tun0 up
2024-12-04 21:17:35 net_addr_ptp_v4_add: 10.1.0.2 peer 10.1.0.1 dev tun0
2024-12-04 21:17:35 qubes-vpn-handler.sh up tun0 1500 0 10.1.0.2 10.1.0.1 init
netlink: Error: Could not process rule: Permission denied
2024-12-04 21:17:35 WARNING: Failed running command (--up/--down): external program exited with error status: 1
2024-12-04 21:17:35 Exiting due to fatal error

I have a hard time understanding why so different between system/rc.local execution and my interactive run (from standard user + sudo). Or is there a systemd slice adding some restrictions? but nothing in /usr/lib/systemd/system/qubes-misc-post.service and else, under system.slice.
And are there so few people using external script with openvpn and Fedora/RedHat?

[DVM]

@juju42

Still openvpn does not start at boot even if no more avc denied logs. Openvpn log returns:

Do you have any other selinux denied logs after this last permission denied error?

[juju42]

Nope. nothing more in selinux denied or anything approaching

[DVM]

Just to rule out selinux with this error, does it still happen if you run setenforce 0 before running the VPN script?

[juju42]

@DVM

setenforce 0

good catch, it IS still selinux.
With above, openvpn starts and works normally.
If restored without, openvpn starts and dies but no denied log…

[barto]

@juju42

good catch, it IS still selinux.

A friend of mine said, about 18-odd-ish years ago: “in 20 years, we’ll be still struggling with SELinux”
Seems that it was a really good prediction.
Mind you, I’m all for SELinux… but my $GOD… it does confuse the non-technical, and even slightly technical, people.