Hello Qubes Community,
I am at my wit’s end after an 8+ hour debugging session trying to set up a standard sys-unbound DNS proxy. I have encountered and seemingly solved a dozen different issues, but I am now facing a final, paradoxical problem that defies all logic. I’m hoping someone with deeper knowledge of the Qubes/Debian internals can shed some light.
The Final State of the System:
- Qube: sys-unbound, based on a fresh debian-12 template.
- Networking: sys-unbound → sys-firewall → sys-net. The VPN has been removed from the chain to minimize variables.
- Qubes Firewall for sys-unbound: Set to allow all outgoing connections to eliminate it as a variable.
- Qubes Services: qubes-network and clocksync are enabled for sys-unbound.
- The Paradox: I have a custom systemd service (unbound-proxy.service) that fails to start with Active: failed (Result: exit-code). However, when I run the exact ExecStart command manually from the terminal, Unbound starts successfully, shows the notice: Start of unbound… message, and then exits cleanly (as expected for a foreground process).
The Debugging Journey (Abridged):
My goal was a standard Unbound setup forwarding to Quad9 over DoT with DNSSEC validation. Here is the chronicle of every major failure and attempted solution:
- Initial SERVFAIL: Started with a Fedora template, ran into numerous issues (SELinux, systemd-resolved, sd_notify). Switched to a fresh debian-12 template to start clean.
- unbound-anchor: command not found: Solved by apt install --reinstall unbound dnsutils in the template after discovering the initial installation was incomplete.
- root.key Download Failures: Discovered that official IANA/Unbound URLs for the trust anchor file returned 404 errors, indicating a potential issue with my ISP or network environment. This was confirmed even in a clean qube connected directly to sys-net.
- Switched to root-anchors.xml: Found the correct root-anchors.xml URL and successfully downloaded it manually.
- chroot and “File Not Found” Errors: The default Debian unbound.service is complex and uses a chroot. This led to a cascade of “file not found” errors for the trust anchor, even though ls -l confirmed the file existed with correct permissions. Attempts to work within the chroot by using relative paths were unsuccessful.
- Permission denied on Port 53: When attempting to run as the unbound user, the service could not bind to port 53. sudo setcap ‘cap_net_bind_service=+ep’ /usr/sbin/unbound was executed, but the service still failed with “Permission denied”, even with SELinux (on the previous Fedora attempt) set to Permissive. This suggests a deeper restriction.
- The “Coup d’État” - Custom Service: To eliminate all of the above, I took the final step of creating a completely custom, clean, and simple setup, which is my current state.
Current (Final) Configuration Files:
This is the setup that produces the final paradox.
/etc/unbound/unbound.conf:
# Clean, chroot-less Unbound Configuration
server:
verbosity: 1
# chroot and directory are disabled to simplify debugging
# directory: “/etc/unbound”
# chroot: “”
user: “unbound” # Unbound should drop privileges after binding the port
interface: 0.0.0.0
port: 53
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
access-control: 0.0.0.0/0 refuse
access-control: 127.0.0.0/8 allow
access-control: 10.137.0.0/16 allow
hide-identity: yes
hide-version: yes
# ... and other hardening options ...
# Using an absolute path since chroot is disabled
trust-anchor-file: "/var/lib/unbound/root-anchors.xml" # File exists with correct content and unbound:unbound permissions
forward-zone:
name: “.”
forward-tls-upstream: yes
forward-addr: 9.9.9.9@853#dns.quad9.net
forward-addr: 149.112.112.112@853#dns.quad9.net
/etc/systemd/system/unbound-proxy.service:
[Unit]
Description=Custom, Stable Unbound DNS Proxy
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
Running as root to bind to port 53; the unbound.conf ‘user’ directive should handle privilege dropping.
ExecStart=/usr/sbin/unbound -d -c /etc/unbound/unbound.conf
Restart=on-failure
[Install]
WantedBy=multi-user.target
The Core Question:
When I run sudo systemctl start unbound-proxy.service, it fails with (code=exited, status=1/FAILURE) and the journal is completely empty.
When I run sudo /usr/sbin/unbound -d -v -c /etc/unbound/unbound.conf manually in the terminal, it prints notice: Start of unbound 1.17.1. and exits without any error, which is the expected behavior for a manual start.
What could possibly cause systemd to fail to start this process, when the exact same command works perfectly when run by hand? Is there a hidden Qubes OS mechanism, AppArmor profile (though it’s a Debian template), or kernel-level restriction that interferes with Unbound’s startup only when launched by systemd? The Qube’s firewall is wide open, so it’s not a network block.
Any insight would be immensely appreciated. After 8+ hours, I am at a complete loss.
Thank you.
