Impossible Unbound exit-code Failure on Debian-12 Template (8+ Hour Debugging Saga)

Hello Qubes Community,

I am at my wit’s end after an 8+ hour debugging session trying to set up a standard sys-unbound DNS proxy. I have encountered and seemingly solved a dozen different issues, but I am now facing a final, paradoxical problem that defies all logic. I’m hoping someone with deeper knowledge of the Qubes/Debian internals can shed some light.

The Final State of the System:

  • Qube: sys-unbound, based on a fresh debian-12 template.
  • Networking: sys-unbound → sys-firewall → sys-net. The VPN has been removed from the chain to minimize variables.
  • Qubes Firewall for sys-unbound: Set to allow all outgoing connections to eliminate it as a variable.
  • Qubes Services: qubes-network and clocksync are enabled for sys-unbound.
  • The Paradox: I have a custom systemd service (unbound-proxy.service) that fails to start with Active: failed (Result: exit-code). However, when I run the exact ExecStart command manually from the terminal, Unbound starts successfully, shows the notice: Start of unbound… message, and then exits cleanly (as expected for a foreground process).

The Debugging Journey (Abridged):
My goal was a standard Unbound setup forwarding to Quad9 over DoT with DNSSEC validation. Here is the chronicle of every major failure and attempted solution:

  1. Initial SERVFAIL: Started with a Fedora template, ran into numerous issues (SELinux, systemd-resolved, sd_notify). Switched to a fresh debian-12 template to start clean.
  2. unbound-anchor: command not found: Solved by apt install --reinstall unbound dnsutils in the template after discovering the initial installation was incomplete.
  3. root.key Download Failures: Discovered that official IANA/Unbound URLs for the trust anchor file returned 404 errors, indicating a potential issue with my ISP or network environment. This was confirmed even in a clean qube connected directly to sys-net.
  4. Switched to root-anchors.xml: Found the correct root-anchors.xml URL and successfully downloaded it manually.
  5. chroot and “File Not Found” Errors: The default Debian unbound.service is complex and uses a chroot. This led to a cascade of “file not found” errors for the trust anchor, even though ls -l confirmed the file existed with correct permissions. Attempts to work within the chroot by using relative paths were unsuccessful.
  6. Permission denied on Port 53: When attempting to run as the unbound user, the service could not bind to port 53. sudo setcap ‘cap_net_bind_service=+ep’ /usr/sbin/unbound was executed, but the service still failed with “Permission denied”, even with SELinux (on the previous Fedora attempt) set to Permissive. This suggests a deeper restriction.
  7. The “Coup d’État” - Custom Service: To eliminate all of the above, I took the final step of creating a completely custom, clean, and simple setup, which is my current state.

Current (Final) Configuration Files:
This is the setup that produces the final paradox.
/etc/unbound/unbound.conf:
# Clean, chroot-less Unbound Configuration
server:
verbosity: 1
# chroot and directory are disabled to simplify debugging
# directory: “/etc/unbound”
# chroot: “”
user: “unbound” # Unbound should drop privileges after binding the port

interface: 0.0.0.0
port: 53
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes

access-control: 0.0.0.0/0 refuse
access-control: 127.0.0.0/8 allow
access-control: 10.137.0.0/16 allow

hide-identity: yes
hide-version: yes
# ... and other hardening options ...

# Using an absolute path since chroot is disabled
trust-anchor-file: "/var/lib/unbound/root-anchors.xml" # File exists with correct content and unbound:unbound permissions

forward-zone:
name: “.”
forward-tls-upstream: yes
forward-addr: 9.9.9.9@853#dns.quad9.net
forward-addr: 149.112.112.112@853#dns.quad9.net

/etc/systemd/system/unbound-proxy.service:
[Unit]
Description=Custom, Stable Unbound DNS Proxy
After=network-online.target
Wants=network-online.target

[Service]
Type=simple

Running as root to bind to port 53; the unbound.conf ‘user’ directive should handle privilege dropping.

ExecStart=/usr/sbin/unbound -d -c /etc/unbound/unbound.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target

The Core Question:
When I run sudo systemctl start unbound-proxy.service, it fails with (code=exited, status=1/FAILURE) and the journal is completely empty.
When I run sudo /usr/sbin/unbound -d -v -c /etc/unbound/unbound.conf manually in the terminal, it prints notice: Start of unbound 1.17.1. and exits without any error, which is the expected behavior for a manual start.

What could possibly cause systemd to fail to start this process, when the exact same command works perfectly when run by hand? Is there a hidden Qubes OS mechanism, AppArmor profile (though it’s a Debian template), or kernel-level restriction that interferes with Unbound’s startup only when launched by systemd? The Qube’s firewall is wide open, so it’s not a network block.
Any insight would be immensely appreciated. After 8+ hours, I am at a complete loss.

Thank you.

I don’t know unbound, but I have had problems where it seemed that ‘network.online’ is not equal to ‘network can actually communicate’.

At the time I used something like the “hacky” ping solution in the comments to this…
https://stackoverflow.com/questions/35805354/systemd-start-service-at-boot-time-after-network-is-really-up-for-wol-purpose
Can it be the same problem? As for the “best” solution - I didn’t go that far.

I use this setup [1] incl. unbound for apparently like 4 years.

However I don’t use it as proxy VM, but rather as regular app VM in a setup such as
sys-net/vpn ↔ sys-fw ↔ client VMs
>
L__> sys-dns

(png at [2])

I also run systemd-resolved as local DNS cache inside sys-fw and pin resolved DNS firewall entries to the IPs used by the Qubes firewall.

[1] GitHub - 3hhh/qubes-dns: DNS VM helper scripts
[2] qvm-ls-mermaid/examples/1.png at master · 3hhh/qvm-ls-mermaid · GitHub