During software development, I need a group of VMs that can access each other over the network, but remain isolated from the Internet. Many guides online are incomplete or confusing, so here’s a clean solution.
Solution Overview
One group of VMs with all-to-all network access.
Each VM in the group that acts as a server may allow incoming connections on all ports.
The network has no Internet access, and the VMs trust each other more or less.
My GIT server is located right in this dev group, btw.
For package installation (npm i, composer install…), use a disposable VM and copy the resulting code back to your dev VMs. Spoiler: it’s still insecure (this is a separate topic for discussion).
Network Setup
By default Qubes OS uses 10.137.x.x IP range, with 10.137.0.x assigned by DHCP.
Use 10.137.1.x for manual IP assignment in your dev group.
Steps
Set sys-firewall as network VM for all AppVMs in the group.
Assign unique IPs for each AppVM (run in Dom0):
qvm-prefs my-vm ip 10.137.1.101
Enable routing between all IPs in the range (run on sys-firewall):
Use this command (see below):
nft add rule ip qubes custom-forward ip saddr 10.137.1.0/24 ip daddr 10.137.1.0/24 ct state new,established,related counter accept
Make it permanent:
Option 1: Add the line to /rw/config/rc.local in default-dvm (template for disposable VM).
Option 2: Clone default-dvm → sys-firewall-dvm, add the line there, and set sys-firewall-dvm as the template for sys-firewall.
Allow server VMs to accept connections. On each VM running a server, add to /rw/config/rc.local:
nft add rule ip qubes custom-forward ip saddr 10.137.1.0/24 ip daddr 10.137.1.0/24 ct state new,established,related counter accept
Restrict Internet access (optional). For any VM you want to keep offline (as I do):
Open Qube Manager → Settings → Firewall rules
Check Limit outgoing connections to…
Add exceptions if needed for intra-group communication.
Reboot all VMs to make sure the settings apply and persist across reboots.
in your guide: at point 4, yo should add rule to the ‘custom-input’ chain, not the custom-forward. - but this is just a copy-paste error, I guess.
Manual IP’s are not required to achieve the goals you have defined. But it makes sense in case of ‘servers’, for sure.
adding the rules by IP is not universal enough, and might end up with side effects, as a Proxy VM is ‘reusing’ a single IP for ALL of it’s interfaces.
The Qubes Firewall - unfortunately - can’t provide you leak proof firewall settings.
My suggestion to fix those issues:
add your netfilter rule based on interface name, instead of IP addresses. So in your firewall/Proxy VM, you only need a single and universal line to allow ALL communications between the VMs attached to this ProxyVM:
nft add rule ip qubes custom-forward iifname vif* oifname vif* ct state new,established,related counter accept
And don’t call it routing, as it has nothing to do with the routing. The routing is already predefined this line is ‘just’ allowing the FORWARDING of the packets in the Linux packetfilter. The routing still needs to tell which interface should those packets go
Because the interface naming is defined by Qubes OS, we can safely relay on those, and it will work regardless of the IP address of your VMs. Means you can assign any disposable VM to this ProxyVM, and automatically will be able to communicate to any other VM attached to the same ProxyVM
Also in your server VMs, you can have a much more general allow rule, based on the interface name: