Hello,
I am trying 4.2 and ran into an issue with networking between qubes, which I sometimes enable temporarily for specific purposes. On 4.1, the commands from Firewall | Qubes OS work fine. However, in 4.2 I am unable to establish a connection between 2 qubes.
iptables -nvL on the NetVM shows that the rules are in place and the packet counters are incrementing as I test, so the rules seem to be working there. I saw the note that iptables is being replaced with nftable, but it looks like the iptables command is delegating to nftable, and when I checked the output of nft list ruleset there were entries identical to the output of the iptables commands. sudo journalctl -u qubes-firewall does not display any interesting messages, just some stuff about the updater. I am testing the connection using ping and netcat. I have tried connecting between 3 different AppVMs (A, B, and C) all using the same NetVM, with the combinations A<->B and B<->C. Qube A is based on a template that was cloned from the default fedora template then customized with additional package installations, both B and C are based on the default fedora template that has been customized with additional package installations. None of them have service changes other than the iptables commands from the documentation.
I’m unsure how to proceed with debugging this. As I understand it, there should only be 3 Qubes required for the communication are the 2 AppVMs and the NetVM that they are both connected to. The sending AppVM sends the data without error (regardless of which qube is sending), the NetVM reports that the packet was accepted, but the receiving AppVM never gets it (tcpdump produces no output). Both of the AppVMs can successfully ping the NetVM, although the NetVM cannot successfully ping the AppVMs. Does anyone know what else I can check?
Thanks,
Skyler
Some more info:
The post at 4.2 equivilant for 4.1 networking commands? mentions that rules should be added to custom-forward. The iptables interface adds them to ip filter. Not sure if this is a bug because I don’t see what the source for that claim is (neither of the linked bugs mention it). However, I tried adding the rules to this chain with nft add rule ip qubes custom-forward ip saddr $SENDING_IP counter. This rule is broader than the the iptables rule I was using (no destination address is specified), but that shouldn’t prevent my target qube from receiving the data. After adding this rule (and deleting the rules from ip filter) the behavior does not change (tcpdump shows the packets in sys-firewall but not the target qube) and the counter for the new rule increments.
Additionally, I monitored the interface on the target qube in wireshark and I saw 2 ICMPv6 router solicitation packets sent to the target qube. One came from the sending qube and the other from an address I do not recognize. Based on this, I repeated the nft add rule ... from above with ip6 instead of ip, but this had no effect on behavior. The counter for the ip6 rule did not increment.
Finally, I tried pinging the target qube’s IPv6 address which led to different results. The sending qube sent several Link-Local Multicast Name Resolution requests but did not receive any replies. The echo requests never appeared in wireshark.
Ok, there were 2 problems with the nft commands listed above. I’ll have a concise list of exact commands to run at the end of this post. First, I added counter to the end but not accept, so the rules were being hit but not doing anything other than collecting metrics. Second, I neglected to convert the rules on the AppVMs into nft rules, so only the sys-firewall qube was set up correctly. Once both of these issues were resolved, pings worked.
Enabling Inter-Qube Networking on 4.2:
IP1=10.x.y.z
IP2=10.x.y.z
# On the NetVM used by both qubes, probably sys-firewall, add rules for both directions
# Note: for all commands here the counter may be omitted, but might be useful to have in for debugging
sudo nft add rule ip qubes custom-forward ip saddr $IP1 ip daddr $IP2 counter accept
sudo nft add rule ip qubes custom-forward ip saddr $IP2 ip daddr $IP1 counter accept
# On the AppVM with $IP1, add a rule to accept data from $IP2
sudo nft add rule ip qubes custom-input ip saddr $IP2 counter accept
# On the AppVM with $IP2, add a rule to accept data from $IP1
sudo nft add rule ip qubes custom-input ip saddr $IP1 counter accept