HVM qube does not have internet when gateway ip is set to the ip specified in qube settings but does have internet when gateway ip is a random ip?

ajsonx · February 10, 2025, 8:43am

Hello, I have a question/issue that I am encountering when I try to route my Kali HVM qube through my Standalone MullvadVPN VM I Will try to explain as best I can in detail What’s happening.

A while back I wanted to create a VPN qube to route some of my VM’s through a VPN and i found this really good guide on here which I followed. The only difference really is the template VM is a debian-12-minimal. I’ve been using this VM for a while now with my appVMs and they work without any issue.

The issue I am encountering is recently I wanted to create a Kali linux HVM and have the net qube be my MullvadVPN qube, So I set up my Kali HVM and set the ip address of the HVM with the ip address in the Kali HVM qube settings which was: 10.137.0.31, I set the netmask to 255.252.0.0 which is what i have to do with my other HVM’s to get internet working (other HVM’s use sys-firewall as net qube) and I set the gateway ip on my Kali HVM to: 10.137.0.28 which is the gateway ip in the kali HVM qube settings when I select the MullvadVPN VM as my net qube and set my DNS to the corresponding ips in my qube settings as well which are: 10.139.1.1, 10.139.1.2. Now usually on my other HVM’s that I’ve setup I should now have internet access and it should be routing my traffic through my specified net qube but when I tried to go on the internet and load up a webpage or ping an ip it would not work. At first I thought it might not be configured properly in the HVM but going through my config files and NetworkManager, And listing my network interface info all was configured correctly. I tried switching netmask to different values and it still was not working, So I switched the net qube to sys-firewall without changing the gateway ip at first and it started working now, But the gateway ip was still on the ip of my MullvadVPN VM. I then switched the gateway ip to sys-firewall ip which was 10.138.7.34 and it was still working, But then i decided to change my net qube back to my MullvadVPN VM without changing the gateway ip in the HVM and rebooted the HVM and saw I now have internet connection and my internet was being routed correctly to my MullvadVPN VM. Which I thought was very odd so then I changed the gateway ip address in my HVM to a random ip 10.137.0.11 and rebooted my HVM and it was still working. I was still routing my connection through my MullvadVPN VM with my outward facing public ip as the server specified in the MullvadVPN App. It is only when I change the gateway ip address to the ip address of the MullvadVPN VM which is specified in the Kali HVM qube settings when i select the MullvadVPN VM as my net qube that my internet stops working. I’ve only exhibited this behavior with my HVM’s and not my other AppVMs that use the MullvadVPN VM as their net qube. I then tried using sys-whonix as my net qube for the kali HVM and it wouldn’t work at all even if i changed the gateway ip to a random ip or to the specified gateway ip in the qube settings or even changing netmask to different values I couldn’t get it to work at all.

I would just like to understand why this is happening and if this is intended behavior for HVMs or specifically for a VPN Standalone VM and a HVM. If anyone can give more insight as to what is happening here and why my HVM does not have internet connection when I configure the gateway ip of the HVM to be the ip of the MullvadVPN VM and instead does work if the gateway ip is set to a incorrect value. Is this safe to do, And use the HVM with the gateway set to the wrong ip yet still is working and routing my traffic through the specified MullvadVPN VM, I would assume so as this is all happening internally but i would like you hear your thoughts on this. I also can’t help to assume maybe it has something to do with the fact that my MullvadVPN VM template is debian-12-minimal which does not include qubes suite of packages but I am not sure. I would also like to know why sys-whonix does not work in any regard if possible, Thanks in advance and apologies for the text wall.

tl;dr: My HVM only has internet connection when my gateway ip is set to a incorrect value and when set to the ip specified in my HVM qube settings it fails to route my traffic through my net qube.

bengee · March 8, 2025, 12:24am

I have a standalone hvm LinuxMintDE 6 “Faye” install similar to your Kali install which I occasionally use for testing some things. I also have a standalone sys-vpn pvh vm running Fedora 40 and mullvad vpn. AppVMs pointing to the sys-vpn work great out-of-the-box (OOB). So I too was a bit puzzled when experiencing the similar behavior. I ended up doing various tests trying to determine what was going on. For what it’s worth, I thought I might share the results. Maybe someone with Qubes technical knowledge can provide internal details.

There is a great forum explanation for HVM networking titled “Setting up Networking for HVMs” on this page https://www.qubes-os.org/doc/standalones-and-hvms/. However, in this article it says to copy the GW also so I don’t know if the behavior seen is a bug that got introduced after that was written, nor why it doesn’t presently work to do so. FYI, I’m on Qubes 4.2.3.

Here is the short version of what I found in testing. (Read this if you want results and don’t want any details.)
In short,

QubesManager’s guest GW setting seems to circumvent or disregard the hvm guest NIC GW settings that from the instructions say to copy into the guest NIC GW settings. And putting the QubesManager hvm guest default gateway setting in the guest’s NIC GW setting breaks the internet access.
Qubes’ IP and DNS IP addresses need to be mirrored into the guest NIC settings. (see the aforementioned article).

Here is the long version. (A more detailed explanation of test results. Sorry about the length… )
When creating a new vm, Qubes sets up the guest IP addressing to tie the vm into the Qubes environment. By setting the “netVM” on this guest to the sys-vpn qube so that you can route through sys-vpn, Qubes automatically puts the sys-vpn IP address into the QubeManager “guest” default gateway (GW) setting. Again, this is a setup that works OOB for appVMs.

Because it was necessary as part of the hvm install to mirror the NIC settings from the QubesManager, I also logically mirrored the IP of sys-vpn into the default GW in the guest nic settings and this is what the instructions say to do. The instructions said “In order to manually configure networking in a qube, one should first find out the IP/netmask/gateway assigned to the particular qube by Qubes. This can be seen, e.g., in the Qube Manager in the qube’s properties:”. I thought “No problem”, if it followed appVM behavior, this should work. However, that was a mistake I later found and the result was no internet. Rechecking the guest nic config showed the nic interface was connected. It looked like a routing issue.

Further testing determined Qubes doesn’t seem to even use the value of the hvm guest NIC GW setting EXCEPT maybe as a placeholder. What I mean by this is you can put just about anything in there as long as it’s NOT the actual GW listed in the QubesManager default GW setting (doing so breaks internet access) and Qubes doesn’t care it’s not an actual valid GW or route. It also doesn’t like an empty GW of course or something like 0.0.0.0 which is the same as empty. Well experiencing this behavior just doesn’t seem right but I can get internet access and verify it is using sys-vpn using just about any gateway when I manually configure the hvm guest NIC GW with any of the following:

with my home router GW IP;
an IP of either of the 2 Qubes DNS servers;
the IP of the GW of sys-vpn (is not the IP of sys-vpn but rather sys-vpn’s netVM);
any random private IP address ie. 172.10.10.10;
any made-up IP address ie. 200.200.200.200, or 1.1.1.1.

This leaves at least the perception that any of these options don’t do anything in that field; they seem to have no effect. And these aren’t even gateways to sys-vpn. Yet the traffic apparently routes there and continues through the vpn on out. Again I verified each of the examples as being put in the NIC default GW and heading out to the Mullvad site where it clearly showed I was using the vpn. In other words, it appears the route to sys-vpn then is set purely with the QubeManager default GW entry for the guest which of course in this case is the actual IP of sys-vpn (or netVM of the guest). This would be the expected behavior especially if I used it in the guest NIC GW setting. The fact that it still gets set correctly isn’t an issue because I would expect Qubes to set that route correctly. But what kind of is not so expected is that putting that IP into the NIC GW kills the internet.

By Qubes setting this route, Qubes DNS also now appears to be available to the hvm guest provided that:

you use a valid placeholder; and
provided you also mirror the Qubes DNS IP’s in the NIC like the instructions say.

For example, if I put a Google DNS server IP 8.8.8.8 in the NIC DNS settings in the same situation, there is no DNS resolution and no internet even though I have a valid “placeholder” set in the NIC GW. Only when I put the Qubes DNS servers in does internet resume working. I would expect that behavior to prevent attempted DNS workarounds. What I’m saying is that all the “correct” Qubes pieces need to be used. Well it seems except for the NIC GW.

To summarize:

using any one of the above 5 scenarios as a choice in the guest NIC GW setting doesn’t seem to prevent the QubesManager guest GW setting from working correctly even if they’re not actually gateways to sys-vpn. Qubes just seems to ignore them. The positive side of this is that Qubes is maintaining control of the GW.
Having an empty NIC GW setting to me would be an expected failure and it does.
But trying to use the guest GW setting from the QubeManager in your guest’s NIC GW setting isn’t allowed. Doing so breaks internet access. This is odd behavior since that is the true GW to sys-vpn. And Qubes maps it to it no matter. Lastly, the instructions support doing so.

So my take on this behavior is possibly a bug, or this is probably by design (a new feature?) and maybe for security reasons so users can’t accidentally break the security or maybe the routing within the guest? This remote possibility is reinforced by the need to mirror the “Qubes determined” networking so the guest vm is using the Qubes environment and not circumventing it. So I suppose…

Well I’ll finally end by saying it doesn’t look right, but you can get it to still do what your goal is which in my case was to send hvm guest internet bound traffic through a vm hosting my vpn; the same as I can do OOB with an standard appVM. And I can verify it is actually using the vpn. My apologies for the length…

ajsonx · March 31, 2025, 11:08pm

Sorry for the late reply I have not touched my QubesOS pc in a couple weeks. I appreciate the in depth break down of the problem much better then i could’ve ever done I also came to the same conclusion it’s either a bug or a feature lol. My concern at the time was if this could be some sort of security risk, But unless there’s a more severe bug also at play I don’t think an attacker can do anything with this. Either way I thought it would be worth mentioning in case this was some sort of bug. Thanks again for the response and your break down of the issue you encountered as well.