Hello,
I have a client to whom I connect via their CheckPoint VPN gateway which involves installing the CheckPoing Mobile Access Portal Agent a.k.a. SSL Network Extender a.k.a. CShell. It creates a new network interface called tunsnx in the qube where it is running.
I run Debian-based qubes and all was well until I upgraded from the debian-10 template in Qubes 4.0 to debian-11 in 4.1. VPN suddenly stopped running. tunsnx still comes up but I can’t connect through it. Not even ping comes through. Wireshark tells me that packets sent to tunsnx have the wrong source address: instead of tunsnx’s own they use the “primary” IP address of the qube, i.e. the one assigned to eth0.
Specifying the source address with ping -I tunsnx or ping -I (tunsx IP address) doesn’t help. When I use this option, ping duly calls bind(2) with the correct IP address - according to strace - but the packet is still sent out with the wrong one. When I don’t use the option, strace logs a getsockname(2) call where the sockaddr struct submitted as argument contains the right source address. I cannot see what getsockname(2) returns, unfortunately.
So I compiled ping from source and tried to debug this but I can’t seem to get the permissions right. I found out I needed to grant the CAP_NET_RAW capability to my compiled executable but that didn’t help either. getcap now displays the same for my file as for /usr/bin/ping yet the latter works and the former still says “Lacking privilege for icmp socket.”
I’m not really a C person so I’m somewhat out of my depth here. I installed the debian-10 template as a workaround and everything works flawlessly there. It’s far from ideal, however, so I’ll appreciate any hints or tips.
z.
I managed to run ping in debug mode and the socket has the correct source address even at the point where sendto(2) is called. So I downloaded the source for the distribution kernel, inserted a few printk()s, compiled, installed, rebooted and found out that the qube is actually running on a Qubes kernel (5.15.81-1.fc32.qubes.x86_64 to be precise). Obvious in hindsight.
Guess I’ll have to learn how to patch that, then.
Having learned how to compile & install a modified Qubes kernel, I proceeded to use printk() to narrow the problem down to an nf_hook_slow() call in netfilter.h - so the culprit is netfilter.
I compared netfilter rulesets between the debian-10 and debian-11 templates and found no change that could be causing the source address swap. More generally, the only rule I found that should be affecting the source address at all uses the masquerade instruction which is supposed to align it with the egress device (i.e. tunsnx) and I see the exact opposite happening.
Disabling netfilter in the qube (sudo nft flush ruleset ip) makes debian-11 behave correctly so that’s the workaround I’ll be using.
After examining how netfilter processes the masquerade rule, I discovered that when choosing the outgoing IP for a packet, the kernel skips the proper address in inet_select_addr() because it lacks global scope. It turns out that when the tunsnx network device gets initialized, its IP address gets a seemingly random scope value somewhere between RT_SCOPE_SITE and RT_SCOPE_LINK. This is apparent even in the output of ip addr - it will say “scope 231” or some such.
I managed to run the VPN executable (/usr/bin/snx) under strace and found that, indeed, it calls sendmsg() on an AF_NETLINK socket with an RTM_NEWADDR message specifying something like ifa_scope=0xf5. That’s under the debian-11 template. Under debian-10 it sends ifa_scope=0 as it should. Also, while it sends no ifa_flags under debian-10, with debian-11 it specifies IFA_F_NODAD|IFA_F_HOMEADDRESS|IFA_F_DEPRECATED|IFA_F_TENTATIVE.
There are other interesting differences between strace outputs under the two templates. Like when snx wants to do something in a child process, it simply calls clone() and then waitpid() under debian-10. Under debian-11, it also calls ugetrlimit(), mmap2(), rt_sigprocmask(), munmap() and it calls wait4() instead of waitpid(). I chalk it down to different versions of libc.
I suppose this is as far as the rabbit hole goes for me, given that snx itself is a closed-source blob and I’m not about to try to decompile it. Given that both the executable and the kernel it runs on are unchanged, there must be something in the debian-11 environment that corrupts tunsnx initialization but I have run out of clues.
I tried adjusting the address scope with ip addr change dev tunsnx scope global but RTNETLINK answers: Operation not supported. I’ll just have to stick with the workaround of disabling the masquerade rule in nftables.
1 Like