Understanding how qvm-connect-tcp works under the hood

User Support
1

qvm-connect-tcp plays an important role in Qubes, so I’m trying to understand it more fully. In particular, I’m confused about how it works to expose a service with a simple port binding. Any insight into the following would be appreciated and hopefully of value to the forum at large.

In general, exposing a service with socat in a server-client setup would look something like

user@server$ socat TCP-LISTEN:55555,reuseaddr,fork,bind=server TCP:localhost:55555

enabling user@client to call on the service at port 5555.

In Qubes, qvm-connect-tcp employs socat under the hood, but it’s not clear to me exactly how it works.

Specifically, if we want to expose a service in a qube, call it Qserver, we would call qvm-connect-tcp ::55555 from another qube, Qclient, which would invoke socat as follows:

Qclient$ socat TCP-LISTEN:55555,reuseaddr,fork EXEC:"qrexec-client-vm '' qubes.ConnectTCP+55555"

so the Qclient qube would listen on port 55555, as the source, and would call qubes.ConnectTCP acting on the Qserver qube via qrexec-client-vm as the destination. (At least this is how I understand it in my admittedly confused state…)

So with a RPC policy of

qubes.ConnectTCP +55555 Qclient @default allow target=Qserver

qvm-connect-tcp ::5555 should create a socat process in Qserver which communicates any input from Qclient to the service listening on port 55555 according to qubes.ConnectTCP+55555 which invokes:

Qserver$ socat STDIO TCP:localhost:55555

However, when I check pstree -ap | grep 55555 in each of these qubes I find the following:

  ├─socat,632 TCP-LISTEN:55555,reuseaddr,fork EXEC:qrexec-client-vm \\'\\' qubes.ConnectTCP+55555
  │   └─socat,728 TCP-LISTEN:55555,reuseaddr,fork ...
  │       └─qrexec-client-v,729 qubes.ConnectTCP+55555

in the Qclient qube and nothing in the Qserver qube. Leading me to question my understanding of how this all works.

Nevertheless, when I check netstat -talnp in each qube I find that they are each listening on port 55555 (the service in Qserver and socat in Qclient) and there are established connections in both directions at 127.0.0.1:5555, as expected.

So I’m left with a few quetions:

  1. At which points above am I misunderstanding something basic?
  2. Why does the Qubes documentation expose a service in Qserver by calling socat via qvm-connect-tcp in Qclient? This seems counterintuitive.
  3. How does EXEC:qrexec-client-vm... work exactly as the destination for socat in the call to qvm-connect-tcp in Qclient?
  4. How would I replicate this setup outside of Qubes? Would it be as simple as
socat TCP-LISTEN:55555,reuseaddr,fork TCP:localhost:55555

on the server, even though Qubes essentially implements this from the client side?

2

/etc/qubes-rpc/qubes.ConnectTCP on the destination side used to be a script invoking socat like you described, but now it’s just a symlink to /dev/tcp/127.0.0.1 which is handled natively by qrexec (see Types of qrexec services)

1 Like
3

I clearly went down the wrong rabbit hole with /etc/qubes-rpc/qubes.ConnectTCP. I suspect most of my fog will clear up once I dig into the qrexec documentation. Thanks for the helpful links!

I take it that qubes.ConnectTCP in the socat destination of qvm-connect-tcp instead refers to /etc/qubes/rpc-config/qubes.ConnectTCP.

4

The client side initiating the qrexec connection makes a request for just the name, e.g. qubes.ConnectTCP. Then the destination side looks for a handler with that name, first in /usr/local/etc/qubes-rpc/ or otherwise in /etc/qubes-rpc/.

/etc/qubes/rpc-config/ is yet another different directory for extra configuration on how to set up the handler - such as wait-for-session=1 telling the system to delay calling the handler until the GUI session is ready.

1 Like
5

Starting to make more sense, or perhaps I’m just confused at a deeper level now. According to the qrexec-internals documentation for domX.

domX: qrexec-client-vm is invoked as follows:

qrexec-client-vm domY qubes.Service [local_program] [params]

If local_program is set, it will be executed in domX and connected to the remote command’s stdin/stdout.

In the case of qvm-connect-tcp ::55555 run on Qclient(domX), local_program is not set by

socat ... EXEC:"qrexec-client-vm '' qubes.connectTCP+55555"

So as I now understand it, the 55555 parameter is passed to bash in Qserver(domY) to complete the sym-link as qubes.ConnectTCP+55555 -> /dev/tcp/127.0.0.1/55555. This will cause the built-in bash script in Qserver to open a TCP socket at 127.0.0.1:55555, while the connection back to Qclient's stdin/stdout is established by qrexec-client.

6

Pretty much. (Except that the destination side doesn’t use Bash to handle /dev/tcp/, even though this notation was inspired by Bash. The TCP connection is opened by qubes_tcp_connect() in libqrexec, after the symlink target was parsed by find_qrexec_service().)