(best) Split SSH setup guide

@deeplow @phl
Ok, got it. This procedure doesn’t work on qubes 4.1 stable.
I just recalled that I was on testing repo both for dom0 and templates previously.
Once I changed to testing and updated dom0 and templates all works as expected.


That’s odd. Probably something that’s not (yet) correctly working in 4.1. Thanks for letting us know!

Short update from my side: Very recently, my Qubes system has started to behave strangely when connecting to SSH hosts via this method.
Fetching keys from my vault VM now takes ~1 minute, while a direct login to a host (either by password or by key which I copied directly into the AppVM for testing) works as fast as expected (~7 seconds for the whole process).

Clearly, the delay must be introduced by some part of the intra-VM connection.
I have yet to find out which part is flawed and how the behaviour was introduces. I have not modified my configuration for quite some time. This has probably been introduced by some recent Fedora update, as I obviously install these regularly.

@whoami, @santorihelix: Has anyone of you observed similar issues?

UPDATE: And I already found a workaround, but wouldn’t really call it a “solution”: I found that I was still running an older setup of my split-SSH, where I used ncat instead of socat, which we chose for the guide. After changing the setup to socat, everything works as fast as I was used to.
So my guess is that some recent update changed the behaviour of ncat, which introduced these huge lags.

I don’t really have time to debug this further right now. so if anyone has performance issues, the recommendation is:
Use socat instead of ncat!

I will check today and come back to you

9 Feb 2021, 11:53 by qubes_os@discoursemail.com:

Nope, here all works super smooth. But for me it is just one sec. : select the vault VM in the pop-up and boom I am in.

Finally followed this through! It all worked in the end :slight_smile:

I was having the same issue, following the process. The solution was to install

  • fedora: libnotify and mate-notification-daemon
  • debian: libnotify-bin and mate-notification-daemon

This is the same issue as described here that broke so many people’s VPN configs.

Maybe a note about this should be added to the guide.


That is interesting and definitely unexpected (at least for me). I thought that an error with the notification should have no impact on the rest of the script because it is executed as a separate command, not related to any other line. We should probably try to avoid such errors instead of forcing users to install components which are ‘only comsetic’ (the popup message has no relevance for functionality).

I just found https://stackoverflow.com/questions/11231937 which discusses ways to avoid the termination of a script due to a single failed command. Maybe we could test some of the approaches listed there?

Another idea would be to just change the order of commands in /etc/qubes-rpc/qubes.SshAgent (socat first, notify-send second). Im not sure if this works, though.
Or we could check for existence of the command before exexuting it (see, e.g., https://stackoverflow.com/questions/592620).

I guess may brain was in usual programming language mode and didn’t even realize was not to be expected. But yes, in bash by default if something breaks, all the rest should be executed… So I’m not sure why this broke it.

I think for security the non-displaying of a notification should stop the rest of the script from executing. This is because the notification actually notifies the user that something happened. If it doesn’t show the user might not notice something has happened in the background.

Imho the security comes from the qrexec protocol itself which allows only those interactions that one has allowed in dom0 (either by explicitly defining “allow” in the rpc policy file or by interacting with a dom0 prompt if “ask” policy is used). The notify-send popup is just a nice bonus.

1 Like

While reading Restricting a Qube to selected websites I learned about qvm-connect-tcp (The Qubes Firewall | Qubes OS) which is probably what we should be using instead of our own netcat/socat approach.

I probably won’t be able to test this myself until in a few weeks.


On 4.1 this additional packages do not fix the problem.

Followed the nice guide on github, tested with both fedora-33 and debian-10 templates but ssh-add -L only lists keys in he vault, ssh-client AppVM always replies with
“error fetching identities: communication with agent failed”

Any ideas what i might be missing?

Edit: github guide works with stable Qubes 4.05

Please double chech all your AppVM names. I’d say in 90% of these cases it is a spelling mistake.

Jun 28, 2021, 18:18 by qubes_os@discoursemail.com:

Just set it up with my stable 4.0 Qubes and it works, i doubt i made typos since i copy pasted from guide into template and AppVMs.
Have been using split-ssh for a few years now on Qubes 4.0.x and switched to socat today.
Nevertheless will go one more time from scratch on the Qubes 4.1.

The guide does not work on 4.1.
Tests with original Marmarek 4.1. Alpha iso from last year and today clean install with latest 4.1. Alpha weekly iso and it just will not work.
Tests were done with a clone from a clean fedora-33 Template (ssh-client & vault-ssh AppVms) which comes with installed libnotify.
Without or with mate-notification-daemon no dice.
Test were done with Stable dom0 and current-testing update and reboot.

Would love to help pinpoint the problem but not sure where to look.

After taking a break from 4.1 testing i did find what the split-ssh guide on github is missing.
The vault vm template /etc/qubes-rpc/qubes.SshaAgent file needs to be made executable,
chmod +x /etc/qubes-rpc/qubes.SshAgent

I guess you’ve a typo here

For the record, this chat has result in this how-to wiki:


Yup. I had also linked it back in a previous post. But it’s a good reminder :slight_smile:

I’ve also edited the original post to include a link to that document.

Could you label it as solution.

1 Like

Yes, I have now moved it out of the “guides” category (since the discussion itself is not the guide) and have marked that as the solution.

1 Like