SSH client with KeePassXC based on a minimal Debian template

User Support General
1

I tried to move my ssh and vault AppVMs based on a (default) Debian 11 template to minimal Debian templates. My setup is based on split-ssh (Forum thread) .

Naming:

  • vault : previous (default) vault based on the standard Debian template
  • ssh : new ssh AppVM based on deb-11-m-ssh
  • deb-11-m-ssh : new minimal Debian 11 template for ssh (only)
  • secrets : new “vault” AppVM based on deb-11-m-secrets
  • deb-11-m-secrets : new minimal Debian 11 template for secrets (only)

Checks I did:
In the ssh terminal, checking the agent identities with:
ssh-add -L
works fine with interconnection to vault - returns my ssh key.

In the ssh terminal, checking the agent identities with:
ssh-add -L
does not work with interconnection to secrets - returns error fetching identities: communication with agent failed.

Therefore, I do expect a missing installation or configuration in either the secrets AppVM or in the deb-11-m-secrets template.

What I did:

Configuration in dom0:

  • in /etc/qubes-rpc/policy/qubes.SshAgent
    @anyvm @anyvm ask

Installed packages in deb-11-m-secrets:

  • qubes-core-agent-networking
  • gnupg
  • policykit-1
  • libblockdev-crypto2
  • ssh
  • ssh-askpass-gnome

Configuration in deb-11-m-secrets:

user@deb-11-m-secrets:~$ sudo gedit /etc/qubes-rpc/qubes.SshAgent

#!/bin/sh
# Qubes App Split SSH Script

# safeguard - Qubes notification bubble for each ssh request
notify-send "[$(qubesdb-read /name)] SSH agent access from: $QREXEC_REMOTE_DOMAIN"

# SSH connection
socat - "UNIX-CONNECT:$SSH_AUTH_SOCK"

user@deb-11-m-secrets:~$ sudo chmod +x /etc/qubes-rpc/qubes.SshAgent

Configuration in ssh:

user@ssh:~$ sudo gedit /rw/config/rc.local

added to file:

# SPLIT SSH CONFIGURATION >>>
# replace "vault" with your AppVM name which stores the ssh private key(s)
SSH_VAULT_VM="secrets"

if [ "$SSH_VAULT_VM" != "" ]; then
  export SSH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
  rm -f "$SSH_SOCK"
  sudo -u user /bin/sh -c "umask 177 && exec socat 'UNIX-LISTEN:$SSH_SOCK,fork' 'EXEC:qrexec-client-vm $SSH_VAULT_VM qubes.SshAgent'" &
fi
# <<< SPLIT SSH CONFIGURATION

and in

user@ssh:~$ gedit ~/.bashrc

added

# SPLIT SSH CONFIGURATION >>>
# replace "vault" with your AppVM name which stores the ssh private key(s)
SSH_VAULT_VM="secrets"

if [ "$SSH_VAULT_VM" != "" ]; then
  export SSH_AUTH_SOCK="/home/user/.SSH_AGENT_$SSH_VAULT_VM"
fi
# <<< SPLIT SSH CONFIGURATION

followed by the KeePassXC configuration. Got graphical feedback with KeePassXC UI “SSH Agent connection is working!” and the pre-test also returns correctly my ssh key (within the secrets AppVM).

Consequently, my guess is that I am missing a package in my deb-11-m-ssh template. What do you think? Any further test I can do?

2 Likes
2

@fsflover any hint what package I miss?

3

Sorry, this is beyond my knowledge of Qubes and Linux. I hope someone else could help you.

4

In a case of a missing package I always try to meld diffs between working and non-working cloned template, then using common sense install one by one in a clone and try it.
VM interconnection setup error like yours should narrow enough your search for a missing package. I’d start with qubes repo packages diffs.

1 Like
5

I have spent 2,5 days now to check, compare install and remove packages and checked 10 times the config files (in the AppVM, TemplateVM). I am still not able to get it working with the minimal template setup. I pause here and hope that someone who has a working minimal debian split-ssh setup read the post. :crossed_fingers:

1 Like
6

I remember having this problem, here’s my fix (run as root in deb-11-m-secrets):

chmod 755 /usr/bin/ssh-agent

Then shutdown the template and restart the vm.

7

Thanks for this tip, unfortunately, this does not fix it on my side.

In my working (ordinary) debian-11 template it is also not set to chmod 755.

  • ssh (appVM based on deb-11-m-ssh) > vault (appVM based on debian-11): works.

  • ssh (appVM based on deb-11-m-ssh) > secrets (appVM based on debian-11): does not work. Returns “error fetching identities: communication with agent failed”

Do you have a working ssh with secrets AppVM based on a minimal Debian? If yes, could you please share your installed packages (for both templates deb-11-m-ssh and deb-11-m-secrets)?

8

@whoami did you ever figure this out? I’m having the same issue.
I may try my hand at comparing packages this next week if I have the time, but until then I guess I’ll just have to base my vault on regular debian-11.

9

No, I pause here and continuing with some work on my automated minimal template script. I would be really happy when you can figuring this out. Please keep us posted. Thank you.

10

Hi @whoami, long time no see ;).

Wild guess (without being able to test this myself currently): might it be that notify-send is not installed in deb-11-m-secrets?

If so, I would expect an error message somewhere in the output of sudo journalctl in your secrets VM. Not sure how such errors would be logged though, which is why I suggest a more simplistic debug method.

Hypothesis: the call from SSH to secrets works in theory, but the code in /etc/qubes-rpc/qubes.SshAgent somehow fails. You could check this hypothesis by inserting a debug command right at the beginning of the file. Something like echo "Debug test" > /home/user/debuglog.txt and then check in secrets VM if debuglog.txt exists and contains the line after you try connecting to the agent from within your SSH VM.

You can insert the command right in your secrets VM, no need to edit the template and restart the appVM. Sure, this will reset when you shutdown secrets but as it is only a temporary debug help that’s no problem.

11

Hi @phl !
nice surprise :slight_smile:

Thanks for your wild guess . I did a quick search and decided to install both packages:

  • libnotify-bin
  • mate-notification-daemon

on deb-11-m-secrets template (restarted template and appVM).

Now, I see a notification popup but split-ssh is still not function as expected (as the vault appVM do).

I cannot edit my previous post; there is a typo on the template name so just for clarification

Testing with ssh-add -L

  • in ssh (based on deb-11-m-ssh) to vault (based on debian-11): works
  • in ssh (based on deb-11-m-ssh) to secrets (based on deb-11-m-secrets): Does not work
  • in vault (based on debian-11): works
  • in secrets (based on deb-11-m-secrets): works

Therefore, the issue points to the secrets appVM or templateVM.

qubes.SsAgent code works: I get a debuglog.txt dummy file.

PS: In both appVMs secrets and vault the KeePassXC settings shows me a working SSH Agent connection is working!

12

That is strange, what else could it be?
I thought it could be related to socat as well but since that is also used in the ssh VM to initiate the connection this shouldn’t be the case…

Nevertheless, what happens if you don’t put the debug log command on the first line, but between the commands or right at the end of the qubes.SshAgent file?

I don’t really see what else could go wrong here, given the behaviour described in your previous posts (works in combination with other VMs but not in this exact scenario).
You could also double-check if all variables are set up correctly by executing env | grep SSH in both secrets and ssh to see if anything unusual shows up.

I’ll try to replicate the situation as soon as I can. Maybe I can spot an error message somewhere in the logs if I am looking at it myself. Remote debugging is always difficult:/.

13

Without putting pressure on you, did you do some tests? Any new insights?

14

I tried to set this up, but until now I didn’t even get to the point where you currently are. My KeePassXC complains that no SSH_AUTH_SOCK variable can be found. It seems that ssh-agent is not set up correctly on boot of the VM.

I realized that your description mentions some things that do not work out of the box in minimal templates. For example you are using sudo in your commands, but passwordless sudo first needs to be set up. Also you mention gedit, which you use for file editing, but this package is not installed by default. Did you make some other changes to the minimal template which could help me to get ssh-agent working?

15

I added to the default Debian minimal template:

qubes-core-agent-passwordless-root
qubes-app-shutdown-idle

Please ignore gedit just use nano.

qubes-core-agent-networking
gnupg
policykit-1
libblockdev-crypto2
ssh
ssh-askpass-gnome

PS: I have made a script to automatically create templateVMs and appVMs. If this helps you to quickly recreate Qubes for further testing just DM me.

16

… it took 9 months but I made it :partying_face:

I simply checked and compared the installed packages for ssh and qubes: default “vault” based on default “debian-11” vs. the minimal Debian “deb-11-m-secrets”

For the log:

apt list --installed *package*

is your best friend :wink:

I installed package by package, rebooted templateVM, checked the agent key request ssh-add -L … and surprisingly

  • qubes-core-agent-networking
  • qubes-core-agent-network-manager

made split-SSH working on my minimal Debian template.

Does this mean split-ssh with Qrexec requires network installation in the template but in the AppVM we can still securely set net qube to none

Credits goes to @enmus SSH client with KeePassXC based on a minimal Debian template - #4 by enmus

I just was not smart enough to strictly follow your suggestion.

17

I doubt you need those packages in your minimal TemplateVM for any ssh vault.

The community guide for split-ssh doesn’t make any mention of how, on Debian at least, Xsession.options control the initialization of the ssh agent per-user, or how to disable that behavior by altering the Xsession options (see man 5 Xsession.options).
Xsession.options is related to startx and, though I don’t profess to know the details myself, in a minimal TemplateVM you don’t really need to have run startx, so you’re not necessarily going to have had the benefit of the same initialization behaviors a non-minimal TemplateVM-derived qube would have had, hence you get different behaviors like your qrexec policy (which is defaulting to user=user in the ssh vault qube it reaches) not finding any ssh-agent already running for user user, if I recall correctly.

More likely by installing more packages in the minimal TemplateVM, you dragged in enough things to drag in something that brought back in the Xsession.options and startx stuff, but you don’t need those! You can just start the ssh-agent yourself and tell it where you want it to keep its socket (somewhere reasonable and predictable like /run/user/1000/split-ssh.socket) by writing a systemd service or maybe starting ssh-agent in rc.local and/or from qubes.SshAgent on demand if the expected socket file doesn’t exist yet on disk, including having qubes.SshAgent run ssh-add to add the identities rather than relying on per-user autostart desktop entries to get run by some unspecified set of packages that the ssh vault really doesn’t need.

Granted, since your interest is the KeepassXC integration however, you do seem to want a graphical environment in your ssh vault, but that environment isn’t otherwise necessary for the vault in particular.

While I’m at it I would also point out the current community split-ssh tutorial also should not be calling notify-send from qubes.SshAgent itself. All qrexec policy actions accept a notify=yes argument which will show a notification with functionally the same information as what that was showing, and there’s just no need to pull in the extra packages for notification-daemon and friends into the minimal TemplateVM for the ssh vault AND the notification itself is pure theater in that it doesn’t actually establish any form of trusted operation as far as I can tell.
The security benefits of split-ssh rest entirely on keeping the secrets secret, so the less packages and theater that goes on in the vault and its TemplateVM, the better.

18

I did some further investigations to the packages which are installed with

  • qubes-core-agent-networking
  • qubes-core-agent-network-manager

Now, I can provide a final solution for SSH client with KeePassXC based on a minimal Debian template (based on split-ssh).

The required packages for the deb-11-m-secrets are:

  • keepassxc
  • ssh-askpass-gnome
  • socat (this was the missing package which comes with qubes-core-agent-networking. Actually, it is pretty obvious when you see this image)

for convenience and additional split-gpg usage I have added:

  • qubes-core-agent-nautilus
  • nautilus
  • zenity
  • gnupg
  • qubes-gpg-split

when using a security token key (i.e. Yubikey):

  • qubes-usb-proxy: required to pass through an USB device
  • xserver-xorg-input-libinput: only required for static passwords
  • policykit-1: only required for challenger response

when using KeepassXC OTP feature:

  • systemd-timesyncd

If you find additional fine tuning please let me know.