[qubes-users] ssh-split issue

haaber · June 21, 2023, 9:45am

I tried to follow carefully the split-ssh instructions on

Qubes-Community/Contents/blob/master/docs/configuration/split-ssh.md

# Qubes Split SSH

Split SSH implements a concept similar to having a smart card with your private SSH keys, except that the role of the “smart card” is played by another Qubes AppVM. 
This Qubes setup allows you to keep your SSH private keys in a vault VM (`vault`) while using an SSH Client VM (`ssh-client`) to access your remote server. 
This is done by using Qubes's [qrexec][qrexec] framework to connect a local SSH Agent socket from your SSH Client VM to the SSH Agent socket within the vault VM. 
This way the compromise of the domain you use to connect to your remote server does not allow the attacker to automatically also steal all your keys. 
(We should make a rather obvious comment here that the so-often-used passphrases on private keys are pretty meaningless because the attacker can easily set up a simple backdoor which would wait until the user enters the passphrase and steal the key then.)

   ![diagram](/attachment/wiki/split-ssh/diagram.svg)

## Security Benefits

In the setup described in this guide, even an attacker who manages to gain access to the `ssh-client`  VM will not be able to obtain the user’s private key since it is simply not there. 
Rather, the private key remains in the `vault` VM, which is extremely unlikely to be compromised if nothing is ever copied or transferred into it. 
In order to gain access to the vault VM, the attacker would require the use of, e.g., a general Xen VM escape exploit or a signed, compromised package which is already installed in the TemplateVM upon which the vault VM is based.

## Overview

1. Make sure the TemplateVM you plan to use is up to date.
2. Create `vault` and `ssh-client` AppVMs.

This file has been truncated. show original

but I experience a stupid error. I did a "plain setup" without keepass.
So when I run

ssh-add -L in vault it does work, and has 1 test-identity.

ssh-add -L in work it does not work "Error connecting to
agent: Connection refused"

even with "-v -v -v" I get no better hint. So I tried to follow the traces;

1.) This happens when I ("manually") run

user@work:~$ bash -x /etc/qubes-rpc/qubes.SSHagent

++ qubesdb-read /name
+ notify-send '[work] SSH agent access from: dom0'
+ socat - UNIX-CONNECT:/home/user/.SSH_AGENT_sshkeys
2023/06/21 11:24:59 socat[1562] E connect(, AF=1
"/home/user/.SSH_AGENT_sshkeys", 34): Connection refused

you may observe that I wrote SSH with 3 capital letters, but I did so
everywhere (I hope :), inclusive the small script snipplets from github
page.

2.) This happens when I query the ssh agent:

systemctl --user status ssh-agent.service
● ssh-agent.service - OpenSSH Agent
Loaded: loaded (/usr/lib/systemd/user/ssh-agent.service; static)
Active: active (running) since Wed 2023-06-21 11:18:46 CEST; 22min ago
Docs: man:ssh-agent(1)
Main PID: 1513 (ssh-agent)
Tasks: 1 (limit: 4618)
Memory: 872.0K
CPU: 3ms
CGroup:
/user.slice/user-1000.slice/user@1000.service/app.slice/ssh-agent.service
└─1513 ssh-agent -D -a /run/user/1000/openssh_agent

Jun 21 11:18:46 work systemd[654]: Started ssh-agent.service - OpenSSH
Agent.
Jun 21 11:18:46 work agent-launch[1515]:
dbus-update-activation-environment: setting
SSH_AUTH_SOCK=/run/user/1000/openssh_ag>
Jun 21 11:18:46 work agent-launch[1515]:
dbus-update-activation-environment: setting SSH_AGENT_LAUNCHER=openssh
Jun 21 11:18:46 work agent-launch[1513]:
SSH_AUTH_SOCK=/run/user/1000/openssh_agent; export SSH_AUTH_SOCK;
Jun 21 11:18:46 work agent-launch[1513]: echo Agent pid 1513;

We observe that the file /run/user/1000/openssh_agent is different from
/home/user/.SSH_AGENT_sshkeys. That may be a problem.

I tried to fix that temporarily with linking one to the other ("ln -s")
. Then ssh-add -L does not fail, but has no identities.

Here I am stuck. Any hints? Thank you, Bernhard

haaber · June 21, 2023, 9:49am

update: somewhere I wrote "vault", somewhere "sshkeys". Correcting this
does NOT resolve the problem :((

mailinglist_bot · June 21, 2023, 11:05am

Running the following command in the work qube should work:
SSH_AUTH_SOCK=/home/user/.SSH_AGENT_vault ssh-add -L

You seem to be running the "ssh-agent.service" in your work qube. This is not part of the linked setup guide. There only one agent is running and that is in the vault qube.

The "clients" (e.g. work qube) only redirect the communication via socat, qubes RPC and the /home/user/.SSH_AGENT_vault file to the ssh-agent in the vault qube.

See: Contents/docs/configuration/split-ssh.md at master · Qubes-Community/Contents · GitHub

haaber · June 21, 2023, 11:54am

Thank you.

>> We observe that the file /run/user/1000/openssh_agent is different
>> from /home/user/.SSH_AGENT_sshkeys. That may be a problem.

You seem to be running the "ssh-agent.service" in your work qube. This
is not part of the linked setup guide. There only one agent is running
and that is in the vault qube.

right, that was a remainder from various tests to debug. I killed it.
Actually, to revert everything to clean setup state, I restarted both
VM's, work and vault.

The "clients" (e.g. work qube) only redirect the communication via
socat, qubes RPC and the /home/user/.SSH_AGENT_vault file to the
ssh-agent in the vault qube.

thank you for clarification.

> Running the following command in the work qube should work:
> SSH_AUTH_SOCK=/home/user/.SSH_AGENT_vault ssh-add -L

error fetching identities: communication with agent failed

That is the answer, with a pop-up message from qubes "denied
qubes.SSHagent from work to vault". Something is odd ... now dom0 log
says "qrexec: qubes.SSHagent: work -> vault: denied: no matching rule
found". I re-checked, my /etc/qubes-rpc/policy/qubes.SSHagent says only
one line, namely

ssh-client vault ask

which I find odd (= I do not understand), since in the nomenclature of
the man-page, ssh-client=work not vault, right?

thank you, Bernhard

haaber · June 21, 2023, 11:58am

Solved. Stupid me!

> I re-checked, my /etc/qubes-rpc/policy/qubes.SSHagent says only

one line, namely

ssh-client vault ask

which I find odd (= I do not understand), since in the nomenclature of
the man-page, ssh-client=work not vault, right?

it is of course either

"@anyvm vault ask" or "work valut ask", right? That does it! Stupid me.

haaber · June 21, 2023, 12:20pm

remaining question: if I want vault to hold several SSH keys, should I best

(a) replace the single ssh-add command by "ssh-add c /path/to/key1 &&
ssh-add -c /path/to/key2"

or

(b) create a ssh-add-my-keys.sh containing

ssh-add -c /path/to/key1

ssh-add -c /path/to/key2

ssh-add -c /path/to/key3

and modify the autostart line to

Exec=/path/to/ssh-add-my-keys.sh