Sshfs over qvm-connect-tcp hangs

User Support
1

Hi, I have setup a simple ssh server qube that provides access to shared resources:

# /etc/ssh/sshd_config
AllowUsers user
PrintLastLog no
AuthenticationMethods publickey
PasswordAuthentication no
PubkeyAuthentication yes
X11Forwarding no
AllowTcpForwarding no
AllowAgentForwarding no
ChallengeResponseAuthentication no
PermitTTY no
ForceCommand internal-sftp
PermitUserRC no

Match User user
  ChrootDirectory /srv/chroot-jail
EOF

Client qube mounts via

qvm-connect-tcp ::22

sshfs \
  -o reconnect,ServerAliveInterval=10,ServerAliveCountMax=1,IdentityFile=/home/user/.ssh/id_ed25519,idmap=user \
  localhost:/ \
  /mnt

But this connection is unstable: Fast-forwarding a 1080p movie can result in hang, same as ffmpeg processing of multiple files at some arbitrary point in time/file. pkill -9 sshfs (not even normal kill) is the only solution here.

sshfs -o debug,loglevel=debug and journalctl -f did not show anything useful.
-o reconnect,ServerAliveInterval=10,ServerAliveCountMax=1 options did not cause a restart of sshfs. Hence I suspect, it is more an issue with qvm-connect-tcp / qrexec.

Are there known issues with larger disk I/O/workload operations over qrexec?

2

I would first try to separate sshfs from the qrexec path. For a quick test, run a plain rsync/scp of a large file over the same qvm-connect-tcp port, and also try sshfs with less aggressive caching, e.g. -o cache=no -o kernel_cache=no. If rsync/scp is stable but sshfs still wedges, it points more at FUSE/sshfs under that workload than at qrexec itself. If both hang, then checking the qrexec service logs on both sides right after the stall is probably the useful next step.

3 Likes
3

Thanks @anna_qubes_berlin ! Sorry for the delay, I tested your suggestions out:

  • Beforehand: journalctl did not show any error about sshfs/qrexec/FUSE with current solution
  • Then completely switched to SMB/CIFS on server qube (Samba) to exclude sshfs/FUSE stack
  • This runs much better than sshfs (no hangs), but quite slow.

Slow performance probably is due to nested CIFS shares:

client qube -- CIFS --> share A on server qube -- CIFS --> share B on external NAS

Not sure, if this security theater (is it?), but:

  • I don’t have control over securing that external NAS incl. its SMB service
  • Hence goal was to use a minimal, hardened SMB/CIFS proxy qube for client access, that itself mounts shares from external NAS.

Feedback is welcome, if anyone has similar setup.

4

Not sure what’s causing your sshfs setup to hang. But you could simplify it by forwarding only the inner plain SFTP protocol layer over qrexec (instead of the outer SSH layer wrapping it), and plug the forwarded stdio streams directly into sshfs’s slave mode. Maybe that would work better?

E.g. add an executable script at /usr/local/etc/qubes-rpc/sftpServer in the server qube:

#!/bin/sh

exec ssh -q -s user@localhost sftp

And connect from the client qube:

socat 'EXEC:sshfs \:/ /mnt -o slave\,idmap=user' \
      'EXEC:qrexec-client-vm @default sftpServer'

This way, SSH authentication and encryption happens entirely inside the server qube. (It would be nice to strip away this now pointless SSH layer and just call ssh’s standalone sftp-server binary in the sftpServer qrexec script, but that makes it more annoying to setup a chroot compared to ssh’s internal-sftp.)

1 Like
5

sshfs worked just fine. But after NAS software got major update, issues seemed to occur. Direct network access without qrexec though works fine so far.

Didn’t know about -o slave, appreciate the reply @rustybird - will try that out.

6

Learned a lesson: Always specify SMB/CIFS version: mount -t cifs -o vers=3.11.

With this addition for server qube → external CIFS share, sshfs → server qube performance is perfectly fine again. Hard failing given unsupported SMB versions or negotiation - yes. But I didn’t consider its absense to be cause for weird hanging issues :wink:

2 Likes