USBIP protocol errors

hax0rbana_adam · February 5, 2024, 9:04pm

I’m trying to use a signet (which is a USB hardware password manager) in Qubes 4.2. It worked fine in 4.1, but after I did a fresh install of 4.2, it doesn’t work. The device passes through and shows up in the target qube, but doesn’t receive any commands sent to /dev/hidraw0 (which I verified with a UART from the device itself).

I believe this is an issue with Qubes 4.2, but I wanted to see if anyone here first. My main two questions are:

Does anyone know about any USB packet filtering for hidraw devices, that started happening in 4.2?
Does anyone know where I should look next to get more information?

Debugging done so far

When the device is first plugged in, I see it in the syslogs of sys-usb using sudo journalctl -f and everything looks fine.

When I pass it through to a qube, it seems to attach just fine. It says “Attaching device” and there’s no errors that pop up.

However, when I attach it, sys-usb immediately starts printing out errors related to usbip:

Feb 05 13:49:06 sys-usb kernel: usbip-host 2-6.3: urb completion with non-zero status -71
Feb 05 13:49:06 sys-usb kernel: usbip-host 2-6.3: urb completion with non-zero status -71
Feb 05 13:49:06 sys-usb kernel: usbip-host 2-6.3: urb completion with non-zero status -71
Feb 05 13:49:07 sys-usb kernel: usbip-host 2-6.3: urb completion with non-zero status -71

Looking at the kernel version, we see this:

Linux sys-usb 6.1.62-1.qubes.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 14 06:16:38 GMT 2023 x86_64 GNU/Linux

Looking at the kernel source for that kernel version, we see that -71 is -EPROTO.

Meanwhile, in the qube, the syslog has errors related to usb:

Feb 05 13:33:18 adam-hax0rbana-no-tor kernel: usb 1-1: reset full-speed USB device number 2 using vhci_hcd
Feb 05 13:33:18 adam-hax0rbana-no-tor kernel: vhci_hcd: vhci_device speed not set
Feb 05 13:33:18 adam-hax0rbana-no-tor kernel: usb 1-1: SetAddress Request (2) to port 0
Feb 05 13:33:18 adam-hax0rbana-no-tor kernel: cdc_acm 1-1:1.0: ttyACM0: USB ACM device
Feb 05 13:33:19 adam-hax0rbana-no-tor kernel: vhci_hcd: vhci_device speed not set
Feb 05 13:33:19 adam-hax0rbana-no-tor kernel: usb 1-1: reset full-speed USB device number 2 using vhci_hcd
Feb 05 13:33:19 adam-hax0rbana-no-tor kernel: vhci_hcd: vhci_device speed not set
Feb 05 13:33:19 adam-hax0rbana-no-tor kernel: usb 1-1: SetAddress Request (2) to port 0
Feb 05 13:33:20 adam-hax0rbana-no-tor kernel: cdc_acm 1-1:1.0: ttyACM0: USB ACM device

Now, if we just ignore these errors, we see that /dev/hidraw0 and /dev/hidraw1 both appear in the target qube, which is the expected behavior. The signet client software opens /dev/signet (which is symlinked to /dev/hidraw0 via a udev rule), it successfully opens the handle and send the startup message to said file handle. It never gets a response.

Using a UART port on the USB device itself, I confirmed that the message never makes it to the device.

Between the errors in the logs, and the message not getting from the target qube to the device, it seem like the attach command is failing in some way which is causing data to not make it through to the qube.

Variations

I’ve tried varying some things to see if I can narrow down the issue, but it persists in all of the cases below:

Plug directly into the computer instead of using a USB hub
Try a different signet device (same model, different physical instance)
Attach it to a different qube
Try the device in another computer (it works fine on a Debian 12 machine)
Run the signet client from sys-usb; it acted just like in the target qube where it could open the device and send data, but then it never gets a reply

apparatus · February 6, 2024, 4:09am

Since it has the same issue in sys-usb then it’s not related to USBIP.
Maybe try to use newer kernel from dom0 kernel-latest-qubes-vm package in sys-usb or try the kernel installed in VM:

hax0rbana_adam · February 6, 2024, 7:18am

Thanks for the suggestion. I tried the “provided by qube” option for the kernel, which got me 6.6.14, but it behaved the same way.

Linux sys-usb 6.6.14-100.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jan 26 20:10:55 UTC 2024 x86_64 GNU/Linux

There aren’t any errors in the syslogs when I attempt to use the device in sys-usb, so that part is different, but the client sending off a packet and never getting a response is the same.

If someone in the US is willing to dig into this with me, I’d be willing to ship you a signet device to whatever name and address you give me. I really want the device to work with Qubes again! (but international shipping is really expensive!)

hax0rbana_adam · February 15, 2024, 11:46pm

I have more information, but no real answers yet.

If I boot into Tails, I also have problems with Signet (I did not do as detailed of debugging to confirm that it’s exactly the same thing, but it acted the same when using the release version of the Signet client). This seems to point to a problem with the USB controller, but I’m already tried two USB controllers on the motherboard and…
Installing a PCI → USB adapter, attaching to it sys-usb, and passing it through to a target qube has the exact same problem! I can see the client send the message, but then it just hangs.

So this one really has me stumped. I did change motherboards when I switched to Qubes 4.2, but since other USB devices seem to work fine, and I chose my motherboard based on the Qubes HCL having multiple reports that the ASRock B450M Pro4 was going to be solid.

My next steps are to test this machine using the PCI card in Tails and try to find another machine where I can install Qubes 4.2. If Signet works fine on another machine with Qubes 4.2, then I know this is a “me problem” and I can figure out if it’s something weird with the motherboard, or maybe a reinstall would fix it, or maybe I’ll figure out something else I can try.

Unfortunately I don’t have the money to just buy another computer consisting of the same components in order to test this. If I still had a high paying tech job, I’d certainly consider that option though!

hax0rbana_adam · March 10, 2024, 7:48pm

I went out and bought a new motherboard (Asus ProArt B550-creator, based on the HCL), and it has this same problem with Qubes 4.2! This is such a mystery!

I’m going to continue testing (the new motherboard with Tails, with Qubes 4.1) to see if I can figure out if I can get it to work, and if so, what the difference is. Just wanted to post an update so people know I’m still very much working on this and trying to find an answer so I can report it back here. The idea of some unknown component of the system messing with hidraw devices in a way that is not understood is… concerning.

hax0rbana_adam · April 15, 2024, 3:32am

After much testing, I found that this is an issue with the USB passthrough system (which may be an upstream bug). I no longer believe it is related to any compile time kernel options or kernel versions.

The problem is exhibited the first time the hidraw device is attached to a qube. Disconnecting the device from the qube and reconnecting it causes the problem to disappear. If the physical hardware is removed and re-inserted, another cycle of attach/detach/attach will be required. Also, the first attachment does not need to be to the same qube as the second attachment.

I can also say that my testing methodology has improved. I can reliably recreate the USB connection failure problem in Qubes 4.2 on multiple computers (I wasn’t able to test Qubes 4.1, but that’ll be EOL soon enough anyway).

I wrote up repro steps on the Signet issue tracker but they won’t be terribly useful unless you have a signet device (or possibly any other USB hidraw device?)

Since I can now reliably repro the problem, I re-tested this on sys-usb itself. The Signet works fine there (no workaround required).

I’m unable to reproduce the problem on any version of Tails, so I’m not sure what was going on there.

My next steps are to write a minimal command line program to demonstrate the issue. Right now I can only demonstrate it with the Signet client, which is a multithreaded QT program that is really gnarly to debug. After I have the minimal example, I’ll see if I can figure out a root cause and open tickets on whatever bug trackers are appropriate.