Windows USB integration with R4.1

[jevank]

@unjust

But I need to patch qemu-stubdom-linux-full-rootfs also?

Correct

[unjust]

Holy smokes, I am indebted to you again. It works!

[joe.blough]

@jevank

It is now available to test ability to attach USB devices to Windows. It works without QWT installation via QEMU emulation. To get it works make this steps:

• update with current-testing repos
  sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing
• install advanced variant linux stubdomain (now available only in current-testing repos):
  sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing xen-hvm-stubdom-linux-full
• update and restart sys-usb with current-testing repos too
• enable feature to windows qube
  qvm-features [windows qube] stubdom-qrexec 1

Any chance these updates work on 4.04?

[jevank]

Not a chance, sorry.

[joe.blough]

@jevank

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

I have numerous No Match for argument lines after entering sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing but I think this is a function of a bad 4.1 migration.

[estoner]

Thank you @jevank

I followed these instructions explicitly for Windows 7 stand-alone HVM. I understand that you don’t need QWT installed to make the USB integration work, but since mine has QWT 4.1.65 installed I wanted to report in that it works with QWT so others can benefit from your work

I’m running latest 4.1 testing

@jevank

• update with current-testing repos
  sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing
• install advanced variant linux stubdomain (now available only in current-testing repos):
  sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing xen-hvm-stubdom-linux-full
• update and restart sys-usb with current-testing repos too
• enable feature to windows qube
  qvm-features [windows qube] stubdom-qrexec 1

For windows 7 it needs to install drivers, we was successful with these

[enmus]

Hello. Thanks jevank for this great contribution. I am Qubes newbie, and I can confirm that this works without QWT - I succesfully attached external usb sound card to Win 7 cube. The sound is “chopy”, or “scratchy” but it’s the same with the integrated sound card. What I haven’t succeeded was to attach external usb HDD, nor any other kind of block device.
Is it not possible yet, or I am doing something wrong? Widget shows that the block device is attached (bold) to the cube, but it isn’t seen in WIn7 cube. I have tried from the cli with qvm-block and qvm-start --hddisk commands as well, but with no luck.

Estoner mentioned qwt 4.1.65, but where I can find it?

Thank you in advance for your help.

EDIT: all my usb devices are connected via sys-usb, and are seen in Linux cubes when attached.

[jevank]

@enmus

Is it not possible yet, or I am doing something wrong? Widget shows that the block device is attached (bold) to the cube, but it isn’t seen in WIn7 cube. I have tried from the cli with qvm-block and qvm-start --hddisk commands as well, but with no luck.

In case USB connectivity you should use devices from ‘USB Devices’ section in widget, nor ‘Block devices’. To get block devices support with Windows it needs to install Xen virtual disk device drivers, but it might (would) be unstable with hot plugging

@enmus

Estoner mentioned qwt 4.1.65, but where I can find it?

Look at github repository, it has instructions to build and this long topic might be useful.

[enmus]

Thanks jevank for clarification and hints for qwt. After testing for several days, here are my findings.

0. Created standalone Win7 SP1 cube. Fully updated, Made a backup of it
1. I have successfully built qwt 4.1.65 and installed it. When starting the Windows VM, a pop-up with the message Failed to execute qubes.NotifyTools from <VMname> to dom0 is displayed. No significant consequences noticed.
2. I was able to attach internal and USB hdd’s<2TB as block devices. Fully operational and functional.
3. Inter-vm interactions worked flawlessly.
4. Couldn’t manage to attach >2TB USB hdd as a block device.
5. Successfully finished procedure for updating xen-hvm-stubdom-linux-full following your advices, coexisting with QWT.
6. Successfully attached all non-storage and non-streaming devices.
7. Couldn’t manage to attach any USB hdd. Win 7 sees it and it starts to find the drivers, and then just gives me the error “device unplugged”?
8. “OK, let’s try to attach all hdd’s via qwt and the rest of usb devices via jevank’s method”. I have read whole “long topic” and applied:

• qvm-prefs --set win7 qrexec_timeout 600, no luck
• qvm-features win7 gui-emulated 0, no luck
• qvm-features win7 gui-emulated 1, no luck

9. Even worse, now I can’t attach any hdd, either as a block device, or as an usb device. If I try as a block device, it starts to attach it, and the cube window just disappears, while Qubes manager shows cube is still running and the widget is showing hdd is attached as block device. After detach it via widget, cube shuts off with a pop-up notification it is halted.
10. Deleted Win 7 cube, and restored the fresh one from the backup, but with the same result, which leads me to think that dom0 is somehow screwed. No Qubes OS restart helps. No other action than stated here was done, I have went through whole history of commands in dom0 terminal.
11. At the end, I tried to restore dom0 cube from the earlier working backup, which was made weeks before I started with win 7. No luck - same results.

I am too desperate, so if you, or anyone else have any thoughts on this, I’d be very grateful

[jevank]

@enmus

Couldn’t manage to attach any USB hdd. Win 7 sees it and it starts to find the drivers, and then just gives me the error “device unplugged”?

Does it works with linux VMs? I know some problems with such devices, but it has same errors on Linux (USB cable is bad or USB streams unsupported).

@enmus

“OK, let’s try to attach all hdd’s via qwt and the rest of usb devices via jevank’s method”. I have read whole “long topic” and applied:
…

What is this about? The ‘gui-emulated’ feature enable window from emulated video card, that couldn’t break smth

@enmus

Even worse, now I can’t attach any hdd, either as a block device, or as an usb device. If I try as a block device, it starts to attach it, and the cube window just disappears, while Qubes manager shows cube is still running and the widget is showing hdd is attached as block device. After detach it via widget, cube shuts off with a pop-up notification it is halted.

Check VM settings to ensure that memory balancing is disabled (Advanced Tab).

[enmus]

Thanks for your response!

1. Hdd’s are fully operational both in fedora, whonix, or debian template based vm’s either via usb widget, or as block devices.
2. Memory balancing was turned off from the very beginning.
3. In despair, I tried emulated, because Win7 cube window disappeared when attaching >2TBB usb hdd in order to try to get it back somehow - no luck as I said.
4. New findings: if I restart Win7 cube and first try to attach <2tb usb hdd, it succeeds and is fully operational. If in addition I try to attach >2TB usb hdd, it tries to attach it, then it unplugs itself, but also it unplugs already attached <2TB usb hdd’s, as well as all other usb devices (sometimes all at once, sometimes one at a time, after additional manual retries to attach >2TB hdd).
5. Once self-detached from win7 cube, >2TB usb hdd is shown in the list of usb devices widget as not bold, but disappears from the list of block devices, so I have to manually unplug-plug it in back in order to be visible in the block devices list.

It is undependable of the controllers. tried them all 2.0 or 3.0, as well as all usb ports…

At this point I’m experimenting only with your method, qwt isn’t installed in the clean win7 cube restored from the backup, in order to eliminate possibility of mixing two of them to cause unpredictable issues.

I can’t conclude anything else, but something is related to dom0, not win7 cube…

[jevank]

@enmus

In despair, I tried emulated, because Win7 cube window disappeared when attaching >2TBB usb hdd in order to try to get it back somehow - no luck as I said.

I got you, but this means that gui-agent or vm in common is broken. In that case you could force stubdomain window to catch BSOD screen if there is internal problem.
qvm-start-gui --force-stubdomain [windows-qube]

@enmus

New findings: if I restart Win7 cube and first try to attach <2tb usb hdd, it succeeds and is fully operational. If in addition I try to attach >2TB usb hdd, it tries to attach it, then it unplugs itself, but also it unplugs already attached <2TB usb hdd’s, as well as all other usb devices (sometimes all at once, sometimes one at a time, after additional manual retries to attach >2TB hdd).

I guess problem is with stubdomain kernel. It is 5.4 version and USBIP might has issues with new devices. I’m afraid you have to live with it some time.

[enmus]

Well, I’ve bee waiting for 4 years v4.1 to come in order to fully transit to Qubes OS (I simply need Windows for some apps that have no alternatives in Linux world, just like Qubes has no alternative in OS world), so what could be “some time” more, haha.
Thanks for your patience and time.

[brendanhoar]

@jevank

First need to quote warning from QEMU docs : “This is an experimental feature. QEMU will slow down when using it. USB devices requiring real time streaming (i.e. USB Video Cameras) are not supported yet.”

Link moved, but warning persists: qemu/docs/system/devices/usb.rst at master · qemu/qemu · GitHub

Just FYI,
Brendan

[brendanhoar]

@jevank

@enmus

New findings: if I restart Win7 cube and first try to attach <2tb usb hdd, it succeeds and is fully operational. If in addition I try to attach >2TB usb hdd, it tries to attach it, then it unplugs itself, but also it unplugs already attached <2TB usb hdd’s, as well as all other usb devices (sometimes all at once, sometimes one at a time, after additional manual retries to attach >2TB hdd).

I guess problem is with stubdomain kernel. It is 5.4 version and USBIP might has issues with new devices. I’m afraid you have to live with it some time.

@jevank -

I haven’t tried to reproduce this, but my hunch is this is due to UAS requirements for > 2TB volumes and lack of UAS support under Windows 7.

Other explanations could exist though…e.g. perhaps a hard limit somewhere in the usbip or qemu stack of 2TB per mass storage device?

Looks like the linux kernel got UAS support in version 3.15.

Might be interesting to disable UAS/UASP for that device in the source VM (sys-usb or dom0) and see if that addresses the issue? E.g. via How to disable USB Attached Storage (UAS) - Leo's Notes

B

[jevank]

Thanks, but I have no ability to edit post. I’ve tried to remove unnecessary testing updates.

[jevank]

We use config with disabled UAS and it helps for some devices with USB Streams support. But this case doesn’t look similar because device works fine with linux-based VMs. Virtual USBIP controller (vhci) in 5.10 doesn’t support USB streams as I remember. So I’m still thinking kernel version might be a problem. I saw PR with stubdom updated components, but have no time to test it yet.

[enmus]

It looks that, due to lack of thorough testing and the set of circumstances, I brought confusion as well as wrong conclusions into the issue, and I apologize for that.

Lack of thorough testing - I wasn’t aware of uas and usb-storage.
Set of circumstances - some of my usb hdds don’t support uas, and they didn’t make any trouble while attaching to AppVms, so I brought wrong conclusion that attaching hdds via USB to Linux AppVMs works, and I am not sure anymore if it ever worked with USB 3.0 hdds. Attaching those uas devices via qvm-block for sure worked to Linux, and for >2TB did not to Win7 qubes, and for the latter qube probably because it was broken somehow (another set of circumstances).

Now I have tested it enough to correct my statement.

Now:

1. It is possible to attach all devices via qvm-block to all AppVMs

2. It is possible to attach all usb-storage devices (USB 2.0 external HDDs) via qvm-usb to all AppVMs regardless of the controller they are attached to.

3. It is not possible to attach any uas devices (USB 3.0 external HDDs) via qvm-usb to any AppVM on some controllers they are attached to (USB 3.0 controllers/hubs/ports).

4. It is not possible to attach some uas devices (USB 3.0 external HDDs, >2TB) via qvm-usb to win 7 qube regardless of the controller they are attached to.

5. It is possible to attach some uas devices (USB 3.0 external HDDs <2TB) via qvm-usb to win 7 qube on some controllers they are attached to (USB 2.0 controllers/hubs/ports).

Now, what might be interesting in order to identify the issue is that (on USB 3.0 controller/port):

1. lsusb -t

in sys-usb gives

Port x: Dev y, If 0, Class=Mass Storage, Driver=uas, 5000M

as well as

lsmod | grep usb

which gives

usb_storage 81920 1 uas

but,

2. Prior to attaching hdd to an AppVM, in that AppVM

lsmod | grep usb

gives

usb_storage 81920 1 uas
usbip_core 40960 1 vhci_hcd

After attaching HDD to that AppVM, it disappears from the block devices list in dom0’s device widget (is this expected? Please answer) and in sys-usb

lsusb -t

now gives

Port x: Dev y, If 0, Class=Mass Storage, Driver=, 5000M

Is this expected? Please explain.

while in AppVM

lsusb -t

it now gives

Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M???

So, why the device is recognized as uas in sys-usb before attaching, and as usb-storage in AppVM? Is this expected? Please explain.

What confuses me even more is that

sudo dmesg | grep “UAS”

in AppVM gives:

[ 3525.722933] usb 2-1: USB controller vhci_hcd.0 does not support streams, which are required by the UAS driver.
[ 3525.722936] usb 2-1: Please try an other USB controller if you wish to use UAS.

In APPVM

sudo fdisk -l, or
sudo lsblk

don’t recognize hdd, so I’m into situation for which I haven’t found a solution on the internet: how to mount a drive that is recognized by lsusb, but not by fdisk or lsblk.

After dettaching hdd from AppVM, in sys-usb hdd is also seen only by lsusb, but not fdisk or lsblk. and the Driver is stil empty, neither uas nor usb-storage. The only way to get it back and make it usable to any qube is to restart sys-usb, or to physically re-plug the hdd.

I’d be very grateful if you could teach me how to mount drives that are seen by lsusb but not by fdisk or lsblk, and to explain dilemmas I asked for above.

[jevank]

In current-testing repo it is now available stubdomain with updated components including kernel, pulseaudio and QEMU (thanks to @fepitre). Quick testing shows much more clear sound with Windows and some VM performance improvements with USB attached devices.

@enmus if you see messages like this in dmesg log while attaching to Linux-based VMs than UAS is a problem. It might be possible to build guest VM kernel w/o UAS support to usb-storage driver instead for all USB storage devices.

[  +0.029211] usb 2-1: USB controller vhci_hcd.0 does not support streams, which are required by the UAS driver.
[  +0.000029] usb 2-1: Please try an other USB controller if you wish to use UAS.

[enmus]

Will that degrade performance of HDDs?