Also, am I correct in my understanding that qvm-usb attach/detach uses USB over IP to share USB devices? (when I qvm-usb attach something, it ends up on the usbip-host driver in the sys-usb vm).
Okay!
I have gotten a command/script keyboard/mouse swap to work!
I run a script in dom0, and it assigns the keyboard / mouse to the Win10 Qube, and it works!
I have a batch file in the Win10 Qube that I run and it returns the keyboard and mouse back to dom0 in the state they start off in before I assigned them to the Win 10 Qube.
NOW then, I have noticed a significant issue with the keyboard/mouse and running apps in the Win 10 Qube. When a 3D program is rendering on the GPU, when I move the mouse, it freezes until I stop moving the mouse. If I am typing on the keyboard, if I move the mouse, it freezes output until I stop moving the mouse, and then it spits out the keys I was typing. Likely, there’s some solution to this already in the forums or elsewhere, I just wanted to post an update, since I have achieved something useful!
Once I fix this stuttering/hanging issue, I am going to get rid of the Qubes Window (it might be the cause for all I know), and just run on the GPU passthrough. If the system remains stable and functional through all the tests I am going to put it through software-wise, I think I am finally in a good place to call the project done!
I plan on documenting with a how-to post for all my efforts at the end of this thread, so anyone else with similar issues can benefit from my pain and sweat. I expect that my solution to get the kvm swap isn’t the best way, and I look forward to any recommendations for doing it better, but right now, it seems to work, and that’s a major first step.
Just a quick summary of what I had to do:
- Modify /usr/lib/qubes/usb-export to un-bind the USB device from the usbip-host driver, and re-bind it back to the driver it originally was bound to at invocation. This is done in the cleanup function, since all the variables are still set with the correct values. (also have to make sure the Template copy is updated so it isn’t lost on sys-usb restart)
- Add a kvmattach script in dom0 which calls qvm-usb attach twice (keyboard and mouse) to the win10 qube.
- Add a kvmdetach script which gets the correct devices and called qvm-usb detach twice (kb & mouse again).
- Create a service script in /etc/qubes-rpc called qubes.KVMDetach which calls the kvmdetach script.
- Create the associated policy file for the service script in /etc/qubes-rpc/policy and set the perms to $anyvm dom0 allow.
- Create a batch file and powershell stub which calls qrexec-client-vm in the Windows 10 qube to send the qubes.KVMDetach service message to dom0. (this was the worst part… qrexec-client-vm’s syntax is strangely different than the linux version, and is not documented properly anywhere; I had to figure it out from another batch file that sets the time).
That’s it. To use it, just run kvmattach from dom0. Wait a few seconds, and boom! the windows 10 qube has the mouse and keyboard. When I want to switch back, run the kvmdetach.bat file, wait a few seconds and boom! back to dom0.
Again, I will post the details in a closing post to this thread. I am still testing it, and there may be some hidden gotchas ( I had one where one of my USB controller PCI devices disappeared on a reboot and would not come back until I completely powered off the system (including the power supply switched off) and let it sit while I made dinner. Dunno if that was just some weirdness that happened during debugging (I muffed a few things in my iteration trying to get it working), or if it will happen again because I am not doing things the right way.
1 Like
So, I set the gui-emulated feature to 0 and the video-model feature to none to get rid of the software emulate vga adapter. So, now I don’t have the Windows GUI inside of Qubes, which is fine. However, it doesn’t affect the mouse movement stuttering problem at all.
It seems like the usbip driver is blocking the execution and driving up CPU utilization across all 8 vcpus. Overall cpu utilization is at around 7% when idle and not moving the mouse. When I rapidly move the mouse overall CPU utilization jumps up to 30-40% and all vCPUs show a massive jump with one going a bit higher than the others. The system isn’t doing anything else, I am just moving the mouse around on the desktop.
I’m running Win11 with NVidia passthroughed to it exclusively on external monitor, smoothly with 2118MB RAM allocated, only xen.xml patched with a 3.5GB RAM patch.
I don’t know if it’s your goal to intentionally passthrough keyboard and mouse to your Win qube, but I’m running mine in a debug mode, and while the debug window (which is on the laptop display) is in focus, my mouse and keyboard are smoothly used inside Win qube on external monitor without any passthrough.
I was thinking of assigning additional set of keyboard/mouse for Win qube, passthroughed, but at the moment I don’t see a point giving too much hardware to an insecure qube, only it’s controller to be used back in Qubes.
Yeha, everyone’s going to have different requirements. I mainly don’t want to run Windows on bare metal anymore, and I want to have the luxury of spinning up VMs to test developed applications on different platforms.
For me, I already run things pretty securely as it is without a HV. This just steps up the game a bit more. I’m not worried about sharing KB/mouse between Win/dom0. This is a desktop system, so anyone with physical access wouldn’t be too challenged by it.
It doesn’t seem to matter how the kb/mouse is passed. Both the input proxy and USB/IP methods of passthrough seem to have the same stuttering issue. I am thinking that it’s probably a QEMU or stubdomain issue. Something not being set right, causing unnecessary blocking of the vm.
Ideally, I would just like to have a hotkey swap so I can jump back and forth between VMs, (and there is a thread here on doing that; I was just waiting on the thread author to respond back) but I am comfortable using this script ping-pong method for the time being. A large majority of the time I will be spending in Windows anyway.
I may just do the PCI USB controller passthrough method eventually if I have to and use a secondary kb/mouse device running on dom0. But, this entire situation is forcing me to learn the guts under the skin, so I can fix and maintain it better in the end, so it’s not all bad. I just would like to get over this initial set of hurdles to get a stable and performant system back running again. It’s been almost 2.5 months now, and I think I am getting pretty close. Seems like just this one last issue (hopefully) and I’ll be set, as long as everything holds together (no crashes or corruption or lockup issues). I haven’t seen much of that so far, which tells me that Qubes/Xen (and ofc Linux) are good and stable; it’s just always the “getting started” part that’s a challenge. Been that way since I started using it back in the late 90s. 
Just FYI, there are KVM and USB switches that have built in support for switching the outputs with a keyboard hotkey like this one:
https://www.amazon.com/Computers-Keyboard-Connections-3840x2160-Supported/dp/B07SX1BZGT/
Yeah, I’ve got one on order. Should take care of the performance issues with USB. Hopefully.
In the meantime, I’m busy restoring data and installing apps, and testing performance and stability.
Maybe I forgot to tell you previously, I use this trick to only use one KB/mouse (KBm) on Xen dom0 -and- domUs :
- use the qemu VGA adapter, and display its window via VNC, even if it does not display the desktop
- let the window “grab” your KBm (click in it)
- now you can use your KBm in the client qube
- use w/e grab keys you configured to stop controlling the qube and give input back to dom0
I dunno how and if it works on Qubes, as I’m not using a PCI-PT GPU.
It works as fast as on bare metal, way better than USB PT (which is laggy even on vanilla Xen last I checked), BUT there’s one drawback if you game a bit : you cannot “keep pressed/hold” a keyboard button, it instantly fires the key and start repeating …
KVMs are ok, but the one I bought is way too chatty : if swapping a lot between hosts, for each swap, it’s a new detection in syslog …
Also if using the HDMI plugs, it can mess with your multi-screen setups, and you may have problems (re)activating the remote screens (some of my RasPis struggle), but it depends on the OS and the KVM brand, YMMV !
Otherwise, I’m eager to read your post with detailed instructions !
It would be very useful for users (like me ^^) already using an HV (Xen or other), and wanting to “move” existing domUs to Qubes !
Yeah, it was working with the window open, but it also seemed to be lagging even then. It also is a weird situation when running GPU passthrough. I was able to sorta get it to make the mouse cursor move on the GPU screen, but I couldn’t move it back to the Qubes window.
Besides, I want my keyboard and mouse to have full functionality, as I have macros and lots of buttons and things (and color profiles 
), so the only way to get that is via USB passthrough.
I have a KM switch on its way, so will see if that clears things up when I dedicate one of the USB PCI devices to Windows and am able to switch it back and forth that way, but right now, it is working “ok”.
I should have everything copied over and reinstalled this weekend, so perf/stab testing will commence next week. One thing I have noticed is that the vdisks even on SSD appear to be much slower loading large data chunks in games than native. May be some kind of performance issue with thin provisioned LVM. Not sure yet.
Strange, maybe the VNC software you were using ?
(PS: I use one on which you can set the “grab keys”, to prevent using an existing shortcut).
Note that everything works this way for me, and faster than normal VNC !
In the domU I can use KB shortcuts/macros, the Fn key, etc.
I have several AHK scripts handling my own shortcuts, including one mimicking Unix-like GUI shortcuts (Alt+mouse to resize/drag, etc).
Ofc with USB PCI-PT, you won’t have any lags !
Maybe, or it could be the sector size/misalignment ? I’m no pro so forget the words.
There is a thread worth reading : Survey: CPU and VM boot time.
Somewhere in the middle, there is a discussion about sector size or sector alignment, 4k sectors stuff on SSDs, or something like that ^^
OK, testing update:
Things have been going pretty well, but I have noticed the following:
- Passed through PCI audio device tends to click and pop a little bit in all apps, and YT videos get desynced (audio gets ahead of video by longer times as the video plays. Pausing and starting it gets it re-synced). Google searches indicated that recent versions of Win10 sometimes have this problem, but I am curious if it is Win10, or Win10 running on Qubes. Since the audio device is a PCI passthrough, I am inclined to think it is mostly in Windows. Running LatencyMon tells me that the system is not reliably fast enough to do real-time audio, due to lag spikes in certain DCP calls. I’m trying to track down which ones.
- Performance is pretty close to the previous windows-only setup I had before. Some things tend to take a longer time to load/start up, and there are still some intermittent latency issues.
- Mouse cursor movement still causes significant cpu usage (jumps by 10-20% on all vCPUs).
- I’ve noticed that even with 1-2 open apps, 200 processes and 3k threads, all vCPUs tend to jump up to 60-80% utilization fairly often.
- 3D game performance is spotty depending on the game, but all seem playable. 30-60FPS on fairly high settings. I figure once I stop passing through the mouse/keyboard and use the external switch onto a passed-through USB3.0 controller PCI device, it will be better.
Also, I have a serious issue when starting up Qubes. it doesn’t seem to be able to keep the same device ids across reboots. This manifests in three separate ways:
- network devices - no matter what, just about every time I boot, I have to term into sys-net and reassign the network connections to the correct devices. I have two ethernet ports, and one wireless one. On boot, the Public connection (going to my router) is on ens6, the next time, it is on ens8, and the wireless moves around as well. I know there is a way to use udev rules to force assignment based on mac address, but the udev rules are on the disposable filesystem, so they aren’t there when sys-net starts up. I suppose I have to put them into the FC37 template. Sometimes, it re-creates a new “Wired connection 1” that I have to delete and reassign. I have made a script to cleanly assign the devices manually, but the fact that it can’t remember this stuff from boot to boot is a pain.
- USB devices - I have assigned persistent usb devices to my Windows Qube. However, different reboots cause them to jump to different bus ids. One boot they are sys-usb:8-x, the next, they are 6-x, and now they are 5-x. I have to detach the old defunct ones, and attach the correct ones. Maddening.
- If a PCI device disappears for any reason (one of my USB 3.0 bus devices did this once), it re-jiggers everything, and both sys-net and qvm-usb/sys-usb have to manually reconfigured.
I guess this issue needs to be turned into a feature request for the linux kernel (and/or the associated USB/PCI device drivers and NetworkManager). It needs to remember device info on particular pci device id (via vendor:product codes and product names and serial numbers/mac addresses or other unique identifier) and not reassign everything to different internal ids so things that are connected logically to those devices don’t break every dang time.
Lastly, I had a networking failure in the Windows Qube tonight, and now when I start it up, it starts up, but the qrexec<>qubesd connection isn’t working, and networking inside Windows isn’t getting configured, so no network is available. I posted this in a new thread here:
On the upside, my fix to usb-export works to return the mouse/keyboard back to dom0 properly, even when the Windows qube crashes or is shut down normally, so I don’t need to use the kvmdetach batch script in Windows to restore it before shutting down. So, +1 for me there. 
Once I get the Windows Qube fixed, I will be working on the summary post for this thread. I think that, unless I see more instability or encounter other issues in the near term, this chapter can be closed.
Anyway, if anyone has any insights on one or more of these issues, please let me know and thanks for continuing to read my little Windows-on-Qubes travel blog. XD
You can add the udev rule in your sys-net disposable template:
You can attach USB devices based on VID:PID:
Ahh, thank you. That should be easier to put the udev rules in to resolve the netman musical chairs with adapters.
As for the USB stuff… that looks pretty hairy… kinda like what I am doing with the kvmattach/kvmdetach stuff. I think this will actually be resolved by the KVM switch and assigning a PCI hub to the Windows VM, and I won’t need the attach/detach anymore (but I will still have it if I need it for something else).
Thankfully, I got the USB switch in today, and am now looking to determine which USB PCI device to pass through. I plan on putting both a hub and the switch on that device so I don’t have to use USB passthrough at all.
One thing is very strange, though. USB devices on different PCI device/controllers seem to be assigned to the same USB bus in sys-usb. I have devices on a USB-2.0 controller and devices on a USB-3.0 controller on the same USB bus. I guess that means that the bus number is a logical one, not a physical one. That’s confusing. I need to finish mapping out all my USB ports to make sure which controllers they are on. Need to find a mapping utility that maps USB busid pairs to the PCI controller they are actually on. I am sure all that info in in the /sys fs, just a matter of organizing it better and following the breadcrumbs/links.
Also, I resolved the network/qrexec issue mentioned in the other thread. So, another problem down, just one more left to go, and hopefully knocks on wood there will be no more of those issues.
Additional discovery!
Reinstalling QWT with 4.1.68.1 (I think I had 4.1.68 before) also seems to have fixed the mouse input lag and sound system popping issues. Still have the rare lag spike out of nowhere, but it is very quick and not very frequent.
Still looking forward to passing through USB controller to further reduce lag and compatibility issues, and going with KMswitch to swap between Windows Qube and Qubes dom0. That’s a task for tomorrow, though!
Apologies for the late reply. I had originally promised to post the updated usbip export script I used when I was trying to get USBIP usb device sharing to work before going down a different road by doing USB hardware passthrough and an external USB KM switch (which has been only slightly less painful… after 3 different pieces of hardware, I still have device pauses and dropout/disconnects).
Anyway, I want to post these changes here in case anyone was having issues with the connect/disconnect of virtual USB devices using the USBIP facility. The changes I made are all in the usb-export script, and all it does is remember what the old setting was, and sets it back when the stub process exits.
Here’s my version; you can compare it with the standard one in the repo:
#!/usr/bin/sh
SYS_USB_DEVICES=/sys/bus/usb/devices
SYS_USBIP_HOST=/sys/bus/usb/drivers/usbip-host
# From /usr/include/linux/usbip.h
SDEV_ST_AVAILABLE=1
SDEV_ST_USED=2
SDEV_ST_ERROR=3
usage() {
echo "$0 device"
}
if [ "$#" -lt 1 ]; then
usage
exit 1
fi
find_by_bus_dev() {
local busnum=$1
local devnum=$2
for devpath in $SYS_USB_DEVICES/*; do
if ! [ -e $devpath/busnum -a -e $devpath/devnum ]; then
# skip individual interfaces etc
continue
fi
if [ "$busnum" -eq "$(cat $devpath/busnum)" -a \
"$devnum" -eq "$(cat $devpath/devnum)" ]; then
echo $devpath
return
fi
done
echo "No matching device found ($bus, $dev)" >&2
exit 1
}
# Resolve device name to sysfs path
resolve_device() {
device="$1"
# handle different device formats
case $device in
0x*.0x*)
device=$(echo "$device" | tr . :)
# make sure there is only one matching device
if [ $(lsusb -d $device | wc -l) -ne 1 ]; then
echo "Multiple or no devices matching $device, aborting!" >&2
exit 1
fi
# Example: Bus 003 Device 002: ID 05e3:0608 Genesys Logic, Inc. Hub
lsusb_output="$(lsusb -d $device)"
bus=$(echo "$lsusb_output" | cut -d ' ' -f 2)
dev=$(echo "$lsusb_output" | cut -d ' ' -f 4 | tr -d :)
find_by_bus_dev "$bus" "$dev"
;;
*-*)
# a single device, but NOT a specific interface
if echo "$device" | grep -q :; then
echo "You cannot export a specific device interface!" >&2
exit 1
fi
if ! [ -d "$SYS_USB_DEVICES/$device" ]; then
echo "No such device: $device" >&2
exit 1
fi
echo "$SYS_USB_DEVICES/$device"
;;
*)
echo "Invalid device format: $device" >&2
exit 1
;;
esac
}
devpath=$(resolve_device "$1")
if [ -z "$devpath" ]; then
exit 1
fi
pidfile="/var/run/qubes/usb-export-$(basename $devpath).pid"
modprobe usbip-host || exit 1
# Request that both IN and OUT be handled on a single (stdin) socket
kill -USR1 "$QREXEC_AGENT_PID" || exit 1
busid=$(basename "$devpath")
# Unbind the device from the driver
if [ -d "$devpath/driver" ]; then
origdevpath=$(readlink -e $devpath/driver)
echo $busid > $origdevpath/unbind || exit 1
fi
# Bind to the usbip-host driver
echo -n add $busid > $SYS_USBIP_HOST/match_busid || exit 1
echo $busid > $SYS_USBIP_HOST/bind || exit 1
# One more safety check - make sure the device is available
if [ "$(cat $devpath/usbip_status)" -ne $SDEV_ST_AVAILABLE ]; then
echo "Device $devpath not available!" >&2
exit 1
fi
busnum=$(cat $devpath/busnum)
devnum=$(cat $devpath/devnum)
devid=$(( $busnum << 16 | $devnum ))
speed=$(cat $devpath/speed)
# Send device details to the other end (usb-import script)
echo "$devid" "$speed" >&0
echo 0 > $devpath/usbip_sockfd || exit 1
exec <&-
echo $$ > "$pidfile"
safe_busid=$(echo ${busid} | tr :. _)
cleanup() {
#--------------
# Rebind device to original driver fix --TL
# (restores usb devices, like mouse/keyboard, back to previous state)
# Unbind from the usbip-host driver
# echo -n add $busid > $SYS_USBIP_HOST/match_busid || exit 1
echo $busid > $SYS_USBIP_HOST/unbind
# Re-bind the device to the original driver, if it existed
if [ "$origdevpath" != "" ]; then
echo $busid > $origdevpath/bind
fi
# -------------
qubesdb-rm \
/qubes-usb-devices/${safe_busid}/connected-to \
/qubes-usb-devices/${safe_busid}/x-pid
# notify dom0
qubesdb-write /qubes-usb-devices ''
exit
}
trap "cleanup" EXIT TERM
qubesdb-write \
/qubes-usb-devices/${safe_busid}/connected-to "${QREXEC_REMOTE_DOMAIN%-dm}" \
/qubes-usb-devices/${safe_busid}/x-pid "$$"
# notify dom0
qubesdb-write /qubes-usb-devices ''
# FIXME this is racy as hell!
while sleep 1; do
# wait while device is "used"
if [ "$(cat $devpath/usbip_status 2>/dev/null || echo 0)" -ne $SDEV_ST_USED ]; then
break
fi
done
# cleanup will be called automatically
I hope someone finds it useful.