I had this issue and I cannot remember what I did to be honest. Ill have to look and see if I omitted something. Make sure your keyboard and mouse are not swapped in the event0 event1 part of the stub domain. Also make sure you are passing the right device in dom0.
Iâve checked and theyâre not being swapped and theyâre the right events, the mouse works fine the clicks also work but when I click it teleports to the top left, did you change anything in the libvirt config? This might have to do with the qubes gui? The only thing Iâve done is set the guivm to none and pass the gpu, also when I pass the same event to sys-gui it works fine
@khmartinsen Turns out the mouse doesnât only go to top left whenever I click or scroll, it teleports to whatever place the âprimaryâ mouse input is. For example if I move the mouse with the window emulated mouse on dom0 (the default way) and than use the evdev mouse that QEMU uses and move it and click or scroll, it goes back to whatever position the other mouse is.
I think having a look at how you set up your input devices on the libvirt config would help but Iâm not sure, hopefully you can tell whatâs going on based on what I tried to explain. (Iâm awful at explaining things)
That is exactly the issue I had. Its from the qubes_drv. In my windows 10 vm I donât need the emulated window that qubes provides so I disable that by using qvm-features. I am not sure if all of these need to be turned off but i the following set:
gui 0
gui-emulated 0
video-model none
I donât think you would want to change anything in libvirt but if you did you would want to remove the tablet input I think. Maybe in the VM you can disable that input device that is related to the qubes gui window tablet inputs.
1 Like
@khmartinsen Thank you simply removing tablet input solves the problem in dom0, for some reason the same setup while using sys-gui doesnât.
Unfortunately I canât remove the emulated window because I need to force MSI on my NVIDIA graphics card in Windows otherwise the display glitches and blacks out (making it unusable) and the VM eventually shutdowns.
Iâve been booting Windows in safe mode and adding a registry entry every time the address changes, but if I change the features you mentioned I wonât be able to force MSI because the GPU address will be different and I wonât be able to change it.
Do you know of an alternative to set my GPU to use MSI by default?
On dom0 lspci -v shows that MSI is disabled:
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Would that capability even be passed to the VM?
I am not sure. I have not used sys-gui yet. I would think there would be some way to just disable the tablet input in whatever VM you are in. So in sys-gui can you disable the tablet input by identifying which device it is in /dev/input/�
Shouldnât removing <input type='tablet' bus='usb'/> from the VM xml be enough? I find it strange that the outcome isnât the same as it is in dom0, I should mention that it doesnât display anything on the screen I think itâs related to the MSI shenanigans, Iâll double check
I would think that is enough. Which xml are you editing? The ones located in /etc/libvirt/libxl/ are auto-generated. You need to follow this: Custom libvirt config â core-admin v4.2.7-0-gf40e8c21-dirty documentation.
It looks like I had messed up somewhere else, it wasnât related to any of your fixes I just had to reinstall the NVIDIA drivers and manually enable MSI again, not really sure how that happened but anyhow itâs working now and the performance seems better than with the usb proxy but that might be placebo, thank you for your patience and for your time!
1 Like
Can you elaborate on step 1?
Specifically:
- Where does the 60-input file go in the directory structure?
- What build target(s) did you use, or just âmake qubesâ to make everything?
Also, can you explain what youâre doing with all the cat-ing?
Thanks for this guide! 
After a deep examination of the qubes-builder tree, it looks like it should go in:
qubes-builder/qubes-src/vmm-xen-stubdom-linux/linux/config
and it can be built just with the component target vmm-xen-stubdom-linux
Is this correct?
Apparently not. I canât get anything to build completely with qubes-builder. Missing internal dependencies, and make qubes dies with a warnerror in qrexec-daemon.c. Iâve probably done something wrong setting up the build environment, or pulled the wrong version of qubes or something. Just going by the qubes-builder doc and guessing at the proper way to achieve this step.
Ultimately, this is pretty much what I am aiming for. A hotkey-reassignable KBM switch that switches between dom0 and configured domUs. I donât need the screen flipping that the libvirt-evdev does, just move my KBM to different VMs, is all.
You are correct on the placement of the 60-input file and using just that part for qubes builder. I also had issues when using qubes builder. I will find my notes. There was a command I had to change when following the official documentation.
I gutted the screen changing part of Libvirt-evdev and it works well for just mouse and keyboard as you described.
Yeah, thatâs what I was thinking of doing as well. Itâs pretty easy to whack that part out and leave the rest.
Thanks for getting back to me! 
Hello, all. Iâve been following the forum for quite some time, and this topic and my quest to enable easy keyboard/mouse sharing with a VM finally pushed me to register here, so I can share the problems and solutions that I have encountered. Hope to help someone with less knowledge to understand things, that are not covered or are unclear, or to save some time for other users like me.
First of all, thank you, khmartinsen, for the guide! This is really one of the cornerstones for the usable GPU passthrough and gaming in Qubes. (Another one is neowutranâs original guide for the GPU passthrough itself.)
Previous solution
Iâve been using barrier/input-leap to pass keyboard/mouse to a gaming VM before. This is a short version of the setup:
- Install
barrierin dom0, configure it to run in server mode. - Install
input-leap(the active fork ofbarrier) to the VM. Run it as a client. - Create a qrexec policy (
qubes.ConnectTCP) to allow connections to barrierâs port in dom0 from the VM (if it has a qrexec client installed). - Use
qvm-connect-tcpin VM to establish the connection to dom0, and connect the input-leapâs client to the opened port.
If the VM does not have a qrexec client (qrexec-client-vm):
- Configure the qrexec connection to dom0 from the netvm of the VM (
sys-firewall, for example). - Open port, used by
qvm-connect-tcp, in the netvm for incoming connections from the VM:nft add rule qubes custom-input ip saddr $VM_IP tcp dport $PORT accept - Connect input-leap to
$NETVM_IP:$PORT.
Now you can move the mouse pointer over the configured border of the screen, and it will âmoveâ to the VM. Pressing ScrollLock locks it there, and this is required to work around the problem of VM getting wrong relative/absolute cursor position in some games, IIRC.
Why it is not enough
There are several problems/inconveniences with this setup.
- I donât really like, from a security standpoint, running a software in dom0, that has full access to my input devices and accepts incoming connections from VMs running proprietary programs (games) (or even Windows, if youâre into that, I play on Linux).
- This is not a general solution, it works only when VM is already running a window system (X11 or Wayland). So it doesnât work during boot, if X/Wayland or input-leapc is crashed/stopped, or when you use a display manager. With KVM GPU passthrough I can easily switch users (administrative/gaming), using a display manager, but with this setup it is not possible or at least not easy and reliable.
- It works bad with non-standard keyboard layout. I need to
setxkbmap usin dom0 before switching to VM and restore the original layout afterwards. - Because of the first point, I didnât setup the qrexec policy rule to
allow, and left it toask, so every time I need to wait for a confirmation window in dom0 to appear and accept an incoming connection. - I use KDE with Wayland in the VM, and it currently has an annoying bug, where you need to confirm every time that input-leap can get permissions to control the wayland session. And the only way to confirm that is using qubes emulated input devices, in other words the VM window with the emulated VGA output must be in focus at that moment in dom0âs window system to receive input. So, considering the previous points, this setup is very convoluted. I can forget one step and then is required to switch back and forth between the dom0 and the VM.
- For some reason, when I try to move in Cyberpunk 2077 (just holding the move button), I get an accelerated motion every other second, as if I push the move button very quickly. So, it is rather unplayable. I donât have that behavior in other games. It can be connected to another issue of some sort of Xen/CPU performance bottleneck in this game (that I am planning to create a topic about).
So, back to the evdev solution. This is my notes/additional info to the main khmartinsenâs guide:
1. Modifying the stub domain kernel
Well, to setup the Qubes builder and to successfully build the stubdom package â it took me some timeâŚ
I used the v2 builder, because⌠it is v2, so it should become the default in the future. The main info is the projectâs README. And it basically explains how to use the builder⌠just in a way that assumes, that you have some basic understanding already. So this is my introduction for those, who donât know anything yet.
I will explain things, how I have understood them. I may be wrong. Corrections are welcome.
qubes-builder v2 introduction
The process is controlled by the configuration file, builder.yml at the top directory by default, which you should create. Look in example-configs/ for examples and read the Qubes builder configuration section of the README for the available YAML keys in the configuration file. You should copy qubes-os-r4.2.yml from examples and edit it.
IIUC, the whole Qubes project is split into âcomponentsâ, that are represented by individual https://github.com/QubesOS/qubes-COMPONENT repositories. They are listed by the key components in the configuration file. The qb, qubes builder executable, will try to build everything by default. You specify required components by the -c COMPONENT arguments. In our case it is -c vmm-xen-stubdom-linux.
Each component source repository have its own .qubesbuilder configuration/description file at the top directory. It is also YAML, but has different set of allowed keys, the section . qubesbuilder and the following examples in the README describe it.
Qubes consists of many Linux distributions. Dom0 is Fedora 37 now, each qube has its own OS: Debian, Whonix or Fedora, which have different versions. Xen stub domains, used with HVM qubes, (it is our target here) also run some distribution inside. Then there are unofficial templates with other distributions. The qubes builder format for naming distributions is host-DISTRIBUTION (distribution for dom0) or vm-DISTRIBUTION (for VMs), and the examples of DISTRIBUTION are fc37 (Fedora 37), fc39 (Fedora 39), bookworm (Debian bookworm).
The configured for the build process distributions are listed in distributions key in the builder configuration file (builder.yml). You can filter, which are selected, by the -d DISTRIBUTION arguments of qb. And each Qubes component describes in its .qubesbuilder file the distributions for which it can be built.
Qubes builder can build individual packages for different distributions, templates for TemplateVMs (used with qvm-template command) and the Qubes installation ISO. These actions are executed by package, template and installer commands for qb. There is also the repository command to manage the package repositories like yum.qubes-os.org and deb.qubes-os.org. Each command is executed in different stages, like fetch, prep, build.
So, for example, to build all packages of a component gui-agent-linux for Fedora 40 VMs, run ./qb -d vm-fc40 -c gui-agent-linux package all. Here the argument all means all stages, not all packages, the builder will always build all possible packages for the specified components and distributions.
With this introduction you should read the real README next, it has all the practical details.
Setup the builder
You need the main qube, that will run the builder, and a disposable template qube, which is a template for DispVMs that will be spawned for each stage of the process. It is the setup with Qubes executor. I didnât try the Docker executor, because I donât have experience with Docker and the qubes-r4.2 example config uses Qubes executor. And with a new PC I donât mind starting many qubes, it is much less painful now.
The main qube may be networkless, you just need to install required dependencies there (see Dependencies section of README) and get the qubes-builderv2 repository (you can run git clone and git submodule update --init in other qube and then qvm-move it to the main qube). But it will store all the data (artifacts), produced in different stages of the process, so it should also have enough storage space.
For the disposable template qube setup, read Qubes executor section of README. The main qube is called work-qubesos there.
You probably also need to import and setup ultimate trust for the Qubes Master Signing Key 427F11FD0FAA4B080123F01CDDFA1A3E36879494 in the main qube. Then you can check the builder sources with git tag -v $(git describe).
The build of vmm-xen-stubdom-linux component produces packages for the dom0. The process will create a build environment, which means it will download some packages of Fedora 37. But that environment is kept inside a DispVM, which will be destroyed after a (successful or not) build. To avoid repeated downloads of the same packages, if you run the build several times, you can create a cache of packages in the main qube with ./qb -d host-fc37 -c vmm-xen-stubdom-linux package init-cache, which will be used automatically in the build.
Modify the sources and build
The whole purpose of using the Qubes builder is to build our own kernel for the xen stub domain, which supports evdev input devices. So we need to modify the sources before the build. At first I downloaded the qubes-vmm-xen-stubdom-linux repo, put the 60-input file in there and tried to make the builder use these local sources instead of the original on github. It means fiddling with git submodules and their urls, url/verification-mode/git-url/sources parameters in the configuration, using local executor and even discovering the git protocol.file.allow configuration. It all wasnât successful. Then I understood, that it is possible to run different build stages separately. So, this is it:
-
Run
./qb -c vmm-xen-stubdom-linux package fetch. It downloads the component sources and other required files. Beware that, if the process is aborted (if, for example, the disk is out of free space) you can end up with not fully downloaded files, and the nextfetchattempt will not check and redownload them. So, the build stage will fail later. You should then check the files inartifacts/distfiles/for corruption manually or just remove all of them and restartfetch. -
Put the linux configuration (
60-inputfile) from the guide into the fetched sources:artifacts/sources/vmm-xen-stubdom-linux/linux/config/. -
Modify the
artifacts/sources/vmm-xen-stubdom-linux/busybox/busybox.config:
change# CONFIG_SU is not setto
CONFIG_SU=yThe
sucommand of busybox is needed for running commands from dom0, using qrexec. The developers disabled many busybox features in commit6863ff1ef3041cf71db8e422e8c24986c475e447, disablingsuwas probably a mistake. -
Run
./qb -c vmm-xen-stubdom-linux package prep build. I am not sure, ifprepstage is required. It creates source packages. But the build stage probably uses them.
Unfortunately, many packages are being downloaded on this step, and I couldnât find, if it is possible to also âcacheâ them. I had many failed attempts to build, and each time the packages were being redownloaded from scratch.
The built packages are artifacts/components/vmm-xen-stubdom-linux/4.2.13-1/host-fc37/build/rpm/xen-hvm-stubdom-linux-*-fc37.x86_64.rpm. Copy the âfullâ version to dom0 and either install it, replacing the original, with rpm --reinstall, or extract with rpm2cpio FILE | cpio -id and move the kernel to /usr/libexec/xen/boot/.
2. Modifying the stub domain rootfs
I have nothing to add to the guide here. It is probably better to make all the modifications in the previous step, in the sources, before the build, and just get the fully prepared package. But I donât know, how the root filesystem is build from sources, and didnât try to find out yet.
3. Pass a mouse and keyboard to the stub domain
Here I got the su: applet not found error message, when tried the qrexec-client command, and found out that busybox in the stubdom rootfs doesnât have su compiled in. See the solution above.
The next bulk of problems came with libvirt-evdev. I decided to leave all the logic (screen input switch and input and usb subsystem monitoring) in the code and just remove the [[screens]] and [usb_switch] sections from the config. At first glance, if the screens list is empty, then screen_input_switch function is just a noop. Well, no screens section in the config at all raises an exception and quits the main loop, but exceptions are not handled. So, the service stops processing input events, but does not quit. Which means, the keyboard is no longer working.
I fixed this and a couple of bugs, that I think the program had. Here are modifications:
--- a/libvirt-evdev.py
+++ b/libvirt-evdev.py
@@ -13,6 +13,8 @@
import pickle
async def screen_input_switch(owner):
+ if not 'screens' in config:
+ return
for screen in config['screens']:
for source in screen['sources']:
if source['owner'] == owner:
@@ -32,7 +34,7 @@
current_mode = {
"host": "guest",
"guest": "host",
- }[mode]
+ }[current_mode]
await screen_input_switch(current_mode)
@@ -49,7 +51,8 @@
target_device.syn()
except:
- pass
+ print("exception in replicate({})".format(source_device))
+ exit(1)
def action_input(action, device):
if action != 'add':
@@ -67,6 +70,8 @@
def action_usb(action, device):
global current_mode
+ if not 'usb_switch' in config:
+ return
if device.get(config['usb_switch']['property_name']) == config['usb_switch']['property_value']:
if action == 'add':
asyncio.run_coroutine_threadsafe(screen_input_switch(current_mode), loop)
@@ -115,13 +120,13 @@
# create host devices
host_devices = {
- key:evdev.UInput(cap)
+ key:evdev.UInput(cap, name="host-" + key)
for (key,cap) in capabilities.items()
}
# create guest devices
guest_devices = {
- key:evdev.UInput(cap)
+ key:evdev.UInput(cap, name="guest-" + key)
for (key,cap) in capabilities.items()
}
@@ -150,7 +155,7 @@
for (device, path) in zip(virtual_devices, virtual_device_paths):
if os.path.exists(path):
subprocess.run(["unlink", "--", path])
- subprocess.run(["ln", "-s", device.device, path])
+ subprocess.run(["ln", "-s", device.device.path, path])
# setup replicate input coroutine
for device in input_devices:
@@ -171,7 +176,7 @@
observer_usb.start()
# notify systemd that start is done
- sd.notify(sd.Notification.READY)
+ sd.notify("READY=1")
loop = asyncio.get_event_loop()
loop.run_forever()
The end
So, it turned out to be mostly a Qubes builder guide, I guess. Still hope it will be useful to somebody.
1 Like
Hello all and psblut:
Many thanks for your work to pass a keyboard and mouse to a windows VM with a gpu. I realize running windows AT ALL in Qubes is a security downgrade, but I am tired of rebooting frequently and having multiple machines.
Although I am nearly ready to take the plunge and go this way to get USB keyboards and mice functional in Windows VMs, it seems to be a lot of complication and perhaps unnecessary. Is there not an easier way? I have explored passing a USB controller to windows, but it is in use by the sys-usb. It seems with could be ignored with a boot parameter as my GPU is, but even this seems a bit clunky and wasteful. I believe I have 2-3 and there does not appear to be a way to identify the controller that a keyboard is connected to.
Is anyone on IRC or discord to explore this with me? Psblut?
Looking forward,
Fellow
Hello, fellow.
If you have several USB controllers, then, sure, passing one of them to Windows qube is an option. But then you will not be able to use USB devices on that controller in the Qubes itself. So, you either need a second pair of USB mouse/keyboard, or replug them manually to USB-ports of a different controller, or some automatic scripts setup is needed.
This is not necessary. You just attach a USB controller as a PCI device to the target qube, and this should just work. Probably, the no-strict-reset=True option will also be needed.
The boot parameter for a GPU is a workaround to avoid its initialization before its passthrough to a VM, because that process is very complex and doesnât work ideally (the GPU is not properly reset when passed to a VM and will be initialized there the second time). And for the security reasons.
There probably is a way to figure that out just by looking at /sys/bus/pci/ and /sys/bus/usb/ in sys-usb, but Iâm not powerful enough user to advise this. (Added later: ok, there is a documentation, but Iâm not sure, that it is correct.)
You can try instead this: create as many test qubes, as there are controllers, and attach a single different controller to each one of them, then start them and see, where your keyboard and mouse appears.
Warning: I donât have a USB keyboard, so everything that I recommend is theoretical and my personal thoughts of how I would do that.
Discovery
-
The easiest way to create these test qubes is just to clone sys-usb that many times, and then selectively remove PCI devices from them, so that each test qube is left with only one, unique, USB controller.
You can do this in GUI. In command line it will be (replace N, X, Y):
qvm-clone sys-usb test-usbN qvm-pci detach test-usbN dom0:XX_XX.X qvm-pci detach test-usbN dom0:YY_YY.Yfor each test qube N.
It is important to not detach any devices from sys-usb itself, because it will be left detached even after a reboot!
-
To allow dom0 accept input from the keyboard and mouse in these qubes, create the same qrexec policies for each test qube, as you have for sys-usb.
grep qubes.Input -R /etc/qubes/policy.d/should give you the current configured policies. So, for example, create a temporary file/etc/qubes/policy.d/50-test-usb.policywith these lines for each test-usbN qube:qubes.InputMouse * test-usbN dom0 allow qubes.InputKeyboard * test-usbN dom0 allow -
(replace A, B, etc):
qvm-shutdown --wait sys-usb; qvm-start test-usbA; qvm-start test-usbBYou should start all test qubes here and everything should be on the same line, because as soon as you hit Enter, sys-usb will be shut down and you will loose the ability to enter new commands without a keyboard.
Now you should see, what qube (with what attached controller) has your keyboard and mouse (in Qubes Devices list or with qvm-usb ls test-usbN). And by replugging them to different USB ports you can discover what ports belong to what controller.
At that point you can still reboot to return to the normal sys-usb setup.
Permanent modifications
You can attach and detach PCI devices only to qubes that are shut down. And only one qube with a specific device can run at the same time.
So, now that both sys-usb and Windows qubes are powered down, you can make necessary detach/attach operations.
To attach a USB controller XX_XX.X to the Windows qube windows:
qvm-pci attach --persistent windows dom0:XX_XX.X -o no-strict-reset=True
Then, decide what to do with sys-usb:
-
Detach that controller from sys-usb, so that it will be attached only to Windows qube, but you can run both qubes at the same time. But that controller will not be available in Qubes itself (so either replugging or second pair of devices is needed).
-
Leave sys-usb as is, but then you will need to shut down sys-usb to run Windows and vice versa each time:
qvm-shutdown --wait sys-usb; qvm-start windows
qvm-shutdown --wait windows; qvm-start sys-usbYeah, if you have a single keyboard and mouse, then you will not be able to enter the second command. Then more advanced scripting and/or libvirt hooks will be needed. Or physical replugging to different ports.
Added later: with
no-strict-resetthis is not a secure option:Ideally, devices for which the
no-strict-resetoption is set are attached once to a VM which isnât shut down until the system is shut down.
- Use new test qubes instead of sys-usb. Turn autostart off for sys-usb, on for test qubes, and then, to start Windows, you have the same choice between 1 and 2 as before, but with the test qube with the controller in question instead of sys-usb.
You can rename âtestâ qubes by cloning them and the removing (when powered off) the original.
To not leave yourself without a keyboard permanently, you should ensure that you always have a 1) working qube 2) with at least one USB controller attached 3) that has autostart enabled and 4) that has qubes.InputKeyboard to dom0 allowed qrexec policy.
USB device attach
Ok, only after writing all of this I thought about a simple âUSB attachâ. You can probably just attach the mouse and keyboard as a USB device to the Windows qube. And then, after the Windows shutdown, get them back available in Qubes. I donât have a USB keyboard, thatâs why havenât considered that option.
Maybe it is not the most efficient setup (but I used barrier, and I think its client/server architecture is more inefficient than USB-passthrough).
I donât understand, how secure/insecure this is: Device handling security | Qubes OS
Attaching a USB device to a VM (USB passthrough) will expose your target qube to most of the security issues associated with the USB-stack.
As far I as understand, target qube here is Windows, so thereâs no problem for sys-usb.
Anyway, I shouldnât probably have written all of this, because there is this document: USB qubes | Qubes OS
And there is this recommendation for you anyway:
When using a USB keyboard on a system with multiple USB controllers, we recommend that you designate one of them exclusively for the keyboard (and possibly the mouse) and keep other devices connected to the other controller(s).
In this case, the designated controller for input devices should remain in dom0 but be limited to input devices only.
Because sys-usb with keyboard have basically full control over the system, and a USB qube is vulnerable to malicious USB devices.