I have noticed at least once a day my usb mouse becomes slow and almost non-responsive. I use a sys-usb with 1 controller and 2 ports dedicated for it.
I often can resolve the problem by restarting the qube - if the mouse can get me there. Unplugging the adapter also works. The problem exist with debian-11. I switched to debian-10 to see if the problem also exists there. I vaguely recall it did when I first used Qubes.
Any ideas? Maybe I need to allocate more RAM (currently 400MB initial / 2 VCPU)?