Slow and non-responsive USB mouse

I have noticed at least once a day my usb mouse becomes slow and almost non-responsive. I use a sys-usb with 1 controller and 2 ports dedicated for it.

I often can resolve the problem by restarting the qube - if the mouse can get me there. Unplugging the adapter also works. The problem exist with debian-11. I switched to debian-10 to see if the problem also exists there. I vaguely recall it did when I first used Qubes.

Any ideas? Maybe I need to allocate more RAM (currently 400MB initial / 2 VCPU)?

I’m leaning on the problem being debian-11. sys-usb under debian-10 seems to be running stable for the last couple days.