Pretty sure qubes automatically locks usb input without that vm active, so you need it to start automatically, there's probably a way to fix that through a rescue terminal or something, but I'd have to look into exactly how that would be done... since it's based on fedora you could probably;
Add rd.break at the end of kernel options
mount -o remount,rw /sysroot
chroot /sysroot
qvm-prefs sys-usb autostart False
exit
exit
-Emlay