Check if all your physical USB ports are connected to the same USB controller or to multiple USB controllers:
Quick script to add BDF to lsusb output to find controller for devices on KVM/HUB
If it’s multiple USB controllers then you can use all USB controllers except for the controller to which you’ve attached the USB disk to create a sys-usb.
Maybe there is a way to use the ctap with device in dom0 using some hack, but it’s not officially supported since it’s insecure.