A tangent, do you have advice for headless sys-usb?
Hm, it seems it needs the GUIVM for those confirmations, so not headless. Sorry.
Other than that, I start from this and install:
qubes-input-proxy-sender
qubes-usb-proxy
Then create a disposable template and base sys-usb on it.
It uses less than 200 MiB without swapping:
root@sys-usb:~ # free -m
total used free shared buff/cache available
Mem: 424 160 112 2 165 263
Swap: 1023 0 1023