Of course you should backup your boot USB. I’d buy 1-2 more USB sticks and clone it(small cheap ones).
That being said you will need to sync them whenever you update one.
Now the risk comes from other USB devices connected to sys-usb. Assuming you boot your computer with other stuff plugged in USB ports connected to USB controllers serviced by sys-usb, it means they would technically be able to compromise your USB drive if it’s also plugged in a USB controller that’s in sys-usb.
Whether that would happen depends a lot on the type of USB devices. A simple mouse or keyboard would hardly have the ability to hack you(unless a dedicated attacker made it on purpose and somehow gave it to you).
Ideally your keyboard and mouse should be PS2 and not USB. If not possible, then it’s best to have 2 USB controllers and have the keyboard/mouse one in Dom0.
Anyhow, assuming you plug other devices in there, like an Android phone, risks get compounded since a phone can be compromised.
If you can always boot clean(no USB devices in sys-usb) other than your USB boot drive. And then unplug the USB boot drive afterwards. That would be ideal.
This way you’re not exposing it.
Your threat model changes a lot on whether it’s a laptop or a desktop.
But I think if you’re worried about someone tampering your computer’s disk, using just a boot USB drive won’t be enough. Look into Anti-Evil Maid.
I’d say you can’t really protect your system against attackers with physical access. There’s always ways.
One way would be for an attacker to put a new EFI partition that would load something malicious before booting from the EFI on your USB drive.
Ideally if you have a BIOS password, you can at least prevent modification of BIOS settings so that whoever wants to do it, has to reset things. And you’d know because your computer won’t ask for a password when you turn it on.
Anyhow, advice can only be based on your threat model:
Laptop or Desktop?
Physically accessible to attackers?
Keyboard/Mouse via PS2 or USB?
More than 1 USB controller or not?
What type of devices are plugged in USB on start?
etc.
Meanwhile to harden it, put BIOS user/admin passwords and look into usbguard.
Also check Anti-Evil maid.