There is always a chance that attacker can access it in certain situations (like during border control search or when you sleep). The same goes for attacker accessing the machine.
Also USB device firmware could be malicious even before it’ll end up in your hands. But the same goes for any other hardware.