Additional warning of you start experimenting with where you attach your USB controller, and rely on a USB keyboard or mouse:
My understanding is that attaching your controller to one VM would detach it from the other VM. If your keyboard or mouse were attached to it, they would get disconnected.