My guess is this:
- During install, the installer adds UEFI boot entry for the USB stick to the list in the BIOS
- When the BIOS starts, it checks the UEFI boot entries, and remove any it finds missing
I’ve not tested it myself, but you could try the suggestion from: