How to avoid flashable firmware compromises via pass-through PCI devices?

General Discussion
1

(To help this conversation I’m starting with what I believe is correct information, to help people identify any possible wrong assumptions I could be starting with)

Many network and USB cards have firmware updates loaded at boot time, which persists until reboot.

Other network and USB cards can have flashable firmware, that is loaded and then persists reboots.

If a disposable sys-net or sys-usb qube that becomes compromised is using the second type (flashable firmware), then potentially the flashable firmware can be attacked allowing the compromise to persist across reboots. If the hardware just has the first type (volatile ram-based firmware), then they should be immune to this.

The word “reboot” is a bit vague. Lets separate “reboot” into 3 types:

  • “Full system reboots”, meaning: “full power down, then power up”
    (note: I have no idea if laptops can do this without removing the battery)
  • “ACPI system reboots”: what happens when you select “reboot” from the dom0 menu
  • “VM reboot”: what happens when you select a qube and click “restart” in the Qube Manager

(Up till now I’ve been referring to full system reboots)

Furthermore, qvm-pci outputs which VM each PCI goes to, and if it does not say “no-strict-reset=True”, then I believe that means that a “reset yourself” signal is sent to the hardware upon qube bootup, meaning that compromised RAM based firmware would only last untill a VM reboot. (assuming they implemented the reset function properly)
(Note: the “permissive” flag may be important here)

So, if this is all true, then the question becomes how to determine which cards have flashable firmware in them. For example, when I confirm that a specific model has RAM based firmware, does that mean that the hardware does not have flashable firmware?
(Note that CPU’s seem to have both flashable and RAM based firmware)

For example, I can image a manufacturer where the default firmware might be flashed by the manufacturer, but they dont give the flashing procedure to the public, and just distribute the firmware updates as RAM-based firmware.

EDIT (added this paragraph): I’ve found a confirmed case of devices that loads its firmware either from flash memory or can be loaded into RAM directly from the driver. The uPD720202 usb controller can load its firmware either from external flash memory or can be directly loaded into RAM by the driver. This means that each PCB manufacturer decides if they want to add persistent state to the card when they design it.

This means as a end user you have to reverse engineer how each PCB design works to figure out if it has state or not. :face_with_diagonal_mouth: (and this isn’t even taking into account the possibility of malicious intent)

More generally, how does one gain the certainty necessary to select network cards and USB cards to purchase for a sys-net or sys-usb qube? The 2 possible methods we have so far are:

  • See if the information is available from the FSF RYF program.
  • Reverse engineer different cards till you find one with no state.

Is this “lack of writable persistent memory” something that gets investigated/verified in “Qubes certified hardware?”

2

If a disposable sys-net or sys-usb qube that becomes compromised is using the second type (flashable firmware), then potentially the flashable firmware can be attacked allowing the compromise to persist across reboots. If the hardware just has the first type (volatile ram-based firmware), then they should be immune to this.

“Firmware is stored in non-volatile memory – either read-only memory (ROM) or programmable memory such as EPROM, EEPROM, or flash.”

The word “reboot” is a bit vague.

Were you looking for the phrase hardware reset?

Furthermore, qvm-pci outputs which VM each PCI goes to, and if it does not say “no-strict-reset=True”, then I believe that means that a “reset yourself” signal is sent to the hardware upon qube bootup, meaning that compromised RAM based firmware would only last untill a VM reboot. (assuming they implemented the reset function properly)

I don’t know if a piece of hardware with compromised firmware (of whatever kind) can be trusted to obey “reset yourself”. If there are no externally (by other system components) verifiable evidence of the reset, it may like and persist the state. If it was made to persist a state, then even a power off may not work (even less if the device itself has some kind of battery, including a capacitor acting as one).

So, if this is all true, then the question becomes how to determine which cards have flashable firmware in them. For example, when I confirm that a specific model has RAM based firmware, does that mean that the hardware does not have flashable firmware?
(Note that CPU’s seem to have both flashable and RAM based firmware)

Or microcode?

As for firmware, it is hard to say without analyzing the hardware itself. Unfortunately, with more and more stuff being integrated into a single chip, that seems next to impossible for the user. Ideally, one would use only proven RYF hardware but that is rare, difficult to get and sometimes doesn’t seem to work with Qubes OS.

Or more generally, how does one gain the certainty necessary to select network cards and USB cards to purchase for a sys-net or sys-usb qube?

Look for RYF hardware.

Is this “lack of writable persistant memory” something that gets investigated/verified in “Qubes certified hardware?”

The computers in the HCL are not stateless.

3

Neat! I was unaware of this. It says that RYF hardware “must give the user, along with the product software source code, the practical capability to install replacement software for any and all of the free software in the device.”
Depending on the details of that clause, it could mean that it has CPU flashable firmware (which we would not want), or it could mean that it requires the user to wire into a JTAG port (which is potentially good). So being RYF hardware does not automatically make it stateless, but it does mean that someone at FSF actually knows the answer! :slight_smile:

Maybe the report from the FSF RYF audit could be obtained somewhere, or FSF would be willing to say if things are stateless from a security perspective or not.

4

Update:
I’ve found a confirmed case of devices that loads its firmware either from flash memory or can be loaded into RAM directly from the driver. (note it’s the same software so defining firmware to mean it counts as “firmware” when the executable code is on the flash but the same executable code counts as “software” when its loaded straight to ram by the driver is not a useful definition for this topic)

Anyway, the uPD720202 usb controller can load its firmware either from external flash memory or can be directly loaded into RAM by the driver. This means that each PCB manufacturer decides if they want to add persistent state to the card when they design it.

This means as a end user you have to reverse engineer how each PCB design works to figure out if it has state or not :face_with_diagonal_mouth: