I found something in the docs about pci devices which raises a question. It says
By default, when a device is detached from a VM (or when a VM with an attached PCI device is shut down), the device is not automatically attached back to dom0.
This is an intended feature.
A device which was previously attached to a VM less trusted than dom0 (which, in Qubes, is all of them) could attack dom0 if it were automatically reattached there.
In order to re-enable the device in dom0, either:
Reboot the physical machine. (Best practice)
In the bottom it says to re-enable the device in dom0 which the docs said just before that it is bad security to re-enable it in dom0 and that’s why it doesn’t do it automatically. But it seems like it does automatically do it when the computer is restarted which most people probably do at least once every day. So when you start the computer next time, you have risked dom0 becoming compromised.
Shouldn’t it be better if it doesn’t re-enable the device in dom0 after reboot? It seems like having it re-enabled in dom0 should be something we don’t want to do at all, ever.
I have read in random posts that it’s common that the built in cam on a laptop is usually attached to dom0 after a fresh install of qubes os. This is fine and normal as I understand but then you will attach it to a qube where you will use it, then the point should be to never re-enable it in dom0 but that means never rebooting the computer?
When you pass through a PCI device into or out of a virtual machine, ideally you want to power cycle it (this means cut electricity to it, then turn it back on, causing it to reset itself).
When you reboot your bare metal computer, incidentally, you also power cycle all your PCI devices, too…
Most PCI devices will power cycle without complaining, but occasionally, you will get PCI devices that will cause the entire system to crash if you try and power cycle them without “additional things”. GPUs are a good example of this, and they do this to prevent display tampering. Broadcom wifi cards, particularly in MacBooks running Qubes OS are also a particularly bad headache in this respect too.
Xen does allow for the passing through of PCI devices without power cycling them, but this has to be manually enabled per PCI device.
Because of this, if the VM using the PCI device shuts down, the device does fall under the ownership of dom0 again, but that doesn’t necessarily mean that the device is allowed to do whatever it wants inside dom0…
The compromise doesn’t come with something nasty being present in the VM. The compromise comes from interacting with it, and gets responses that manipulate and hijack the things those processes then do after the interactions.
It’s the same reason why when someone sends malware over the internet, and it infects someone’s computer, it doesn’t obliterate all the network infrastructure it passed through en route.
Having a malicious PCI device present in dom0 isn’t the end of the world, unless it gets initialised with firmware, and the kernel grants it permissions.
Unfortunately, if dom0 doesn’t know it’s there, it can’t pass it through
Dom0 also has A LOT of system components removed from it, for this very reason. Vanilla dom0 doesn’t have a network stack at all in it. In the sysfs, you’ll notice that all the drivers for all the PCI devices are overridden to pciback, which ensures that as much as possible, the Linux kernel doesn’t initialise any firmware for devices it detects.
EDIT:
Dom0 doesn’t actually override pci devices to pciback. On my own machines, I wrote a systemd service for it so long ago that seemed to persist across version upgrades, so my mistake for assuming that was part of a vanilla install.
Essentially, dom0 only knows how to enforce policies and pass through.
On almost every laptop in existence, the internal camera is hard-wired into one of the onboard USB controllers. It’s just cheaper and easier that way for the manufacturers.
It’s extremely rare nowadays to get a PCI camera. In the 90’s, maybe, but not after the invention of the USB protocol.
Thank you for such a great explanation. You are becoming one of my favorite users here. I just want to make sure I have understood everything you said.
Are you sure you’re not exaggerating how few modern laptops have pci cam?
If we go with the example of a pci cam, it’s not the end of the world as you say because qubes os and linux kernel prevents it from running its firmware. It’s only when we use the cam that the firmware gets executed that’s when there is a high threat but not for dom0 because we wouldn’t use the cam in dom 0, we would attach it (pass through) to a qube first. So after doing a reboot and it’s re-enabled in dom0, it again is prevented by qubes os and linux kernel from running its firmware.
Even if an adversary has a 0day and flashes that cam’s firmware from a qube it’s attached to and being used in, it won’t compromise dom0 because even after reboot the cam’s firmware is prevented from running. The cam would still be compromised but it can only compromise the qube it’s attached to and being used in.
I also have no idea how difficult it would be for an adversary to flash the cam’s firmware from a compromised qube. But as explained, even if it would happen then it is not going to compromise dom0 even after reboot.
It would probably be unnecessary to remove a hardwired pci cam from the laptop.
Well, it depends on personal choice and your circumstances.
You’re going to hear that a lot, and I know it sounds unhelpful, but it’s actually people just reminding you that every situation is unique, that there are usually a LOT more variables to consider than what are stated here, and that “cookie-cutter security” without fully grasping everything that’s going on can be dangerous.
None of this in intended to be security advice. It’s intended to help people get their baseline understanding to a level of abstraction that is sufficiently detailed enough to be able to engage in healthy debate.
The camera can send literally whatever signals it wants to your computer. The determining factor is how your computer responds to those signals once it receives them.
Sidenote: The camera would likely be connected to a USB controller, which is a PCI device, so this example isn’t the best fit. A better example would be the wifi card before sys-net starts up.
A zero-day exploit is just a way that someone has found to introduce another variable into a program’s existing instructions/operations that changes the end result in some way, usually that does something beneficial to whoever introduces the variable.
The general gist of an exploit is that:
It says the right thing(s)
In the right way
At the right time
To the right programs/hardware that are already listening
To make them do something that was unintended
This doesn’t necessarily have to always be malicious. In fact, this is how people end up “unlocking” their devices to get more functionality out of them, extending their device life.
But yes, it’s a such a cliche, but it only takes one, as the old saying goes.
dom0 actually doesn’t assign every device to pciback, because that would require hard-coding a bunch of information about the system topology. It only hides devices that appear to be network or USB controllers. So this is a valid attack vector, sadly.
I have proposed solutions in the past, but they haven’t been implemented because they generally require firmware support and nobody has provided funding to implement that yet.
With Demi’s contribution it seems that a compromised pci device, lets continue with cam as the example, the malicious firmware that was flashed from a compromised qube can be run to compromise dom0 after rebooting the laptop. This results in it being a valid existing attack vector against qubes os that have a pci device.
Even if we would have to knowledge to write a systemd service to prevent pci devices firmware from being run on dom0, it would mean that every qube you attach the cam to could become compromised. And even if you only attach to disposables, the adversary could be able to see what you are doing with the cam while using it. I know it’s a lot of “could” and it requires a 0-day to be found in the first place.
But all this makes me at least lean more towards it being worth to remove the pci device such as a cam.
A cam connected to the usb controller would show up in qubes os as a pci device? Did I understand you correct? and are you sure? Because wouldn’t that be the same for the mic then? Shouldn’t both the cam and the mic be attached to dom0 on a fresh install? Not only the cam while the mic is attached to sys-usb?
Maybe I shouldn’t have marked this topic as solved already because it’s actually an existing attack vector. I will leave it up to moderators to decide if it should be solved or not then they can make the edit if they think it shouldn’t be solved.
Funding is used at the developers’ discretion, so earmarking for one feature or another is unfortunately not possible, even if the devs could put a price tag on it.
Any PCI device for networking (like that Wi-Fi card) should be taken care of automatically by the qubes-pciback dracut module during early boot. Supposedly that happens before a driver for it is loaded in dom0.
But it’s still a problem in general, e.g. with a (PCI based) SD card reader. Even if you’ve manually assigned it to a VM, it’s easy to forget to also add an appropriate rd.qubes.hide-pci= argument to /etc/default/grub.
Not a holistic solution by any means, and I’m not sure how the lower level attack surface works/how much this helps with that- but what I’ve done to avoid this happening by accident, because it’s annoying and inconvenient- is add a bunch of modprobe.conf.d files to explicitly prevent a long list of drivers for devices on my system from being loaded
After physically moving PCI devices around, or just making a typo in hide pci, I’ve had several different PCI devices be assigned drivers in dom0. That was my good enough solution
Easy, hacky way to get a decent list of modules to block for your system, from memory, roughly:
lspci -vvvv | grep driver | sort -u
With -vvv, each device has a “kernel driver” and “loaded driver” line, something similarly worded. Reference the “kernel driver” one, that’s what it would load if it was floating about, not assigned to pciback
Anything in that list that you don’t want to ever load in dom0 can be explicitly blocked
You’ll have to be a little familiar with driver names but it shouldn’t be difficult for most
Great post. I was planning on doing some research today about adding pci devices to qubes-pciback. The solution you posted seems to be doing roughly the same thing or rather have the same result? So if for whatever reason qubes-pciback failed to “keep the pci back” then your solution would be like a backup protection?
I’ve done more research. I couldn’t find anything in the docs about qubes-pciback or…
or…
or…
Maybe it is fine there is nothing in the docs about that, i will explain now why.
All PCI devices should be resettable via the hypervisor. When I created this topic I didn’t understand what that means exactly because it was explained with power cycling it on and off and I just didn’t understand how that affects security. Now I understand it completely reinitialises the device. That sounds like it is restored to factory settings and that any malicious flashed firmware from a compromised qube will be removed when the pci device resets at reboot. And the docs say “This ensures that any device that was attached to a compromised VM, even if that VM was able to use bugs in the PCI device to inject malicious code, can be trusted again. (Or at least as trusted as it was when Qubes booted.)”
So if that is the case, that an adversary can flash the pci device with malicious firmware from a compromised qube but the malicious firmware will not be persistent, the firmware will be gone after reboot and the original OEM firmware will be restored. That ensures when the pci device is attached to dom0 again after reboot, it won’t attack dom0 with malicious firmware, unless the OEM wrote malicious firmware.
This makes it very safe to attach for example a pci cam to a qube, risk it becoming compromised and then reboot the computer so the pci cam attaches itself to dom0 again.
For the flashed malicious firmware from a compromised qube to persist after reboot (reset), a 0-day would need to be found in the pci-device when it resets, a bug which can be abused to make the device not reset itself.
This means the adversary must first find a 0-day to compromise the linux kernel in a qube. Then another 0day to flash firmware on the attached pci device. Then a third 0day to make the pci device not reset itself after reboot. That is a lot of 0days.
All that is why I think it is probably fine the docs don’t write anything about qubes-pciback or the other quoted configurations.
But in this topic I have had a few replies where you are saying you have make such configurations on your own system or that it’s something that is an option to protect from this security threat. That makes me wonder if the PCI device resetting itself is a strong enough security because I only understand all this from an abstract level from reading the docs and the replies here. It would be wonderful if someone can explain a bit more about this and why it would be in mine or someone else interest to learn how to make the custom qubes-pciback configuation or the other quoted configurations.
I don’t think PCI reset deals with persistent state like firmware. IIUC it’s more comparable to merely taking a PCI card out of its socket and plugging it in again, if even that.
Also, if the PCI device itself is not malicious, dom0 might nevertheless expose itself to malicious data by interacting with it. Take for example a PCI-connected SD card reader: After loading the driver for it, dom0 will now see the (potentially malicious) data stored on an inserted SD card. The kernel will automatically parse it for partitions, udev will automatically parse it for filesystems etc.; both of these interactions could be exploitable. Hiding the SD card reader from dom0 with rd.qubes.hide-pci=... is supposed to prevent this.
ok, in that case it’s best to either remove the PCI-device or to do what you suggested with “add an appropriate rd.qubes.hide-pci= argument to /etc/default/grub”. I’m guessing it is the BFD (or was it called BDF?) which should be after the =. That would ensure any firmware from a pci-device can’t be run on dom0, it’s almost as if the device doesn’t exist?
Another thing we haven’t brought up is Intel Boot Guard. I guess it’s because it’s a proprietary black box. But it’s advertised to make sure the firmware on the ROM and any devices that comes with the laptop can’t be changed without the OEM key so they can make micro code updates. In theory shouldn’t that protect against an adversary trying to flash malicious firmware to a pci device that came with the laptop such as the cam? I mean not prevent it from happening but it would detect it then prevent the computer from being able to boot. But yes, it’s proprietary black box and we don’t know how many bugs and back doors there is and how bad opsec the OEM has with the key to Boot Guard.
Lastly it would be interesting if anyone could compare the different configurations to protect against this threat. The methods I quoted in my previous post in this topic, I won’t quote them again in this reply so you don’t get needlessly spammed notifications.
They will both prevent the device-specific kernel driver from taking control of the device, yes
Unnecessary unless you’re making physical changes to your PCI cards or, rarely, if you changed certain major settings in BIOS - if I remember correctly, I once got burned by disabling the onboard bluetooth, as it caused devices on the bus to be enumerated differently. Enabling/disabling ACS may also have this effect though I’m not certain
Point is that it’s rarely necessary, unless you’re iterating the configuration on a new build or experimenting in some other way that a casual user shouldn’t
I’ve been studying this pci security stuff so much and have had a lot of help from alzer89 in a private conversation as well to understand. My biggest obstacle to learning this is probably that the docs describe this attack vector as very serious but alzer89 is trying to explain to me that it’s almost not an attack vector at all but either way now I am equipped with more knowledge to ask better questions.
You say dom0 will see the malicious data after loading a driver for it. But isn’t the driver loaded in a VM that the PCI device is attached to (PCI passthrough)? The PCI device can’t be attached to both dom0 and the VM at the same time. So how is dom0 being exposed while the PCI device is attached to a VM? And after a reboot the pci-device is reset.
Also, with the rd.qubes.hide-pci= configuration, wouldn’t that prevent me from using the PCI device? Can’t passthrough to a VM anymore?
As mentioned before, this doesn’t truly “hide” the device. More precisely, it prevents kernel drivers in dom0 from binding to it, by explicitly seizing control of it using a sort of stub/dummy driver (pciback).
The “rd” in Qubes refers to ramdisk (the initramfs stage) which is the early stage of the boot process. In this stage, a script parses the list of devices provided and explicitly binds each “hidden” device to the pciback driver, which holds onto it until asked to release it for assignment to a VM.
Yes I understand you better now. It’s a bit challenging to learn from different sources at the same time because sometimes I can misunderstand something and the misunderstanding results in what seems like contradicting info from the different sources. But at the same time even though it’s challenging learning from different sources at the same time, I think it’s more secure when you can get the different sources info to be on the same page, but it can take more time to put all the info from different sources together like that.
So this is how it works: You have a PCI device, lets stop this pci camera nonsense I was doing before and lets do SD card now instead. I am not sure how to rate the trustworthiness of the sd card reader or the sd card. I guess it just comes down to unavoidable root of trust.
One thing for example, when you go to buy something for cash locally, you say you want an sd card and they go into the back room to get it for you. 99% of their sd cards are fine probably but maybe they do something dirty and decide you are a good target to give one of their malicious sd cards.
This isn’t something I know much about but in the docs it says
In many cases a malicious device can choose what driver will be loaded (for example by manipulating device metadata like vendor and product identifiers) - even if the intended driver is sufficiently secure, the device may try to attack a different, less secure driver.
So maybe it does something like that. But if I have done the rd.qubes.hide-pci= configuration then dom0 should be protected from that at least that attack vector. But the qube I attach the card reader to will have that attack vector still.
Or the SD card could have built in security problems which a malicious driver can take advantage of. This turns the SD card into a trojan horse or backdoor and the malicious driver is the key to the backdoor.
Then there’s the problem for the adversaries about what to do when they have succeeded with their attack. Because they can’t attack dom0 if we have done the rd.qubes.hide-pci= configuration but they can still attack sys-usb and any qube we attach the SD card to. And how big threat is this? depends on if there will be net access to that qube. I could have the most top secret data in the world in that qube but if there is no net access then the adversaries can’t retrieve that data. Maybe they can if they have physical access, they can just come and take the SD card by force from me which has stored the top secret data on it.
I say, that’s some very liberal paraphrasing, esquire
I said it wasn’t as simple as “regardless of what’s running or not running on your machine, abra-kadabra, you are now COMPROMISED!” like the movies would have you believe
Think “sacrificial protection”, or “You want a driver? Here, little munchkin, munch on this *hands dummy driver to PCI device*…”.
The dummy driver essentially becomes like a lazy parent, and pats the PCI device on the head and says “Yes…yes…mmm…really? Wow…that’s amazing…I agree…” whenever it says anything.
Attack vectors are not simply:
The USB stick is “malicious”.
They’re more like:
The on-device firmware on the USB stick will take advantage of the fact that the host machine asks it what it is, and it can declare itself as whatever it wants, and the host machine has no way of confirming if this is the truth or not, so it has to blindly accept whatever it’s told.
So now you’ve identified the attack vector in sufficient detail, you can start to devise effective countermeasures that operate as 1-to-1 countermeasures to neutralise any avenue of attack
There are usb drives which you can insert sd cards into (adapters). I guess that could work, so the sd card gets handled by qubes os as a usb device in sys-usb and never touches dom0. Would that be a good way to handle SD cards securely? Or maybe it doesn’t work like I think? I haven’t tested it.
