Hey, I’m actually surprised a search did not return any results for this… I want to utilize a UHS-II (for slightly faster speeds) SD card to store parts of the OS (and ideally the template VMs, and system VMs) on a write-blocked SD card. I have been searching for information on this and I’m actually surprised that Google does not pull up any information - maybe I’m searching for the wrong thing?
Anyway, my question is: Which paths/files should I partition off and store on a write-blocked SD card during the QubesOS installation? The SD card will be writable during the installation but as soon as it is finished, I will switch the hardware switch on the SD card which could, I imagine, improve the security of Qubes significantly.
FYI, for those of you who are also interested in pursuing this avenue / experimenting with me, there are faster solutions available than SD cards. I just like the SD card method because my future laptop will have an integrated SD card slot.
Here is basically the only seemingly legit supplier of hardware-switch write-blocked NVMe / SSD drives:
I am not affiliated with the company. I am however amazed that they somehow managed to integrate a virus scanner into a USB stick - I imagine it has to have an integrated computer on board – pretty confused about it actually.
The other question I have in case any sleuths stumble upon this question: If I manage to figure out which parts of the file system can be stored on write-protected media, then how can I prevent Qubes from slowing down? The SD cards that fit into traditional card readers max out at about 250MB/s whereas modern PCIe drives can do over 3000MB/s. I imagine, hopefully without too many tradeoffs, we can load the read-only stuff onto a RAM disk or even overwrite a disk paritition during the boot process.
Why isn’t this the way computers work? Shouldn’t system files / libraries / binaries be locked down to prevent malicious substitutes by forcing hardware authentication (in the form of switching the write protection) to modify system files? Or am I fundamentally missing how modern malware/actors that are capable of penetrating QubesOS work?
It seems that you’re talking about hardware Lock switch on SD card. That switch is not preventing software from writing on SD card, it just says a recommendation for software not to write on it. I’ve written more info here:
At least protecting boot partition with it makes sense.
Not sure about whole dom0 root on write-protected SD card
Maybe you can just install Qubes on SD card and add secondary storage pool for your non-persistent VMs and then lock SD card from write.
You can copy dom0 root to the RAM or your SSD and boot from it. You can read about it here:
Because of usability for average user I guess. You need to update dom0, if you need to add more VMs/persistently change their settings then you need to update dom0 root as well.
Hey @tzwcfq – thanks for writing a detailed response.
Are you sure about this? Everywhere I read online it seems to imply that the hardware lock on the SD card prevents writes (https://www.amazon.com/SanDisk-64GB-Extreme-UHS-I-SDSDXXY-064G-GN4IN/dp/B07H9J1YXN/ref=sr_1_3 is an example of a card with a hardware switch that I was referring to). It says here: Support | Western Digital that “You will not be able to modify or delete the contents on the memory card if it is locked.” I did manage to find one site that claims the hardware switch does nothing here: SD Write Protect Switch is a Dummy — I’m really confused about this now. You’re basically telling me that the write-protect switch is a conspiracy that SanDisk, for instance (at least whoever wrote their documentation), is actively supporting.
But you said write-protected SD cards aren’t actually write-protected so I’m still seeing an issue. Sorry if I’m coming off brash but I have a compulsive need to do a Qubes-level job of acquiring and keeping essential software pristine. Are there support groups for people with Qubes-level paranoia?
Thanks for the tmpfs link!!!
I’d feel a lot better if dom0 was restored from a write-protected media every reboot that can be optionally disabled during updates. This is what we’re starting to do with passwords and keys (i.e. YubiKey). Maybe the industry has some sort of unconscious moral compass that is preventing them from making computers boot idempotently.
You can’t restrict controller from writing on SD card by Lock switch but you can restrict it by writing in the SD card protection register. I’ve written about it at the linked post. You can search for more info about SD card registers and MMC block device and read more about it.
If you have have non-persistent VMs on secondary storage that may be infected and they have a way to infect dom0 then there is no point in persistent dom0 as it will be infected again on every boot and in the end will be infected on your SD card when you’ll want to update dom0.
If in your particular use case a real write protect is important then it is possible to use an Opal 2.0 SED drive with a write protected range defined over top of the partition containing your files. The range will be read-only and enforced by the drive itself so no software can bypass this enforcement.
Someone with the the proper key could of course modify the range back to r/w status but that’s not going to happen unless you deliberately make that key available to the software actually mounting the drive.
SSD’s may be faster than SD cards, and more durable. The Opal write protect mechanism is of course more difficult to manage than a physical switch.
Can you link to a reader that mounts as mmcblc? The one I have mounts as sd. If possible, can you link to one that connects via USB?
Yeah, I don’t expect it to be able to block any malware. But doesn’t it make sense to write-block any files on a system that don’t need to be dynamic? It could potentially block a malware that relies on a persistent modification to anything the write-block protects. Any idea on what other major components of Qubes we can write-block in addition to /boot?
This is cool. I didn’t realize these exist. I think this is the solution I would choose. Two concerns I would have are: Can I use RAID? And what would it look like to load as much from the partition as possible? tzwcfq mentioned that we should do this with /boot. What other areas of the file system can we lock down? Also, do you know if it would be possible to write-block only specific files on the partition? I’m trying to figure out how to use Ventoy to boot ISOs but allow persistent Live CD data.
You can search for RTS5129 or RTS5139 based card readers.
I think the better idea is to have persistent dom0 root on write-protected SD card and volatile dom0 files (VM configs, some Qubes settings maybe) stored on writable secondary storage like SSD. Then on boot you can copy this dom0 root from SD card in RAM or on SSD and then copy/overwrite the volatile files from secondary storage there. Maybe you’ll need to restart some services for new configs to work.
So the question is what volatile files from dom0 do you need to save on writable secondary storage to be used by dom0.
I don’t know what files these will be and where are they stored but you can run some software in dom0 to log all filesystem changes to see what files/directories are used by dom0 to store some volatile data or VM configs and configure dom0 to save/restore the needed ones.
If you plan to use write-protected SSD then you can just mount the needed volatile files/directories from secondary storage and don’t copy add dom0 to RAM/SSD.
If in your particular use case a real write protect is important then it is possible to use an Opal 2.0 SED drive with a write protected range defined over top of the partition containing your files. The range will be read-only and enforced by the drive itself so no software can bypass this enforcement.
This is cool. I didn’t realize these exist. I think this is the solution I would choose. Two concerns I would have are: Can I use RAID? And what would it look like to load as much from the partition as possible? tzwcfq mentioned that we should do this with /boot. What other areas of the file system can we lock down? Also, do you know if it would be possible to write-block only specific files on the partition? I’m trying to figure out how to use Ventoy to boot ISOs but allow persistent Live CD data.
Many SDD’s come with this built in. My initial thought was to do this as an anti-evil-maid setup as it would certainly prevent ANY modification of the boot partition. I was even working on this on a special project at work before I was forced to retire for medical reasons. I am now running a nonprofit for researching my disease (hdri.org) and have not had the time to go back and finish this work.
I don’t know much about RAID systems and how they would even deal with read only media, infact I’m not sure there would even be a benifit on read-only media for RAID since you could just read once and cache the data in one go, not needing/able to rewrite anything on that partition. Each drive in a RAID system would need its own keys to make it r/o on the device level so this could get very complicated.
Care will need to be taken when performing updates that need to write back to the /boot partition, obviously. That would be the only time you would be vulnerable to attack by having the keys/password exposed in memory. In my case I was planning to use the memory storage inside a TPM for holding this information but my machine at home doesn’t even have one. swtpm doesn’t have any true benifit in that regard but a vault VM could isolate that data for use during updates and do some kind of split gpg or something like that for unlocking keys to unlock the r/o data. How deep down the rabbit hole of security you want to go is based entirely on your personal use case and who your adversary is.
What other areas? Anything that can be desired to be set to r/o. You get to define four seperate regions to play with, and each can be marked r/o or just encrypted with its own key. Locking crypto currency data or a split gpg database/container behind its own keys would make sense to me. There would be many potential uses of Opal technology within Qubes but alas many people wearing tin foil hats won’t trust Opal/TPM technology. I get that, but I actually worked with the person who helped set the Opal standards and who even wrote the book on Opal, and so I do trust Opal to do what it promises. I actually wrote software and tested it for use by a government agency. If anyone could break Opal 2.0 it would be them. It works.
I just signed up for your smile.amazon.com. I wish you success in your endeavors.
Should I run a checksum on every file, do some heavy use, and then run a checksum again and lock any file that has the same checksum? Would I run this checksum report in dom0?
If they will not trust Opal/TMP, what will they trust? Isn’t just another tool that should be added to the arsenal?
Your hdri.org site could use some work. If you need a pro-bono savvy web-dev, email me at brian@megabyte.space - I can help you update your WordPress theme, get a few logos to choose from, and add a contact form.
Thank you! Every little bit helps to hopefully develop a test and find a cure.
The r/o feature of Opal ranges work at the block level. From cylinder number X to cylinder number Y. Typically you will define a range that covers an entire partition. I do know of one person who was incrementally enlarging a r/o range as a log file grew within a partition but that was a rather unusual use case.
True. People are generally afraid of black boxes that they can not verify for themselves. I get that, but almost every device these days has embedded code running inside of it and yet we trust those things, or at least we have to. Any non Opal drive has code too. The Opal standard is actually tested by a non-government organization to be sure it meets the very strict specifications. Each drive model from every manufacturer has to pass a test in order to use the Opal label branding on their device. Its not an easy test and its very thorough. Is there a bug in one of these drives? Likely, but at some point one needs to trust something. My take is if you don’t trust Opal to encrypt your data then add your own software encryption on top of that. The Opal drive processor will carry the overhead of its own encryption and your CPU can handle your software based encryption. It will be double encrypted then. I had one project I designed with double encryption where my supervisor joked about sending a drive to Russia just to watch them waste their time trying to break it.
Proprietary encryption is one of the things I’ll never put my trust into. Especially hardware encryption. There are just too many factors in hardware implementation that may lead to bypassing the encryption.
Here is an example of such improper implementations:
I finally found this thread continuation and read the paper. The steps used in the paper are basically to remove the physical drive from your machine, attach a JTAG device (the easiest way), extract the drives firmware, reverse engineer the processor code in the firmware with IDA Pro, find a flaw in that existing code, rebuild the firmware with the new exploit, and rewrite that modified firmware back to the device while somehow bypassing the firmware signature verification process built into the drive. What could be simpler?
This is not a general attack. This is a monumental undertaking for each and every physical drive. While I do know people who could do all this (me, if I wanted to) its not a practical attack unless its a nation state agency with many millions of dollars to throw at hacking your specific drive. The authors had to find a completely different set of flaws for each model drive that they included in their study. In some drives they had to use ATA commands to talk directly to some of the chips on the board to trick it into modifying the firmware, while others used JTAG. Once your own firmware is installed you then have access to the key-derivation routines, crypto blob, and the salt (entropy) values.
Bottom line: This is Physical Security problem we are talking about. If a person gets their hands on any piece of electronic equipment and they have both the time and desire to reverse engineer that device, its “game over”. This is not specific to SED/Opal drives. This also is not your typical “Evil Maid” attack. This isn’t even a general hack that can be deployed by any script kiddy, so unless NSA is after you, specifically you, then you are probably safe from this. I’m not sure that even the FBI would go through this much trouble for just one person.
I disagree with you.
The point is that if you use hardware encryption it’s possible for some non-ideal realizations to bypass the cryptography protection and in case of software encryption it’s impossible. This paper was focused on this aspect:
Machine off, awareness. The adversary has physical access
to a powered-down machine, and the victim is aware of this.
Therefore, from that point onward, the victim is unwilling to
enter key information into the machine. In this scenario, given
that the implementation is sound, software full-disk encryption
offers full confidentiality of the data, and hardware encryption
supposedly does so as well. In this paper, we focus on this
attacker model.
The work in that paper was done by two researchers (maybe there were more people but in any case it was a small group) and it didn’t cost them millions of dollars to do this. Yes, my specific drive with hardware encryption could be hard to crack but there’s still a possibility that it will be cracked or similar drive already was cracked before.
No, if you used software full-disk encryption to encrypt your drive then if anyone got their hands on this drive there would be no possible way to bypass the cryptography protection.
Pretty cool that people who use IDA Pro can literally say, “I da pro.” Makes me wonder about other subtle nuances are hidden in plain sight.
You’re right. Except AI is becoming very real. I imagine there is already some top secret AI that can do nifty stuff, perhaps automated IDA Pro routines. I went to the Amsterdam World Summit AI and a best-seller on Amazon said something memorable - he said comparing us to AI will be like comparing a fly to Einstein by the year 2045. And it doesn’t take Einstein to hack a computer and corrupt something that is meant to be read-only.
Unless… you can physically disable the write access. That’s why I opened this thread. The idea of write protecting with Opal and a physical switch sounds good to me though. I once hired a hacker to hack one of my websites and he gave up because my website was returning 418 HTTP codes (stands for the Teapot Protocol - officially released as a joke). Now, I imagine a hacker could potentially IDA Pro me but still face another hurdle which may be enough to have them give up on compromising the read-only section of the computer (and eventually the source of truth for other computers).
It may be impossible to crack the encryption when it stands alone after you boot the computer. But what about the scenario when the computer is actively being used, browsing the internet, downloading and installing system packages from HTTP endpoints, and, in my particular scenario, being used to respond to this thread.
This thread may be turning into a discussion about using encryption to protect information, which is cool. But the reason I opened it was to figure out a way of reliably ensuring bits and bytes remain intact, regardless of the environment or the “adversaries”.
And honestly, I’d love for the NSA to break through multiple levels of encryption, only to find an empty physically write-protected drive. Edit: I’d love for the NSA to cautiously decrypt the information they are looking for while keeping my firmware intact. Make them do this a few times, and then you can actually hide whatever it is that needs the encryption. For me, it’s the Qubes ISO.
With software encryption your keys are always in memory, with SED its never in memory ( unless you did it wrong). The password is in memory for only a split second, for you to unlock the drive and then scrub that memory, but even then the password is not the key needed to decrypt anything.
I also advise the belt-and-suspenders methodology when it comes to protecting important information from specific adversaries. I once designed a system with Opal and three layers of software encryption for specific purposes. The hardware encryption you get for free, and you can add just as many layers of software encryption to that as you need on top of that but then key management is a bear. It all depends on who you are protecting it from. Who is your nemesis.
But I think we have strayed far enough off topic at this point. All I wanted to say is Opal/SED can be used to write protect the unencrypted boot partition on Qubes. Nobody is going to mess with your UEFI/Grub configuration if they can’t even write to the drive. Protecting hardware against someone reflashing your physical drive is a little out of scope even for a Qubes forum. Feel free to contact me off list if you would like to extend the conversation.
IDA Pro has a great plugin system, so yes you can extend it with your AI if you like. There are already plugins that will do fancy things like extract keys and other things from memory. There are lots of fun things you can do with it, if you can afford it these days.
In case of Qubes someone has to first break away from Xen virtualization to get into dom0 to get the encryption key.
If attacker is already in dom0 then he also can install malware in your firmware to have a keylogger that will allow him to get your hardware encryption password.
Yes, it’ll be harder than to just get the key from RAM as it’ll be with software encryption but it’s still possible.
But as you’ve said it all depends on your threat model. If you’re Snowden and want to cross the border with an encrypted drive full of data that will land you in prison for life in best case then you won’t want to use hardware encryption that will be possible to be bypassed.
| 
slcoleman: