RAM-based Qubes OS over PXE Network Boot

[qstateless]

This is a sub-topic fork of @alzer89’s Automated Qubes OS Installation using Kickstart and/or PXE Network Boot, focused on creating a repeatable guide to PXE Network Boot a RAM-based Qubes OS.

Basic concept: Your Qubes OS installation sits on another computer with the PXE server software. You can then have as many client computers as you want PXE network boot from this RAM-based Qubes OS using only their network connection and no drives needed to boot from. Your client computers reboot to a 100% clean Qubes OS state each time. Scales to many client computers at once.

RAM-based Qubes OS over PXE Network Boot is very useful for:

• Running further stateless endpoint computers without storage drives in them.

• Anti-forensics. Nothing stored on endpoint computers, except firmware of the hardware parts. No drives necessary at all.

• Disposable Qubes OS. Just easily reboot to instantly make and boot a new clean install of Qubes OS many times per day.

• Quick centralized management & deployment of Qubes OS configurations/updates to multiple endpoint computers.

Interesting initial conversations and a general Qubes OS PXE installer implementation in the general topic thread mentioned above.

Note: If you’d alternatively just like to run a RAM-based Qubes OS from a normal boot drive, then this is already available using @xuy’s Qubes in tmpfs.

[qstateless]

@alzer89 continuing in this forked sub-topic…

@alzer89

I meant the initial loading of the xen.gz , vmlinuz and initrd.img files. The PXE server seems to not be able to serve those files in parallel to multiple machines simultaneously.

Once you serve the ISO over another protocol (I used NFS), there’s no bottleneck whatsoever

At least, that’s what I encountered in my testing of getting 6 legacy boot laptops and 6 UEFI boot laptops to PXE boot the Qubes ISO simultaneously…

Ah, better than I thought. Understood.

@alzer89

@qstateless

Benefits:

• Prevents drive firmware attacks / drive firmware persistence on the client computers.

I would respectfully argue that these attacks are just moved to a different place…

Yes & No. For my setup, the path to get to attacking my file server is a meaningfully harder one for attackers.

If going after a client machine drive of mine, they would have to breakout of Xen and attack the locally installed drive.

If going after a LAN-networked file server drive of mine, they would have to breakout of local Xen, then successfully break into the file server’s DomU, and then breakout of that server’s Xen and attack the remotely installed drive. More chaining of exploits needed.

Also, on the client endpoint machines, lots of random code, apps, websites, scripts, etc get executed regularly. Where as, on the file server, files just sit there unexecuted, and are served up through whatever networking protocols. I think it is an important added security barrier to have the files stored on a separate machine like this who’s processor does not directly run untrustworthy code.

@alzer89

Many would argue that this is more cumbersome than connecting a physical drive to a SATA/NVME/PCI slot on a motherboard, but it appears tht this approach better suits your use case.

Just be sure to factor all of this in when you use it

No worries. The efficiencies really add up the more machines this solution simultaneously serves. In my case, dozens of computers, so I expect it to greatly ease my already very cumbersome management & maintenance tasks across all of these machines.

@alzer89

There is potential for VLANs to alleviate some of these issues, allowing you to use a single NIC, but I’d need to investigate a lot deeper into this than I already have…

Yeah, VLANs are a possibility, but I tend not to like VLANs. The use of VLANs still runs mixed purposes over the same nodes & wires. I prefer good old fashioned hardware isolation, where the bits flow through entirely different hardware. We’ve seen too many processor and memory level vulnerabilities in recent years to know that any local software protocol alone can be compromised or interfered with by a deeper hardware exploit.

@alzer89

@qstateless

However, if a RAM-based Qubes is entirely loaded into the client’s RAM first, before Xen/Qubes Dom0 boots up, then maybe the connection to the PXE server can be cutoff once the Qubes OS is loaded into client RAM and the NIC could be used normally by Qubes in a sys-net?

That would definitely be possible, but if you were to do that to a current Qubes OS install without any modifications, do you realise how much RAM you’d actually need…?

The dom0 partition in the current standard Qubes OS install is at least 24GB.
That means:

• You’d lose at least 24GB of the client’s RAM to a dom0 ramdisk
  • This would only really be viable on server motherboards with a billion and one RAM slots (mad respect if you have such a motherboard, I’m super jealous!)
  • Your options for doing this are basically including a complete dom0 in the initrd.img , essentially making the initramfs 25GB, which would likely take 30-90+ minutes to serve via PXE

I thought Qubes Dom0 was much smaller than this, as I am presently running Qubes Dom0 from RAM in only about 6GB of RAM using @xuy’s Qubes in tmpfs.

In My Dom0:

sudo df -m

Filesystem     1M-blocks  Used Available Use% Mounted on
devtmpfs            4933     0      4933   0% /dev
tmpfs               4953     0      4953   0% /dev/shm
tmpfs               4953     4      4949   1% /run
none                9905  6065      3840  62% /
tmpfs               4953     0      4953   0% /tmp
xenstore            4953     1      4952   1% /var/lib/xenstored
/dev/sda1            974    72       836   8% /boot
tmpfs                991     1       991   1% /run/user/1000

@xuy

It takes 6Gb of extra 𝚁𝙰𝙼 (for store root filesystem files)

My experience seems to match @xuy’s guidance on 6GB of RAM for the Dom0 root filesystem in RAM. This filesystem in RAM also initially loads and boots as closer to ~3.5GB before expansion from usage.

Maybe I’m not counting something properly?

@alzer89

@qstateless

Nooooo way. Wouldn’t let that happen. Their machines can’t touch the file server shares my work VM contents is stored upon. Physically isolated network. Also, their own AppVMs would have separate credentials (per PC and per VM) to access different shares on the file server, depending upon who’s specific machine and what specific AppVM is accessing the file server.

So then I guess you’ve decided that each machine gets their own Qubes OS install on the server

Not necessarily. I could do that, but would ideally like to consolidate down to one generic Qubes OS installation on the PXE server, if I can, and then handle the unique AppVM state separately that “customizes” each RAM-running instance of Qubes OS on the various client computers.

@alzer89

@qstateless

what do you think is the best way to make a RAM-based Qubes OS ready to use in sys-pxe?

Off the top of my head, I would say:

1. PXE serve xen.gz , vmlinuz , and initrd.img

• Have the initrd.img contain a complete read-only dom0

OR

• Have a dom0 served via NFS as an image file once the initrd.img was served via PXE, allowing faster boot times of client machines, particularly if multiple clients are booting simultaneously

Interesting. I’m not yet familiar with these xen.gz, vmlinuz, and initrd.img files and their function/purposes.

[alzer89]

@qstateless

I’m not yet familiar with these xen.gz , vmlinuz , and initrd.img files and their function/purposes.

They boot your machine

The Vmlinuz Kernel: A Critical Part Of The Linux Operating System – Systran Box

Initial ramdisk

@qstateless

Yes & No. For my setup, the path to get to attacking my file server is a meaningfully harder one for attackers.

It’s no different than any other attack. Just because it might not be connected to the internet doesn’t mean that the network link (means of transport) doesn’t exist. Maybe it’s hiding in one of those files you open.

Just saying. It’s not as clear-cut as you’re thinking it is (and I wish it was….).

@qstateless

If going after a client machine drive of mine, they would have to breakout of Xen and attack the locally installed drive.

This is the part where it enters a grey area….

If you’re going to load dom0 entirely in RAM, but have a constant network stream for every single VM (eg. an NFS share), then you’ve still got a way out of the VMs that you can’t turn off (your machine would stop working if you did….).

@qstateless

Where as, on the file server, files just sit there unexecuted, and are served up through whatever networking protocols. I think it is an important added security barrier to have the files stored on a separate machine like this who’s processor does not directly run untrustworthy code.

Well, that’s the plan, at least. You’d just have to make sure the serving machine was properly configured.

@qstateless

I thought Qubes Dom0 was much smaller than this

Well, in any case, if dom0 was too large, it would restrict client machine hardware to absolute beasts with a rackmount amount of RAM.

But there will be a way.

——-

I do like the idea of dom0 being an image loaded into RAM.

I’d be inclined to design a system that would be flexible enough to become “Qubes Air” or “Qubes in the Cloud”, meaning that this design would be able to be used by many other people.

The issue is how to make a personalised Qubes OS config from a generic image server via PXE.

[renehoj]

@qstateless

Anti-forensics. Nothing stored on endpoint computers, except firmware of the hardware parts. No drives necessary at all.

Why not just use Tails, if this is what you want?

You would need to have both dom0 and all domUs in memory at the same time.

Each domU (and dom0) needs a writable root file system, they are going to take a lot more memory than just the memory need by the vm.

Just booting would need to load dom0 + sys-net + sys-firewall + sys-usb + template into memory, with each sys having a writable copy of the template in memory. If you also add sys-vpn + user-browser + user-email + user-personal + user-work + 1-2 extra templates, it really starts to add up.

I don’t know how much memory you are going to need, but I wouldn’t be surprised if you need 64 GB just to boot the system.

[alzer89]

@renehoj

Why not just use Tails, if this is what you want?

That’s what I said, but @qstateless wishes to use Qubes, and who am I to say no to that…?

@renehoj

You would need to have both dom0 and all domUs in memory at the same time.

That’s one way of doing it.

But I see this as an opportunity to map out “Qubes Air”, in the sense that some VMs could be run locally, some could be run remotely and streamed via VNC or X11 forwarding, and some could be a hybrid of the two.

VMs with the Qubes packages installed would behave similarly to locally-run VMs, and even when hosted remotely, could potentially behave similarly to the current Qubes OS experience.

@renehoj

Each domU (and dom0) needs a writable root file system, they are going to take a lot more memory than just the memory need by the vm.

Definitely. That can be achieved in RAM, or with syncs to a remote host.

In any case, further investigation would be needed.

@renehoj

Just booting would need to load dom0 + sys-net + sys-firewall + sys-usb + template into memory, with each sys having a writable copy of the template in memory. If you also add sys-vpn + user-browser + user-email + user-personal + user-work + 1-2 extra templates, it really starts to add up.

I don’t know how much memory you are going to need, but I wouldn’t be surprised if you need 64 GB just to boot the system.

Well, I have a ProLiant 8 with 192GB of ECC RAM ready to go for this.

Once it “works”, we can start tweaking it to use less RAM.

(Yes. It’s inefficient and definitely not the way a project would go in a business environment, but it’s better than shelving this. This could potentially morph into “Qubes Air”, and I’m kind of really hoping that it does )

[renehoj]

@alzer89

But I see this as an opportunity to map out “Qubes Air”, in the sense that some VMs could be run locally, some could be run remotely and streamed via VNC or X11 forwarding, and some could be a hybrid of the two.

Qubes Air is something completely different, from using PXE and running the OS in memory.

You can also get mainboards that has iscsi initiator in the firmware, but also not the same as running the OS from memory, but it would allow you to run qubes without local disks.

[alzer89]

@renehoj

Qubes Air is something completely different, from using PXE and running the OS in memory.

I respectfully disagree with this statement.

We aren’t even sure what Qubes Air would look like, and I genuinely think this could be one of the potentially viable forms, especially if the VMs were not on the same host as the TFTP server (which would not be a requirement of this endeavour).

The fact that the VMs will be remote (or a mix of remote and local) leaves the same door open for dom0, and I think that should be included in the same category.

In any case, regardless of the outcome, it would provide valuable insight into Qubes Air, further progressing it.

@renehoj

You can also get mainboards that has iscsi initiator in the firmware, but also not the same as running the OS from memory, but it would allow you to run qubes without local disks.

Very cool idea, but you’d still have a local disk somewhere, which might not be good enough for @qstateless…

But you’ve given me a thing to try over the weekend

[renehoj]

@alzer89

We aren’t even sure what Qubes Air would look like

It’s not going to be a PXE booted system running from memory.

You can enable sys-gui, and you get an idea of what it’s going to look like.

[qstateless]

@alzer89

I do like the idea of dom0 being an image loaded into RAM.

I’d be inclined to design a system that would be flexible enough to become “Qubes Air” or “Qubes in the Cloud”, meaning that this design would be able to be used by many other people.

The issue is how to make a personalised Qubes OS config from a generic image server via PXE.

@alzer89

We aren’t even sure what Qubes Air would look like, and I genuinely think this could be one of the potentially viable forms, especially if the VMs were not on the same host as the TFTP server (which would not be a requirement of this endeavour).

The fact that the VMs will be remote (or a mix of remote and local) leaves the same door open for dom0, and I think that should be included in the same category.

In any case, regardless of the outcome, it would provide valuable insight into Qubes Air, further progressing it.

@alzer89 Yes! Would love to see your Qubes Air like implementation come out of this work. I’d most certainly make heavy use of it. Official Qubes Air is likely going to take several years to come into existence and I believe at the recent Qubes OS Summit they even said sys-gui might not be stable for the next couple versions, so lots of time-space open and available for such community innovations to flourish.

[qstateless]

@renehoj

@renehoj

Why not just use Tails, if this is what you want?

A lot of people instinctively think Tails, just because it is known for running live in RAM. However, Tails is horrible for security, when compared to Qubes. Just because Tails efficiently runs in RAM, does not mean it is securing your apps & files & hardware very well as it runs. Regardless of running from a drive, or live in RAM, once up and running, the security strength of Xen/Qubes is what is also important over Debian/Tails. Qubes OS wouldn’t be a necessary or important security project, if monolithic Linux systems were good enough for strong security. Qubes’ original Arch Spec whitepaper, as well as Joanna’s talks and blog posts, are a good source for understanding the general weaknesses of monolithic Linux systems and why Xen/Qubes is fundamentally stronger security (not perfect or without bugs, but fundamentally stronger architecture). Getting live RAM-based properties for Qubes OS is what is very important for me (which Qubes in tmpfs provides), as well as getting rid of regularly cloning and managing the local hard drives across the dozens of machines I manage (which is what sys-pxe will hopefully provide). Huge time savings for me, with some subtle but meaningful security benefits thrown in.

@renehoj

You would need to have both dom0 and all domUs in memory at the same time.

Yes. That is totally fine with me. I personally love running Qubes OS in a more stateless and disposable way from RAM. Super clean. Just reboot to an “instant” clean install of the whole Qubes OS. Across multiple machines, the hard drives and updates/maintenance start becoming a serious nuisance though (hours and hours of regular maintenance tasks for me), so sys-pxe to the rescue for central management and deployment across multiple machines!

@renehoj

Each domU (and dom0) needs a writable root file system, they are going to take a lot more memory than just the memory need by the vm.

Yes. No problem. Computers have RAM and specs are growing into the future. Already successfully running RAM-based Qubes in tmpfs this way, and it takes ~6GB RAM for holding Dom0 in RAM, so no problem. PXE network booting a RAM-based Qubes OS just changes the boot source from local hard drives to a centralized image stored on the LAN.

---

This becomes a solution to a natural problem for just about anyone if you do run a RAM-based OS across several machines, whether Qubes, Tails, etc (doesn’t matter the OS).

Regularly updating the OS and recloning the drives, over and over for a lifetime, is a heavy ongoing maintenance burden across dozens of machines.

PXE network booting will be a big releaver of pain, as you then just need one central OS image that all of the client computers can network boot from. Just update the one central image on the PXE server and reboot all your machines. Fully updated and clean state for all. End of maintenance. Done.

---

People are so used to running their OS from local hard drives.

Running from RAM and PXE network booting are just a couple different ways of operating & thinking about interacting with your OS.

Cutting the umbilical cord to the hard drives brings freedom to a superior way of computing that is closer to fully “stateless”.

It’s like the freedom gained when you first learn to run more of your tasks through DisposableVMs.

Except you get to treat the entire Qubes OS as disposable, which is quite freeing once you get it and experience it regularly.

Thinking about whole Qubes OS machines as disposable nodes to work with is pretty neat and useful.

[alzer89]

@renehoj

It’s not going to be a PXE booted system running from memory.

But it could be…

There’s nothing inhibiting it from being like this, as well as a standard on-drive installation…

@renehoj

You can enable sys-gui, and you get an idea of what it’s going to look like.

Way ahead of you on that one. Enabled it six months ago, and regularly check up on it’s progress. It looks good, and mad respect to the creators of it, but it wasn’t quite what I had in mind for this application…I wanted to do something better than a “glorified Apache Guacamole” (no offence meant, but that’s essentially what it is: a VM designed to remove the GPU from dom0, which, given the size and “blackbox-ness” of some of today’s GPU drivers, is definitely warranted )…

I envisaged a way to keep your VMs off-site (eg, the infamous “cloud” that everyone seems so excited about ), and connect to them and interact with tehm in a way that the hoster of your VMs can’t make sense of them. This would allow for any Qubes OS install, regular or RAM-boot, to “add a remote Qube” to their menu.

People could add that Qube to their Q-menu of their daily driver Qubes OS, and it would behave just like a local Qube. And people like @qstateless could also PXE boot a generic Qubes OS image, load some sort of config that loaded their remote Qubes, and the experience would be the same.

[renehoj]

@alzer89

But it could be…

https://www.qubes-os.org/news/2020/03/18/gui-domain/

Finally, decoupling support for VNC and other remote desktop capabilities opens the door to various server-based solutions in which Qubes can run on a remote server, and we can delegate some or all of our domains to other machines (potentially with faster harder and more resources). This is a another step toward Qubes Air.

Judging from the official statement, I think it’s a lot more likely sys-gui is going to turn into Qubes Air, then a system based on PXE booting qubes from memory.

[alzer89]

@renehoj

I think it’s a lot more likely sys-gui is going to turn into Qubes Air, then a system based on PXE booting qubes from memory.

I agree. So why not both? They’d both have their use cases…

[sly0bvio]

This all seems similar to an idea of my own. Essentially a home, central server is used for most of the larger processing needs, and work is divided up between the device and main host (or the host and other processing servers like cloud services and AI services) in order to deliver a reasonably consistent experience.

[qstateless]

FYI: I haven’t forgot about this RAM-based Qubes OS over PXE Network Boot project and intend to make this work and publish a Step-by-Step guide in the future.

Life continues to happen and so I haven’t had the time yet. I expect to accomplish this project “sometime” in 2024.

TODO:

• Get @alzer89’s sys-pxe system working.
• Implement Qubes Stateless with sys-pxe.
• Setup PXE boot capability on my PCs.
• Create & Publish Step-by-Step guide for everyone.

If sys-pxe is made ready earlier, by another person, then this could shortcut the time for me finishing this project. See prior posts in this thread for more info.

In the meantime, I just posted new Step-by-Step instructions for running Qubes 4.2 stateless from RAM:

Qubes Stateless 4.2: Step-by-Step Instructions

Looking forward to getting rid of the cumbersome maintenance of dozens of Qubes OS hard drive installations, by then PXE network booting a RAM-based Qubes Stateless image from a central boot server! Should be very clean and secure.