Automated Qubes OS Installation using Kickstart and/or PXE Network Boot

[qstateless]

Thanks for your sharing your expertise, @alzer89!

@alzer89

I was about to that that this would likely be the easiest and most straightforward way for you to achieve what you want.

It would get particularly messy if you wanted multiple clients to be able to interact with the same AppVMs simultaneously, and there would likely be a lot of unforeseen implosions and the like that I wouldn’t be able to anticipate at this point…

Agreed. Separate client computers would have their own separate AppVMs.

@alzer89

But things start to get a little restrictive if you want to boot a read-only ISO. You basically either load the entire thing into RAM, turning your RAM into a boot drive; or you load it into RAM of a network server and “transmit” the parts the client machine needs on demand…

I think loading the entire thing into the client computer’s RAM would be best. That’s what I’m currently doing with Qubes in tmpfs. Hopefully, I could then even cutoff the network connection to the PXE server once Qubes Dom0 is loaded into the client computer’s RAM.

@alzer89

The 3.1 version of Qubes had a LiveUSB ISO, but it ended up being paravirtualised, making it potentially problematic to rely on, especially if you regularly used it on a variety of hardware configurations…

Yeah, I’m aware of the old Qubes Live 3.1 ISO, and wouldn’t want to rely upon that as you say.

@alzer89

@qstateless

Does the OS need to be remotely loaded from an .ISO (iso9660) formatted image? For example, you seem to use the Qubes Installation .ISO for your PXE setup.

Hahaha. No, it doesn’t

I just used the ISO image because it achieved what I was trying to achieve, which was create a way to “deploy” an automated Qubes OS install.

Basically, PXE boot needs to load the xen/kernel/initrd first, and then root= states where to find the OS drive.

So that parameter on most current Qubes OS installs is root=/dev/sda3 , root=/dev/nvme0n1p3 or root=UUID=s0m3-r4nd0m-uu1d .

But you could easily change it to root=nfs:192.168.69.69:/nfs-exports/laptops/wife-laptop if you wanted to

Or even root=nfs:awesome-qubes-os-share.example.com:/ninja-turtles/michelangelo/cowabunga-dude if you were game enough to do it over WAN
(with encryption, obviously…)

Wow, that is awesome. For some reason, I falsely associated PXE booting with use of ISOs, but this flexibility of where to load the OS from is awesome.

@alzer89

Because You’d need to make sure that each of the client machines wouldn’t be running duplicate instances of the same daemons, messing with .so files, system logs, file permissions, and the like.

You can fix this by having a separate image/drive for each machine, but that means you’re just moving the hard drive to another computer…

PXE can also only serve one machine at a time, so there would be a long queue. Thankfully NFS doesn’t have this bottleneck, otherwise it would be a nightmare…

I see. If PXE is limited to serve one client computer at a time, then I would just try duplicating the PXE software setup many times on the PXE server, running each PXE server instance in a separate qube with a different LAN IP on the network. So PXE-1 serves Client-1, PXE-2 serves Client-2, PXE-3 serves Client-3, etc.

@alzer89

Well this can already be done with PXE booting a unique disk image for each machine (or restricting the disk image to one machine at a time)…

Yes.

@alzer89

Mind you, this wouldn’t really provide any “security” benefits over having the drive locally, other than sounding really cool to your boss

I think there are some subtle security benefits, although maybe not so important to most people.

The primary feature gained can be lack of any OS state saved if using a client-RAM-based OS image.

Benefits:

• Prevents drive firmware attacks / drive firmware persistence on the client computers. Reduces the storage drives footprint down to my PXE server and my file server.

• Enables greater anti-forensics on the client computers (can be important even for common situations like civil lawsuits where everyday lawyers try to subpoena your drives, or hackers plant stuff on them remotely and SWAT you, etc). Less drives to manage and worry about as potential liabilities with something nasty sitting on them that I can’t conveniently audit. Reduces the storage drives footprint down to my PXE server and my file server, rather than also having dozens of endpoint computers containing storage drives.

• Adds convenience of quickly wiping to a known clean OS state on client computers, which can then easily be done many times per day, compared to a major pain that doesn’t hardly ever get done otherwise. Like having DispVM capability for the entire Qubes OS running on your machine. Just reboot to a fresh read-only instance of RAM-based Qubes OS provided from the PXE server. 99.9% Trustworthy as clean. Would never reinstall Qubes OS multiple times per day like this to a known clean state, but network booting could make this easy.

@alzer89

It would be more of a convenience for you.

Huge convenience, yes!

@alzer89

• Your Qubes OS install would be tied to another machine via network
  • I have no idea how sys-net would react to having the machine booted via PXE, and then having the same NIC passed into a VM…
  • It would probably be similar to booting Qubes OS from a USB drive, and then opening sys-usb : OS implosion
• Your NIC would continually be inside dom0 in some way, shape or form (your dom0 would need to come from somewhere , right…?)
  • Depending on how your PXE firmware was loaded, you may have a gaping hole in your Qubes OS install
  • I’d be curious to try it, though

Having 2 or 3 network cards per client computer might be necessary, where one NIC is sacrificed for the PXE network boot of the Qubes OS, which is okay for me.

However, if a RAM-based Qubes is entirely loaded into the client’s RAM first, before Xen/Qubes Dom0 boots up, then maybe the connection to the PXE server can be cutoff once the Qubes OS is loaded into client RAM and the NIC could be used normally by Qubes in a sys-net? It appears with Qubes in tmpfs, that the Dom0 is loaded into RAM like this before booting Xen/Qubes Dom0, so maybe it wouldn’t need any connection/resources/etc tied back to the PXE server once loaded and booted entirely into the client computer’s RAM?

@alzer89

Well, dom0 could be a read-only ISO, and the VMs could be NFS/CIFS/SSHFS (or SSHFS OVER TOR mind blown) shares. That would work, and allow the dom0 to load into RAM…

The only thing is, this would require every single one of your machines to constantly have an ethernet cable plugged into it. If you’re ok with that, then let’s make this happen

Yes! I am game with you for achieving this setup! I have several duplicate machines ready to go for pursuing this setup.

@alzer89

Encrypt it, please

Yes, I will be looking to encrypt by default. Thanks for the reminder. I also use physical network segmentation in addition, because I don’t like mixing cross-purpose or cross-security-domain stuff into the same physical network.

@alzer89

Then you’d need a secondary LAN for internet access (and I’m sure you’ve got a plan for that )

Yes, I have several different independently segmented networks setup here, and multiple physical NICs on separate sys-nets per Qubes machine.

@alzer89

I’m currently restoring a Compaq Evo N150c with Gentoo, and the thing takes 3 minutes to fully boot (and the record for a complete world update was SIX WEEKS).

I feel your pain…

Wow, 6 weeks. That’s pain.

@alzer89

Well, that was probably what renehoj was asking about. The configurations for the VMs will have to come from somewhere. Will you bake them into the ISOs (meaning adding and removing VMs would be impossible), or would you allow this to change, bearing in mind that if one client changed it, then those changes would come up on the next reboot to any other client machine that PXE booted?

Presently, with Qubes in tmpfs, I have made custom scripts that restore persistent AppVM configuration and personal file access, upon startup of Dom0 and AppVMs.

If need be, I would just try duplicating the PXE software setup many times on the PXE server, running each PXE server instance in a separate qube with a different LAN IP on the network. So PXE-1 serves Client-1, PXE-2 serves Client-2, PXE-3 serves Client-3, etc.

Also, I could have the PXE server write-protect or, or if need be, just overwrite the Qubes OS install regularly.

Through a combination of tactics, it seems feasible to create a RAM-based Qubes OS PXE boot setup, for multiple client computers, that does not have to change with each reboot.

@alzer89

For example, your family members could purge your work VMs, and you wouldn’t be able to stop them…

Nooooo way. Wouldn’t let that happen. Their machines can’t touch the file server shares my work VM contents is stored upon. Physically isolated network. Also, their own AppVMs would have separate credentials (per PC and per VM) to access different shares on the file server, depending upon who’s specific machine and what specific AppVM is accessing the file server.

---

@alzer89, what do you think is the best way to make a RAM-based Qubes OS ready to use in sys-pxe?

Maybe this would work…

• 1. Install Qubes OS normally onto a drive or RAW image.
• 2. Add Qubes in tmpfs modifications to that installation.
• 3. Copy that drive or RAW image to the sys-pxe server.
• 4. Point the PXE to this volume or RAW image file containing Qubes in tmpfs.

Then expect the client computer to boot Qubes in tmpfs and run Dom0 from the client computer’s RAM?