Updating an air-gapped/network-gapped Qubes computer

User Support
1

I have a secure Qubes computer that is on an isolated LAN network (network-gapped) and am wondering how would be the best way to regularly update it, including updates Qubes Dom0 and Qubes Templates (Debian, Fedora, Whonix, etc), with the stipulation that this Qubes computer cannot ever connect out to the live internet.

Here’s some general potential approaches I have thought of:

  • A. Update Qubes templates on another Qubes computer and backup/restore these templates via something like a USB thumbdrive. Would seemingly only work for templates though, and not Dom0. Could script this to make more convenient than manual process.

  • B. Implement some kind of updates caching system on another Qubes computer that stores all the update files (maybe inside of a Qubes UpdateVM?). Then backup/restore that cache/UpdateVM via something like a USB thumbdrive, and have the air-gapped Qubes computer use it to install updates from.

  • C. Setup an updates web server mirror on the LAN that connects to Qubes and distro update servers, rsyncs all the update files, and then somehow setup local DNS to route updates to this. Theoretically the Qubes updates process would use this local web server as the update server without issues?

I’m speculating at this point, and am not familiar with how to actually achieve this. Has Qubes air-gapped/net-gapped updates even been solved/implemented before by someone?

Any advice/insights or specific solutions/instructions you may have for implementing updates for an air-gapped/network-gapped Qubes computer would be great.

Cheers.

1 Like
2

Could this work?

If you run a clone from the very beginning that you then update, pull the log from the update & then parse this onwards to download all packets then you could then run update on the airgapped one based on the downloaded packets.

Should not be very complicated to extract + download the packages themselves, some regexps and so on. Added convenience would be that you’d get a good opportunity to look through it all before running it on the airgapped one.

Bet someone has written some scripts for this kinda functionality too?

3

Mirror a Qubes Repo host it locally edit repo files ?

1 Like
4

The requirement is to update dom0 and templates, (by which I
understand update packages in the templates, not the templates
themselves.)

Debian has a nice tool called apt-offline which will update the
databases and download packages ready for you to transfer to an
offline system.

For Arch, I’ve used http://github.com/cookiengineer/pacman-backup

For Fedora? You can use dnf repoquery --upgrade to get a list of
available updates - then dnf download --resolve to get the packages.
For dom0 substitute qubes-dom0-update --action="repoquery --upgrade".
Transfer the files to offline machine and install with dnf install

You can use a caching proxy to grab packages and metadata, but you
will have to configure the proxy on the offline machine not to look for
updates to the metadata.
In the proxy I use, apt-cacher-ng, this is quite straightforward.

5

but with this, you wouldnt use the recommended salt based qubes updater or?

6

Thank you all so far for the ideas and insights.

I’ve done a bit more thinking about general approaches:

  • UpdateVM with Offline Caching: The most straight-forward solution would theoretically be to drop a caching program inside of the Qubes UpdateVM that catches all metadata and package files in a way that can be used for updating offline after shutdown/restart of the VM. Then one would just run an additional/clone online Qubes computer without holding any user data inside; run all of the Qubes Dom0 & Qubes Template package updates; have the UpdateVM catch/cache them inside of itself, then backup, transfer, restore this entire Qubes UpdateVM to the air-gapped Qubes computer; and then on the air-gapped Qubes computer run the normal Qubes update process through this restored UpdateVM with cached metadata and package files for all Dom0 & Template packages. Theoretically, the recommended more-secure “salt” based method of running Qubes updates should work with such an imported UpdateVM cache too.

I don’t know if such a Qubes-compatible UpdateVM caching program exists or not. Maybe rustybird’s “qubes-updates-cache” or unman’s “apt-cacher-ng” programs (linked below) could achieve this?

Regarding other approaches, here are some of my thoughts:

  • Running a Mirror Updates Web Server: I’m not familiar enough with Linux distro update authentication processes, but since many people/organisations create public HTTP update mirrors, then it should seemingly work to host the same type of HTTP update mirror on the LAN-network and point the Dom0 & Template repo files to the LAN-network IP address of one’s own update mirror web server. This seems to be the 2nd most straight forward solution. However, the cost involved would be the initial configuration and maintaining the web server setup that provides this mirror service between online and offline networks. Not a huge cost, but some cost, if a more ideal UpdateVM cache can’t be easily achieved.

  • Manually Scripted Downloader and/or Installer of Update Files: While this could seemingly work, this approach seems to have even more downsides and costs to it. Depending on the method of implementation, there may be a higher cost in the initial configuration/scripting setup. Additionally, my concern is with the security/authentication/verification of the updates. Experts might know how to manually process or script each step along the way without risk, but I don’t expect to become an expert in this subject matter any time soon. Potentially, a pre-existing script from a more knowledgeable expert would work fine here, and maybe some of the built-in protection mechanisms of Qubes, Fedora, Debian, etc would help to mitigate exploits upon installation of updates, but I know there have been several issues with update vulnerabilities that Qubes has even recently had to independently harden its update process against, so to not only rely upon Fedora/DNF etc.

Specific Personal Replies:

Brainhack - Thanks for your ideas. My thoughts on the general approach of your proposed solution are written above in the section “Manually Scripted Downloader and/or Installer of Update Files”.

ymy - Thanks for your ideas. My thoughts on the general approach of your proposed solution are written above in the section “Running a Mirror Updates Web Server”. Also, your concern about the security matter of the “salt” based method of running Qubes updates has been noted. Here is another Qubes forum discussion 4131 about that matter: Recommendation against "qubes-dom0-update" (use "Qubes Updater" instead)

@unman - Thanks for your ideas. Yes, your presumption of me having meant packages inside the templates is correct. Although, whole template updates would be an interesting optional extension too. I looked up some of your Qubes updates caching scripts (linked below). I haven’t tested them out yet and at first glance am not yet able determine how close they get to the more ideal UpdateVM cache I described. I have bookmarked them for further review/testing.

Existing Qubes Update Caching Tools Links:

qubes-update-cache by rustybird
Squid-based package update cache for Qubes:

apt-cacher-ng by unman
Using apt-cacher-ng as caching proxy:
github(dot)com/unman/notes/blob/master/apt-cacher-ng

salt-based apt-cacher-ng by unman
This is a caching proxy, based on apt-cacher-ng:
github(dot)com/unman/shaker/tree/main/cacher

Any further ideas, insights, resources, are always appreciated!

Cheers.

7

@unman Your note about “you will have to configure the proxy on the offline machine not to look for updates to the metadata” seems important for implementing your apt-catcher-ng scripts offline.

Would you be able to share the commands necessary to turn off the metadata?

Cheers.

salt-based apt-cacher-ng by unman
This is a caching proxy, based on apt-cacher-ng:

apt-cacher-ng by unman
Using apt-cacher-ng as caching proxy:

8

Depends on which method you use.
If you are using a cache, you could still use salt.

1 Like
9

apt-cacher-ng has an option in the config file for offline use.
That should do it. (should)