Is there a way for disposables created by qrexec policy target=@dispvm:VMNAME to create the disposable without setting any netvm for it, even when the disposable template has a netvm set?
Network and firewall settings for disposable templates can be set as they can for a normal qube. By default a disposable will inherit the network and firewall settings of the disposable template on which it is based.
If there is a way to force the disposables of a disposable template to not use the same netvm as the disposable template, it would save me a lot of trouble for my particular situation: disposable template runs Nix to build a cache of packages and tarballs, then other offline qubes want to use this cache on demand but don’t want anything to do with having a network connection.
alternatives I’ve already considered
I have in past lived with cloning the disposable template and removed the netvm pref from the clone.
This duplicated the private image, which was quite large for this use case.
The need to periodically delete and reclone was very cumbersome and led to being stuck on stale and outdated data.
I have considered using a named disposable instead of a disposable.
The multiple qubes using this qrexec service do not have any reason to be sharing the same destination qube however, and sharing the same qube does mean that no qube can receive an updated version of the underlying data until everyone stops using the service and I happen to notice and shutdown the named disposable. qubes-app-shutdown-idle isn’t safe to use in this instance either out of the box, so the lifetime issue is somewhat prohibitive.
I have considered having the disposable template have the netvm prefs the disposable needs to have and just using another qube or a named disposable to push the data into the disposable template, but that approach seems like a more involved and risky equivalent to my original approach of relying on clones.
And for Nix in particular I am not confident that I know what data I would need to copy over myself to actually replicate the same state. The nix store is one thing, but there’s evidently other bits of data that nix squirrels away on the system that I don’t want to lose in translation.
Right, but the caching outside the store does matter at least for nix channels. I don’t profess to know how the caching is implemented or tracked, but while the nix channel tarballs do get unpacked to the store, the cache where the tarballs are downloaded to is not in the store, and there is some notion of a time-to-live for those cached tarballs as well.
I haven’t been keeping up with how flakes and flake registries might add new forms of caching in future or present, so at this point I’m just more inclined to bet that whatever plan I could come up with to ferry data out would miss something, or maybe there would be a database file storing metadata on the cached files making simple relocation of cached files a no-go without understanding how to merge and update records or something.
Hence my hopes to be able to use disposables from a disposable template just without the same netvm pref rather than going through a complicated procedure to clone or shuffle data to accomplish something similar.
If you pin your tarballs, like downloading a specific commit tarball of nixpkgs, there is no need to cache because nix can check if it’s already in the store
In the interest of obtaining security updates, I do want the nixos-23.05 channel, so reproducibility in the presence of caching does matter for my use case.
For channels where I can work from a specific commit of nixpkgs I have in past downloaded a tarball of that commit to the qube and setup the channel using a file:// URL so that caching is not prohibitive and deleting and recreating the channel programmatically is straightforward even offline.
There’s a hacky way to achieve this if you’re using a ‘file-reflink’ driver Qubes OS storage pool (as in the Btrfs installation layout) to host e.g. a nix-dvm with network access, a placeholder nix-offline-dvm without network access (which will never be started, and whose ‘private’ storage volume will be a symlink), and ultimately the disp1234 DisposableVMs based on nix-offline-dvm:
# qvm-clone nix-dvm nix-offline-dvm # just for the VM metadata
# qvm-prefs nix-offline-dvm netvm ''
# qvm-prefs nix-offline-dvm include_in_backups False
# cd /var/lib/qubes/appvms/nix-offline-dvm # or wherever the pool is
# rm private*.img*
# ln -s ../nix-dvm/private.img
# chattr +i .
@rustybird
Is the symlinking the private image something that requires the file-reflink driver or would it possibly work with the default driver?
I haven’t had a chance to setup a storage pool with the file-reflink driver to try out what you recommended, but I’m going to go ahead and mark your post as the solution anyway despite not having tried it out to confirm it personally, as I won’t be hanging around these forums.
It might be possible to do something similar in a storage driver agnostic way, by stopping qubesd.service and editing qubes.xml to change the value of the vid attribute for the <volume name="private" ...> element of nix-offline-dvm to that of nix-dvm. But I haven’t tested this at all, and I don’t know how to prevent accidentally starting nix-offline-dvm (like chattr +i . did for the file-reflink setup) which would probably cause all kinds of weirdness and even data loss, especially if nix-dvm is also running.