Do you need an option allowing to save backups in multiple files (one file per qube)? I think it would be helpful for everyone, because one would be able to manage old backups more easily:
One could see which qubes are backed up without starting Qubes Backup (or even Qubes at all if necessary);
One could easily manage the old backups: delete qubes one not longer considers valuable (e.g. Debian-8, Fedora 28 templates) to save space, make additional copies of valuable qubes with simple copy-paste;
Potentially faster access to the data by only decrypting the needed qubes.
@adw thinks it is confusing for regular users and is only useful for power users, but I do not consider myself the latter. For this reason he closed the Github issue:
My sense is this is not a mainstream use case.
Do you agree with him?
P.S. Sorry for not commenting in the Github issue. I do not have a Github account and do not want to agree to Microsoft ToS (including possible more crappy versions of it in the future).
As I mentioned on the issue, this already exists: You can simply use qvm-backup to back up a single VM. Repeat as many times as you like. Backup profiles and scripting can make this automatic. You now have one-file-per-qube backups.
You can already see this in the Qube Manager. It tells you the date and time at which each qube was last backed up.
This concerns managing the backup files after qvm-backup creates them. Thatās something an external program could do, or something you could do yourself with a naming scheme and a policy for how you manage the files.
This, again, concerns how you use the available tools.
The idea is that the sorts of users who wants to do fine-grained manual management of their backup files (and who want more files to manage) are generally the sorts of users who should be willing to run qvm-backup multiple times when they want multiple outputs, especially since they donāt actually have to run it multiple times manually. They can use backup profiles and, optionally, a simple script to make it easy to get exactly what they want. All these tools are already built in to qvm-backup.
Now, if the claim is that there are a lot of users who want this functionality but who donāt know how to use the available tools to do it themselves, then that might be sufficient justification to spend scarce developer time on it. Iām somewhat skeptical that there are a lot of users who fall under this description, but itās an empirical question how many there are, and Iām open to learning the answer.
As I mentioned, I do not consider myself an advanced user. Although I sometimes use terminal, I cannot, without a ton of web searching, get this working (and even then, Iām afraid of mistakes, because backups are critical). Care to share your scripts with automation?
Yes, this is exactly why I started this topic. I believe that more users need this option than you expect (but I cannot know for sure of course). I even actually think that this option should be the default, because why is a single file better than multiple files in a folder at all? Managing your backups with your favorite file manager is definitely more convenient than with Qubes Backup tool.
Do you mean document them here? We surely dont want Qubes documentation
filled with this sort of helper script.
Hereās a trivial bash script.
#!/usr/bin/bash
cd /home/user/backups
for qube in `qvm-ls --raw-list`; do
qvm-backup -q -y --passwordfile /home/user/backups/passwordfile /home/user/backups $qube
if [ $? -eq 0 ] ; then
last=`ls -tr|tail -n1`
mv $last $qube"-"$last
unset $last
fi
done
If you want a subset of available qubes. then create a text file with
qube names, one on each line, and replace the 3rd line with:
for qube in `cat qube_list`; do
What should happen if the backup fails - Stop the script?
Generate a big warning message, or a terminal message?
Note too that qvm-backup is incredibly noisy, even with the -q flag set.
I donāt use a script to make one-qube-per-file backups. I just occasionally do that manually, when Iām done with a VM and want to archive it.
If I did, it would probably just be a less sophisticated version of what @unman already shared.
In general, qubes are only usable together in a single integrated system. I canāt take a single qube backup and use it anywhere else besides another Qubes installation or for emergency backup recovery (and, even in both of those cases, I can still select one qube out of a multi-qube backup). In particular, ServiceVMs, TemplateVMs, and dom0 are mainly there to support your use of AppVMs. Given that these things are only useful when theyāre all together, it makes sense that they would be bundled together in a backup.
Iād also guess that most mainstream users would rather deal with one file than a whole bunch of files, but thatās another empirical question for user research.
Thereās also a confidentiality problem with per-qube backups. Qube names are fairly well guarded inside of a Qubes OS installation. Itās not intended that they would be exposed in the outer filenames of Qubes backups, since those backups could be in an untrusted environment. (This is why qvm-backup-restore is so careful to authenticate the backup before processing it.) You could obfuscate the filenames, but that would make them incomprehensible to humans, negating several of your use cases. You could let users opt in to exposing qube names in filenames, but now we have a potentially dangerous option in qvm-backup that introduces a new layer of complexity.
I mean itās trivial bash - not very good, either, but poc.
qvm-ls --raw-list generates a list of qube names.
Create a loop over the results of that command ( the back ticks run the
command and return the result), and for each qube name in the list run
the qvm-backup command.
Check the exit status of the qvm-backup command. $? gives you this. If
the command completed successfully, the exit status will be 0: the if
statement tests this.
In that case, find the name of the last created file, and prepend the
qube name (stored in the $qube variable). Throw away the variable storing
the original name.
Go to the next item in the list of qube names, and qvm-backup that.
And yes, @adw is right that you may not want to expose the qube name in
the filename, but if you are generating large numbers of these files,
you will probably need to.
Thank you for the detailed and very reasonable reply. Indeed, qube names are better to keep private.
I only now have just one problem. Is there any way to remove specific qubes from my earlier backups? I do not need to keep dozens of debian templates (each 10+ GB), which differ only by updatesā¦
I donāt know, but I can offer some speculative thoughts: On the one hand, the entire backup file is one big archive, so, in principle, itās possible to manipulate the contents (see Emergency Backup Recovery without Qubes for some hints). On the other hand, since the entire backup file is HMACed, any such manipulation would break your ability to authenticate the entire backup with qvm-backup-restore.
My personal approach to this is to back up templates separately from other VMs.
I wrote a bash script that will scan for volumes in sys-usb, mount that volume (doesnāt do luks pw), and then scan dom0 to see what vmās have been modified since their last backup. It optionally shuts down running vmās and optionally halts the system when done. It logs to a file in dom0 which could be scanned upon boot up to check for errors.
The volume containing the directory āQubesBackupsā will have one directory created for each vm, and multiple copies will be stored in each directory. Before running a backup the script will run a script in sys-usb to prune old backups until there is enough room on the drive for the vmās that need to be backed up, and always tries to keep a minimum number of archives for each vm.
This works well for me but needs improvement. In fact I had my computer hardware die today and I just finished restoring everything on a laptop so I could shop for a new desktop system.
If I can find time tomorrow I might be able to share that script for others to play with. The latest pruneāing script in sys-usb might have been lost because you canāt backup a running sys-usb. Hopefully I saved another copy of that one someplace. I will have it in a monolithic archive somewhere.
Its all a work in progress but if it can help others I will be happy to share what I have. If nothing else it will show some techniques for doing the various steps if someone wanted to write a better one.
Do you have a clean way to get the last modified time of VMs? (Only way I can think of is to look at the latest of the last modified times of the various logs associated with a VM).
I donāt know how to grab them programmatically (which is what I think youāre asking), but theyāre clearly visible in Qube Manager in the āLast Backupā column.
Unfortunately, it doesnāt look like theyāre available from qvm-ls. That might be a good feature request.
The last backup timestamp can be found as:
function last_backup()
{
vmbt=$1
last_back=qvm-prefs --get $vmbt backup_timestamp
if [ $? -ne 0 ] ; then
echo 0
else
echo $last_back
fi
}
I tried to upload my code here but it wonāt allow that for some reason. Cut and paste of the whole thing would be too messy to reconstruct from. Iām looking for another way right now.
Here is a temporary link for someone to grab. Iāll leave it here for a couple of months but not forever.
If you are unable to download this zip file please let me know and we will find another way.
There are three scripts:
doBackup which does the backup from dom0.
doBackupUSB uses sys-usb to perform the backup
trimBackups will iterate across the directory tree and prune out older backups trying to make enough room for the current projected space.
This is still a work in progress and ideally I wanted to rewrite it in python3 but never had the time, and it was working well enough for me.
The space calculations are currently bogus, so if someone wants to fix that part that would be great.
The sys-usb version is what I was using, and it has additional command arguments which will (-s) shutdown running VMās and (-h) halt the system when done. Running it with no arguments will skip running VMās and backup any non-running VMās which are due for a backup. I donāt remember if the dom0 version does the same since its been a while since I did it that way.
I just purchased a new machine (Dell XPS 8930, 8-core, 64-Gigs ram) and loaded R4.0.4 and it didnāt create a sys-usb so for the time being Iāll be dusting that one off again. Hopefully not for long though. So, Iām in the process of rebuilding my life after loosing my last workstation, but Iāll try to answer any questions if I can.
Hope this helps. It saved me, since it made backing things up so easy. All I had to do at the end of the day is click an icon on my desktop and go to bed.
To clarify, I was wondering if it was possible to get the last modification time of a VM (the mtime of its private storage LVM volume), not the time of last backup.
(Useful for skipping backup of VMs that havenāt changed since the last backup. For this purpose, time-of-last-shutdown of each VM would also work.)