Questions about Wyng

[behemothwerecat]

I’d like to use Wyng, I’m basically a noob, so I’d like to run some questions by @tasket about how to do so. The reason that I’m switching to Wyng from the official tool is that the latter is too slow.

My Qubes install is standard, which I understand to mean that I’m using thin LVM. From reading the README for wyng-util-qubes and wyng-backup, this is what I understand I need to do (please correct if it could be improved):

Set up:

• Verify with git and then copy both folders to dom0
• Plug in my external SSD, and in (disposable) sys-usb, decrypt the LUKS volume.
• In dom0, create a config file at /etc/wyng/wyng.ini

[var-global-default]
dedup = 1
authmin = 10
dest = qubes://sys-usb/media/user/VOLUME-NAME/backups/laptop.backup

• Create the fresh archive:
  sudo wyng arch-init

Prune, backup, then verify

Every time I want to do a backup:

• Prune. I usually keep my last four backups, and make a backup approximately each week. The closest I have come to this goal is for no backups older than 60 days to be kept, and for any backups older than a week, there’s only 1 per week (though I don’t understand how autoprune could fit into this):
  sudo wyng-util-qubes prune --apdays=60:7:7:1
• Backup of every qube configued to be backed up by default:
  sudo wyng-util-qubes backup --includes
• Verify the session you just made:
  sudo wyng-util-qubes verify --session=YYYYMMDD-HHMMSS

Remaining questions:

• What is the practical difference between verify and arch-check for this use case? Is it sufficient to just verify the session you just made as described above, or is there something else that should be done occassionally or regularly?
• To restore the qube “personal” without overwriting the current one, is the best way to do this to simply rename the current “personal” qube prior to restore? I like to do occassional test restores for a sanity check, but don’t actually want to overwrite the current qube.
• I don’t understand the point about lvextend, is it applicable to my use case? If so, what is the command I should run?
• Is there any concievable way that running either of the two tools from dom0 could break the Qubes OS installation? (If the dest is always in an App qube and not local to dom0, as described above, so it’s not adding space to the dom0 filesystem)

I plan to make a community guide called “Wyng for noobs” once these questions are answered, for this simple use case.

[kenosen]

@behemothwerecat

I don’t understand the point about lvextend, is it applicable to my use case? If so, what is the command I should run?

If you’re using the standard install LVM partitioning, then these responses/threads may be helpful in answering that question:

Unable start any qube after forcefully canceling wyng backup

Yes, I would say 79% usage for the metadata volume is very concerning/risky. The basic issue here is that metadata use can fluctuate much quicker than data use when complex operations are causing fragmentation, even if only momentarily. Therefore, I stopped following my data use % as an indicator and got better results by keeping metadata use well below 50%. You can do this with lvextend --poolmetadatasize <size> qubes_dom0/vm-pool. In your case I would at least double the current size. You…

ANN: Wyng incremental backup, new version!

Yes, restore would also need that option… if you need automated/scripted operation. If you prefer, an env variable can be set with the passphrase and then echo the var (instead of a tmp file). Only restore has this restriction (it will even stop the VMs you are restoring). With backup, it uses the Qubes snapshot the system took when you last started/stopped that VM. So if the VMs is running during backup it won’t see changes you made since you started the VM, but otherwise there is no confli…

@behemothwerecat

To restore the qube “personal” without overwriting the current one, is the best way to do this to simply rename the current “personal” qube prior to restore?

Exactly.

@behemothwerecat

Is there any concievable way that running either of the two tools from dom0 could break the Qubes OS installation? (If the dest is always in an App qube and not local to dom0, as described above, so it’s not adding space to the dom0 filesystem)

It can cause issues only insofar as the metadata and snapshot space is concerned (hence the recommendation for lvextend), but nothing irreversible. So the simple answer to this question is “no.”

@behemothwerecat

I plan to make a community guide called “Wyng for noobs” once these questions are answered, for this simple use case.

There is one here (though note this is for BTRFS install, though that makes little difference in the use of the tool other than pool storage locations), in Part 3:

Qubes Salt Beginner's Guide

Qubes Salt Beginner’s Guide Part 1: Creating our first qubes As a beginner, Salt seemed daunting to me at first. It took some effort to learn but it was worth it! I’m writing this guide for beginners who enjoy an hands-on introduction with examples. 1.1 Creating personal state configuration directories Our journey starts with a file found in the base Salt configuration directory in dom0: /srv/salt/qubes/README.rst (GitHub link). In this file we can read: qubes.user-dirs Install and maintain …

[behemothwerecat]

Thanks for the insights. I need some more clarification about lvextend. My current understanding is that the command that I ultimately need to run in dom0 is as follows, which ends with the argument <volgroup/logicalvolume>:
lvextend --poolmetadatasize <size> qubes_dom0/vm-pool

To fill in the <size> argument, I first need to know the current poolmetadatasize of the logical volume vm-pool so that I can calculate 3X it, but I’m not sure how to do that. As far as I got was this command:
sudo lvdisplay qubes_dom0/vm-pool

However, in the output I see Allocated metadata: 38.35%, but not the total poolmetadatasize. How do I display the current poolmetadata size so that I can multiply it by 3?

Additionally, how do I determine if I must first free up space in the volgroup before running lvextend? sudo vgs outputs VFree, so should I just make sure that the difference in poolmetadatasize (new value - old value) is smaller than VFree?

[kenosen]

How about
lvs -o+metadata_size,metadata_used qubes_dom0/vm-pool

[HPOA909]

Please write this guide properly before putting this into Community Guides category.

[behemothwerecat]

Thanks for that, the field wasn’t mentioned in the man pages for lvs.

sudo lvs -o+metadata_size qubes_dom0/vm-pool

outputs that Msize is 100.00m. Which means that the command I want to run is:

lvextend --poolmetadatasize 300.00m qubes_dom0/vm-pool

@tasket

You might need to free up space in the volgroup before you can lvextend; in that case what I’ve done is to temporarily turn off dom0 swap with swapoff then use lvresize to shrink the swap lv a bit, and finally use mkswap to reformat the swap lv and then swapon -a.

Finally, how do I determine if I must free up space in the volgroup before I can lvextend? I’m extending by 200 megabytes. sudo vgs outputs several gigabytes for VFree. Would I only need to free up space if VFree was below 200m?

[behemothwerecat]

I changed the title to better reflect that this post is not intended to be a community guide.

[tasket]

Every time I want to do a backup:

Try this:

sudo wyng-util-qubes backup --includes --autoprune=full -w apdays=60:7:7:1
sudo wyng arch-check --session=newest

The apdays option has no effect with prune command unless you also add the autoprune option. Those two options may also be used with wyng send (or wyng-util-qubes backup) instead of issuing a separate prune command.

You can also add either apdays or autoprune option, or both, to wyng.ini.

The verify and arch-check commands are quite different: Verify is like verifying a trail along a map (the ‘trail’ must be followed to assemble a complete snapshot), while arch-check is like verifying points on the map (all of them by default). That is not to say that verify is always quicker; some options can make arch-check quicker or more convenient. For example, wyng arch-check --session=newest will check only the most recent session data. FWIW, a little part of arch-check always runs every time you access an archive; the metadata at least gets some sanity checks each time. Whether you want to extend that with an explicit arch-check after each send, or weekly, monthly, etc is up to you. Personally, I only use it rarely.

Verify can be used from wyng-util-qubes while arch-check cannot; the latter must be accessed via Wyng directly.

To restore the qube “personal” without overwriting the current one, is the best way to do this to simply rename the current “personal” qube prior to restore?

Yes, either rename or clone the qube first. Cloning can yield a better result since Wyng’s sparse-write mode will then only change the regions necessary to make it match the archived version; this saves disk space. (The util uses sparse-write mode by default.)

I don’t understand the point about lvextend, is it applicable to my use case? If so, what is the command I should run?

The advice in the other responses will give you a good idea, however the amount varies from system to system its hard to tell about someone else’s system partly because either the OS installer made a guess for the appropriate chunk size which has a very big effect on the overall metadata size. If lvextend could handle percentages then it would be simple. Its best to look at what you got and double or triple that. If lvextend doesn’t let you extend due to lack of space, you can free up some space in the vg by temporarily disabling dom0 swap, shrink the swap lv, reformat the swap lv (via its LUKS mapping), then turn swap back on. The Qubes installer typically sets up a lot more dom0 swap than is needed so taking 300-1000mb is usually fine. (I think this is a good example of why Btrfs setups are better: simple resizing is always possible and the “danger” from metadata overuse is slower performance.)

As to the ‘why’ for all of this, one reason is that Qubes kept expanding its practice of “versions to keep”; IMO it keeps excessive amounts of old snapshots by default. Wyng must also keep one snapshot for each volume, and must also temporarily make a second snapshot during send or monitor. So the high point of metadata stress will be during those Wyng commands (and wyng-util-qubes backup).

Is there any concievable way that running either of the two tools from dom0 could break the Qubes OS installation?

Yes, I think this is true for any backup/restore system. Using wyng receive directly can result in it writing over files; even if you only use it for LVM if --local is specified incorrectly it might think you are trying to restore disk image files instead (also --saveto is file oriented so it can be similarly misused). With wyng-util-qubes restore this risk basically disappears because it asks Qubes where to save the data.

There is also the risk that metadata stress from using snapshots (see prior explanation) could break your Thin LVM pool. This can and has happened simply from using Qubes without Wyng, but Wyng adds its own snapshots. IMO, the best way to avoid this is to use Btrfs and then the only metadata problem you are likely to encounter is when to run online defrag to prevent the system from slowing down; Second-best way is to fuss with extending the Thin LVM metadata. BTW, its possible on a Qubes LVM installation to migrate all VMs to using Btrfs by default; you just have to know how to add a Btrfs volume to the dom0 system (possibly re-allocating disk space) and do the qvm-pool setup and set the default pool.

[behemothwerecat]

@behemothwerecat

lvextend --poolmetadatasize 300.00m qubes_dom0/vm-pool

Upon running this, it works but also outputs:
WARNING: Sum of all thin volume sizes (<1.49 TiB) exceeds the size of thin pools and the size of whole volume group (487.92 GiB).

Can I disregard this or is it conveying something I need to act on? @kenosen

[apparatus]

This warning is normal, you can disregard it.

[behemothwerecat]

Thank you for the extensive answer! Two clarifying questions remain:

1. What is the difference between sudo wyng arch-check --session=newest and sudo wyng-util-qubes verify --session=newest ? If there is a difference, which is better suited to doing regularly?
2. Should sudo wyng arch-check​ be run without args occassionally, and if so, how often would you recommend?

[tasket]

1. It comes down to the nature of using an incremental backup system: You have a choice of verifying original objects (volumes) in their entirety, or the updates those volumes. From a speed/economy standpoint, the latter (arch-check) makes more sense if you’re going to check frequently.
2. I probably wouldn’t run arch-check without args (reading the whole archive) more than 2-4 times a year. How often you arch-check or verify simply depends on your level of confidence in Wyng and your storage media. I personally wouldn’t consider arch-check a part of the main instructions in a tutorial, instead preferring to mention it in a note at the end or adding a comment to it like # optional.