ANN: Wyng incremental backup, new version!

General Discussion
1

Introduction

Wyng is able to deliver faster, more efficient incremental backups for logical volumes. It accesses logical volume metadata (instead of re-scanning data over and over) to instantly find which data has changed since the last backup. Wyng also has fast trimming of older backups to recover space, meaning you only ever have to do a full backup once and can send incremental backups to the same archive indefinitely and frequently–many times a day, or many times an hour.

Who Is It For?

Anyone with an intermediate level of skill in the Linux shell and a desire to keep your system backed up; If this is you, then you can use Wyng comfortably to backup snapshots of your VMs including dom0 in a minimal amount of time. One caveat is that Qubes integration does not yet save/restore VM settings, just the VM data.

A Brief Example on Qubes

Assuming you have a LUKS encrypted volume attached to sys-usb, mount it in dom0:

$ qvm-block attach dom0 sys-usb:sda0
$ sudo cryptsetup luksOpen /dev/xvdi backups
$ sudo mount /dev/mapper/backups /mnt/backups

Create an archive and add volumes to the configuration, in this case the data volumes for the VMs ‘personal’ and ‘work’:

$ sudo wyng arch-init --local qubes_dom0/pool00 --dest internal:/mnt/backups
$ sudo wyng add vm-personal-private vm-work-private

…and perform a backup!

$ sudo wyng send

What’s New

Beta release for new v0.3 branch.

A number of helpful features such as snapshot tagging, exclusions, sparse mode, Zstandard compression and LVM optimizations were recently added to Wyng. There is also a new Qubes integration tool under development. They do not affect the core backup code, but are numerous enough to make a version number increase appropriate. Currently Wyng sits at v0.3beta2 and expect only a single release candidate version before full release.

The main branch also has v0.2 release candidate 10. I recommend trying the beta anyway, especially if you’re already on Qubes 4.1 as you will be able to use faster modes such as Zstandard.

Future

Integrated encryption is first and foremost on the horizon for v0.4; this would enable creating encrypted backups without relying on an intermediary such as LUKS or fscrypt. The Wyng issue tracker also lists some other possibilities like volume cloning, differential modes and cloud transfer protocols.

If you have any questions about Wyng feel free to post here in the thread!

6 Likes
2

Thanks, will certainly give it a try while I’m in the learning phase!

Is it feasible to clone VMs across Qubes installs using Wyng too? Or is that blocked by Qube itself?

3

Wyng will recognize where different volumes have the same data and save that space in deduplication mode. In that sense it supports VM cloning, where if you clone a VM that has already been backed up then add the clone to the backup, Wyng will only need to send the differences (if any) between the two volumes to the archive.

Also, if you do things like delete a backed-up VM in Qubes and replace it with a related VM (such as by renaming the VM to the old name), Wyng will still only record the differences on the next backup.

Of course, when you ask it to restore (receive) it will compose the entire volume exactly as it was at the point in time when it was backed up.

1 Like
4

BTW,

Version 0.3.0 Release candidate 1 is now available! :rocket:

2 Likes
5

@tasket Do you think you will have some time soon to work on wyng? :smiley:

6

I think I do.

Version 0.3 is released, and 0.4 is almost ready for its next alpha iteration.

2 Likes
7

Interesting project. Will definitely give it a try.

How does it compare to timeshift?
Is the focus on system or user files?

8

I can only go by what the Timeshift readme says for now, but here are my impressions:

  • It has an integrated GUI which looks rather nice.

  • It functions like an ‘rsnapshot’ that is focused on “only system files and settings”. It concerns me this is being attempted as there is no officially targeted OS and the subject of restoring working configs is much more complex and error-prone (and distro-specific) than restoring data volumes.

    There are a number of technical caveats that imply “cannot” relating to various distros’ idiosyncrasies. Seems you can only be confident of it working on Ubuntu installs that did not have custom install options.

    TBH, the cross-distro restore feature/idea its readme is pushing looks like administrative hot-dogging to me. :slight_smile: With all the caveats about using timeshift on non-Ubuntu distros, why even bother.

  • Uses either rsync or btrfs-send underneath. This suggests its format and dynamics are a poor fit. For example: If you choose btrfs mode then it almost certainly means you must have a functional btrfs volume on the destination; that’s a big constraint if the destination is remote.

    OTOH, Wyng’s requirement for the destination is basic POSIX filesystem functions which almost every common storage service supports.

  • For the above reason, there is probably no concept of domain isolation in timeshift, needing to handle the destination/backup data directly. Wyng avoids this issue rather effectively.

  • Seems to be no way to specify a VM or network destination for the backup data? In Wyng, you can specify local fs, ssh, a qube, or ssh via a qube.

  • LVM handling: Based on a quick look through the code, it is concerned with LVM only for the purpose of restoring configurations. That suggests during backup operations its mounting the LVM volumes locally and using rsync on the contents. Wyng uses LVM-specific tools to compare LV snapshots at the block level when backing up; its faster and also more secure.

Summary: Timeshift looks like a conventional backup-snapshots-to-local-disk utility that tries to preserve system configuration data in a comprehensive way. Its backup file format appears to be an entire btrfs volume, which it must access directly. Readme says its ill-suited to Fedora, probably doubly ill-suited to Qubes (which is based on Fedora).

3 Likes
9

Thank you for taking the time to write such a thorough explanation, I appreciate it very much.

10

Update: The next release of Wyng v0.4 has moved into final alpha. A number of big changes have been made including Btrfs & XFS support and integrated encryption – which makes the LUKS steps in the OP unnecessary!

Overall people should find it much easier to use as well, as you can specify different archive URLs at will without worrying about Wyng metadata locations, and most Wyng commands now let you specify more than one volume at once.

6 Likes
11

Update: The new version of Wyng now has a beta release. Look for a separate announcement soon…

6 Likes
12

Very cool. This got me thinking about backup security in general: IIUC the regular QubesOS backup tool also uses qvm-block to attach a block device to dom0, right? Are there any caveats regarding untrusted backup storage devices, such as an external hard drive that is also used to connect to untrusted systems?

Hoping for Wyng-backup integration into Qubes 4.2!

13

I don’t think it works that way. You attach it to a certain qube and the dom0 sends the data to that disk in that qube.

1 Like
14

Well I was going by @marmarek 's comment from 2013 here, though I realize that this was early days and just a discussion about how things should work. So is it implemented via QubesRPC then, with the destination qube taking the stream of data and handing it off to the block device mounted there?

Do you know of any security issues resulting from attaching an untrusted partition via qvm-block to a qube (or dom0)? I only know that it’s safer than attaching the full disk and much safer than attaching the full device, but I have no clue about the exact technical details of what goes on during qvm-block and mount.

I ask because currently Wyng requires doing this in dom0, so I’m wondering whether I’d have to get a dedicated external hard drive that I only use for backups, so that it stays more trusted, rather than using existing external storage that I also use for other purposes with other systems.

15

qvm-block is used by dom0 to attach block devices to other qubes, not itself: How to use block storage devices | Qubes OS

When you use Qubes backup, you have to mount the block device somewhere, either locally to dom0 or a qube and then point backup/restore to that mount point from the tool How to back up, restore, and migrate | Qubes OS

wyng-backup does a similar thing, asking you explicitly where the archive will be stored to/restored from, and expects that directory to be already containing the mounted filesystem, abstracting it. It would be either a local dest from dom0 or a qube url as described here, reflecting docs at time of writing those lines https://github.com/tasket/wyng-backup/blob/eb0776854e37242d9a4fc590afa4bba6af4e6534/README.md?plain=1#L376

3 Likes
16

@tasket thanks for this write-up. And what are the restore commands if one wants to restore or recreate vm-personal-private ?

I understand this would grab the old VM, but does it get called into the pool and require anything further from the user? sudo wyng-util-qubes restore vm-personal-private --dest=[wherever the backup was saved]

Does the user need to make a new AppVM or will one be created by virtue or restoring the backup image?

17

@tasket
Can wyng-backup be used to back up a running qube?
How would I find the previous two versions of a specific file and restore
them to a target qube?

I never presume to speak for the Qubes team. When I comment in the Forum I speak for myself.
18

@kenosen Using wyng-util-qubes removes the need to refer to volume names, so referring to the qube name is the thing to do…

sudo wyng-util-qubes restore personal --dest=<archive location>

(If, for some reason, you wish to restore a particular volume without any other part of the qube, then I’d suggest using wyng receive directly.)

The util has been evolving to require minimal input from the user, so that should be all you need. A new qube/VM will be created if one doesn’t already exist. You could specify the storage --pool for restore, but that would only affect where a restored qube was created if it didn’t already exist in the system. Otherwise, it will pick the pool that the qube already uses (or create a qube in Qubes’ default pool).

I should note that wyng-util-qubes v0.8 won’t work with more than one storage pool at a time, although this is probably not an issue for most people using the Qubes default. The v0.9 branch does already support multiple pools in addition to supporting Btrfs.

1 Like
19

@tasket This is great, thank you. You noted in a separate thread how to reference a password cmd; does the restoration process also require -w 'passcmd=command' to unencrypt the restored backups?

I think the only downside is, if correct, I can’t be working in a qube while backing up. Though it is certainly faster than the qvm-backup tool. My last experience with wyng led to the pool becoming bloated with .ticks and .tocks, preventing any qube from launching.

20

Yes, restore would also need that option… if you need automated/scripted operation. If you prefer, an env variable can be set with the passphrase and then echo the var (instead of a tmp file).

Only restore has this restriction (it will even stop the VMs you are restoring). With backup, it uses the Qubes snapshot the system took when you last started/stopped that VM. So if the VMs is running during backup it won’t see changes you made since you started the VM, but otherwise there is no conflict in backing up a running VM.

Thin LVM (or ‘lvmthin’) tends to grossly underestimate the amount of metadata it will need to track changes as lv snapshots are saved and written to. Although newer Qubes installers have tried to address this by extending the ‘tmeta’ by a certain percentage, I’ve found that about 3X the original metadata size (which can be increased with lvextend --poolmetadatasize) is required to handle both Qubes internal and backup-related snapshots without incident. BTW, the ‘tock’ snapshots are temporary and generally should not hang around (they do only when there is a crash, and Wyng removes them when possible); the ‘tick’ snapshots are the ones that persist, one per volume. I think a factor in all this is that Qubes now keeps more of it own snapshots than it did in the past.

My general recommendation on Qubes is to use Btrfs instead of (or as the default pool in addition to) LVM. In general there is far less hassle and risk.

Also, see my notes that I’ll be posting to the Wyng issue 184.