Incremental backup possibilities

[distopia]

Thanks a lot for spawning this fast in this thread explaining about your motives !

@tasket

The problem with existing tools is they assume a local storage system with no CoW analysis features so they re-read all of the data in each file to find which chunks of the file have changed

To great surprise, it’s my finding too, for both tools discussed previously …

Restic and Borg doesn't seem to leverage CoW snapshots metadata

• Restic : Someone fiddling around with the restic --use-fs-snapshot for ZFS but not working and seemingly no other fs than VSS.

• Borg : Found an issue from the Borg lead thinking about introducing this snapshot diff concept in it a couple of years back but just unassigned himself from the - still open - issue later on. More interestingly he referenced Wyng and Qubes main back-up GH issue in it.

.

I finally went through the incremental backup GH thread where you and many other people (including Qubes team and core members and Borg lead) were debating on since 2015.

Also read many other sparse and rapidly-closing threads on the forum. They are so many of them, with some people having corrupted backups and so on. Even with someone who created guides for cron reminders to perform full back-ups

The most interesting thread I have come across IMO is :

How do you organize your backups?

Before I discovered “Qubes OS” (it’s smart now I’m addicted), I was using Debian and my backups were done with rsync on two external media and with “duplicati” to a remote server. The advantage is that my backups were incremental and I didn’t have to do a full backup every time. If I understand correctly, I have to make a full backup (of each vm) each time, which can quickly become time consuming if my vm occupy a lot of space. I would like to have your point of view on how you organize your b…

.

At the risk of saying something obvious : It is important (not to say crucial) for many users to have an easy to use disaster-recovery mecanism. Which Qubes doesn’t really provide as of now …

.

Came across interesting tools on the way :

• Borg off-site backup scripts written by @Rudd-O - on which not many people talked about.
• Qube sync + zpaqfranz and/or syncthing over qrexec both from @unman

How Wyng could compare to these approach, were most maintenance work would be held by upstream ?

Really don’t want to be rude here, just thinking one-man software maintenance is maybe a blocker for which Qubes team is not ready to rely on Wyng.

.

I didn’t find this statement (increased security than then existing tool) in the GH thread and, if this is true, could greatly help get adoption :

@tasket

From a security perspective, Wyng also has advantages. Unlike qvm-backup it doesn’t parse any un-verified tar archives or other metadata in dom0 (qvm-backup creates archives with two tar layers, the outer layer is un-verified when its processed)– in an OS security model where even reading partition tables in dom0 is considered risky, this is significant.

Are you sure there isn’t a signing mecanism that prevent unknown package parsing in existing Qubes backup tool ? Or a safe parsing function (like a safe_ftprintf, if you catch the ref ) ?

Found this performance and security flaw report here (probably related to what you explained)

.

As far as I am concerned, my back-up habits have been screwed since I have been using Qubes as a daily driver. My almost Tb of data takes ages to back-up and with usb performance bottleneck, ages to be sent to an external drive ; so much that my last full system back-up has already 6 months …

Hopefully having other off-site strategies for critical data.

.

In the hope this particular thread will bring back discussion - may it be divergence of opinions - around incremental backup topic

since GH threads became more silent after moderation

andrewdavidwong on Oct 20, 2022

Just a friendly reminder that this issue tracker (qubes-issues) is not intended to serve as a discussion venue. Instead, we’ve created a designated forum for discussion and support. (By contrast, the issue tracker is more of a technical tool intended to support our developers in their work.) Thank you for your understanding!

.

And please anyone feels free to also talk about your usage and other tools and strategies related to incremental back-up (Bacula, Duplicati and others). Or to link interesting and related discussion.