[qubes-users] Recover data from 'private-cow.img'

Stickstoff · April 17, 2021, 9:03pm

Hello everyone,

I lost a somewhat important file from a software crash in an appvm.
Within the VM, I couldn't find a way to recover it. I copied the appvm
filesystem containers ('private.img', 'private-cow.img',
'private-cow.img.old').
As I understand it, 'private-cow.img' contains the old version of files
that were changed since last appvm startup.
However, '-cow.img' files contain no filesystem, but "binary patch"
data, thus can't be mounted or read directly or without their
corresponding'.img' files.
I found only little info online. Qubes documentation [1] says about
reverting template 'root-cow.img' changes:

    1. Ensure that no other VMs uses this template.
    2. Prepare snapshot device with root-cow.img.old instead of root-cow.img (/etc/xen/scripts/block-snapshot prepare).
    3. Replace snapshot device-mapper target with snapshot-merge, other parameters (chunk size etc) remains untouched. Now kernel starts merging changes stored in root-cow.img.old into root.img. d-m device can be used normally (if needed).
    4. Waits for merge completed: dmsetup status shows used snapshot blocks – it should be equal to metadata size when completed.
    5. Replace snapshot-merge d-m target back to snapshot.
    6. Cleanup snapshot device (if nobody uses it at the moment).
    7. Move root-cow.img.old to root-cow.img (overriding existing file).

I tried to apply these steps to 'private.img' and 'private-cow.img', but
already failed with the second step, where I couldn't really figure out
how to use the '/etc/xen/scripts/block-snapshot' script.

Is there any hope to recover an older version of a changed/deleted file
from 'private-cow.img'?
Can anyone give me a pointer how to access its content, aka "apply its
content as a patch to 'private.ig'"?

Thank you in advance,

Stickstoff

[1] Template implementation | Qubes OS

tripleh · April 18, 2021, 9:23am

Hello everyone,

I lost a somewhat important file from a software crash in an appvm.
Within the VM, I couldn't find a way to recover it. I copied the appvm
filesystem containers ('private.img', 'private-cow.img',
'private-cow.img.old').
As I understand it, 'private-cow.img' contains the old version of files
that were changed since last appvm startup.
However, '-cow.img' files contain no filesystem, but "binary patch"
data, thus can't be mounted or read directly or without their
corresponding'.img' files.
I found only little info online. Qubes documentation [1] says about
reverting template 'root-cow.img' changes:

     1. Ensure that no other VMs uses this template.
     2. Prepare snapshot device with root-cow.img.old instead of root-cow.img (/etc/xen/scripts/block-snapshot prepare).
     3. Replace snapshot device-mapper target with snapshot-merge, other parameters (chunk size etc) remains untouched. Now kernel starts merging changes stored in root-cow.img.old into root.img. d-m device can be used normally (if needed).
     4. Waits for merge completed: dmsetup status shows used snapshot blocks – it should be equal to metadata size when completed.
     5. Replace snapshot-merge d-m target back to snapshot.
     6. Cleanup snapshot device (if nobody uses it at the moment).
     7. Move root-cow.img.old to root-cow.img (overriding existing file).

I tried to apply these steps to 'private.img' and 'private-cow.img', but
already failed with the second step, where I couldn't really figure out
how to use the '/etc/xen/scripts/block-snapshot' script.

Is there any hope to recover an older version of a changed/deleted file
from 'private-cow.img'?
Can anyone give me a pointer how to access its content, aka "apply its
content as a patch to 'private.ig'"?

Use `qvm-volume revert` to go back to previous snapshots for a VM.

For the file pool you cited above there's just one previous revision available though. It should indeed be in the files you mentioned, but the other doc is somewhat unrelated.

If you start the VM, the old snaphot will be deleted, i.e. better create a backup before applying your changes.

haaber · April 18, 2021, 9:25am

I lost a somewhat important file from a software crash in an appvm.

However, '-cow.img' files contain no filesystem, but "binary patch"
data, thus can't be mounted or read directly or without their
corresponding'.img' files.

These are real disc-image files! There is a filesystem, but it is not in
sector 1 The trick is to mount it with an offset (see mount command).
To get the right offset, fdisk the file (it should have an old-style
MBR). If fdisk does not accept files (I forgot), try either cfdisk or
simple loop it in a device:

   losetup /dev/loop42 imagefile.img
   fdisk -p /dev/loop42
   losetup -d /dev/loop42

This gives the starting sector of the partition, that is than handeld
over to mount as offset. And then you can grab data. Good luck!

Stickstoff · April 18, 2021, 3:14pm

Thank you David and haaber for your help, it is much appreciated!

Use `qvm-volume revert` to go back to previous snapshots for a VM.

Oh, I saw that page [1] at some point, but didn't try it before. Thanks
for the reminder.

[dom0] qvm-volume info vmname:private

Available revisions (for revert): old

looking good!

however:
[dom0] qvm-volume revert vmname:private old

Got empty response from qubesd. See journalctl in dom0 for details.

Journal says:

unhandled exception while calling src=b' dom0' meth=b' \
admin.vm.volume.Revert' dest=b'vmname' arg=b' private' \
len(untrusted_payload)=3

[..]

NotImplementedError: Volume Filevolume has revert() not implemented

Both the revert command as well as the entry in journal are the same no
matter if I include or omit 'old' in the revert command.

These are real disc-image files! There is a filesystem, but it is not
in sector 1

Interesting and promising!
After way too long I figured out that the regular 'private.img' seems to
contain its filesystem beginning right at sector 1,
'private-cow.img.old' apparently as well, and 'private-cow.img' at
sector 560.
Testdisk could find them and correctly determine they are ext4
filesystems. But no matter if I loop-mount with offset, or deep-scan
with testdisk, or fsck, I can't find any data on the 'cow' images. The
regular 'private.img' is no problem here. The most I could get is

/dev/xvdbc contains a DM_snapshot_cow file system

from fsck when letting it work on a rw copy of 'private-cow.img'.

Honestly, I don't understand why the last step fails so hard, after all
the files aren't corrupted, filesystems are detected, and the data
should just pop out by now?

I would be very grateful for hints

Thank you,

Stickstoff

[1] Volume backup and revert | Qubes OS

rustybird · April 19, 2021, 5:03pm

Stickstoff:

[dom0] qvm-volume revert vmname:private old
> Got empty response from qubesd. See journalctl in dom0 for details.

Journal says:
> unhandled exception while calling src=b' dom0' meth=b' \
> admin.vm.volume.Revert' dest=b'vmname' arg=b' private' \
> len(untrusted_payload)=3
[..]
> NotImplementedError: Volume Filevolume has revert() not implemented

The legacy 'file' storage driver just doesn't implement the required
functionality for 'qvm-volume revert' - one of the many reasons it
will be deprecated:

github.com/QubesOS/qubes-issues

Deprecate 'file' storage driver in R4.1, remove in R5.0

opened 02:15AM - 12 Feb 21 UTC

marmarek

C: core T: task release notes P: default C: storage

**Qubes OS version (if applicable)** R4.1 **Affected component(s) or fun…ctionality (if applicable)** core **Brief summary** 1. Add some verbose notification about using deprecated storage driver in R4.1. I think this can be a notification at the system startup (can be added to the code of one of the widgets, for example qui-updates one). Also, add a warning when creating new "file" pool. Maybe also when listing storage pools? 2. Do not create "file" pool automatically during system installation. But do create file-reflink for `/var/lib/qubes` directory, for compatible filesystems. 3. Document migration off the file driver (either to file-reflink or to lvm), and link to it from release notes and the notification. In-place migration is preferred if possible, otherwise document reinstall method. **Additional context** The file storage driver is very inefficient and also some features are impossible to achieve using its old architecture (it simulates CoW volumes using dm-snapshot, which limits snapshots levels to 1). Example issues caused by its inferior features: - #4324 - #6329 - #3255

> These are real disc-image files! There is a filesystem, but it is not
> in sector 1

After way too long I figured out that the regular 'private.img' seems to
contain its filesystem beginning right at sector 1,
'private-cow.img.old' apparently as well, and 'private-cow.img' at
sector 560.

private.img is a full disk image, but private-cow.img(.old) is more
like a patch as you said, and isn't mountable.

You could use this script (after backing up private.img and
private-cow.img.old):

github.com

rustybird/qubes-stuff/blob/master/dom0/bin/qvm-legacy-filevolume-revert

#!/usr/bin/sudo bash
# shellcheck shell=bash
# vim: filetype=bash

set -e -u -o pipefail -o errtrace
trap 'exit $?' ERR

# parse arguments
[[ $# == 2 ]] || { echo "Usage: ${0##*/} vm volume" >&2; exit 1; }
set -x
vm=$1
volume=$2

# sanity checks
if qvm-check --running -- "$vm"; then exit 1; fi
qvm-volume info -- "$vm:$volume" pool |
xargs -d '\n' qvm-pool --info |
grep -xE 'driver +file'

# find image files

This file has been truncated. show original

$ qvm-legacy-filevolume-revert vmname private

Rusty

haaber · April 20, 2021, 9:04am

The legacy 'file' storage driver just doesn't implement the required
functionality for 'qvm-volume revert' - one of the many reasons it
will be deprecated:

Deprecate 'file' storage driver in R4.1, remove in R5.0 · Issue #6399 · QubesOS/qubes-issues · GitHub

Awesome! Thank yu for that hint. When/how will it be changed ?? That
seems quite troublesome to change the internal storage type within a
running system ... will need re-install in some further time??