[qubes-users] Recover data from 'private-cow.img'

Hello everyone,

I lost a somewhat important file from a software crash in an appvm.
Within the VM, I couldn't find a way to recover it. I copied the appvm
filesystem containers ('private.img', 'private-cow.img',
'private-cow.img.old').
As I understand it, 'private-cow.img' contains the old version of files
that were changed since last appvm startup.
However, '-cow.img' files contain no filesystem, but "binary patch"
data, thus can't be mounted or read directly or without their
corresponding'.img' files.
I found only little info online. Qubes documentation [1] says about
reverting template 'root-cow.img' changes:

    1. Ensure that no other VMs uses this template.
    2. Prepare snapshot device with root-cow.img.old instead of root-cow.img (/etc/xen/scripts/block-snapshot prepare).
    3. Replace snapshot device-mapper target with snapshot-merge, other parameters (chunk size etc) remains untouched. Now kernel starts merging changes stored in root-cow.img.old into root.img. d-m device can be used normally (if needed).
    4. Waits for merge completed: dmsetup status shows used snapshot blocks – it should be equal to metadata size when completed.
    5. Replace snapshot-merge d-m target back to snapshot.
    6. Cleanup snapshot device (if nobody uses it at the moment).
    7. Move root-cow.img.old to root-cow.img (overriding existing file).

I tried to apply these steps to 'private.img' and 'private-cow.img', but
already failed with the second step, where I couldn't really figure out
how to use the '/etc/xen/scripts/block-snapshot' script.

Is there any hope to recover an older version of a changed/deleted file
from 'private-cow.img'?
Can anyone give me a pointer how to access its content, aka "apply its
content as a patch to 'private.ig'"?

Thank you in advance,

Stickstoff

[1] Template Implementation | Qubes OS

Hello everyone,

I lost a somewhat important file from a software crash in an appvm.
Within the VM, I couldn't find a way to recover it. I copied the appvm
filesystem containers ('private.img', 'private-cow.img',
'private-cow.img.old').
As I understand it, 'private-cow.img' contains the old version of files
that were changed since last appvm startup.
However, '-cow.img' files contain no filesystem, but "binary patch"
data, thus can't be mounted or read directly or without their
corresponding'.img' files.
I found only little info online. Qubes documentation [1] says about
reverting template 'root-cow.img' changes:

     1. Ensure that no other VMs uses this template.
     2. Prepare snapshot device with root-cow.img.old instead of root-cow.img (/etc/xen/scripts/block-snapshot prepare).
     3. Replace snapshot device-mapper target with snapshot-merge, other parameters (chunk size etc) remains untouched. Now kernel starts merging changes stored in root-cow.img.old into root.img. d-m device can be used normally (if needed).
     4. Waits for merge completed: dmsetup status shows used snapshot blocks – it should be equal to metadata size when completed.
     5. Replace snapshot-merge d-m target back to snapshot.
     6. Cleanup snapshot device (if nobody uses it at the moment).
     7. Move root-cow.img.old to root-cow.img (overriding existing file).

I tried to apply these steps to 'private.img' and 'private-cow.img', but
already failed with the second step, where I couldn't really figure out
how to use the '/etc/xen/scripts/block-snapshot' script.

Is there any hope to recover an older version of a changed/deleted file
from 'private-cow.img'?
Can anyone give me a pointer how to access its content, aka "apply its
content as a patch to 'private.ig'"?

Use `qvm-volume revert` to go back to previous snapshots for a VM.

For the file pool you cited above there's just one previous revision available though. It should indeed be in the files you mentioned, but the other doc is somewhat unrelated.

If you start the VM, the old snaphot will be deleted, i.e. better create a backup before applying your changes.

I lost a somewhat important file from a software crash in an appvm.

However, '-cow.img' files contain no filesystem, but "binary patch"
data, thus can't be mounted or read directly or without their
corresponding'.img' files.

These are real disc-image files! There is a filesystem, but it is not in
sector 1 :slight_smile: The trick is to mount it with an offset (see mount command).
To get the right offset, fdisk the file (it should have an old-style
MBR). If fdisk does not accept files (I forgot), try either cfdisk or
simple loop it in a device:

   losetup /dev/loop42 imagefile.img
   fdisk -p /dev/loop42
   losetup -d /dev/loop42

This gives the starting sector of the partition, that is than handeld
over to mount as offset. And then you can grab data. Good luck!

Thank you David and haaber for your help, it is much appreciated!

Use `qvm-volume revert` to go back to previous snapshots for a VM.

Oh, I saw that page [1] at some point, but didn't try it before. Thanks
for the reminder.

[dom0] qvm-volume info vmname:private

Available revisions (for revert): old

looking good!

however:
[dom0] qvm-volume revert vmname:private old

Got empty response from qubesd. See journalctl in dom0 for details.

Journal says:

unhandled exception while calling src=b' dom0' meth=b' \
admin.vm.volume.Revert' dest=b'vmname' arg=b' private' \
len(untrusted_payload)=3

[..]

NotImplementedError: Volume Filevolume has revert() not implemented

Both the revert command as well as the entry in journal are the same no
matter if I include or omit 'old' in the revert command.

These are real disc-image files! There is a filesystem, but it is not
in sector 1 :slight_smile:

Interesting and promising!
After way too long I figured out that the regular 'private.img' seems to
contain its filesystem beginning right at sector 1,
'private-cow.img.old' apparently as well, and 'private-cow.img' at
sector 560.
Testdisk could find them and correctly determine they are ext4
filesystems. But no matter if I loop-mount with offset, or deep-scan
with testdisk, or fsck, I can't find any data on the 'cow' images. The
regular 'private.img' is no problem here. The most I could get is

/dev/xvdbc contains a DM_snapshot_cow file system

from fsck when letting it work on a rw copy of 'private-cow.img'.

Honestly, I don't understand why the last step fails so hard, after all
the files aren't corrupted, filesystems are detected, and the data
should just pop out by now?

I would be very grateful for hints :slight_smile:

Thank you,

Stickstoff

[1] https://www.qubes-os.org/doc/volume-backup-revert

Stickstoff:

[dom0] qvm-volume revert vmname:private old
> Got empty response from qubesd. See journalctl in dom0 for details.

Journal says:
> unhandled exception while calling src=b' dom0' meth=b' \
> admin.vm.volume.Revert' dest=b'vmname' arg=b' private' \
> len(untrusted_payload)=3
[..]
> NotImplementedError: Volume Filevolume has revert() not implemented

The legacy 'file' storage driver just doesn't implement the required
functionality for 'qvm-volume revert' - one of the many reasons it
will be deprecated:

> These are real disc-image files! There is a filesystem, but it is not
> in sector 1 :slight_smile:

After way too long I figured out that the regular 'private.img' seems to
contain its filesystem beginning right at sector 1,
'private-cow.img.old' apparently as well, and 'private-cow.img' at
sector 560.

private.img is a full disk image, but private-cow.img(.old) is more
like a patch as you said, and isn't mountable.

You could use this script (after backing up private.img and
private-cow.img.old):

$ qvm-legacy-filevolume-revert vmname private

Rusty

The legacy 'file' storage driver just doesn't implement the required
functionality for 'qvm-volume revert' - one of the many reasons it
will be deprecated:

Deprecate 'file' storage driver in R4.1, remove in R4.2 · Issue #6399 · QubesOS/qubes-issues · GitHub

Awesome! Thank yu for that hint. When/how will it be changed ?? That
seems quite troublesome to change the internal storage type within a
running system ... will need re-install in some further time??