Potential bug in backup restore resulting in empty restored VMs

I’m having some trouble with restoring a backup. I made a fairly large backup (~60GB) with the intent of carrying it over into a fresh install. The restore was successful (I initially had the same issue described here: Restoring R4.0_rc3 backups >53.0 GB fail with error · Issue #3468 · QubesOS/qubes-issues · GitHub but was able to solve it by following the advice of breaking the backup into smaller restores) and the backed-up vms were generated, but none of their data seems to have been restored; the restored templatevms aren’t taking up any disk space, the restored appvms’ home directories are empty, and the only thing that seemed to restore correctly was each vm’s available services.
I thought that something had gone wrong with the backup itself, but I attempted to back up a new appvm with a test file in the home directory and I got the same result; while the restored appvm does take up a small amount of disk space (~10MB while the original was ~100MB), the test file doesn’t appear to have been restored.
Any advice on this issue would be greatly appreciated; I’m fairly new to using Qubes, so it very well could be something obvious that I’m missing.

2 Likes

I am having the same problem and tried posting to the mailing list for some help:

I have tried to restore close to 10 different backups I have made and every single one is giving me problems. I have tried to restore to two different computers (one fresh install, and one I have had for awhile). Almost every install restores without errors, but restores empty appvms, that is they show up as being 0mb which seems accurate as when I open them none of the data that was supposed to be there is present. While the appvms are restoring as 0mb, the backup files are anywhere from 30-100gb?

When I try to verify the backup I get the following:
Finished with errors!
failed to decrypt /var/tmp/restorekyip2gjv/vm9/private.img.000.enc: b’scrypt: Input is not valid scrypt-encrypted block\n’
Partially restored files left in /var/tmp/restore_*, investigate them and/or clean them up

and when I check /tmp/ all the restore* dirs are empty.

I have now started to try to extract the contents manually which is a bit of a mind bender (the instructions on the Qubes docs page that is). As it takes awhile, and was the easiest to understand, I have tried to untar first and seem to be getting errors I dont understand. below are the untar’ing errors I have gotten for two of the backups:

[user@disp9624 appvms2salvage]$ sudo tar -i -xvf qubes-2019-12-14T162028_shuttle -C ~/Templates/
backup-header
backup-header.hmac
qubes.xml.000.enc
dom0-home/bob.000.enc
tar: Skipping to next header
tar: Archive contains ‘Q\352\334\\\275h\274\202B*\275s’ where numeric off_t value expected
tar: Archive contains ‘I\221\272\031\205\325\314۴5\373\377’ where numeric off_t value expected
tar: Archive contains ‘\037\2571\344\323\377ȧD\204\t"’ where numeric off_t value expected
tar: Archive contains ‘p\177\246\257\371\215\243Bqp[\350’ where numeric off_t value expected
tar: Archive contains ‘\210\265\225\265[\313\311i\205!TI’ where numeric off_t value expected
tar: Archive contains ‘\255\220.\217\232\254j\247\326\325\355\303’ where numeric off_t value expected
tar: Archive contains ‘\032q&\345\377k\005\342.<w/’ where numeric off_t value expected
tar: Archive contains ‘ע8"]T\310pr'\0\353’ where numeric off_t value expected
tar: Archive contains ‘Q\241\371x\216\244AȞ\350?\343’ where numeric off_t value expected
tar: Archive contains ‘a\023\2478\223\320\363S\231\275\345\320’ where numeric off_t value expected
tar: Archive contains ‘\034v\320\313j7\262v\362\033G\002’ where numeric off_t value expected
tar: Archive contains ‘\317\r\367Z\241.\256\017\036r[\342’ where numeric off_t value expected
vm3/root.img.025.enc
tar: Skipping to next header
tar: Archive contains ‘!\017\257\f\317f\356\r\273\377\271\t’ where numeric off_t value expected
tar: Archive contains ‘ؽY--f\212\317\025%\371\177’ where numeric off_t value expected
tar: Archive contains ‘\241\227w\376\235f\347\263\366\025\342M’ where numeric off_t value expected
tar: Archive contains ‘\331(\330n\372\352\033\343ٷ߉’ where numeric off_t value expected
tar: Archive contains ‘\337\323\002h\315\371\002w\242\334\355\031’ where numeric off_t value expected
tar: Archive contains ‘\001\327£\375\034[-QV7\036’ where numeric off_t value expected
tar: Archive contains ‘k\020\327\023\210Cq\b[5\334 ’ where numeric off_t value expected
tar: Archive contains ‘\255\273Y\273\360\004u\276\360\027\270M’ where numeric off_t value expected
tar: Archive contains ‘[-&t\244\206\260(&\220\246)’ where numeric off_t value expected
tar: Archive contains ‘\a\002\251\354 ݓ-t\031h\037’ where numeric off_t value expected
tar: Archive contains ‘8\362\241\251\360Ah\255-\272\f^’ where numeric off_t value expected
tar: Archive contains ‘\353\250\317\342\260\\\356\362\301;\303\311’ where numeric off_t value expected
tar: Archive contains ‘\001Z\223\220ɜ\327j\360\235fA’ where numeric off_t value expected
tar: Archive contains ‘Ő\267\033z0_\264\326_-\326’ where numeric off_t value expected
tar: Archive contains ‘\316\354\330Țv|w\027\311%\370’ where numeric off_t value expected
tar: Archive contains ‘\233\215\351K\226\303BH\241\250<\337’ where numeric off_t value expected
tar: Archive contains ‘\376:{\204\362\312$\267\223\205\310p’ where numeric off_t value expected
tar: Archive contains ‘\236\226\3258+~\245\033\003~\376\022’ where numeric off_t value expected
tar: Archive contains ‘\260\341D\266\241[\224~\303\037\220\263’ where numeric off_t value expected
tar: Archive contains ‘w\360<\215\023\212\315:p:^\003’ where numeric off_t value expected
tar: Archive contains ‘\227\177\212\211FF\375\360Oe\201\232’ where numeric off_t value expected
tar: Archive contains ‘\247;\347P\306M\035a\213\350-\221’ where numeric off_t value expected
tar: Archive contains ‘\356b\372\212\326S\230H\320s\261w’ where numeric off_t value expected
tar: Archive contains ‘J\263\236[\034&"ƭ\344%\336’ where numeric off_t value expected
tar: Archive contains ‘\323Uy\260\225X/a\314\020] ’ where numeric off_t value expected
tar: Archive contains ‘P;\a\300\214E\304\026\373\314D\353’ where numeric off_t value expected
tar: Archive contains ‘nW\273B"\224\366*\334|\241R’ where numeric off_t value expected
tar: Archive contains ‘@\214\027D*\342-u\034}\214x’ where numeric off_t value expected
tar: Archive contains ‘\016\364\236DV\314\020\302]N\035y’ where numeric off_t value expected
tar: Archive contains ‘\362\307"\031\277y\330څ\021\304\320’ where numeric off_t value expected
tar: Archive contains ‘\375k\374\206\235j+\016W=w\215’ where numeric off_t value expected
tar: Exiting with failure status due to previous errors
[user@disp9624 appvms2salvage]$ sudo tar -i -xvf qubes-2019-09-12T202430 -C ~/Templates/qubes-2019-09-12T202430/
backup-header
backup-header.hmac
qubes.xml.000.enc
tar: Skipping to next header
tar: Archive contains ‘"\246\350\270\004\324c\025\2020\371)’ where numeric off_t value expected
tar: Archive contains ‘\372\340e"\273'\023k\260\256\001\373’ where numeric off_t value expected
tar: Archive contains ‘c\236|>Ӵ\325\373\030\026D\021’ where numeric off_t value expected
tar: Archive contains ‘\nfT\244`(\b\316o\2341\214’ where numeric off_t value expected
tar: Archive contains ‘\331\300\250P\t\370\347b’ where numeric mode_t value expected
tar: Archive contains ‘\245\345\303\006\006\216\322Ѣ\227\342\031’ where numeric time_t value expected
tar: Archive contains ‘\271\035\356\370f\207\374H’ where numeric uid_t value expected
tar: Archive contains ‘\374\371|V\343!Aa’ where numeric gid_t value expected
\301Y\f\nҼy\036\343\204caM\373\3749\r\206\355\225u\375~\255U\2412\240\237D\276.à\177\023\n\221a\003\373t\366\235G\004\036ۆ!\3275c\177\253\026\025\025\320g\327A\240\\[#NR\214\265]\264\334\035\017\251\231\226\202^\354\345iOc\366q\001Z\2325\022P\022\237\267\375\237\276\272
tar: \301Y\f\nҼy\036\343\204caM\373\3749\r\206\355\225u\375~\255U\2412\240\237D\276.à\177\023\n\221a\003\373t\366\235G\004\036ۆ!\3275c\177\253\026\025\025\320g\327A\240\\[#NR\214\265]\264\334\035\017\251\231\226\202^\354\345iOc\366q\001Z\2325\022P\022\237\267\375\237\276\272: Cannot hard link to ‘O$I\2624\262X\204\332\357=\3774#N\306\315\366"\306\350\206\324N\025ÝVѹ=\272\350\004\310b\374v_\233\252\327\256\367\261Y[\210\2174$\bnΪj\036\375\257\263\255\226\255k﹘\036AZ\234\362W\224Rh\347Ғ\3348Q\354\273\337\374B=TJk{m\213{45mT\233’: No such file or directory
tar: Skipping to next header
tar: Archive contains ‘\351\242\375\246\265\v\2123$<\252e’ where numeric off_t value expected
vm1/root.img.028.enc
tar: Skipping to next header
tar: Archive contains ‘E\361X\033/\301\003a\314\306\340\203’ where numeric off_t value expected
tar: Archive contains ‘\350\310H6\306\332\350\210|\002s\242’ where numeric off_t value expected
tar: Archive contains ‘\0\256\354*\371-?\252Po\005y’ where numeric off_t value expected
tar: Archive contains ‘\271n-\352\226T\330\017\372i|\\’ where numeric off_t value expected
tar: Archive contains ‘\236\277\220\006\\\370KbW5s\305’ where numeric off_t value expected
tar: Archive contains ‘#\342l\211;\341\236P\3050\373\243’ where numeric off_t value expected
tar: Archive contains ‘\373/\254\317\356\335\004~\026\\\245\032’ where numeric off_t value expected
tar: Archive contains ‘9\243Z*\n\240\256\v\004\021\316\177’ where numeric off_t value expected
tar: Archive contains obsolescent base-64 headers
tar: Archive contains ‘+;\0309\fL\220H\277N\363\343’ where numeric off_t value expected
tar: Archive contains ‘6\314%l\357\230>Az\003u\026’ where numeric off_t value expected
tar: Archive contains ‘\306\300J/\005\226i\273x\376\301\006’ where numeric off_t value expected
tar: Archive contains ‘k\207\026\bo\001"\037C\205\266'’ where numeric off_t value expected
tar: Archive contains ‘\350G\251\031\301\342~\305\023j\200\366’ where numeric off_t value expected
tar: Archive contains ‘\177\023_\234\267z\002\306PT6o’ where numeric off_t value expected
tar: Archive contains ‘6\232\245\025\2564\220\261[\205\210\334’ where numeric off_t value expected
tar: Archive contains ‘4\356\274\003s\304\322Ӡf;#’ where numeric off_t value expected
tar: Archive contains ‘\271\261\231x;6\025vZ\021\374?’ where numeric off_t value expected
tar: Archive contains ‘\023z\037\255\337&e\375\274\255[\261’ where numeric off_t value expected
tar: Archive contains ‘6\032G\272~\226\2120\022\f\0234’ where numeric off_t value expected
tar: Archive contains ‘S+pĪ0\n\016\f\005\211\\’ where numeric off_t value expected
tar: Archive contains ‘Ɔ\203\317b\3462C9KE+’ where numeric off_t value expected
tar: Archive contains ‘\233(\036S&\020 \235\024vQx’ where numeric off_t value expected
tar: Archive contains ‘m\227\264qv.\217?\375f&\366’ where numeric off_t value expected
tar: Archive contains ‘&\3669\036\020Ch_\031\375\375]’ where numeric off_t value expected
tar: Archive contains ‘\rO\242\371\f;d\003l_$\221’ where numeric off_t value expected
tar: Archive contains ‘\006\320\343\351xGb\204\275\324\024w’ where numeric off_t value expected
tar: Archive contains ‘\003GWTk1\360\231\355\346\257\342’ where numeric off_t value expected
tar: Archive contains ‘\\\265\243\223\256\306\0246\360jd$’ where numeric off_t value expected
tar: Archive contains ‘\001G\350\345a\204!\020\367\205z\247’ where numeric off_t value expected
tar: Archive contains ‘ہs\210K\2473\226ܷ&f’ where numeric off_t value expected
tar: Archive contains ‘ijo\004\203E\312\001\211\303\347\327’ where numeric off_t value expected
tar: Archive contains ‘E\212\337\001\271yb\344\350pH\036’ where numeric off_t value expected
tar: Archive contains ‘u\226\024\237c\377\326`\371\212\255\327’ where numeric off_t value expected
tar: Archive contains ‘\306`M\350jR\034\343\362J3\210’ where numeric off_t value expected
tar: Archive contains ‘\304\343\330 )T\025s2\231\325\304’ where numeric off_t value expected
tar: Exiting with failure status due to previous errors

If I said any help would be appreciated I’d be massively understating it.

2 Likes

Really sorry to hear this, although your problems are not the norm.

I’d start by testing the archive with file.
I’ve seen the “numeric off_t value expected” error where I’ve used later
libraries than those used to create the archive, so check what is the
result on a fedora 25 machine.
If the restore tool is failing, then it’s best to take the manual route
slowly, using (if possible) an environment as close as possible to that
used when the archives were made.

3 Likes

Thanks @unman I honestly hadnt even thought of that.
So I should be able to install an older version of qubes and possibly be able to restore those backups? While a bit of a pain it would be totally worth it if i could get at least 1-2 of the backups back. I have an old laptop that I can try that on, will report back.
Cheers (and thanks again).

2 Likes

Do report back, and dont despair.
Even when archives are hopeleslly corrupted, it’s usually possible to
recover something.

1 Like

I just thought I’d give a quick update, but I don’t know how much it’ll help. I had an earlier version of Qubes installed on another drive, so I attempted to restore my backup there just to see what would happen and got the same empty vms as before. Next, I went back to my current installation and updated dom0 using the current-testing and security-testing repositories before attempting to restore backups again. After updating, I attempted to restore my original backup as well as the test backups from Sunday, but once again: empty restored vms. Since I had updated, I figured that it’d probably be best to start over completely and see if the updates made any difference. I placed my test file into a new vm, backed it up and restored it, and for some reason it worked! the test files were present in this restored vm.

I wanted to test it a bit more thoroughly, so I repeated the test in ANOTHER new vm just to be sure. This time, however, it didn’t work and I once again got an empty restored vm. During the backup process, I noticed the backup’s text prompt saying saying that the target vm was still on and the backup will contain its current state, so I shut it off, did another backup, and this new backup contained the test files.

To close, I’m not sure why the backups made today were successful and those from before failed. I’d like to chock it up to a user error on my part and say that they failed because I had failed to close the vms before backing them up, but this doesn’t make much sense to me; the original vms had been opened and closed dozens of times before I attempted to back them up, so I would assume that at least some of their files would’ve carried over.

2 Likes

Happy to hear it’s working now. @Blorbo I was trying to rename this thread to something more concrete but I’m having a hard time. It’ll help others in the same situation find this thread. If you could sum it up in a few words, what would they be? If you can help, feel free to change the title, or let me know what I should change it to.

Thanks

1 Like

@deeplow To better clarify the issue, I’d rename it to something like “Potential error in backup restoration process that results in empty restored VMs.”

Renamed it for you :slight_smile:

2 Likes