I was hoping someone could help me in recovering my Qubes OS. I recently attempted to upgrade my Qubes 4.0 to 4.1 using this guide and I believe there were errors that occurred that are now affecting my OS. I first tried to install each stage sequentially but in hindsight, after reading qrexec agent not starting after vm update, I feel like that may not have been the best course of action to take. I had initially tried the --all option and the first attempt stopped after Stage 2. I then proceeded to attempt them individually (sequentially) and got through them as best as my machine could. After trying to complete Stages 0-5 (the six in total before the restart) I restarted and found that only dom0 had started. I can no longer start any Template or App VM Qubes (with the exception of one standalone Debian template that wasn’t upgraded during the update process). sys-net will not start and neither with sys-firewall (even taking the network off), nor will any template VMs. I get the error “qrexec-daemon startup failed” with all VMs and upon further investigation I’m seeing:
Failed to start Qubes remote exec agent. See ‘sytemctl status qubes-qrexec-agent.service’ for details.
Upon checking the status I get:
“Unit qubes-qrexec-agent.service could not be found.”
Also:
“Incompatible agent protocol version (remote 3, local 2)
Unable to init server: Couldn not connect: Connection refused
(zenity:9543): Gtk-WARNING **: cannot open display:
Connection to the VM failed”
I’m assuming that because I didn’t update the Template VMs in time that dom0 is using the 4.1 qrexec-agent and the templates are trying to use the 4.0 qrexec-agent?
I have read that people had luck being able to download a fresh template and were able to recover some of their App VMs, however, I don’t have internet anymore. Nor is my sys-usb working, so I feel as though downloading the template and manually installing is off the table as well. I just have one standalone Debian template with no internet access at the moment. As well, I tried the Qubes backup route but I had not personally made a backup (unfortunately…) since the install of Qubes. Unless Qubes may have initially made one and I’m unable to locate the .xml file, I’m not sure that is an option?
Sorry to hear of your problem - that tool should carry a significant
health warning and a requirement that users take a backup before use,
not just a strong recommendation.
If you don’t have any data worth saving, I’d suggest running a clean
install.
If you do have data you want to keep, I would take backup now. You can
do this from dom0, although you may have to change the boot parameters
and reboot if you want to use a USB device attached to dom0. (check
kernel parameter rd.qubes.hide_all_usb )
If you have another device you should be able to download a template,
check the signature, and then transfer it in to dom0 from a USB stick or
similar. You should then be able to install the template using rpm -i
Whether this will be enough to get you back on track, it’s difficult to
say, but it should be a start.
Maybe you’ve already resolved the problem? Good. Remember to take
backups in the future.
Hahaha, I will definitely be making backups from now on!! I really appreciate the reply. You actually already helped me solve it based on a previous post you made, so thank you very much for that! I did a little write-up for anyone like me with minimal linux experience, so hopefully it helps someone in some way. Going to copy that below.
Solved (more or less) and credit goes to unman:
So, I was unable to correct what had happened. I ended up just recovering my files in the most secure ways I could and did a fresh install.
Since I had one qube that I could actually access (despite the fact that I would have preferred using a disposable VM for each qube I backed up) I was able to recover the majority of things via the Backup Qubes utility and completing that process with as many qubes as I could. (If I recall correctly, once I was able to connect a USB drive to dom0 I was able to mount to the standalone). I believe best practice, in this case, would be for anyone going about backing up their qubes to use a separate disposable VM and storage device, per backup. I would also recommend that the drive be brand new and bought in-person.
I backed up as many qubes as I could but was unable to backup one in particular. This led me to need to mount a device directly to dom0 (which is highly unrecommended for security reasons, but considering I’m going ahead with a fresh install and it’s only one qube being accessed, I think it’s relatively okay to do) in order to recover the last of the files that I needed.
In order to be able to mount a USB device directly to dom0 you need to:
After you reboot go ahead and mount your device to dom0. And if you don’t know how, open a dom0 terminal and check to see that your device is recognized:
lsblk
Check for your device and verify what it’s mounted as. (e.g. sdb, sdc, etc.) (Device partitions are labeled with a number, e.g. sdb1, sdc1, etc.)
Make a couple directories just to simplify things and make the process easier:
sudo mount /dev/(your device partition) /mnt/destination
Check for the name of the qube you’re trying to recover files from:
cd /dev/qubes_dom0/
ls
Find the name of the qube you need. (It will end with -private)
Mount the qube to the source directory you just made:
sudo mount /dev/qubes_dom0/(name of needed qube) /mnt/source
If you’re unable to mount the device to dom0, it may be due to the file system type of your device. In my experience exFAT is not recognized, ext3 copied very slowly, FAT32 would have worked if I did not need to copy files larger than 4gb, and oddly enough (I thought I remembered NTFS not being recognized on a qube before) NTFS worked well. You can do this a number of ways, of which, I think parted and fdisk are the most popular solutions.
If you want to check the file system type of your device:
With parted:
sudo parted /dev/(your device)
print
quit
With fdisk:
sudo fdisk /dev/(your device)
p
q
If you need to change the file system type of your device, look up online how to partition & format a drive in linux using either parted or fdisk.
Check for the directory/directories you may need to copy to your device:
cd /mnt/source/
You’ll probably only need what’s in your home folder, if so:
cd /mnt/source/home/user/
List directories:
ls
If you would like a specific folder (cp is copy, -vr represents the options for verbose and recursive):
(There are other commands to copy data between locations as well.)
sudo cp -vr /mnt/source/home/user/(name of folder) /mnt/destination
If you would like all of these folders:
sudo cp -vr /mnt/source/home /mnt/destination
Once you run either command, you’ll see the source of the file being copied as well as the destination.
To confirm your files are there:
cd /mnt/destination
ls
Then just navigate through the folders to ensure everything went smoothly.
Unmount your device:
sudo umount /dev/(your device)
If it says the device is busy, try closing the terminal and opening a new dom0 terminal.
In the event it’s still busy and will not let you unmount, I’m sure rebooting would be better than forcing it to unmount with the below command:
sudo umount -l /dev/(your device)
If you get a read-only error you may need to check the device’s file system type again (this error popped when I attempted FAT32 [lol]). Partition and format it one more time if you feel you may need to or you may need to remount each (your usb device & the qube that was previously mounted) with read and write properties. Search online for how to remount a read-only device as read and write, you should find something. Hopefully this method worked, if not keep at it.
An additional method I was going to try was to download a template along with the signing keys and try to install it via a USB drive. I didn’t get very far with that, other than reading a little on the website, but if you’d like to try this method to potentially get a working qube to rectify the qrexec problem:
I’m a casual linux user so I was able to learn a little bit more than I previously knew from this experience, which I am grateful for. Qubes is an amazing OS and I’m very happy the Qubes Team decided to pursue this project. It is, without a doubt, one of the greatest assets out there in regards to security and privacy. The new app menu looks amazing too. And, again, credit for me being able to fix this problem goes to unman. Thank you!