Major OS error, all VMs unusable, network manager disappeared

I have encountered some very major issues making my whole OS unusable. Upon boot I saw a few [failed] processes, then when I logged in the network manager Icon has disappeared and all my VMs will not start they give me the error below ( I will post a picture also)

Domain sys-net has failed to start: qrexec-daemon start failed: connection to the VM

Then Depending on the VM I try to start :

Domain [ VM name ] has failed to start: qrexec-daemon start failed: connection to the VM failed

Additionally when I try to manually open the Network manager from the system tools it is all grayed out and not accepting inputs.
Also when I logged in all my desktop icons were randomly rearranged.

I have no idea what to do to fix this or what caused it, possibly a recent update?

Please advise me on any thoughts you could or ways to revert to a previously working version.
Any help is super appreciated, I have some personally valuable information stored in a VM so I can not just reinstall the OS.

Maybe changing kernel would work, what kernel you using now ?

1 Like

Thanks for the advice

I do think it has something to do with the kernel failing, upon shut down I get the error message

[Failed] failed to start Load Kernel Modules.
see systemctl status system

I will try and change the Kernel, I am using currently 5.4.107-1.fc25

I am not sure how to change it or what version to change it to though? Do you have any recommendations?

i always use kernel-latest and kernel-latest-qubes-vm and make sure that you use current-testing repository, because kernel is always latest.

and based on your screenshot, are you installing nvidia driver?

The message about Loading Kernel Modules is irrelevant here - it’s a
stock error, and is in the FAQ. If you check the status you will likely
find it succeeded (in the end).

The Network Manager from System Tools is also irrelevant, I think. Isn’t
that loading from dom0?
You haven’t said if this is a new error following an update, template
change, or similar. Can you provide that basic information?
A simple way to test if this is related to your sys-net is to see if you
can start a qube that has no network connection - like the vault.
Then see if you can start sys-net from the command line - qvm-start sys-net,
and see if you have error messages. Also check the logs for sys-net.

If you did make a dom0 update, you can always load a previous kernel by
selecting at the grub menu.
If you updated the template you are using for sys-net, you can choose an
earlier one at the GUI by looking on the Advanced tab.
If you changed the template used for sys-net you can always change the
template used on the Basic tab of the qube settings GUI.

I am completely new with all this thanks for helping me out. I am at a complete loss my heart sank when I realized what had happened and making everything unusable.

The error about the kernels on shut down and network manager are probably irrelevant as you say, sorry I just didn’t realize and wanted to include anything possible.

At your advice I attempted to launch Vault but was unfortunately meet with the same error message. I am seemingly unable to successfully start anything.

When I start to start says-net from the command line it continues to give me the error

As of yesterday , I had installed some automatically suggested updates. If I recall correctly it was all the templates except Debian-10. This is a new issue and has never occurred before, I think this may be the first reboot after the updates yesterday.

When I attempt to change the Kernel I am only seeing one boot entry option listed ( the current default kernel) and no other kernels to choose from. Is there a way to see all available kernels?

Thanks for the response, I am unable to update or check for a updated kernel because of the networking issues.

I am not installing any Nividia driver that I know of. Is there a way to view what drivers I have installed?

@sun oh i remember this,
could you check in qube manager, then check all vm how much you assign ram ?
initial ram should be at least 10% of the max ram you give.

I’m in Qubes Manager but I can’t seem to find the listed Ram of the VMs. Could you kindly point out to me where the assigned ram is located and how to change it if necessary? Sorry for the basic questions , I tried to google it but couldn’t seem to find a clear answer

you can right click and open vm settings. then move to advanced tab.

Thank you , I have changed all their setting of the initial memory from 10% to 25%

However when I try to start any VM I still get the same error message is there something else I could do or try?

Here are the logs for sys-net if anyone could tell anything from them

That qubes machine is completely useless it wont even recognize anything in the USB ports now so i can’t use a drive to transfer it and i can’t transmit because it has no networking.
So I had to just manually take pictures and upload them to google drive

the first 2 links are the first 2 pages then then the 3rd link folder is the next 15 pages please let me know if more

IMG_2653.HEIC - Google Drive
IMG_2654.HEIC - Google Drive
Logs - Google Drive

Okay… based on the log i still can’t find your problem, my recommend is reinstalling qubes to 4.1 and use current-testing repository, then use kernel-latest, kernel-lastest-qubes-vm and do update, i have using this for 4 months and don’t have any problem.

Let’s back to problem, i have a solution from our discussion before, you just need gpg file right? or lets say any file in vm.

  1. in dom0 terminal type sudo fdisk -l and find your vm name disk, in this case i use sllieal.

  1. mount disk sudo mount /dev/mapper/qubes_dom0-vm--YOUR-VM--private /mnt

  1. enter your private keys dir cd /mnt/home/user/.gnupg/private-keys-v1.d/

note : you may use ls command if not sure

  1. copy that secret key into your gpg private file location

example of my case : cp *.key ~/.gnupg/private-keys-v1.d/

4

note : use CP command not MV, in case you doing wrong you still have copy file and from here i use sys-net as example because its disposable vm, you can go with dom0 its okay.

  1. write down your public key file ( i’m sorry :frowning: ) and save it, then import your gpg, after that export private keys to ascii so you can write it down again in other system ( :frowning: )

5

6

  1. Final step before you wiped out the system, make sure the gpg files are work :slight_smile:

you can use qubes rescue os for easy copying file.

1 Like

Please provide some logs:

  • xl dmesg
  • dmesg

Some of a failing VM, e.g.:

  • /var/log/xen/console/guest-vault.log
  • /var/log/qubes/vm-vault.log

Have you checked your bios settings to be sure that virtualization has not been disabled due to default options restored from failing bios battery?

Thank you so much for the detailed instructions, I spent the day transcribing the keys but when I went to import them to a new machine, it is unrecognizable. I must have made a typo somewhere or something.
However I am not completely ready to give up on the whole OS yet.
I have a interesting new development, I switched the default kernel and can can now open VMs that do not use networking or if I switch the networking off I can now launch the VM.

Is there any things else I could do so I can maybe get the USB port functional so I can transfer files out of this machine before I have to reinstall the OS? As of right now nothing is recognized in any USB port whether it be a USB drive or mouse

Have you try with qubes rescue? 1 flash drive contains qubes installer and another flashdrive to move your file from system to here.

This is the second and last time I ask you to please provide logs if you want some help from Qubes team. See my previous message.

Thanks you for the offer to check the logs, this was just getting to be too much for me as an inexperienced user to fix and I didn’t trust the current install I had not to fail. again I ended up reinstalling and it seems to be fine now.

Thanks you for the support and checking the logs, this was just getting to be too much for me as an inexperienced user to fix and I didn’t trust the current install I had not to fail. again I ended up reinstalling and it seems to be fine now.

Hey guys.

After updating my system yesterday I encountered the exact same issues. I`m running a very supported x230 for years, 0 issues.

The issue only occurs on Debian 10 based templates. Fedora 33 is fine.

It says it cannot connect to qrexec-agent for 60 seconds.
Xen logs show that Network Manager service fails to load on the vm.

Could you guys point me out on a debuging direction? Really not in the mood to reinstall Qubes since this is a Heads installation.

1 Like