QubesOS freeze, crash and reboots

[tanky0u]

Last night had 3 consecutive total freezes. The first one was out of nowhere, I was simply reading some RSS feeds on my newsboat program. Then, having heard that these freezes might be connected to having qubes os updates (either to the dom0 or to the domU’s) I have initiated the command line updates, using sudo qubesctl --show-output state.sls update.qubes-dom0 && sudo qubesctl --show-output --skip-dom0 --templates state.sls update.qubes-vm and pretty much a few seconds after pressing enter, I had another total system freeze. I had to press down on the power button to shut down the machine and restart qubes os. Then I repeated the same update command, and I had another freeze. At the third try, the update command went through without freezes and I saw that qubes os had downloaded updates to the whonix templates.

So, it indeed appears that these total system freezes are connected to fetching or downloading the updates. And perhaps doing these updates over the tor network? As I have my system download the system updates over Tor.

For the record, I am running vanilla QubesOS 4.1 with XFCE on a coreboot’ed Thinkpad X220 i5 CPU with 16 GB RAM.

[enmus]

August 22nd

QubesOS freeze, crash and reboots

And it looks like it isn’t related to these kernels only. Just after writing my previous post, my Qubes simply crashed and rebooted, since I was updating dom0 while writing that post. Today it happened again, so I chose lower version kernel (5.15.52 if I can recall) and update went smoothly. After update I set the kernel-latest as default kernel again. Have no courage to try to update dom0 again under it. xen_version : 4.14.5 Linux 5.18.16-1.fc32.qubes.x86_64

[tripleh]

Did anyone with the updater crashes ever test whether it’s reproducible from either
a) high CPU load (e.g. via stress)
b) high disk I/O (e.g. via dd if=/dev/urandom of=[somewhere on fs] bs=1M) c) high network I/O (e.g. via curl [huge file]`)
d) high GPU load (some 3D application)
?

Please also keep an eye on the CPU temperature during the process to rule out any crashes or freezes due to hardware overheating.

[Klappstock]

@howfuniscrashing to be more precise, I said that I had no crashes before I ever connected to the internet. After my first update I got crashes.

@tanky0u I do not update over tor.

Also, my system is running on a Dell t550 server with 2 xeon CPUs and 80-something GB RAM. Its not easy for a desktop environment to get the machine sweating. So there was no high CPU load. Also those CPUs don’t heat up that easily. Regarding the GPU IDK, though.

[Isolator]

@taradiddles

The issue is that there’s tearing/glitches/artifacts/corruption with the fb driver when there’s no compositing (which i3wm doesn’t provide - probably explaining the high proportion of reports from i3wm users). People then switch to the intel driver because that’s the solution mentioned in posts to fix those glitches - but for some (yet to be found) reason intel is unstable for some people - from oopses to hard freeze/reboot.

In my case I’d get a few random reboots a day. I eventually made the connection and reverted to fb and so far haven’t encountered a single crash. In hindsight, given all the problems reported in this thread, I’m wondering which proportion is due to people using the intel driver instead of fb and being hit by bugs in intel…

Guess this is my case too as I switched back to fb in November and haven’t seen a crash since. Visual glitches also became less pronounced with updates - when I first installed 4.1 it took a good couple of seconds for mouse pointer to change its shape - now it’s barely noticeable with some split second rendering glitches in menu and at window borders. Doesn’t affect rendering of content in VMs window, so not a deal breaker.

I should mention all this relates to default xfce.

[tanky0u]

Sounds familiar:

Since I updated from 4.0 to 4.1 in August last year, I’ve had random gui freezes with no possibility to recover, I just have to hard reset the machine every time (I’ve had 4 such freezes trying to write this on a stock new Qubes install, but that’s another story…)

Not bootable after freeze/reset during dom0 update

Since I updated from 4.0 to 4.1 in August last year, I’ve had random gui freezes with no possibility to recover, I just have to hard reset the machine every time (I’ve had 4 such freezes trying to write this on a stock new Qubes install, but that’s another story…) I had a freeze a few days ago while dom0 was updating and hard reset, now upon reboot I just the blank grub prompt (grub>). The rescue tool finds my root partition and decrypts but doesn’t mount with error message “You don’t have any …

[tanky0u]

I was checking some github issues and stumbled upon this: Qubes Update tool freezes machine · Issue #6156 · QubesOS/qubes-issues · GitHub

Pretty much the same as my reported freezing issues.

Also, update on this “freezing issues”: I seem to get 1 total system freeze per week now. Definitely better than what I was getting with i3wm + “screen tearing fix” I had previously with QubesOS. Back then, the freezing frequency per week was higher, in my anecdotal evidence.

[khal]

I’ve watched these freezes for 6 months and it seems to me to be associated with whonix-ws-16. At least, the most consistent freezes occur when I start anon-whonix and use for a few minutes. I very rarely get freezes without anon-whonix being opened, but it does happen. I can’t figure rhyme or reason why it happens.

[tanky0u]

Sharing an anecdotal experience of mine: the frequency of the freezes I reported in this thread seems to have decreased. I haven’t got a total system freeze on me for some weeks now, even though during the time I had my machine working day and night, doing backups, working idle, etc.

So that’s encouraging. Thanks to the QubesOS dev team!

[ruza]

The only freeze and hard reboot I am experiencing now happened when doing qubes-dom0-update. Otherwise my Qubes 4.1.1 is stable. Using 6.1.12-1 kernel a xen 4.14.5 (after reboot).

[oh_no]

Qubes OS if very unstable in my case. Unfortunately, I’ve had enough, I can’t recommend Qubes.

First, desiring to get rid of the reboots, I changed my RAM, CPU, motherboard, and power supply. Now I understand that the problem is in the OS. It’s crashing without any signals in the log, mainly on virtual machine start/stop.

So I still can’t boot into the system at the first time! Usually I am trying to boot five or more times to start all my appVMs. Then I am trying to keep my computer uptime 24/7, that is usually a week without the accident reboot. Starting an appVM turned into a reboot lottery.

[unman]

Qubes OS if very unstable in my case. Unfortunately, I’ve had enough, I can’t recommend Qubes.
First, desiring to get rid of the reboots, I changed my RAM, CPU, motherboard, and power supply. Now I understand that the problem is in the OS. It’s crashing without any signals in the log, mainly on virtual machine start/stop.

I’m sorry you have had these problems. Thanks for trying so hard.
It would be helpful if you could share details of the hardware you have
used.

There is a hard regression in 4.1 that results in these problems for
some users. In some cases this seems to be linked to Qubes
starting/shutting down, and perhaps memory usage. In some cases there is
no log evidence.
It may be that these problems are resolved in 4.2, so it may be worth
your while checking in now and again and trying that release.
I hope to see you again.

I never presume to speak for the Qubes team. When I comment in the Forum or in the mailing lists I speak for myself.

[ChrisA]

Have you tried/tested kernel-latest? - if no, would you be willing to test it?

sudo qubes-dom0-update --disablerepo=* --enablerepo=qubes-dom0-current-testing kernel-latest

(in Dom0)

If you have issues booting, could you try temporally adding:

qubes.skip_autostart

to the kernel line at boot (hit e to edit the GRUB config, use the arrow keys to locate where it says quiet and replace it with qubes.skip_autostart)?

[tripleh]

Maybe relevant for some coreboot users: Bug #121: T520: Hangs in OS - coreboot - Issue Tracker

[tanky0u]

I would like to report that my intel i7-3520M CPU (on Thinkpad X230) has been rock-solid stable since a month. I am on Qubes 4.1 with the latest updates installed. I have been using my setup for video encoding day and night and never had a freeze so far.

[kenosen]

Ditto. On a Librem 14. Used to be that I had to have power-adapted plugged in for updates, but rock solid now.

[oh_no]

The Qubes OS stability has increased significantly since last dom0 update. No single crash and reboot since then.
I’m even nervous about that

user@dom0 ~ % cat /var/log/dnf.log
...
2023-07-14T13:54:33+0100 DEBUG --> Starting dependency resolution
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel.x86_64 1000:6.1.35-1.qubes.fc32 will be installed
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-devel.x86_64 1000:6.1.35-1.qubes.fc32 will be installed
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-latest.x86_64 1000:6.3.9-1.qubes.fc32 will be installed
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-latest-modules.x86_64 1000:6.3.9-1.qubes.fc32 will be installed
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-modules.x86_64 1000:6.1.35-1.qubes.fc32 will be installed
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-qubes-vm.x86_64 1000:6.1.35-1.qubes.fc32 will be installed
2023-07-14T13:54:33+0100 DEBUG ---> Package python3-qubesadmin.noarch 4.1.27-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package python3-qubesadmin.noarch 4.1.31-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-admin-client.noarch 4.1.27-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-admin-client.noarch 4.1.31-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-dom0.noarch 4.1.30-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-dom0.noarch 4.1.32-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-qrexec.x86_64 4.1.21-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-qrexec.x86_64 4.1.22-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-qrexec-dom0.x86_64 4.1.21-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-qrexec-dom0.x86_64 4.1.22-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-qrexec-libs.x86_64 4.1.21-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-core-qrexec-libs.x86_64 4.1.22-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-gpg-split-dom0.x86_64 2.0.67-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-gpg-split-dom0.x86_64 2.0.68-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-manager.noarch 4.1.28-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-manager.noarch 4.1.30-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-mgmt-salt-dom0-update.noarch 4.1.10-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-mgmt-salt-dom0-update.noarch 4.1.11-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-pdf-converter-dom0.x86_64 2.1.16-1.fc32 will be upgraded
2023-07-14T13:54:33+0100 DEBUG ---> Package qubes-pdf-converter-dom0.x86_64 2.1.18-1.fc32 will be an upgrade
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel.x86_64 1000:5.15.89-1.fc32.qubes will be erased
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-devel.x86_64 1000:5.15.89-1.fc32.qubes will be erased
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-latest.x86_64 1000:6.2.6-1.qubes.fc32 will be erased
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-latest-modules.x86_64 1000:6.2.6-1.qubes.fc32 will be erased
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-modules.x86_64 1000:5.15.89-1.fc32.qubes will be erased
2023-07-14T13:54:33+0100 DEBUG ---> Package kernel-qubes-vm.x86_64 1000:5.15.89-1.fc32.qubes will be erased
2023-07-14T13:54:33+0100 DEBUG --> Finished dependency resolution
2023-07-14T13:54:33+0100 DDEBUG timer: depsolve: 16 ms
2023-07-14T13:54:33+0100 INFO Dependencies resolved.
2023-07-14T13:54:33+0100 INFO ==============================================================================================================
 Package                          Arch        Version                           Repository               Size
==============================================================================================================
Installing:
 kernel                           x86_64      1000:6.1.35-1.qubes.fc32          qubes-dom0-cached        11 M
 kernel-devel                     x86_64      1000:6.1.35-1.qubes.fc32          qubes-dom0-cached        14 M
 kernel-latest                    x86_64      1000:6.3.9-1.qubes.fc32           qubes-dom0-cached        12 M
 kernel-latest-modules            x86_64      1000:6.3.9-1.qubes.fc32           qubes-dom0-cached        72 M
 kernel-modules                   x86_64      1000:6.1.35-1.qubes.fc32          qubes-dom0-cached        65 M
 kernel-qubes-vm                  x86_64      1000:6.1.35-1.qubes.fc32          qubes-dom0-cached        16 M
Upgrading:
 python3-qubesadmin               noarch      4.1.31-1.fc32                     qubes-dom0-cached       446 k
 qubes-core-admin-client          noarch      4.1.31-1.fc32                     qubes-dom0-cached        67 k
 qubes-core-dom0                  noarch      4.1.32-1.fc32                     qubes-dom0-cached       703 k
 qubes-core-qrexec                x86_64      4.1.22-1.fc32                     qubes-dom0-cached       170 k
 qubes-core-qrexec-dom0           x86_64      4.1.22-1.fc32                     qubes-dom0-cached        37 k
 qubes-core-qrexec-libs           x86_64      4.1.22-1.fc32                     qubes-dom0-cached        25 k
 qubes-gpg-split-dom0             x86_64      2.0.68-1.fc32                     qubes-dom0-cached        23 k
 qubes-manager                    noarch      4.1.30-1.fc32                     qubes-dom0-cached       1.8 M
 qubes-mgmt-salt-dom0-update      noarch      4.1.11-1.fc32                     qubes-dom0-cached        32 k
 qubes-pdf-converter-dom0         x86_64      2.1.18-1.fc32                     qubes-dom0-cached        27 k
Removing:
 kernel                           x86_64      1000:5.15.89-1.fc32.qubes         @qubes-dom0-cached       35 M
 kernel-devel                     x86_64      1000:5.15.89-1.fc32.qubes         @qubes-dom0-cached       60 M
 kernel-latest                    x86_64      1000:6.2.6-1.qubes.fc32           @qubes-dom0-cached       39 M
 kernel-latest-modules            x86_64      1000:6.2.6-1.qubes.fc32           @qubes-dom0-cached      411 M
 kernel-modules                   x86_64      1000:5.15.89-1.fc32.qubes         @qubes-dom0-cached      309 M
 kernel-qubes-vm                  x86_64      1000:5.15.89-1.fc32.qubes         @qubes-dom0-cached      515 M

Transaction Summary
==============================================================================================================
Install   6 Packages
Upgrade  10 Packages
Remove    6 Packages

2023-07-14T13:54:33+0100 INFO Total size: 193 M
...

user@dom0 ~ % uname -a
Linux dom0 6.3.9-1.qubes.fc32.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jun 23 19:49:58 CEST 2023 x86_64 x86_64 x86_64 GNU/Linux