Overnight idling had another total system freeze. Also, just now had another system freeze out of nowhere. This time was different in the sense that it froze minutes after the OS boot/ Normally my experience with freezes have been occurring over hours of idling.
Hereās a good update on this: I have disabled all the updater checks from Qubes Global Settings. And since then (~2 days) the total qubes os freeze issues that were plaguing my computing experience HAS GONE.
I have left the machine idling overnight, I left it idling during the day, I have used it semi-heavily for hours, and during all these I havenāt observed the freezes, yet.
System: running vanilla QubesOS 4.1 with XFCE, without the āintel fixā that many of the i3wm users employ.
1 Like
I can kind of confirm this. I did a fresh install of 4.1.1 via USB and kept the machine separated from the internet. cron.daily did not changeā¦
I also kept the machine idle by running videos (from the HD) on many vms and kept this going for 2 days. No crashes.
Then I plugged in the internet and after some time I allowed update of whonix and debian (I have no fedora) . At this point dom0 was not updated and cron.daily was still unchanged.
But now a crash happened pretty soon after the update. After restart dom0 update was done too and cron.daily changed. Then another crashā¦
If there is interest I can send the full dmesk from installation until today. Its not that long because the machine was not doing anything other then running the videos, updating and crashing
2 Likes
Sure probably mentioned in this mammoth thread and doesnāt apply to those posting now I believe but just in case nouveau driver seems to be the only thing I have come across to cause system wide crashes and this is fixed by adding nouveau.noaccel=1 to GRUB_CMDLINE_LINUX in /etc/default/grub and then doing a grub2-mkconfig. Also to get the nouveau driver to actually use your video card on more than the very lowest setting at least for older nvidia cards you need to add a nouveau.config=NvClkMode=15 as well in the same spot. Havenāt seen a crash since made those changes. The system does still hang if I do a restart instead of a shutdown and then restart manually but disks get unmounted so no harm no foul. Pretty sure probably a setting I can look into but rarely reboot so havenāt bothered.
There is a now a strong indication that the crashing is caused by some part of the update process.
There are multiple people in this thread who are willing to put forward logs and I dare say would be willing to perform any steps that are needed to further debug this.
Can we please, please take advantage of this @marmarek @unman?
Letās make Qubes usable againā¦
4 Likes
Hello everyone, I experienced the same crashes as some of you: freeze 1-2s ā black screen ā power off & for 1-2 times a day. Most of the time because of window movement in i3 (I guess), but also happened at random times. I tried the different Intel video driver configurations but it only changes the frequence of crashes (if not just pure randomness).
However, I am freeze free (like not a single one) for around 3 months now. Not sure if pure coincidence or if it was actually related, but it happened after modifying the Bios power configuration to the following:
It probably means that it is related to temperature even though no error messages suggested this nor the captorsā value that I started monitoring.
I first dug into software related bugs and remediations but could not find any solutionā¦ It was driving me crazy, so I ended up trying all possible BIOS configurations and finally used that one.
As I said, I am not entirely sure it is the reason the crashes stopped. I will try reverting the setting someday, but since Qubes OS is my daily driver, it is quite sensitive to crashes (says the one that worked for several months with 2 crashes per day). I will keep you updated.
2 Likes
Hey thatās cool to know, thanks for reporting. Unfortunately, I have corebootāed my BIOS, so I have no detailed settings as you have, thus I am not sure if I can even disable built-in battery on my machine.
Apart from that, I am continuing using the vanilla setup of QubesOS 4.1 with XFCE4. Disabling automatimc update checks have helped me with the total system freezes. However, I have experienced one freeze a few days ago. I guess there is still some background update checking process remaining that might keep freezing my system once a week or so. I will keep reporting.
1 Like
We already know that for some people the update process can trigger a
system failure.
But we need to do testing to establish whether the fault is in the
update process, in the operation of salt, or in the repeated cycling of
qubes starting and closing. My feeling is that the errors I see on the
old thinkpads are triggered by qubes starting and shutting down, i.e.
not by the update process per se.
Itās good that multiple people are willing to put forward logs.
When Iāve tried to get user input in the he past Iāve had zero response,
even at a trivial level.
If you are willing to help out here, PM or email me with details of
your hardware, Qubes configuration including DE, when you see these
crashes in use, when you started to see this adverse behaviour.
If you see crashes during updates and are willing to run some tests, I
have some simple scripts that will test various elements of the update
process to get focus on what might be the root cause of the issue.
If you want to help please reply here or PM me.
@unman Iām willing to run whatever is necessary on my dell t550. I donāt use the machine for anything else currently. In my cases ācrashesā are freezes, where the pointer and keyboard are unresponsive. The server does not turn off thoughā¦
I can do a fresh install as often as needed (within boundaries of course) and run your script.
As I wrote above, I think everything works fine as long as the machine stays disconnected from the internet after a fresh install. But to test if starting/stopping many qubes is the actual issue I could run a script that tests this.
1 Like
Hi Klappstock
Thanks for the offer.
Can you finger anything specific that is triggering these freezes?
Is it the act of connecting to the internet, or do you update after going
online, and then see the freezes?
As I wrote the freezes only happen after the update. If its due to many starts and stops of VMS, IDK
1 Like
Well, after 6 months (not for me, except very few reboots and freezes), the only what is left is some Xen 0day, since people obviously cannot be more specific, hahaha.
@Klappstock is saying that when heās offline thereās no crashes/freezing. Does the update process have an early check for being offline or does it still run through the entire process and eventually fail? If itās the latter then that would mean the VM starting/stopping isnāt the issue because crashes would still be expected even when offline. This report indicates to me that the crash is being triggered after the update has been fetched/is being applied. Has this been investigated? @unman
1 Like
I just donāt see how this could be resolved when itās not reproducibleā¦ If it was, it would be on Githubā¦
I had a freezing / crashing probleme and it was one of my RAM slote that was half-dead.
You can do a memtest just in case.
A good and open-source memtest
My post on my crashing probleme
1 Like
Last night had 3 consecutive total freezes. The first one was out of nowhere, I was simply reading some RSS feeds on my newsboat program. Then, having heard that these freezes might be connected to having qubes os updates (either to the dom0 or to the domUās) I have initiated the command line updates, using sudo qubesctl --show-output state.sls update.qubes-dom0 && sudo qubesctl --show-output --skip-dom0 --templates state.sls update.qubes-vm and pretty much a few seconds after pressing enter, I had another total system freeze. I had to press down on the power button to shut down the machine and restart qubes os. Then I repeated the same update command, and I had another freeze. At the third try, the update command went through without freezes and I saw that qubes os had downloaded updates to the whonix templates.
So, it indeed appears that these total system freezes are connected to fetching or downloading the updates. And perhaps doing these updates over the tor network? As I have my system download the system updates over Tor.
For the record, I am running vanilla QubesOS 4.1 with XFCE on a corebootāed Thinkpad X220 i5 CPU with 16 GB RAM.
1 Like
August 22nd
Did anyone with the updater crashes ever test whether itās reproducible from either
a) high CPU load (e.g. via stress)
b) high disk I/O (e.g. via dd if=/dev/urandom of=[somewhere on fs] bs=1M) c) high network I/O (e.g. via curl [huge file]`)
d) high GPU load (some 3D application)
?
Please also keep an eye on the CPU temperature during the process to rule out any crashes or freezes due to hardware overheating.
@howfuniscrashing to be more precise, I said that I had no crashes before I ever connected to the internet. After my first update I got crashes.
@tanky0u I do not update over tor.
Also, my system is running on a Dell t550 server with 2 xeon CPUs and 80-something GB RAM. Its not easy for a desktop environment to get the machine sweating. So there was no high CPU load. Also those CPUs donāt heat up that easily. Regarding the GPU IDK, though.
1 Like
Guess this is my case too as I switched back to fb in November and havenāt seen a crash since. Visual glitches also became less pronounced with updates - when I first installed 4.1 it took a good couple of seconds for mouse pointer to change its shape - now itās barely noticeable with some split second rendering glitches in menu and at window borders. Doesnāt affect rendering of content in VMs window, so not a deal breaker.
I should mention all this relates to default xfce.