Endless Issues with Qubes 4.1.0, 4.0.4 on Librem 14v1

User Support General
61

The issue I have with LibreOffice isn’t that bad. It’s not a CPU hog, it’s just slow (e.g. moving from cell-to-cell or entering a value takes ~200-300 ms, which is enough lag to make it annoying and difficult to be productive in).

I’m using kernel 5.15.52-1.fc32.

I don’t make any modifications like this. My OS is essentially stock, which is why all of these issues are so confusing. I recently discovered that fedora-34 was at EOL, so I’m trying fedora-36 to see if this will help solve some of these issues. I think many of the sudden qube shutdowns may be networking-related.

62

Is the xen-acpi-processor configuration change still helpful with Qubes 4.1.1 on Librem 14?

63

Are you certain your updates are actually being installed? I had a ton of issues until I reinstalled and decided not to do the tor updates after reading about the troubles people were having with them. Thru tor My updates were failing every time, on clearnet they worked first time and solved a load of issues I was having.

Have you dont anything is your Bios? I would suggest turning off instant boot if you have that option and see if that helps.

Have you ran a memory test? are you certain you seated your memory properly? Many of those issues could be a bad stick of ram.

I would look into a bios or hardware issue. Qubes 4.1 works flawlessly for me other than issues with me not knowing how to use Linux very well. You either you have a corrupted qubes install USB(easily verifiable) or it is a hardware/bios issue. Have you researched your motherboard for ran compatibility?

Did you write zeros over your entire drive when you reinstalled each time? and when you say suspend you mean suspend to ram not suspend to disk/hibernation right?

I had a lot of issues when I first installed qubes and I was able to fix most of them by going thru my bios and disabling the stuff made to work with windows and switching to clearnet updates. Since then it has been working flawlessly other than user error.

good luck, and I would start with a memory test if you have not done one already. Also maybe try a memory test while shaking the laptop a bit if it is hard to open it up to check that you seated your memory properly.

64

I’m not sure whom you’re addressing, but I’ll provide an update for anyone who’s interested.

I struggled with these issues for several months, at the cost of hundreds of hours of lost productivity. Mind you, my device (Librem 14v1) is supposedly compatible with Qubes OS and even came with Qubes OS installed on it from the factory. I have not made any BIOS changes (using PureBoot) as this should be configured from the factory to work with Qubes. If not, I’m not sure what I would change anyway.

I decided that perhaps my Qubes OS v4.1.0 was corrupted after several months of crashes, update issues, etc., so I sought to wipe the whole thing and install Qubes OS v4.1.1. I did this, following the instructions for downloading, verifying, and installing the OS to a T. I even went through the trouble of verifying the integrity of the OS image on the flash drive itself, and everything checked out.

v4.1.1 installed without issue. When I first set up the OS and logged in, I made two changes:

  1. Set up Wi-Fi
  2. Changed touchpad to register a tap as a click

However, I immediately started running into major issues:

  1. System crash on bootups 2-5 after LUKS screen
  2. dom0 updates repeatedly failing (although I believe the first one I tried “worked” as in it gave me the green checkmark)
  3. whonix-ws-16 and whonix-gw-16 updates failed a few times, including 1st attempt
  4. Fresh fedora-36 AppVM exhibited the same journalctl errors as previous installation (e.g. pulseaudio errors). Note that I don’t see at least the pulseaudio errors in Fedora 36 Workstation, which I replaced Qubes OS with (see below), which doesn’t make any sense to me.
  5. When I opened the Cheese app, the system crashed

My next thought was, “perhaps I have a hardware issue.” To address this, I installed Fedora 36 Workstation, which has since proven to be orders of magnitude more stable than Qubes OS v4.1.1. Admittedly, Fedora 36 (and now 37 as I have since upgraded) does crash* on me ~once per week, but this is a far cry from Qubes (or AppVMs within Qubes) crashing on me several times per day. As far as I’m concerned, this rules out faulty hardware or that I just can’t “do Linux.”

My best guess is there is something about my M.2 NVMe drive (Samsung 970 EVO 2TB) that Qubes doesn’t like - perhaps a bad driver. Since Fedora 37 is much more stable than Qubes, this would imply a software issue rather than a hardware one. Does anyone know if Qubes is using an older driver for such a drive?

*Slightly off topic, but some of these crashes are while I’m working, but I do have a consistent problem with Fedora 36 and 37 crashing on me when I leave my system idle with rsync running an operation to a USB drive encrypted with VeraCrypt. I say slightly because I had a similar issue in Qubes as well. Let me know if anyone knows anything about this because it’s frustrating. I’ve even disabled auto screen locking and turning off, but it still freezes, leaving me to have to be working on the computer while I run a backup.

65

I would also like to point out that Fedora 36 and 37 Workstation successfully boots up with my USB-C dock plugged in, whereas Qubes OS would crash before I could get to the LUKS prompt ~90% of the time. I was never able to figure this one out either, but this further rules out hardware issues.

66

Wow, this sounds like a very bad experience.

On my side, I had no issues at all with my L14 (just the minor graphical issue with the LUKS password).

Did you use disposable sys-net?

What Cheese app do you mean?

Maybe a memory issue. Did your Qubes run in suspend mode?

1 Like
67

This looks like you are experiencing this problem:

However, I personally never experienced any crash on a Librem 14. You should indeed check your RAM and SSD.

1 Like
68

you certainly have a hardware issues. It is impossible that my qubes 4.1 works flawlessly and your doesn’t as a result of software. Well actually that is not exactly true because I did not install the Fedora templates so in theory I could have a better experience because of that. you could try an install with only Debian 11 but honestly I would not bother because you obviously have a hardware issue.

Well I guess there is another possible software issue which is that the green check mark when updating with the Qubes updater does not mean it actually worked. It can mean that the update timed out and did not install any update. Like I said I had this issue causing lots of problems like the ones you explained and I fixed it by reinstalling and doing my updates over clear net.
over tor I could update and get the green checks and immediately update again and I could have an infinite loop of updates that would always be available. this appears to be a update over tor issue so I set updated to clear net and they worked the first time.

If Fedora is crashing on you too it sounds like you have a bad stick of ram, what you describe is exactly what would be expected with a bad stick of ram. It would happen much more often with qubes because qubes uses a lot of ram where as with Fedora you could go a lot longer before your PC calls on the bad sector of the ram and it crashes.
you also may not have seated your ram properly when installed it which can cause a crash if you bump or move the PC while using it.

Also suspension problems are very common with linux. If you are suspending to disk that will crash almost every time if you do not have a large enough swap partition or file. If you have a 4gb swap and your PC is using less than 4gb ram at the moment you can suspend to disk and it should work but you said you have 64GB of ram so the only way to have suspend to disk aka hibernation work 100% of the time is to have more than 64GB of swap space.
This is a simplification as their is some compression involved but there is a 100% chance that hibernation/suspend to disk will cause your PC to crash if you did the guided partitioning during your install as your swap partition will be like 1GB something tiny like that.

It sounds like you either have a bad stick of ram or your trying to hibernate 64 gb of ram onto a 1gb swap partition. If you are trying to suspend to ram then you probably have a bad stick of ram because qubes suspend to ram works perfectly well for me and I dont have a swap at all.

With 64gb of ram you should hot have a swap either unless you are trying to use hibernation in which case you need a swap partition or file around 65GB if you want it to work 100% of the time. You can get it to work like 99% of the time if you make your swap about the size of the maximum amount of ram you typically use.

I would suggest no swap file for you as suspend to ram is great and hibernation is only useful if you want to suspend a lap top for weeks or months without plugging it in. So it is basically useless. If you will be not using your laptop and cant plug it in for days or weeks just turn it off and save yourself the hassle of trouble shooting hibernation.

But yea run a ramtest and see how that goes and look thru your bios and duck duck go the settings and turn off the ones that dont work with qubes. Instant boot for example will give Linux all kinds of problems and can be a default.

This is obviously not a qubes or Fedora issue. Computers dont have a mind of their own, if yours is not doing what it is supposed to do it is either set up wrong or something is broken. Simple as.

1 Like
69

Are you using PureBoot or CoreBoot?

If the problem was faulty RAM or SSD, wouldn’t I be having frequent issues on my now-installed Fedora 37 Workstation installation? I’m not opposed to checking this, but do you know of any good, trustworthy tools for this, preferably built into Fedora or available in its Software store?

70

Are you running CoreBoot or PureBoot?

I was not using a disposable sys-net.

Cheese is a default camera app in Fedora. Look in your Fedora AppVMs.

Maybe my memory is faulty, but if that is the case, why does Fedora 37 Workstation, which I’ve been running for the past few months, not seem to have issues with my memory? I’m not sure what Qubes suspend mode is, but I never suspend or put my PC in hibernation if that’s what you’re asking. Do you know of any good, trustworthy tools for this, preferably built into Fedora or available in its Software store?

71

PureBoot.

Memory check is built into the Qubes OS installer (which is based on Fedora 32).

Yes, although not necessary as frequent, because Qubes uses RAM in a much more heavy way. And it seems you do:

1 Like
72

I will try the memory check in the Qubes installer.

I do have a consistent issue on Fedora 36/37 when leaving it idle, but I’m not sure that’s indicative of faulty hardware because:

  1. It’s consistent, whereas hardware issues are usually intermittent
  2. I’ve seen this in Qubes, other Linux distributions, and my desktop PC, so I believe this particular stability issue I’m referring to is a common Linux problem, not a hardware problem.

However, it is a good point that Qubes is more memory-intensive, so I will look into this.

73

The memory test from the Qubes OS v4.1.1 installer would not run.

This is what happened:

  1. ISO was downloaded and verified per Qubes instructions
  2. ISO was written to USB via dd per Qubes instructions
  3. ISO was verified on USB to have the same sha256sum
  4. Booted to USB and chose the memtest option
  5. Output:
    Loading the new kernel:
    kexec -l /media/isolinux/memtest --append="intel_iommu=igfx_off "
    Cannot determine the file type of /media/isolinux/memtest
    Failed to load the new kernel
    !!! Failed to boot w/ options: Run a memory test|elf|kernel /isolinux/memtest
    !!! Something failed during USB boot

Then, I was taken to recovery shell. Any ideas?

74

You should have a memory test built right into your bios. It should be able to run without you even needing a hard drive installed or an operating system. If you search for or have the manual that cam with the motherboard you should be able to find it pretty easily.

With bad ram you will only have a crash when the bad sector of ram is called upon to remember something. It could be once in a year or once a day depending on how much ram you have and how much ram you are using. With qubes you will be using a ton of ram so Fedora crashing once a week and qubes crashing daily is entirely consistent with a bad sector of ram. That would be the first thing I checked out because it sounds very likely.
Did you buy the ram new or used? often people sell broken ram online because some people will never even notice it is broken and not be able to figure it out because of how random the crashes will be. If you dont use a lot of ram for your computing you could never even have an issue if you get lucky. With qubes you are probably going to crash often if you run a lot of qubes.

good luck!

2 Likes
75

I was eventually able to get debian “memtester” to run on my system by booting into Purism’s EC updater image and installing/running memtester. I have indeed been suffering from faulty memory.

Unfortunately, since I’m not running Qubes at the moment, and may not for some time due to my current work requirements, I can’t attest to which of these issues were caused be the faulty RAM. However, based on what I’ve read from other Librem 14 users, virtually all of these issues were likely due to the RAM issue (except for the graphical issues on the LUKS prompt and perhaps other minor issues). @fsflover and @anon11917472 appear to be wise to suggest that my memory has appeared to be stable on Fedora Workstation as compared to Qubes because Qubes is more memory-intensive, thus increasing the likelihood of reading/writing faulty sectors of the RAM.

For any prospective users, don’t let any of this discourage you from trying Qubes OS. This is a great operating system and I miss using it. Every time I download a PDF or image, for example, I wish I could quickly open it in a disposable VM (DVM). Every time I want to use Tor, I wish I could fire up a quick Whonix DVM. Someday.

For thread participants, thank you for your help and I apologize for wasting your time. I feel like an idiot for not checking this sooner, and have this on my list to check for any new hardware. Fortunately or unfortunately, in my decades of using computers on a daily basis, I’ve never dealt with faulty RAM. Being aware that Qubes can be picky about hardware, and the Librem 14 being relatively new and uncommon at the time, it seemed unlikely to me that this could have been the issue, but here we are. Better late than never I suppose.

@whoami, @fsflover: Thank you for your responses and confirming your lack of issues on the Librem 14.

2 Likes
76

Thanks for your help. Unfortunately, and surprisingly, there is no memory tester in PureBoot, nor can you boot most memory testers. Purism has confirmed that PureBoot users should switch to coreboot/SeaBIOS to be able to boot to something like memtest86+ for now, which is lame. However, one could boot some other text-based Linux image and run a memory test application, which is what I did. This is not a 1:1 replacement for something like memtest86+, but was fortunately able to confirm my RAM was faulty.

77

Update on this one - I no longer have this graphical issue on 4.1.0 with Kernel 5.15.81-1.fc32

2 Likes