Qubes Memory Allocation Slow?

User Support General
1

I have several Xen VM that feels slower in past month. Mostly tied to browsers (firefox, chromium) and programming (vscode).
Resetting preferences helps, but watching xl info, xl top, and VM behavior (huge slow down/irresponsive 5-15min aka swapping), it seems VM preferences are not followed as I increased memory for those.
If I remember correctly, changed from Initial memory 400MB to 1G and maximum to 10 times that.
I have the impression that VM are never growing beyond the given initial memory.
VM console displays “system-journald: Under memory pressure, flushing cashes” and often get an oom-killer
xentop for the VM still show a MAXMEM around initial memory value at least for my non-disposable VM.
disposable VM seems to get their maximum value more often.
Or is it by load/VM start order?
I have sys-firewall fully allocated at 4GB whereas it has 400MB-GB as initial-max values and checking free -m inside 500-1024MB would be enough
dom0 is also at 4GB where 2GB is likely enough but can’t edit settings there
console disp get 4GB too even if started last…
I decrease a few of the non-disp ones and will see where it goes.

Got worse in recent weeks, maybe months.
Better after host reboot.

Similar threads

1 Like
2

What you see is most probably the notorious SELinux label bug which affects Fedora 41 templates. Most of the bugs are fixed in testing repositories. You could confirm it by switching to Fedora 40 or Debian templates and see if the issue remains or is solved. For more info:

And if you follow my weekly updates review newsletters (on forum), you will find the technical details and related patches.

3

Good catch. Thanks!
Trying latest reco sudo semodule -Bn and will see how it goes

1 Like
4

From past week, it seems not enough as issue is still happening after some time.
Reboot host as only workaround

5

How much RAM does your host have?
If you are using Firefox and Chromium then yes, you will have issues depending how many tabs you have open. I’m glad you didn’t say Chrome.

I have about 80 tabs oped myself, but not all are active. Runs in 2GB RAM per guest I have for it.
I run Vivaldi. It’s much better than any of the spyware bloated browsers.
It even has things removed from Chromium which makes it faster and better still, in many cases.

I generally run Debian, but I have not encountered your issue in Fedora when running Vivaldi.

Domain-0 can run with 512 MB RAM just fine. But I found that I needed to run it with 1 GB just to make things easier.

Are you running out of drive space?

What is your drive setup? is it SATA or NVME or PCIE?

You should check your hard drives as well. Run smartctl on them and see if they are failing. You may have either a drive failure, or even just a few bad sectors that need to be assigned so they aren’t utilised.

My VSCode guests I have set to 8 GB RAM each, since I use QEMU inside them to run certain virtuals for testing software.

Also, have you openned the window manager tweak settings to turn off and on the composer?

If you could, please provide your system specifications. (Example Below)
It will help us with some basics to maybe know the issue.
CPU:
Mobo:
RAM:
GPU:
Drive1:
Drive2:
Drive3:
Drive4:

---- EXAMPLE ----
CPU: AMD Ryzen 5 5600G
Mobo: MSI B550 Mortar II v.1.3
RAM: Corsair 3600Hz DDR4 Tripple Channel 4x8GB Dual Sided
GPU: NVIDIA 2060 6GB PCIEx16 Gen 4 running at x16
Drive1: ADATA SP900 250GB SATA [BOOT,O/S,SWAP]
Drive2: SAMSUNG 850EVO 1TB [QUBES (Mirrored)]
Drive3: SAMSUNG 850EVO 1TB [QUBES (Mirrored)]
Drive4: Seagate 4TB Seahawk [STORAGE GUEST]

6

Host is 32GB RAM with an average 10-15 instance including sys-*/basic stuff, 3-5 browsers.
Web stuff is mostly Firefox and sometimes chromium, occasionally brave, vivaldi and other but not decided. sometimes want to go to lynx/elinks but that’s a bit too much :wink:
disk space is fine.
no window manager tweak
host is about 10y old. without going to full details: i7-4790K, 32GB RAM, SSD, Integrated Video

7

Saying SSD, what SSD is it?
SATA? PCIE?

How many guests are doing a lot of activity at any time?

my sys-net has 512 MB RAM and 1 thread.
my firewall is the same.

My browser guests have 2GB assigned, static.
My VSCode has 8GB assigned.

I generally have the same number of guests dong things at all times as well.

I have multiple CPU pools assigned. I have the first 2 threads assigned to Domain-0, the next 4 threads are available to all the browsers and such, the last 6 are dedicated to my important guests such as my programming and compiling (Visual Studio Code, Lazarus, and others)

I run an AMD 2600X as well as 32 GB of DDR4 RAM, I run 2 SATA drives, one is O/S 240GB, second is QUBES drive. I run 3 monitors.
The machine is about 10 years old. Or getting close to.

I would possibly recommend you do a stress test on your machine to make sure nothing is overheating or causing issues, to find the bottleneck.

The CPU, specs wise, should be fine. Depending how much you are doing.
So I would advise assigning some pools for the guests and splitting them between the 2 schedulers like I do.

Much easier for things. I’ll get the instructions on how to do it for you along with my scripts to automate it when a guest is about to start.

The pools and schedulers aren’t saved when the system is rebooted or they are set, so it has to be done every time the host starts and the guests are booted they need to be assigned.

You are running DDR3.

More details on the PC would be advantageous.

An HCL would be good, a DMI would be fantastic.

If you could get an HCL and upload it for your machine that would be very helpful.

8

HCL uploaded Asus H87M
I put a partial dmidecode here Send
SSD like Solid State Drive
Not using pools historically (not needed…) - What is a qvm pool? Default VM POOL? - #3 by Lace ?

1 Like
9

Well, aparently you don’t have a drive in your system.

Can you send another DMIDECODE that isn’t partial? that one has no information about the system to assist with knowing anything much at all.

You haven’t told me what sort of SSD it is, you just say that it is an SSD. The DMI didn’t contain any drives at all. So either you have no drives at all in your system, which would make it all very slow if you are running everything from USB.

Just pm it to me will be fine if you don’t want to upload it to a third party spyware website like “vis.ee”.
I’m not talking about storage pools, I’m talking about CPU pooling.
They are different things.

10

Is the 860 an EVO or QVO or PRO? Is it M.2 or SATA connection? Is it an M or K key?

Lots of information that is unknown for answering questions.

Also, can you provide the partition layout of your drives please?

11

Disk does not appear in dmidecode for unknown reason. I have a drive.
From dmesg, “scsi 0:0:0:0: Direct-Access ATA Samsung SSD 860 2B6Q PQ: 0 ANSI: 5” . Not M2, too old for that.

$ sudo fdisk -l /dev/sda
Disk /dev/sda: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 860 
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: [REDACTED]

Device       Start       End   Sectors   Size Type
/dev/sda1     2048   1230847   1228800   600M EFI System
/dev/sda2  1230848   3327999   2097152     1G Linux filesystem
/dev/sda3  3328000 976773119 973445120 464.2G Linux filesystem
$ sudo lvdisplay -C
  WARNING: PV /dev/sdb1 in VG qubes_dom000 is using an old PV header, modify the VG to update.
  LV                                               VG           Attr       LSize   Pool      Origin                                           Data%  Meta%  Move Log Cpy%Sync Convert
  root                                             qubes_dom0   Vwi-aotz--  20,00g root-pool                                                  45,82                                  
  root-pool                                        qubes_dom0   twi-aotz--  24,00g                                                            38,19  18,29                           
  swap                                             qubes_dom0   -wi-ao----   3,93g                                                                                                   
  vm-Email-private                                 qubes_dom0   Vwi-a-tz-- <19,58g vm-pool   vm-Email-private-1741715155-back                 10,54
[...]
  vm-pool                                          qubes_dom0   twi-aotz-- 395,93g                                                            91,79  57,38

removed the various vm root/private/volatile…
I believe it’s installer auto format but may be wrong.

12

What version of Qubes are you currently running on it?

13

Current R4.2.4

14

I’ve got a couple of qubes installs on different computers and they have very different appvm startup times. Some are ridiculously slow. Initial memory stage for quite a while, also hangs waiting for terminal to pop up. Used to be that my installs were pretty similar across computers. They are all updated

15

@corny
There are times when it’s performing it’s regular updates check that things can get slow, but there are ways to prevent some slowness.
I often have RAM set as static, because they always use the maximum if you have them set to 4 GB anyway. I have some set to 1 and 2 GB of RAM.
They often take a few moments to boot up since I’m running SATA SSD on 6 Gbps SATA ports and I’m normally running multiple guests at once.

NetVM and FWVM I have with only 512 MB of RAM, and I have no issues on them.

Once the guests start, they then have the login and script execution times, this is where you may be seeing it be slow because of the script delay using all the CPU power for that guest before the terminal process starts and displays.

So for low RAM usage guests, just have it static. I think 95% of my guests are set as static because they all use 4GB for Fedora or Debian or Whonix anyway.
If they are a gateway or firewall, then 512 MB is enough. 1 GB if you feel you need to. I also trimmed down those guests as well, so they are small enough to not use much anyway, since all they do is route traffic.

I have the following pools…
CPU 0-1 (Domain-0) Scheduler 1
CPU 2-3 (Networking guests) Scheduler 1
CPU 4-6 (Web browsers and unimportant things) Scheduler 1
CPU 7-11 (Development/work/important guests) Scheduler 2

I generally run 8+ guests at any time.
512MB - NetVM (with NIC).
512MB - FW connected to NetVM.
512MB - InterVM guest connected to firewall for inter-guest communications.
512MB - Work Office VPN
1GB - Storage guest.
2GB - Dom0 (Sometimes I have it set to 1GB when I’m not going much on Domain-0)

Then I have the browsers…
2GB - Qubes Forum Guest
2GB - Temporary 1
2GB - Temporary 2
2GB - LinkedIN
2GB - Searcher
And others…
8GB - AI Development
2GB - Email Guest
2GB - Work Remote Connection
2GB - Work Office Storage

Generally on any guest the terminal takes about 3 seconds after the “started” message.
UNLESS they are installing packages after startup like my Viber guest. After it starts, the script installs Viber and dependencies from the local storage. I give it about 30 seconds to finish the DPKG runs for the installation.

Some others do other software as well. So the timing is extended because of the CPU utilisation from those guests. TOP normally shows that they hit 5 or 6 in a delay while they are doing the installs.

@juju42
I have a 4th generation PC here, I’ll install onto it and see where things go on that.

I still have no idea why you are having these issues though.

I’ll check it out and let you know what happens.

1 Like
16

Thanks a lot for all the information.

I think the slowness is memory related and not CPU. Unless fedora 40/41 is a lot more demanding on CPU than 38-39

Edit: At first, seemed faster now that initial and max are same higher number. I also turned off memory balancing on some. But when I tested a little later, it was back to being slow. On a subsequent test, I opened several disposables and it took forever, and half of them just failed and never started. I noticed that they were allocated 2000mb memory but then reduced automatically to 1984mb and then opened the terminal. During this time they were only at 10-20% cpu

What would be the purpose of default being to have very low initial ? If the default was higher, couldn’t memory management just be used as needed ? It seems like extra work for xen with little benefit (that I can perceive)

17

I have Fedora 40 and 41 as well, and have had no issues.

Memory managements is more for a guest that you use a general use, not one where you only have little uses for and set them for what you know they need.

But since they are using Debian and Fedora, it’s going to use whatever you have it set to because the installs of the systems is not that great.

If they used less resource heavy systems then it can work just fine.

I have my systems that I have 4GB maximum set for that only use 780 MB when here is a min of 512 MB.

So you have to take into consideration the many factors. But Linux in general, will run with 2-4 GB of RAM for anything you are doing normally.

Only the resource heavy machines you are using should be given a lot more.

If you have one guest that you use for many things, from coding to watching videos and browsing the web, then you may need a min/max. But in general you don’t need it. It only causes the operating system to load more modules for Qubes any way.

If you don’t use balooning then you don’t need to load those modules in the guest.

Honestly, you really don’t need that automatic management for anything.

Just set it to 4 GB in general as default and leave it at that…

All the SystemD variants of Linux are heavy on CPU and RAM compared to a non SystemD version. SystemD only causes issues and problems.

But they won’t change away from Fedora and other SystemD variants.

Still trying to convert the templates away from SystemD and have things still work properly.

1 Like