NUC10i3FNK / installer: python3 storage job running out of memory

User Support General
1

Tried the latest ISO:

Qubes-20210424-x86_64.iso
24-Apr-2021 04:53
5408555008

and the previous ISO:

Qubes-20210417-x86_64.iso
17-Apr-2021 06:49
5408555008

In the installer every few minutes the python3 storage job

python3 -m pyanaconda.modules.storage

runs out of memory. In top on tty2 (shell) I see it running at 95+ % CPU usage and increasing memory percentage until it crashes and restarts.

I have 16 GB installed and it uses almost 4 GB for the installer.

From dmesg:

[  975.060113] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/anaconda.service,task=python3,pid=5829,uid=0
[  975.060146] Out of memory: Killed process 5829 (python3) total-vm:3374476kB, anon-rss:3100836kB, file-rss:5960kB, shmem-rss:0kB, UID:0 pgtables:6248kB oom_score_adj:0
[  976.409429] audit: type=1131 audit(1619353452.886:505): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=anaconda-tmux@tty1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[  976.416786] audit: type=1130 audit(1619353452.893:506): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=anaconda-tmux@tty1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[  976.416822] audit: type=1131 audit(1619353452.894:507): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=anaconda-tmux@tty1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[  976.599788] oom_reaper: reaped process 5829 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

checking for other details (syslog, storage log, . . .)

It logs that it found 56 devices and then it fails after processing md8. It never logs md9, the last item in the list.

Some raid devices are inactive. Then it waits longer before it processes the next.

Hans

2

I have added 16GiB so the total installed memory is 32GiB, now.

Now it can complete the storage module job and it was using 11 GiB.
Some time later 15GiB :

free -m

_____ total used free shared buff/cache available
Mem: 31117 23390 1735 249 5991 7174
Swap: 0 0 0

and most of that is used by the python3 process:

top | head -n 10

top - 12:10:14 up 26 min, 0 users, load average: 1.06, 0.95, 0.86
Tasks: 175 total, 2 running, 172 sleeping, 0 stopped, 1 zombie
%Cpu(s): 24.6 us, 1.5 sy, 0.0 ni, 73.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 31117.3 total, 2551.4 free, 22112.3 used, 6453.6 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 8450.0 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1870 root 20 0 21.2g 21.0g 21184 R 106.7 69.0 19:03.89 python3
1 root 20 0 170580 13088 9416 S 0.0 0.0 0:12.88 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd

Attached the anaconda.log (17.0 KB) , which shows that it took some 18 minutes to find out what storage exists. ( “56 devices”, see previous comment)

Maybe I can try running the same on a dom0 shell in Qubes to see what happens ?

Hans

3

Here is the output of xl info (with log extension to allow upload): xl-info.log (1.3 KB)

I specified this as xen command-line in the Grub boot menu:

dom0_mem=8GiB console=vga

Without options I still got the 4 GiB installer shell even though I had 32 GiB installed.

Hans