Persistant logging in Dom0 - please help

User Support
1

Hi all,

I’m currently trying to resolve an issue with my QubesOS install.

Presently when I boot my Librem Mini v2 with 2 monitors connected I get a freeze in the boot process about 90% of the time. I initially thought this issue was related to the bootloader, however it was later pointed out to me that the corruption seen on the screen occurs as Qubes is initializing i915.

Here is a link to my Github issue, which includes an image of the issue: https://github.com/QubesOS/qubes-issues/issues/8467

I now find myself trying to find the issue in logs, however that is turning out to be difficult.

As suggested I enabled debug verbose loglevel=7 in my grub at the end of GRUB_CMDLINE_LINUX, here is what my grub now looks like:

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=false
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-d1ab3523-ce8e-43e8-869d-dc0a54985a6f rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles intel_iommu=igfx_off 6.6.21-1.qubes.fc37.x86_64 x86_64 rhgb quiet debug verbose loglevel=7"
GRUB_DISABLE_RECOVERY="true"
GRUB_THEME="/boot/grub2/themes/qubes/theme.txt"
GRUB_CMDLINE_XEN_DEFAULT="console=none dom0_mem=min:1024M dom0_mem=max:4096M ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096"
GRUB_DISABLE_OS_PROBER="true"
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX usbcore.authorized_default=0"

I’m now trying to produce a log using journalctl, however this is where I am encountering another issue.

Assume i boot the computer 3 times:

  • Boot 1: This boot is fine with no issues
  • Boot 2: the issue occurs which forces me to have to restart the computer by holding down the power button until it turns off, then pressing the power button again to turn it on
  • Boot 3: This boot is fine with no issues (this is my current session)

I produce logs for this session for comparison:

journalctl -b > good_boot.txt

Then I want to produce logs for the bad boot for analysis:

journalctl -b -1 > bad_boot.txt

However journalctl -b -1 produces logs of Boot 1, I’m guessing because I do a hard shutoff via the power button it doesn’t persist the logs.

I then assumed the issue could be to do with persistent logging, therefore I enabled persistent logging by adding Storage=persistent to /etc/systemd/journald.conf, my journald.conf now is as follows:

[Journal]
Storage=persistent
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
#ReadKMsg=yes
#Audit=yes

However the issue still persists and I am unable to produce the logs of the ‘bad boot’ to diagnose the issue. Could someone please point me in the right direction as I am very stuck.

Is there a way to persist the journald logs?
Or is there another set of logs that I’m unaware of that would have captured the issue?

Thanks.

2

what if you browse journalctl output without parameter, can you find some traces of previous boots?

3

Sorry I don’t know what to look for if just type journalctl - can you tell me specifically what to do?

I can list the boots with journalctl --list-boots if that’s what you mean? Of course the logs from the bad boot are not present as I previously described

4

AFAIK journalctl already has persistence in /var/log/journal, so I am not
clear why those logs are not present.

You could try printing by date:
journalctl --since '2024-04-20' --until '2024-04-22
and see if anything is relevant.

I never presume to speak for the Qubes team. When I comment in the Forum I speak for myself.
5

I just tried your suggestion now and the ‘bad boots’ are missing from there aswell.

What else can I try?
Really need to get this issue resolved this issue is driving me crazy.

6

Do I understand correctly that when the problem occurs, you aren’t able to see/pass the LUKS prompt? If you type in the decryption passphrase blindly, does it seem like the system continues to boot (e.g. according to LED activity)?

/var/log/journal/ is on the dom0 root filesystem, so without somehow unlocking LUKS, journald just can’t save its logs to disk. Maybe try a USB debugging cable instead:

1 Like
7

Correct, I don’t even get to the LUKS prompt. If you take a look at my github issue there is an image posted showing the freeze occurring right as the boot screen is supposed to disappear Librem Mini v2 with dual monitors often freezes on boot · Issue #8467 · QubesOS/qubes-issues · GitHub.

However, I should have added this for context, what I typically do is turn on the computer with one monitor connected, wait for boot, then login/ pass the LUKS prompt, then right before I login to my user account I connect the second monitor and login to my user account.

If I plug in my second monitor at the LUKS prompt sometimes the screen will freeze or at least appear to freeze, I can type in my LUKS password hit enter and advance to the user login but I dont see any characters being typed into the LUKS password field as I’m typing. So to answer your question yes it does seem to continue to boot.

This is a good suggestion.

If I understand you correctly a possible reason for this is that logs arent being stored because saving to /var/log/journal/ is blocked by the LUKS encryption.

Could I not just reformat my computer with a fresh install of Qubes without LUKS, then diagnose the issue as the logs would be saved?

Or is there something else I’m not considering - I just want to check before I try a reformat?

8

That should work too, assuming that the installer currently lets you create an unencrypted installation. It’s been buggy in the past.

But why bother if you can already unlock LUKS by blindly typing in the passphrase? After that, either cleanly shut down the system somehow or wait a few minutes before hard powering it off to ensure the logs have been written to disk.

9

@rustybird So I reformatted and followed the same procedure as before with no LUKS this time and the logs from the ‘bad boots’ are still missing.

What else could it be causing the logs not to persist? if you take a look at my first comment in this post you will see my GRUB settings as well as my journald settings, is everything there correct ? Any other ideas?

10

Maybe the kernel is having an issue while booting from initramfs before even mounting your dom0 rootfs.
Do you have the same issue with Qubes OS 4.2.1 and latest kernel?
You can try to enable the debug output:

Or maybe there is a way to enable the debug output for kexec.

11

That’s the only thing I can think of as well. Although apparently @reknies can reach the login screen at least some of the time. :thinking:

Adding a plymouth.enable=0 kernel parameter (it might be necessary to also remove the rhgb parameter) to disable the boot process GUI is worth a try. Other than that, I’d get the debugging cable.

12

Hi all, just thought I’d update.
I tried updating Pureboot (I have done this multiple times in the past to no avail) and it seems to have resolved the problem.
The issue seems to stem from from using higher resolution displays.

Thanks all for your input was greatly appreciated.

1 Like