Lenovo ThinkPad T14 with AMD Ryzen CPU freezes when idle

[montgomery]

Hi,

I would like to ask for your help!

I have a Lenovo ThinkPad T14 Gen2 laptop with AMD Ryzen 7 pro 5850U CPU which I have been using since the end of 2022. Something like 6 months ago (or maybe 10, I really can’t remember) it randomly started to freeze. It freezed completely and had to push the power button for 5-10 secs to turn off the laptop. I tried to look for the cause in the journal file in dom0 but I did not find anything, even using loglvl=all guest_loglvl=all kernel parameters on startup.

After that I slowly started to recognize that the freezes are not random, it happens when the system was idle. Typical circumstances when the system freezes:

• after boot when waiting for the LUKS password (it is especially bad, when the laptop itself is cold, like going into the office from outside in winter. In those cases sometimes I do not have enough time to type in my password)
• at the login screen when waiting for the user password
• when I just leave the laptop turned on after I finished my work
• reading a static website
• watching the Sensor Viewer

Interestingly the system does not freeze:

• when I have a Windows VM on (I do not even have to log in, just start one)
• when I boot from a Windows installer USB
• when I tried Ubuntu Studio live USB
• before loading the linux kernel

However it freezed every single time:

• when I tried with Tails live USB
• when I tried Qubes 4.3 installer

I am ready to buy a new laptop if this is a hardware error, however I would like to understand first if this is a hardware error and if so is it a replaceable one like memory. I would appreciate if someone could help me to diagnose the problem.

Thanks
Monty

[corny]

Relevant ?

AMD iGPU issue after recent update

After a seemingly random amount of time, my display will cut out and typically not recover. The cutout doesn’t seem to correlate to any particular activity or lack of activity. Audio is unaffected which is essentially the only way I’m able to differentiate this from a system crash. I recently updated dom0 and switched to fedora 42 templates. I assume the template switch is unrelated and that the dom0 update is the cause. I’m using an AMD processor with the iGPU being used for output. Kernel ve…

[de_dust2]

Run https://www.memtest86.com/ on it and see if any memory errors are detected. You may have a hardware problem on your hands but replacing the machine may not be necessary, just the memory modules.

[montgomery]

I’m not sure, but don’t think so. My screen doesn’t goes blank, just freezes and there is no entry in the journal. It is like the whole system just stops working.

[montgomery]

I made the memory test, also made Lenovo’s built in memory test and none of them raised any issues.

[ChrisA]

Nice that memtest didn’t find any problems.

As I recall, some of the Lenovo Thinkpads with AMD CPUs got memory soldered to the mainboard – in that case, a faulty memory module could require a new mainboard … :-/

[KitsuneNoBaka]

Check BIOS for igfx turbo/power save. It made me such problem on intel igfx with modesetting driver and iris dri.

[HardcodedNonce]

@montgomery

However it freezed every single time:

• when I tried with Tails live USB

As the system freezes with tails and as the symptoms got worse if the system is cold, i would recommend using memtest and letting that run for 3 days if you can afford it.

Not to blame lenovos buildin RAM testing, but it is so bazingly fast that i doubt the results compared to a multi day memtest run.

[xone]

Hello, I have the same problem but different Thinkpad series. I have the exact same CPU. It was more noticeable around the time Qubes 4.3 released. I then decided to have a fresh install from 4.2 to 4.3 but still happens.
It is not consistent as I am able to leave my system idle but it has become more frequent. Additionally may not be relevant but I notice a new problem where my display goes blank and restarts randomly. Journalctl does not show anything anything for both problems. Did you figure out what happened?

[BigQ]

Same issue with my P14s Gen2 AMD, which has the same mainboard and cpu. Reproducable with Xubuntu or Windows.
Seems like an “production error” with that cpu and/or chipset?!

Or did you find a solution?

[montgomery]

Ok, I will do so in the upcoming weeks and get back once I have the results.

[montgomery]

I also have the new symptom of the screen going blank then a random restart. Still have no idea what’s going on.

[montgomery]

not yet. i am thinking of bringing it to a computer store to diagnose and resolve (if possible) the issue.

[xone]

Hello, I have good news. After installing to a new BIOS update, neither of the issues I had has occurred since.

[kxra]

Unfortunately for me (P14s Gen 3), a recent firmware update has reintroduced the problem. I reported it in the Lenovo Community:

https://forums.lenovo.com/t5/ThinkPad-P-and-W-Series-Mobile-Workstations/BIOS-Regression-CPU-idle-causes-random-freezes-lockup-System-unusable/m-p/10030329

and since this is the top search result for it, I am posting logs for greater reach:

The system journal with your kernel logs may read something like:

May 04 14:20:12 default-user audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-clean comm="systemd" exe="/usr/lib/systemd/systemd" >
May 04 14:20:12 default-user audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-clean comm="systemd" exe="/usr/lib/systemd/systemd" h>
May 04 14:23:05 default-user kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
May 04 14:23:05 default-user kernel: rcu:     Tasks blocked on level-0 rcu_node (CPUs 0-15): P321/1:b..l
May 04 14:23:05 default-user kernel: rcu:     (detected by 6, t=60002 jiffies, g=189157, q=15350 ncpus=16)
May 04 14:23:05 default-user kernel: task:kworker/6:1H    state:R  running task     stack:0     pid:321   tgid:321   ppid:2      task_flags:0x4208060 flags:0x00080000
May 04 14:23:05 default-user kernel: Workqueue: kblockd blk_mq_timeout_work
May 04 14:23:05 default-user kernel: Call Trace:
May 04 14:23:05 default-user kernel:  
May 04 14:23:05 default-user kernel:  __schedule+0x2b2/0x5d0
May 04 14:23:05 default-user kernel:  preempt_schedule_irq+0x38/0x60
May 04 14:23:05 default-user kernel:  asm_common_interrupt+0x26/0x40
May 04 14:23:05 default-user kernel: RIP: 0010:blk_mq_timeout_work+0x53/0x1e0
May 04 14:23:05 default-user kernel: Code: 00 48 c7 44 24 08 00 00 00 00 48 8b 05 b6 49 da 01 48 89 44 24 10 e8 3c f6 86 ff 48 8b 85 88 fe ff ff a8 03 0f 85 3b 01 00 00 <48> 89 5c 24 20 65 48 ff 00 e8 5f 5b 8>
May 04 14:23:05 default-user kernel: RSP: 0018:ffffca4dc07e3e08 EFLAGS: 00000246
May 04 14:23:05 default-user kernel: RAX: ffffffffa716d6b8 RBX: ffff893a41713e00 RCX: ffff893a41713e40
May 04 14:23:05 default-user kernel: RDX: 0000000000000000 RSI: ffff893a6b054b60 RDI: ffff893a6b054b58
May 04 14:23:05 default-user kernel: RBP: ffff893a6b054b58 R08: 8080808080808080 R09: ffff893a41713e80
May 04 14:23:05 default-user kernel: R10: ffff893a401d9ac0 R11: fefefefefefefeff R12: ffff8940a1d32358
May 04 14:23:05 default-user kernel: R13: ffff893a417d0205 R14: 0000000000000000 R15: ffff893a6b054b58
May 04 14:23:05 default-user kernel:  ? blk_mq_timeout_work+0x44/0x1e0
May 04 14:23:05 default-user kernel:  ? __pfx_blk_mq_timeout_work+0x10/0x10
May 04 14:23:05 default-user kernel:  process_one_work+0x190/0x350
May 04 14:23:05 default-user kernel:  worker_thread+0x18d/0x2f0
May 04 14:23:05 default-user kernel:  ? __pfx_worker_thread+0x10/0x10
May 04 14:23:05 default-user kernel:  kthread+0xfa/0x240
May 04 14:23:05 default-user kernel:  ? finish_task_switch.isra.0+0x82/0x2a0
May 04 14:23:05 default-user kernel:  ? __pfx_kthread+0x10/0x10
May 04 14:23:05 default-user kernel:  ? __pfx_kthread+0x10/0x10
May 04 14:23:05 default-user kernel:  ret_from_fork+0x130/0x1a0
May 04 14:23:05 default-user kernel:  ? __pfx_kthread+0x10/0x10
May 04 14:23:05 default-user kernel:  ret_from_fork_asm+0x1a/0x30
May 04 14:23:05 default-user kernel:  </TASK>

or a tty with repeating:

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu:    6-....: (1 GPs behind) idle=7afc/1/0x4000000000000000 softirq-261861/261862 fqs=120756
rcu:    (t=60000 jiffies g=734565 q=515 ncpus=16)
CPU#6 Utilization every 4000ms during lockup:
        #1:   1% system,          9% softirq,    92% hardirq,     0% idle
        #2:   1% system,          9% softirq,    92% hardirq,     0% idle
        #3:   1% system,          9% softirq,    92% hardirq,     0% idle
        #4:   1% system,          9% softirq,    92% hardirq,     0% idle
        #5:   1% system,          9% softirq,    92% hardirq,     0% idle
CPU#6 Detect HardIRQ Time exteeds 50%, Most frequent HardIRQs: 
        #1: 3790790     irq#7
        #2:3790790      irq#77

[BigQ]

What Thinkpad / model and which BIOS Version? Please as precise as possible

My P14s AMD Gen 2 (21A1) still freezes with BIOS 1.33.
As it didn’t froze in the Qubes Installer (idle for about 2h) as before (max 15min idle in the Installer and 100% freeze), I was already euphoric, just to end up frozen after a few minutes idle on the Qubes Desktop.

Have to try out BIOS 1.34.

[BigQ]

Okay, after installing Qubes 4.3 with Option “… latest Kernel” and Debian as default instead of Fedora, the Thinkpad had no freeze for about 1-2h just idle with firefox open, without any updates.
Now I updated to R4.3.1-rc1 and Kernel 6.19.14-1 and leave it idle for some more time … keep fingers crossed!

Edit: Still on BIOS 1.33, I gave it a last try.

[537390726]

Maybe this is related?

https://gist.github.com/dlqqq/876d74d030f80dc899fc58a244b72df0

Try adding this to /etc/default/grub (in dom0):

# Fix Ryzen system freeze bug by limiting C-State
GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX processor.max_cstate=1"

Then, in dom0, regenerate the grub configuration file with:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot your machine, and the freezing will be gone.

[BigQ]

I don’t think so … for me the issue started after more then 3 years without any problems, something changed. The mentioned bug exists for 6 years. And with Windows (and Ubuntu) my Device had the exact same idle freezes.

Over 4h idle and no freeze yet. I continue testing …

[BigQ]

After 15h sleep/standby the P14s resumed as it should and still had no idle freeze!!! I rebooted (willingly ^^) to check BIOS and saw that I had set the UMA Frame Buffer to 4GB instead of Auto. I reduced that to 2GB and still no freezes. I don’t think that matters, just to be sure and to document all settings/changes.

The P14s drains around 1-2% per hour while sleeping, but still much better then my E16 G3 with 4-5% (>48h vs max 24h standby). Thought it was better, but I’m maybe just confusing things.