Ron387
January 12, 2024, 12:35pm
1
Hi there,
on Friday I tried to upgrade in-place (from 4.1) to Q4.2. Within Q4.2 the T490 then had several full freezes (screen is shown as normal, but no reaction to mouse movement or keystrokes at all).
Sometimes the system froze again right during forced reboot of the system, sometimes it was functional for hours until the next full freeze. Looking at journalctl I did not notice a recognizable pattern of log entries before the freeze - which may obviously be caused by the freeze to block the logged lines from being written to disk.
So I decided to give Q4.2 a second chance with a clean install (and import of appVMs). Sadly, I noticed the same kind of freezes. Now I replaced the M.2 and installed Q4.1 again, restoring a backup from friday. I did not notice freezes since then, but I only used the device for a few hours yet.
It looks like there is some compatibility problem of T490 and Q4.2. As looking at the logs did not help I have no idea how to debug this issue, so I write this here. As you know there will be now Q4.1 updates after this summer, so I obviously need to transition to Q4.2 somehow
thanks
Ron387
January 12, 2024, 12:54pm
3
Interestingly, there is a report of T490 failing to install at all at a specific step - I did not have issues during the Q4,2 setup on my T490
opened 03:17AM - 03 Jan 24 UTC
closed 03:30AM - 04 Jan 24 UTC
T: bug
R: duplicate
C: installer
P: default
hardware support
affects-4.2
[How to file a helpful issue](https://www.qubes-os.org/doc/issue-tracking/)
#… ## Qubes OS release
R4.2.0
### Brief summary
On a 10th-gen ThinkPad t490, the same one listed in [this HCL entry](https://www.qubes-os.org/hcl/#lenovo_thinkpad-t490-20rys0m200_i5-10210u_integrated-graphics-uhd-620_kevin-o-gorman_r4-0),
The Qubes install process fails to make it to the graphical installer, freezing with a black screen with the default boot option, and during/after initramfs and systemd starts as shown in the attached photo.
Some notes from troubleshooting:
- The same Thinkpad works with Qubes 4.0-4.1 with only the minor issues noted in the HCL (
- The result is the same when booting in Legacy or UEFI mode.
- The install USB was properly verified and used successfully to install Qubes 4.2.0 on another system (a 6th-gen X1).
- The t490's BIOS is up-to-date.
- The [UEFI troubleshooting docs](https://www.qubes-os.org/doc/uefi-troubleshooting/) look to be out-of-date (eg. there is no longer an `EFI/BOOT/BOOTX64.cfg` file to modify, just `grub.cfg`).
- I played around with kernel parameters to no real effect. Disabling modules changed the text output a bit, but the system still froze at around the same point.
### Steps to reproduce
- Create 4.2.0 install USB
- Attempt to install on Thinkpad T490 model num. 20RYS0M200
### Expected behavior
- Installation completes.
### Actual behavior
- Install process freezes before graphical installer is displayed.
Example of freeze point in verbose boot:

Ron387
January 12, 2024, 12:55pm
4
Thanks. When the Q 4.2 is bootable again I’ll try that kernel parameter.
Ron387
January 12, 2024, 2:08pm
5
ok, so I added the parameter to what I think is the xen commandline in /boot/grub2/grub.cfg right (before ${xen_rm_opt}, seperated by spaces ) How can I verify that the running system actually has hwp=off active?
Check scaling_driver
in the output of this command in dom0 terminal:
xenpm get-cpufreq-para
Ron387
January 12, 2024, 3:12pm
7
I get “failed to get cpufreq parameter” for CPU 1,3,5 and 7-15.
4 report acpi-cpufreq (I assume core 0,2,4,6).
The system has 4 physical cores. As qubes disables SMT for security reasons 4 cores is what I expected. likely 1,3,5,7 are the disabled SMT-cores, but I have no idea where 8-15 are from?!
With hwp enabled the scaling_driver
would be hwp-cpufreq
, since your scaling_driver
is acpi-cpufreq
it means hwp is disabled.
Ron387
January 12, 2024, 3:22pm
9
perfect, thank you. Any idea why 16 cpus are listed?
Ron387
January 12, 2024, 3:46pm
11
Ok. Thank you. I’m curios to see whether freezing is solved for T490 as well.
Ron387
January 21, 2024, 9:26pm
12
This solved the freezing for me, however it takes a noticeable amount of energy. With Qubes 4.1 after 3h of Idle I have about 60% battery remaining, with Qubes 4.2 and this Xen tweaks its down to about 40%.
You can report the issue here:
opened 10:23PM - 12 Dec 18 UTC
T: bug
C: Xen
P: major
r4.1-buster-stable
hardware support
r4.1-bullseye-stable
r4.1-dom0-stable
needs diagnosis
pr submitted
r4.1-centos-stream8-cur-test
r4.1-bookworm-stable
r4.2-host-cur-test
affects-4.1
affects-4.2
### Qubes OS version:
<!-- (e.g., `R3.2`)
You can get it from the dom0 te… rminal with the command
`cat /etc/qubes-release`
Type below this line. -->
R4.0
### Affected component(s):
intel_pstate
acpi-cpufreq
xenpm
---
### Steps to reproduce the behavior:
<!-- Use single backticks (`) for in-line code snippets and
triple backticks (```) for code blocks.
Type below this line. -->
Tested on:
- Lenovo T480s
- Lenovo X1 Carbon Gen. 6
- Huawei Matebook X Pro.
All with Intel i7-8550U.
Latest BIOS revisions for the respective systems as of Dec. 2018
Kernel: 4.19.2-3.pvops.qubes.x86_64.
EFI install.
In dom0, `sudo xenpm get-cpufreq-para`
### Expected behavior:
The processor is rated at 1.8 GHz (4.0 turbo), so we would expect to see appropriate scaling in that range, available frequencies from 1800000 - 4000000.
Further, we would expect to see `scaling_driver = intel_pstate`.
### Actual behavior:
The CPU frequencies do not scale correctly. Why?
Frequencies are pinned at 2 GHz max, 400 MHz min, across all cores.
```
# xenpm cpu-freq-para
...
cpu id : 0
affected_cpus : 0
cpuinfo frequency : max [2001000] min [400000] cur [2001000]
scaling_driver : acpi-cpufreq
scaling_avail_gov : userspace performance powersave ondemand
current_governor : ondemand
ondemand specific :
sampling_rate : max [10000000] min [10000] cur [20000]
up_threshold : 80
scaling_avail_freq : 2001000 2000000 1900000 1800000 1700000 1500000 1400000 1300000 1200000 1100000 1000000 800000 700000 600000 500000 *400000
scaling frequency : max [2001000] min [400000] cur [400000]
turbo mode : enabled
...
```
Confirmed with `watch -n1 "cat /proc/cpuinfo | grep \"[c]pu MHz\""`
`xenpm set-scaling-maxfreq` and `-minfreq` have no effect.
`xenpm get-cpufreq-states` shows 16 total/usable P-states.
Changing the governor to `performance` has no effect. Default is `ondemand`
`dmidecode` reports a max of 2 GHz on the Lenovos, and an apparently erroneous speed on the Huawei (~ 8 GHz).
```
# dmidecode | grep -i speed
Speed: 2400 MT/s
Configured Clock Speed: 2400 MT/s
Speed: 2400 MT/s
Configured Clock Speed: 2400 MT/s
Speed: Unknown
Speed: Unknown
Speed: Unknown
Max Speed: 2000 MHz
Current Speed: 1800 MHz
```
The `scaling_driver` is legacy `acpi-cpufreq`. Interestingly, `intel_pstate` can be seen initializing during boot, but it does not take over handling anything. Attempting to `blacklist acpi-cpufreq` in `modprobe.d` has no effect.
```
# dmesg | grep pstate
[ 5.067624] intel_pstate: Intel P-state driver initializing
```
`/sys/devices/system/cpu/intel_pstate/` contains the expected attributes, but as mentioned in the "related issue" linked below, `no_turbo`, `num_pstates`, and `turbo_pct` error `Resource temporarily unavailable`.
`/sys/devices/system/cpu/intel_pstate/status` always returns `off`, and does not respond to `echo "active" >`. This behavior has been tested with various kernel command line parameters, including `intel_pstate=force`, `intel_pstate=disabled`, `intel_pstate=no_hwp`, `intel_pstate=enable` with no change in performance aside from `../cpu/intel_pstate/` attributes disappearing when `no_hwp` or `disabled` were in effect. Also tried `processor.ignore_ppc=1`.
Strangely, none of the appropriate attributes for `cpufreq` exist in `/sys/devices/system/cpu/cpu*/`.
```
# ls /sys/devices/system/cpu/cpu0/
acpi_cppc driver hotplug power topology
cache firmware_node node0 subsystem uevent
```
`lsmod | grep cpufreq` shows no results, trying to `modprobe acpi-cpufreq` or `cpufreq-xen` returns errors. `xen_acpi_processor` is loaded.
```
# modprobe acpi-cpufreq
modprobe: ERROR: could not insert 'acpi_cpufreq': No such device
# modprobe cpufreq-xen
modprobe: FATAL: Module cpufreq-xen not found in directory /lib/modules/4.19.2-3.pvops.qubes.x86_64
```
`cpupower frequency-info` is completely unresponsive, with zero information available about the processor.
```
analyzing CPU 0:
no or unknown cpufreq driver is active on this CPU
CPUs which run at the same hardware frequency: Not Available
CPUs which need to have their frequency coordinated by software: Not Available
maximum transition latency: Cannot determine or is not supported.
Not Available
available cpufreq governors: Not Available
Unable to determine current policy
current CPU frequency: Unable to call hardware
current CPU frequency: Unable to call to kernel
boost state support:
Supported: yes
Active: yes
```
Though it shouldn't have any effect, testing was attempted with `smt=on` and `off`, and `Hyperthreading` enabled/disabled in the BIOS appropriately.
Testing was also performed while toggling various BIOS settings.
- enable/disable `Intel SpeedStep`
- power settings at `Maximum Performance` vs. `Balanced`
It does not appear to be a thermal throttling issue, with idle ~ 37*C and under load ~60*C observed consistently.
`tlp` was tested with no effect on the frequency scaling, regardless of being enabled or disabled. `tlp-stat` yields minimal additional info, with what seems to be an outdated recommendation for the Lenovos to install `tp-smapi kernel modules`, that are in fact deprecated in favor of `thinkpad_acpi`, which appears to be active on the Thinkpads.
```
dmesg | grep thinkpad
[ 19.589434] thinkpad_acpi: ThinkPad ACPI Extras v0.26
[ 19.589439] thinkpad_acpi: http://ibm-acpi.sf.net/
[ 19.589440] thinkpad_acpi: ThinkPad BIOS N22ET50W (1.27 ), EC unknown
[ 19.589441] thinkpad_acpi: Lenovo ThinkPad T480s, model 20L7CTO1WW
[ 19.591883] thinkpad_acpi: radio switch found; radios are enabled
[ 19.591898] thinkpad_acpi: This ThinkPad has standard ACPI backlight brightness control, supported by the ACPI video driver
[ 19.591899] thinkpad_acpi: Disabling thinkpad-acpi brightness events by default...
[ 19.612278] thinkpad_acpi: rfkill switch tpacpi_wwan_sw: radio is unblocked
[ 19.643468] thinkpad_acpi: Standard ACPI backlight interface available, not loading native one
[ 19.674512] thinkpad_acpi: battery 1 registered (start 0, stop 100)
[ 19.674576] input: ThinkPad Extra Buttons as /devices/platform/thinkpad_acpi/
```
`thermald` is not loaded.
### General notes:
https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_pstate.html
- This link suggests removing `irqbalance` but I'm skeptical.
https://askubuntu.com/questions/1067866/ubuntu-18-04-steam-games-frame-rate-drop/1073353#1073353?newreg=c7c120f373da4effb7317104571cd573
- https://cateee.net/lkddb/web-lkddb/XEN_ACPI_PROCESSOR.html
Regarding xen_acpi_processor: "It also registers itself as the SMM so that other drivers (such as ACPI cpufreq scaling driver) will not load."
How could `lsmod` report `xen_acpi_processor` as loaded but `xenpm` shows the scaling driver `acpi-cpufreq` ? This might make sense as to the missing `/sys/devices/.../cpufreq` entries.
- The following exchange is dubious at best, the final post gets down to the point of disabling intel microcode. They also suggest the use of `msr-tools`, but that really shouldn't be necessary.
https://bbs.archlinux.org/viewtopic.php?id=231077
- This is good work, but in my opinion, running a script every few seconds in dom0 isn't a legitimate fix.
https://github.com/erpalma/lenovo-throttling-fix
---
### Related issues:
https://github.com/QubesOS/qubes-issues/issues/4491
https://github.com/QubesOS/qubes-issues/issues/450