I’m using the Lenovo ThinkPad X1 Carbon Gen 11 . The BIOS doesn’t support s3 sleep so the only option is s2idle.
Out of the box, Qubes OS 4.2 RC3 works well but doesn’t suspend. I tried upgrading dom0 to kernel-latest but the problem persists. Screen turns off, fans stop spinning, then screen flashes on once or twice, fans start spinning again, and the lock screen appears.
Kernel logs show that the suspend was cancelled due to a hardware issue:
Sep 23 17:20:20 dom0 kernel: PM: suspend exit
Sep 23 17:20:20 dom0 kernel: PM: suspend entry (s2idle)
Sep 23 17:20:20 dom0 kernel: Filesystems sync: 0.007 seconds
Sep 23 17:20:21 dom0 kernel: Freezing user space processes
Sep 23 17:20:21 dom0 kernel: Freezing user space processes completed (elapsed 0.002 seconds)
Sep 23 17:20:21 dom0 kernel: OOM killer disabled.
Sep 23 17:20:21 dom0 kernel: Freezing remaining freezable tasks
Sep 23 17:20:21 dom0 kernel: Freezing remaining freezable tasks completed (elapsed 0.059 seconds)
Sep 23 17:20:21 dom0 kernel: printk: Suspending console(s) (use no_console_suspend to debug)
Sep 23 17:20:21 dom0 kernel: proc_thermal_pci 0000:00:04.0: failed to save offset (-5)
Sep 23 17:20:21 dom0 kernel: iosm 0000:08:00.0: msg timeout
Sep 23 17:20:21 dom0 kernel: PM: suspend devices took 0.698 seconds
Sep 23 17:20:21 dom0 kernel: intel_pmc_core INT33A1:00: PM: dpm_run_callback(): acpi_subsys_suspend_late+0x0/0x50 returns -5
Sep 23 17:20:21 dom0 kernel: intel_pmc_core INT33A1:00: PM: failed to suspend late: error -5
Sep 23 17:20:21 dom0 kernel: PM: late suspend of devices failed
...
Sep 23 17:20:21 dom0 kernel: i915 0000:00:02.0: [drm] GT0: GuC firmware i915/adlp_guc_70.bin version 70.5.1
Sep 23 17:20:21 dom0 kernel: i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
Sep 23 17:20:21 dom0 kernel: nvme nvme0: Shutdown timeout set to 10 seconds
Sep 23 17:20:21 dom0 kernel: nvme nvme0: 10/0/0 default/read/poll queues
Sep 23 17:20:21 dom0 kernel: i915 0000:00:02.0: [drm] GT0: HuC: authenticated!
Sep 23 17:20:21 dom0 kernel: i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
Sep 23 17:20:21 dom0 kernel: i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
Sep 23 17:20:21 dom0 kernel: i915 0000:00:02.0: [drm] GT0: GUC: RC enabled
Sep 23 17:20:21 dom0 kernel: iosm 0000:08:00.0: msg timeout
Sep 23 17:20:21 dom0 kernel: PM: resume devices took 0.510 seconds
Sep 23 17:20:21 dom0 kernel: mei_hdcp 0000:00:16.0-b638ab7e-94e2-4ea2-a552-d1c54b627f04: bound 0000:00:02.0 (ops i915_hdcp_ops [i915])
Sep 23 17:20:21 dom0 kernel: OOM killer enabled.
Sep 23 17:20:21 dom0 kernel: Restarting tasks ...
Sep 23 17:20:21 dom0 kernel: mei_pxp 0000:00:16.0-fbf6fcf1-96cf-4e2e-a6a6-1bab8cbe36b1: bound 0000:00:02.0 (ops i915_pxp_tee_component_ops [i915])
Sep 23 17:20:21 dom0 kernel: done.
Sep 23 17:20:21 dom0 kernel: random: crng reseeded on system resumption
Sep 23 17:20:22 dom0 systemd-sleep[14852]: Failed to put system to sleep. System resumed again: Input/output error
The most interesting parts to me are:
acpi_subsys_suspend_late+0x0/0x50 returns -5
intel_pmc_core INT33A1:00: PM: failed to suspend late: error -5
But I don’t know what device this could be that’s failing to suspend. I tried disabling all optional devices in BIOS(webcam, fingerprint reader, WiFi, etc.) without success.
Does anyone have ideas on how I could troubleshoot this further?
balko
September 23, 2023, 3:43pm
2
Updated: sorry, I misunderstood your question, I though you were trying to make S0x suspend work which is still not supported.
Hmm, according to kernel docs s2idle maps to ACPI state S0. Does this mean your comment still applies and I should expect s2idle is fully unsupported for now?
balko
September 23, 2023, 6:37pm
4
Yes, it seems to be the same thing, just different names (Intel calls it S0ix).
Unfortunately it is unsupported in Qubes OS, nothing can be done at this point by usual user.
Follow the progress here:
opened 02:15PM - 17 Feb 21 UTC
T: enhancement
C: Xen
P: major
hardware support
waiting for upstream
C: power management
***Editor's note:** The original issue description below is somewhat out-of-date… . This issue is now narrowly focused on support for the S0ix sleep state (aka "Modern Standby," "Low Power S0 Idle," "InstantGo," "Connected Standby," etc.) found in newer CPUs (especially Tiger Lake and later).*
-----
**The problem you're addressing (if any)**
It appears Qubes does not support Intel's latest version of processors.
**Describe the solution you'd like**
Support for Intel's latest version of their processors (i.e., tiger lake)
**Where is the value to a user, and who might that user be?**
People who want to benefit from the significant performance upgrades of Tiger Lake
**Describe alternatives you've considered**
N/A
**Additional context**
N/A
**Relevant [documentation](https://www.qubes-os.org/doc/) you've consulted**
N/A (I've checked GitHub for any open issues on this but either I've missed it or my SearchFu is weak)
**Related, [non-duplicate](https://www.qubes-os.org/doc/reporting-bugs/#new-issues-should-not-be-duplicates-of-existing-issues) issues**
N/A
Thanks for the link, I will follow the progress.
Sad to hear this is not supported, because the newer ThinkPad machines like mine don’t have any option for S3 suspend so the only option we have on these devices is a full shutdown.
balko
September 24, 2023, 10:47am
6
Actually they do have S3, e.g. Thinkpad T16 Gen1 and many others. The available BIOS/EFI settings (including S3 support) can be check on the Lenovo web site, they have BIOS/EFI emulator available in the browser.
The Lenovo BIOS simulator is new to me, thanks for sharing that!
Yes I see that the T16 Gen1 and Gen2 have the option Config > Power > Sleep State in the BIOS. Looks like this:
Even the X1 Carbon Gen 10 has this option, but unfortunately not the Gen 11.
Mark Pearson from the Lenovo team shared some background on why the S3 option was removed :
Most of our platforms (with the exception of a couple of workstations) are now S0ix only.
We did dual sleep support with both S3 and S0ix as an option in the BIOS (with S3 as ‘best effort’ for users who didn’t want to switch) on our Linux certified Intel platforms for a few years - and it was a nightmare.
…
We found many S3 issues would creep in with FW updates - devices would stop working on resume, system wouldn’t sleep properly and battery drain in a few cases were horrible. Getting fixes done took forever and we couldn’t delay FW updates for a sleep mode that was supposed to be ‘best effort’. Users were frustrated (understandably) and it was not a good experience for anybody. We were honestly trying to do the right thing - but it wasn’t working.
We made the decision to stop doing S3 support last year and to remove the option.
So it looks like issue #6411 that you shared will be increasingly important for Lenovo laptop users in future.
balko
September 25, 2023, 5:55am
9
I think, it is already through the roof for laptop users.
As far as I know, Intel said something similar to “Intel CPUs of 11-12 Gen will not support S3”, but it was a lie, as we see S3 on some Thinkpads with Intel 12 Gen CPUs, including T16. But those Thinkpads are exceptions nowadays, most of laptop manufactures decided to drop S3 support as Intel advised.
Not obvious what/whom he blames, their own firmware developers of their own laptop’s hardware and devices they choose? I mean S3 existed for years, working perfectly well on many devices (but not all). Why would one blame S3 instead of firmware of devices they choose and use for being too unreliable and full of bugs within deep sleep? Looks a bit strange to me.
Thanks!
So I got the gen 11 now too, I can boot from the qubes installer from the flash drive and go through the install, but once its done installing and I’m asked to reboot, nothing happens - I then get stranded in a Lenovo boot menu like no OS is installed.
Any advice? Did you have any similar issues?
Did you install Qubes OS on removable disk?
If yes then follow these:
opened 09:26AM - 02 Jun 22 UTC
closed 03:55AM - 12 Mar 24 UTC
P: default
pr submitted
r4.2-host-stable
C: boot
[How to file a helpful issue](https://www.qubes-os.org/doc/issue-tracking/)
#… ## Qubes OS release
Latest as of posting, 4.1.0
### Brief summary
I manually unplugged every drive on my computer except for the target drive for the installation, which was all free space unpartitioned. Installer ran typically, chose automatic for the installation type. Booted into the OS' configuration and then the full OS.
GRUB is verified to have worked because I managed to see it when initially booting for configuration and because of a later restart due to a display artificating and then system lockup problem I had that I attribute to Nvidia.
After using another drive and then unplugging it, and plugging the Qubes drive back, there was no detectable boot source.
I booted a live OS and found that the partition structure was intact but didn't investigate further and reinstalled Qubes.
I then got the idea to see if this issue happened if I BIOS disabled drives instead of unplugging them. I disabled my Qubes drive and enabled another, and booted into it fine. I then disabled my other drive and reenabled the Qubes drive and the same issue happened again where my system does not see any boot entries.
The system does see the drive, and switching on CSM and legacy boot I can boot from the drive but get an error screen saying to boot from proper media.
This second time, I _kind of_ fixed it but I must do these steps every time I remove the drive or disable it in BIOS. I lose the option to boot without the Xen hypervisor and the boot hiccups. I essentially have to boot my install media, run Anaconda rescue and:
1. Mount the bootloader partition
2. Copy the contents of /mnt/EFI/qubes/ to /mnt/EFI/BOOT/
3. Rename grubx64.efi to bootx64.efi
4. Rename grub.cfg to bootx64.cfg
5. efibootmgr -v -c -u -L Qubes2 -l /EFI/BOOT/bootx64.efi -d /dev/sda -p 1
### Steps to reproduce
I believe the best way to reproduce this issue is to do a fresh install of Qubes on hardware similar to mine:
Asus Prime Z-390-A Motherboard
Intel i7-9700K CPU
16GB DDR4 RAM
1TB HDD
Most notably, this motherboard. I will be filing a compatibility report soon, but I believe this motherboard may have something to do with how it reads the GRUB bootloader as it's configured for Qubes.
Further, these are the only non-stock BIOS settings:
- Disabling the Intel LAN Controller (it causes a PCI reset error and ends the last step of the configuration, doesn't matter since I don't use it, and I believe the fact that it's unplugged is the reason this issue happens)
- Enabling Virtualization
- Enabling Vt-d
- Disabling all other drives
### Expected behavior
The GRUB bootloader is supposed to appear after the BIOS initializes regardless of whether the drive was previously unplugged from the system or not (safely of course).
### Actual behavior
Please read above.
After unplugging the drive or disabling it in BIOS, the entry for GRUB is missing.
opened 02:53PM - 15 Jul 23 UTC
closed 03:55AM - 12 Mar 24 UTC
P: default
diagnosed
pr submitted
r4.2-host-stable
affects-4.1
affects-4.2
C: boot
### Qubes OS release
R4.1 and R4.2
### Brief summary
When making a raw … disk backup from a Qubes installed to an internal hard drive to an external hard drive, the external hard drive is unbootable.
raw disk backup means a backup using `dd` or 1 to 1 exact copy.
### Steps to reproduce
1. install Qubes normally on a computer that only support EFI booting on the internal harddrive
2. reboot
3. brief test that Qubes is working normally (yes)
4. shutdown Qubes
5. boot from an external drive, boot a live DVD or live USB such as Debian Live
6. perform a raw disk backup from the internal disk to the external disk
7. unplug that disk and try to boot from it in a different computer (or the same Qubes computer)
([example instructions for raw disk backups](https://www.kicksecure.com/wiki/Raw_Disk_Backup))
### Expected behavior
The raw disk backup of Qubes is bootable.
### Actual behavior
The raw disk backup of Qubes is unbootable.
### Additional information
According to my research that might be because of missing entries in the EFI firmware's NVRAM which is stored on the motherboard. Unfortunately, the EFI boot process doesn't seem by default to be self-contained on 1 harddrive but require extra settings stored outside harddrives on the motherboard (EFI NVRAM).
Using `grub2-install` with options `--removable` / `--force-extra-removable` during Qubes installation might help?
> [--removable](https://manpages.debian.org/bookworm/grub2-common/grub-install.8.en.html#removable)
> the installation device is removable. This option is only available on EFI.
> [--force-extra-removable](https://manpages.debian.org/bookworm/grub2-common/grub-install.8.en.html#force~2)
> force installation to the removable media path also. This option is only available on EFI.
Qubes does not have good support for multiboot support anyhow:
* https://github.com/QubesOS/qubes-issues/issues/8351
* https://www.qubes-os.org/faq/#can-i-install-qubes-os-together-with-other-operating-system-dual-bootmulti-boot
* It's "patches welcome". (And that's okay.)
This ticket is not a feature request to improve multiboot support. Why do I mention this? Because otherwise, when considering options `--removable` / `--force-extra-removable` one *might* argue "but that breaks mutliboot support". I would argue that being able to boot a raw disk backup of Qubes is more important than mutliboot support.
[Why do I like full raw disk backups? See this link.](https://forum.qubes-os.org/t/how-do-you-organize-your-backups/3986/16?u=adrelanos)
Manually updating the NVRAM can be challenging:
* It's vendor dependent because of many different EFI BIOS versions.
* Some BIOS don't even have such an option.
* Might require booting into an operating system installed on USB and running console commands to fix it.
* Difficult, impossible for far most users to do or even to find instructions for it.
Qubes with legacy BIOS booting as far as I remember didn't have this issue. However, never notebooks sometimes (or often, dunno) don't even support legacy BIOS booting anymore. Therefore "use legacy BIOS booting" is a non-solution. Also not a good long term solution in either case because of Qubes planned Secure Boot support.
This bug might also make Qubes "non-portable". Meaning, Qubes installed to an external drive such as a USB SSD might be bootable on the computer where it was installed but unbootable when attempting to boot the same Qubes USB SSD on another computer. I didn't test this very part mentioned in this very chapter. Anaconda by Fedora might be using `grub2-install` with options `--removable` / `--force-extra-removable` already when installing to external devices (USB) instead of internal harddrive.
Where is the Qubes code which defines bootloader / grub2 / EFI installation? Or is this currently all done by upstream's Anaconda?
If it’s internal disk then follow this:
If you use uefi :
get a qubes os media installation.
boot to qubes os rescue mode.
skip to shell.
[shell] : efibootmgr -v -c -u -L "Qubes OS" -l /EFI/qubes/grubx64.efi -d /dev/nvme0n1 -p 1
reboot
change -d <which drive?> and -p <which efi partition?>.
if you want you can make sure first by issuing efibootmgr -v in dom0
abbedda
January 10, 2024, 10:45am
13
Thank you! Its installeres on an interval disk, my next question is How to apply these changes, I found the menu but my prompts are not working
Then follow the second guide, this one:
If you use uefi :
get a qubes os media installation.
boot to qubes os rescue mode.
skip to shell.
[shell] : efibootmgr -v -c -u -L "Qubes OS" -l /EFI/qubes/grubx64.efi -d /dev/nvme0n1 -p 1
reboot
change -d <which drive?> and -p <which efi partition?>.
if you want you can make sure first by issuing efibootmgr -v in dom0
Not sure what are you talking about. Can you describe in more details?
abbedda
January 10, 2024, 11:40am
15
I follow the guide, write reboot and it ends up the same place. Here are some pics of what I see
The screen showing when I turn on the computer (before and after following the guide)
abbedda
January 10, 2024, 11:40am
16
My input and answer when following the guide
abbedda
January 10, 2024, 11:41am
17
My boot settings
Boot order is:
Harddrive
USB HDD
USB FDD
Windows boot manager
Intel Vt-d is turned on
Boot order lock is on
Boot mode is: Diagnostics
Did you make sure that /dev/nvme0n1 is your Qubes OS drive?
Did you check it? Using fdisk -l / blkid / tried to mount it / or something else.
abbedda
January 10, 2024, 11:47am
19
No, im quite new to Linux, how can I do that?
As I see it I can not even boot into the Qubes OS environment from where I can open a terminal
Run these commands:
fdisk -l
blkid
In the rescue mode shell before running efibootmgr and check the output.
The output of fdisk for your Qubes OS drive with default partitioning should have 3 partitions and look like this:
It could be a missing EFI Boot entry. Can you boot from an USB drive check the output from
fdisk -l /dev/sda
fdisk -l /dev/sdb
fdisk -l /dev/nvme0n1
?
– one of those is probably the USB stick … but the other could be your Qubes OS drive. If one of your drives looks like:
Device Start End Sectors Size Type
/dev/nvme0n1p1 2048 1230847 1228800 600M EFI System
/dev/nvme0n1p2 1230848 3327999 2097152 1G Linux filesystem
/dev/nvme0n1p3 3328000 976773119 97344512…
abbedda
January 10, 2024, 11:57am
21
It looks like that
Any other suggestions, @apparatus ?
And just so you know it, I highly appreciate your help here - thanks SO much
ChrisA
January 10, 2024, 2:56pm
22
I assume you have the installation stick around still .. so could you try:
Connect the USB installation stick to the computer
Select the Installation stick as the boot medium
When you see the “Install Qubes OS R…”/“Test media and …”/“Troubleshooting - …” menu, hit c to get a grub> prompt
Type configfile (hd0,gpt1)/EFI/qubes/grub and hit the Tab key on your keyboard – did it complete the line to: grub.cfg?
– if so, hit enter .. if no, try with: configfile (hd1,gpt1)/EFI/qubes/grub and hit Tab
and report/share the result?