QubesOS freeze, crash and reboots

FudgingDonut · October 28, 2022, 1:47pm

5.18.16 seems pretty stable to me too. I’ve had a system running it for a couple of days with no freezing issues. I used to get crashes often while running salt states, but I’ve ran a lot of them without any issues on 5.18.16. I still have sysrq enabled so I can avoid corruption from a hard reset.

Isolator · October 29, 2022, 1:09am

Yes, I’ve experienced both a freeze and a crash and I think those are two unrelated issues that exhibit different behaviors.
A freeze leaves no logs and results in a massive system load for whatever reason. That’s when people report temperature issues. I always have cpu temp and rpm sensors on my panel - It idles at around 32 and never goes above 50 under stress. But when qubes froze my fans immediately started spinning like never before.

A crash on the other hand does not result in any noticeable system load whatsoever, and it leaves logs related to grant tables xen issue like here

seberm:

The trace I am getting is following:

Oct 17 15:47:53 dom0 kernel: BUG: Bad page map: 992 messages suppressed
Oct 17 15:47:53 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000adaf14365 pmd:135f77067
Oct 17 15:47:53 dom0 kernel: page:0000000074dfd1dd refcount:1025 mapcount:-1 mapping:0000000000000000 index:0x0 pfn:0x195d94
Oct 17 15:47:53 dom0 kernel: flags: 0x27ffffc0003408(dirty|owner_priv_1|reserved|private|node=0|zone=4|lastcpupid=0x1fffff)
Oct 17 15:47:53 dom0 kernel: raw: 0027ffffc0003408 ffff88810376d300 ffffea0006576540 0000000000000000
Oct 17 15:47:53 dom0 kernel: raw: 0000000000000000 0000134500000007 00000401fffffffe 0000000000000000
Oct 17 15:47:53 dom0 kernel: page dumped because: bad pte
Oct 17 15:47:53 dom0 kernel: addr:00007ed79a637000 vm_flags:1c0600f9 anon_vma:0000000000000000 mapping:ffff888100290508 index:7af
Oct 17 15:47:53 dom0 kernel: file:gntdev fault:0x0 mmap:gntdev_mmap [xen_gntdev] readpage:0x0
Oct 17 15:47:53 dom0 kernel: CPU: 2 PID: 6715 Comm: Xorg Tainted: G    B   W         5.18.16-1.fc32.qubes.x86_64 #1
Oct 17 15:47:53 dom0 kernel: Hardware name: LENOVO 20TJS2F44A/20TJS2F44A, BIOS N2VET37W (1.22 ) 01/18/2022
Oct 17 15:47:53 dom0 kernel: Call Trace:
Oct 17 15:47:53 dom0 kernel:  <TASK>
Oct 17 15:47:53 dom0 kernel:  dump_stack_lvl+0x45/0x5e
Oct 17 15:47:53 dom0 kernel:  print_bad_pte.cold+0x6a/0xc5
Oct 17 15:47:53 dom0 kernel:  zap_pte_range+0x430/0x8b0
Oct 17 15:47:53 dom0 kernel:  ? __raw_callee_save_xen_pmd_val+0x11/0x22
Oct 17 15:47:53 dom0 kernel:  zap_pmd_range.isra.0+0x1b8/0x2f0
Oct 17 15:47:53 dom0 kernel:  zap_pud_range.isra.0+0xa9/0x1e0
Oct 17 15:47:53 dom0 kernel:  unmap_page_range+0x16c/0x200
Oct 17 15:47:53 dom0 kernel:  unmap_vmas+0x83/0x100
Oct 17 15:47:53 dom0 kernel:  unmap_region+0xbd/0x120
Oct 17 15:47:53 dom0 kernel:  __do_munmap+0x177/0x350
Oct 17 15:47:53 dom0 kernel:  __vm_munmap+0x75/0x120
Oct 17 15:47:53 dom0 kernel:  __x64_sys_munmap+0x17/0x20
Oct 17 15:47:53 dom0 kernel:  do_syscall_64+0x59/0x90
Oct 17 15:47:53 dom0 kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Oct 17 15:47:53 dom0 kernel: RIP: 0033:0x7ed7a34e237b
Oct 17 15:47:53 dom0 kernel: Code: 8b 15 21 6b 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb 89 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 0b 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed 6a 0c 00 f7 d8 64 89 01 48
Oct 17 15:47:53 dom0 kernel: RSP: 002b:00007fff098b4488 EFLAGS: 00000206 ORIG_RAX: 000000000000000b
Oct 17 15:47:53 dom0 kernel: RAX: ffffffffffffffda RBX: 0000000000000055 RCX: 00007ed7a34e237b
Oct 17 15:47:53 dom0 kernel: RDX: 00007fff098b44a0 RSI: 0000000000055000 RDI: 00007ed79a637000
Oct 17 15:47:53 dom0 kernel: RBP: 00007ed79a637000 R08: 0000000000000008 R09: 0000000000000000
Oct 17 15:47:53 dom0 kernel: R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000000009
Oct 17 15:47:53 dom0 kernel: R13: 00006080c7ff49d8 R14: 000000000000005f R15: 00006080c6bc9e00
Oct 17 15:47:53 dom0 kernel:  </TASK>

My journalctl keeps choking on these kinda messages even tho I’ve not experienced a crash or a freeze since sep 28. I can’t seem to find any pattern as to what triggers them either.

Here are all my messages related to this issue since my last boot which was 2022-10-24 19:11:06

Oct 25 03:40:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000001a8d0d365 pmd:1065e8067
Oct 25 03:40:49 dom0 kernel: WARNING: CPU: 1 PID: 9175 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 03:40:49 dom0 kernel: WARNING: CPU: 1 PID: 9175 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 25 03:40:49 dom0 kernel: WARNING: CPU: 1 PID: 9175 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 06:36:24 dom0 kernel: BUG: Bad page map: 90 messages suppressed
Oct 25 06:36:24 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000003d4637365 pmd:10da34067

<a bunch of the same repeated messages>

Oct 25 06:36:26 dom0 kernel: BUG: Bad page map in process Xorg  pte:800000034a414365 pmd:10da34067
Oct 25 06:36:26 dom0 kernel: WARNING: CPU: 3 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 25 06:36:26 dom0 kernel: WARNING: CPU: 3 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 06:38:54 dom0 kernel: BUG: Bad page map: 113 messages suppressed
Oct 25 06:38:54 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000003f544f365 pmd:1500c1067
Oct 25 06:38:54 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000001fc87a365 pmd:1500c1067
Oct 25 06:38:54 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000002aaa49365 pmd:1500c1067
Oct 25 06:38:54 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000002aaa4a365 pmd:1500c1067
Oct 25 06:38:54 dom0 kernel: WARNING: CPU: 5 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 25 06:38:54 dom0 kernel: WARNING: CPU: 5 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 07:15:30 dom0 kernel: BUG: Bad page map: 91 messages suppressed
Oct 25 07:15:30 dom0 kernel: BUG: Bad page map in process Xorg  pte:800000036ddf1365 pmd:10357a067

<a bunch of the same repeated messages>

Oct 25 07:15:30 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000001ccfc2365 pmd:10357a067
Oct 25 07:15:33 dom0 kernel: WARNING: CPU: 2 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 25 07:15:33 dom0 kernel: WARNING: CPU: 2 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 16:36:54 dom0 kernel: BUG: Bad page map: 119 messages suppressed
Oct 25 16:36:54 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000000000364 pmd:11143e067
Oct 25 16:36:54 dom0 kernel: WARNING: CPU: 2 PID: 142 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 16:36:54 dom0 kernel: WARNING: CPU: 2 PID: 142 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 25 16:36:54 dom0 kernel: WARNING: CPU: 2 PID: 142 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 22:11:44 dom0 kernel: BUG: Bad page map: 91 messages suppressed
Oct 25 22:11:44 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000000000364 pmd:1558f0067

<a bunch of the same repeated messages>

Oct 25 22:11:45 dom0 kernel: BUG: Bad page map in process Xorg  pte:800000055e8db365 pmd:1558f0067
Oct 25 22:11:47 dom0 kernel: WARNING: CPU: 4 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 25 22:11:47 dom0 kernel: WARNING: CPU: 4 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 22:58:59 dom0 kernel: BUG: Bad page map: 91 messages suppressed
Oct 25 22:58:59 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000003698b0365 pmd:15006f067

<a bunch of the same repeated messages>

Oct 25 22:59:00 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000005615fd365 pmd:15006f067
Oct 25 22:59:02 dom0 kernel: WARNING: CPU: 2 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 22:59:02 dom0 kernel: WARNING: CPU: 2 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 25 22:59:02 dom0 kernel: WARNING: CPU: 2 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 25 23:10:34 dom0 kernel: BUG: Bad page map: 72 messages suppressed
Oct 25 23:10:34 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000000000364 pmd:130e63067
Oct 25 23:10:34 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000000000364 pmd:130e63067

<a bunch of the same repeated messages>

Oct 25 23:10:36 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000000000364 pmd:130e63067
Oct 25 23:10:36 dom0 kernel: BUG: Bad page map in process Xorg  pte:800000038b072365 pmd:130e63067
Oct 25 23:10:36 dom0 kernel: WARNING: CPU: 1 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 25 23:10:36 dom0 kernel: WARNING: CPU: 1 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 00:07:59 dom0 kernel: BUG: Bad page map: 91 messages suppressed
Oct 26 00:07:59 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000000000364 pmd:15010b067
Oct 26 00:07:59 dom0 kernel: WARNING: CPU: 0 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 26 00:07:59 dom0 kernel: WARNING: CPU: 0 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 03:50:19 dom0 kernel: BUG: Bad page map: 91 messages suppressed
Oct 26 03:50:19 dom0 kernel: BUG: Bad page map in process Xorg  pte:800000025c682365 pmd:1115f9067
Oct 26 03:50:19 dom0 kernel: WARNING: CPU: 0 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 03:50:19 dom0 kernel: WARNING: CPU: 0 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 26 03:50:19 dom0 kernel: WARNING: CPU: 0 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 04:28:25 dom0 kernel: BUG: Bad page map: 126 messages suppressed
Oct 26 04:28:25 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000791520365 pmd:150acd067
Oct 26 04:28:25 dom0 kernel: WARNING: CPU: 1 PID: 9175 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 04:28:25 dom0 kernel: WARNING: CPU: 1 PID: 9175 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 26 04:28:25 dom0 kernel: WARNING: CPU: 1 PID: 9175 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 05:04:43 dom0 kernel: BUG: Bad page map: 72 messages suppressed
Oct 26 05:04:43 dom0 kernel: BUG: Bad page map in process Xorg  pte:800000071f635365 pmd:1640b2067

<a bunch of the same repeated messages>

Oct 26 05:04:43 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000002054bb365 pmd:1640b2067
Oct 26 05:04:45 dom0 kernel: WARNING: CPU: 0 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 26 05:04:45 dom0 kernel: WARNING: CPU: 0 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map: 91 messages suppressed
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000003b94eb365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000005205cd365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000792753365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000396be6365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000396bf0365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:800000071e16b365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:80000007343e8365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:800000072c59c365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000310642365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000536a2a365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: BUG: Bad page map in process Xorg  pte:800000079107d365 pmd:150acd067
Oct 26 05:12:49 dom0 kernel: WARNING: CPU: 3 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 05:12:49 dom0 kernel: WARNING: CPU: 3 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 26 05:12:49 dom0 kernel: WARNING: CPU: 3 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 05:12:49 dom0 kernel: WARNING: CPU: 3 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 05:47:47 dom0 kernel: BUG: Bad page map: 91 messages suppressed
Oct 26 05:47:47 dom0 kernel: BUG: Bad page map in process Xorg  pte:8000000734dac365 pmd:105291067
Oct 26 05:47:47 dom0 kernel: WARNING: CPU: 1 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]
Oct 26 05:47:47 dom0 kernel: WARNING: CPU: 1 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

<a bunch of the same repeated messages>

Oct 26 05:47:47 dom0 kernel: WARNING: CPU: 1 PID: 2259 at drivers/xen/gntdev.c:399 __unmap_grant_pages_done+0x5b/0x110 [xen_gntdev]

my uptime is currently 4+ days and as can be seen for the past two days I had no messages related to this issue even tho my usage pattern remained the same.

This is my dom0 update history

ID     | Command line                                           | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------------------------------------
    14 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-10-24 17:42 | Upgrade        |   10 EE
    13 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-10-19 14:40 | Upgrade        |    6   
    12 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-09-19 13:42 | E, I, U        |   13 EE
    11 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-09-10 22:07 | Upgrade        |    4   
    10 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-09-06 01:13 | E, I, U        |   25 EE
     9 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-08-16 05:08 | E, I, U        |   13 EE
     8 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-07-26 06:01 | Upgrade        |    2   
     7 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-07-22 01:21 | Upgrade        |    2 EE
     6 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-07-20 07:37 | I, U           |   10 EE
     5 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-07-15 03:43 | Upgrade        |    6 EE
     4 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-06-28 19:52 | Upgrade        |   11 EE
     3 | --exclude=qubes-template-* install qubes-windows-tools | 2022-06-10 11:53 | Install        |    1   
     2 | --exclude=qubes-template-* --quiet -y upgrade          | 2022-05-26 23:09 | I, U           |   48 EE
     1 |                                                        | 2022-05-26 14:13 | Install        | 1023 EE

And here are transaction details

Transaction ID : 14
Begin time     : Mon 24 Oct 2022
Releasever     : 4.1
Packages Altered:
    Upgrade  linux-firmware-20220913-135.fc32.noarch        @qubes-dom0-cached
    Upgraded linux-firmware-20220610-135.fc32.noarch        @@System
    Upgrade  linux-firmware-whence-20220913-135.fc32.noarch @qubes-dom0-cached
    Upgraded linux-firmware-whence-20220610-135.fc32.noarch @@System
    Upgrade  python3-qasync-0.23.0-2.fc32.noarch            @qubes-dom0-cached
    Upgraded python3-qasync-0.23.0-1.fc32.noarch            @@System
    Upgrade  python3-qubesadmin-4.1.25-1.fc32.noarch        @qubes-dom0-cached
    Upgraded python3-qubesadmin-4.1.24-1.fc32.noarch        @@System
    Upgrade  qubes-audio-daemon-4.1.25-1.fc32.x86_64        @qubes-dom0-cached
    Upgraded qubes-audio-daemon-4.1.24-1.fc32.x86_64        @@System
    Upgrade  qubes-audio-dom0-4.1.25-1.fc32.x86_64          @qubes-dom0-cached
    Upgraded qubes-audio-dom0-4.1.24-1.fc32.x86_64          @@System
    Upgrade  qubes-core-admin-client-4.1.25-1.fc32.noarch   @qubes-dom0-cached
    Upgraded qubes-core-admin-client-4.1.24-1.fc32.noarch   @@System
    Upgrade  qubes-gui-daemon-4.1.25-1.fc32.x86_64          @qubes-dom0-cached
    Upgraded qubes-gui-daemon-4.1.24-1.fc32.x86_64          @@System
    Upgrade  qubes-gui-dom0-4.1.25-1.fc32.x86_64            @qubes-dom0-cached
    Upgraded qubes-gui-dom0-4.1.24-1.fc32.x86_64            @@System
    Upgrade  zlib-1.2.12-5.fc32.x86_64                      @qubes-dom0-cached
    Upgraded zlib-1.2.11-21.fc32.x86_64                     @@System

Transaction ID : 13
Begin time     : Wed 19 Oct 2022
Releasever     : 4.1
Packages Altered:
    Upgrade  qubes-input-proxy-1.0.28-1.fc32.x86_64          @qubes-dom0-cached
    Upgraded qubes-input-proxy-1.0.26-1.fc32.x86_64          @@System
    Upgrade  qubes-input-proxy-receiver-1.0.28-1.fc32.x86_64 @qubes-dom0-cached
    Upgraded qubes-input-proxy-receiver-1.0.26-1.fc32.x86_64 @@System
    Upgrade  qubes-input-proxy-sender-1.0.28-1.fc32.x86_64   @qubes-dom0-cached
    Upgraded qubes-input-proxy-sender-1.0.26-1.fc32.x86_64   @@System
    Upgrade  xen-hvm-stubdom-linux-1.2.5-1.fc32.x86_64       @qubes-dom0-cached
    Upgraded xen-hvm-stubdom-linux-1.2.4-1.fc32.x86_64       @@System
    Upgrade  xen-hvm-stubdom-linux-full-1.2.5-1.fc32.x86_64  @qubes-dom0-cached
    Upgraded xen-hvm-stubdom-linux-full-1.2.4-1.fc32.x86_64  @@System
    Upgrade  xfwm4-1000:4.14.2-3.fc32.x86_64                 @qubes-dom0-cached
    Upgraded xfwm4-1000:4.14.2-2.fc32.x86_64                 @@System

Transaction ID : 12
Begin time     : Mon 19 Sep 2022
Releasever     : 4.1
Packages Altered:
    Install  kernel-1000:5.15.64-1.fc32.qubes.x86_64          @qubes-dom0-cached
    Install  kernel-qubes-vm-1000:5.15.64-1.fc32.qubes.x86_64 @qubes-dom0-cached
    Upgrade  qubes-audio-daemon-4.1.24-1.fc32.x86_64          @qubes-dom0-cached
    Upgraded qubes-audio-daemon-4.1.23-1.fc32.x86_64          @@System
    Upgrade  qubes-audio-dom0-4.1.24-1.fc32.x86_64            @qubes-dom0-cached
    Upgraded qubes-audio-dom0-4.1.23-1.fc32.x86_64            @@System
    Upgrade  qubes-gui-daemon-4.1.24-1.fc32.x86_64            @qubes-dom0-cached
    Upgraded qubes-gui-daemon-4.1.23-1.fc32.x86_64            @@System
    Upgrade  qubes-gui-dom0-4.1.24-1.fc32.x86_64              @qubes-dom0-cached
    Upgraded qubes-gui-dom0-4.1.23-1.fc32.x86_64              @@System
    Upgrade  qubes-libvchan-xen-4.1.8-1.fc32.x86_64           @qubes-dom0-cached
    Upgraded qubes-libvchan-xen-4.1.7-1.fc32.x86_64           @@System
    Upgrade  qubes-mgmt-salt-4.1.16-1.fc32.noarch             @qubes-dom0-cached
    Upgraded qubes-mgmt-salt-4.1.15-1.fc32.noarch             @@System
    Upgrade  qubes-mgmt-salt-admin-tools-4.1.16-1.fc32.noarch @qubes-dom0-cached
    Upgraded qubes-mgmt-salt-admin-tools-4.1.15-1.fc32.noarch @@System
    Upgrade  qubes-mgmt-salt-config-4.1.16-1.fc32.noarch      @qubes-dom0-cached
    Upgraded qubes-mgmt-salt-config-4.1.15-1.fc32.noarch      @@System
    Upgrade  qubes-mgmt-salt-dom0-4.1.16-1.fc32.noarch        @qubes-dom0-cached
    Upgraded qubes-mgmt-salt-dom0-4.1.15-1.fc32.noarch        @@System
    Removed  kernel-1000:5.15.52-1.fc32.qubes.x86_64          @@System
    Removed  kernel-qubes-vm-1000:5.15.52-1.fc32.qubes.x86_64 @@System

Transaction ID : 11
Begin time     : Sat 10 Sep 2022
Releasever     : 4.1
Packages Altered:
    Upgrade  qubes-mgmt-salt-4.1.15-1.fc32.noarch             @qubes-dom0-cached
    Upgraded qubes-mgmt-salt-4.1.14-1.fc32.noarch             @@System
    Upgrade  qubes-mgmt-salt-admin-tools-4.1.15-1.fc32.noarch @qubes-dom0-cached
    Upgraded qubes-mgmt-salt-admin-tools-4.1.14-1.fc32.noarch @@System
    Upgrade  qubes-mgmt-salt-config-4.1.15-1.fc32.noarch      @qubes-dom0-cached
    Upgraded qubes-mgmt-salt-config-4.1.14-1.fc32.noarch      @@System
    Upgrade  qubes-mgmt-salt-dom0-4.1.15-1.fc32.noarch        @qubes-dom0-cached
    Upgraded qubes-mgmt-salt-dom0-4.1.14-1.fc32.noarch        @@System

Transaction ID : 10
Begin time     : Tue 06 Sep 2022
Releasever     : 4.1
Packages Altered:
    Install  kernel-1000:5.15.63-1.fc32.qubes.x86_64                  @qubes-dom0-cached
    Install  kernel-qubes-vm-1000:5.15.63-1.fc32.qubes.x86_64         @qubes-dom0-cached
    Upgrade  alsa-lib-1.2.4-5.fc32.x86_64                             @qubes-dom0-cached
    Upgraded alsa-lib-1.2.3.2-1.fc32.x86_64                           @@System
    Upgrade  alsa-sof-firmware-2.1.1-1.fc32.noarch                    @qubes-dom0-cached
    Upgraded alsa-sof-firmware-1.5-1.fc32.noarch                      @@System
    Upgrade  alsa-ucm-1.2.4-5.fc32.noarch                             @qubes-dom0-cached
    Upgraded alsa-ucm-1.2.3.2-1.fc32.noarch                           @@System
    Upgrade  alsa-utils-1.2.4-2.fc32.x86_64                           @qubes-dom0-cached
    Upgraded alsa-utils-1.2.3-4.fc32.x86_64                           @@System
    Upgrade  libvirt-bash-completion-1000:6.6.0-7.fc32.x86_64         @qubes-dom0-cached
    Upgraded libvirt-bash-completion-1000:6.6.0-5.fc32.x86_64         @@System
    Upgrade  libvirt-client-1000:6.6.0-7.fc32.x86_64                  @qubes-dom0-cached
    Upgraded libvirt-client-1000:6.6.0-5.fc32.x86_64                  @@System
    Upgrade  libvirt-daemon-1000:6.6.0-7.fc32.x86_64                  @qubes-dom0-cached
    Upgraded libvirt-daemon-1000:6.6.0-5.fc32.x86_64                  @@System
    Upgrade  libvirt-daemon-driver-interface-1000:6.6.0-7.fc32.x86_64 @qubes-dom0-cached
    Upgraded libvirt-daemon-driver-interface-1000:6.6.0-5.fc32.x86_64 @@System
    Upgrade  libvirt-daemon-driver-libxl-1000:6.6.0-7.fc32.x86_64     @qubes-dom0-cached
    Upgraded libvirt-daemon-driver-libxl-1000:6.6.0-5.fc32.x86_64     @@System
    Upgrade  libvirt-daemon-driver-nodedev-1000:6.6.0-7.fc32.x86_64   @qubes-dom0-cached
    Upgraded libvirt-daemon-driver-nodedev-1000:6.6.0-5.fc32.x86_64   @@System
    Upgrade  libvirt-daemon-xen-1000:6.6.0-7.fc32.x86_64              @qubes-dom0-cached
    Upgraded libvirt-daemon-xen-1000:6.6.0-5.fc32.x86_64              @@System
    Upgrade  libvirt-libs-1000:6.6.0-7.fc32.x86_64                    @qubes-dom0-cached
    Upgraded libvirt-libs-1000:6.6.0-5.fc32.x86_64                    @@System
    Upgrade  python3-libvirt-1000:6.6.0-7.fc32.x86_64                 @qubes-dom0-cached
    Upgraded python3-libvirt-1000:6.6.0-5.fc32.x86_64                 @@System
    Upgrade  python3-xen-2001:4.14.5-7.fc32.x86_64                    @qubes-dom0-cached
    Upgraded python3-xen-2001:4.14.5-6.fc32.x86_64                    @@System
    Upgrade  xen-2001:4.14.5-7.fc32.x86_64                            @qubes-dom0-cached
    Upgraded xen-2001:4.14.5-6.fc32.x86_64                            @@System
    Upgrade  xen-hvm-stubdom-linux-1.2.4-1.fc32.x86_64                @qubes-dom0-cached
    Upgraded xen-hvm-stubdom-linux-1.2.3-1.fc32.x86_64                @@System
    Upgrade  xen-hvm-stubdom-linux-full-1.2.4-1.fc32.x86_64           @qubes-dom0-cached
    Upgraded xen-hvm-stubdom-linux-full-1.2.3-1.fc32.x86_64           @@System
    Upgrade  xen-hypervisor-2001:4.14.5-7.fc32.x86_64                 @qubes-dom0-cached
    Upgraded xen-hypervisor-2001:4.14.5-6.fc32.x86_64                 @@System
    Upgrade  xen-libs-2001:4.14.5-7.fc32.x86_64                       @qubes-dom0-cached
    Upgraded xen-libs-2001:4.14.5-6.fc32.x86_64                       @@System
    Upgrade  xen-licenses-2001:4.14.5-7.fc32.x86_64                   @qubes-dom0-cached
    Upgraded xen-licenses-2001:4.14.5-6.fc32.x86_64                   @@System
    Upgrade  xen-runtime-2001:4.14.5-7.fc32.x86_64                    @qubes-dom0-cached
    Upgraded xen-runtime-2001:4.14.5-6.fc32.x86_64                    @@System
    Removed  kernel-1000:5.10.112-1.fc32.qubes.x86_64                 @@System
    Removed  kernel-qubes-vm-1000:5.10.112-1.fc32.qubes.x86_64        @@System

Transaction ID : 9
Begin time     : Tue 16 Aug 2022
Releasever     : 4.1
Packages Altered:
    Install  kernel-1000:5.15.57-1.fc32.qubes.x86_64          @qubes-dom0-cached
    Install  kernel-qubes-vm-1000:5.15.57-1.fc32.qubes.x86_64 @qubes-dom0-cached
    Upgrade  python3-qubesadmin-4.1.24-1.fc32.noarch          @qubes-dom0-cached
    Upgraded python3-qubesadmin-4.1.23-1.fc32.noarch          @@System
    Upgrade  qubes-audio-daemon-4.1.23-1.fc32.x86_64          @qubes-dom0-cached
    Upgraded qubes-audio-daemon-4.1.21-1.fc32.x86_64          @@System
    Upgrade  qubes-audio-dom0-4.1.23-1.fc32.x86_64            @qubes-dom0-cached
    Upgraded qubes-audio-dom0-4.1.21-1.fc32.x86_64            @@System
    Upgrade  qubes-core-admin-client-4.1.24-1.fc32.noarch     @qubes-dom0-cached
    Upgraded qubes-core-admin-client-4.1.23-1.fc32.noarch     @@System
    Upgrade  qubes-core-dom0-4.1.28-1.fc32.noarch             @qubes-dom0-cached
    Upgraded qubes-core-dom0-4.1.27-1.fc32.noarch             @@System
    Upgrade  qubes-gpg-split-dom0-2.0.63-1.fc32.x86_64        @qubes-dom0-cached
    Upgraded qubes-gpg-split-dom0-2.0.60-1.fc32.x86_64        @@System
    Upgrade  qubes-gui-daemon-4.1.23-1.fc32.x86_64            @qubes-dom0-cached
    Upgraded qubes-gui-daemon-4.1.21-1.fc32.x86_64            @@System
    Upgrade  qubes-gui-dom0-4.1.23-1.fc32.x86_64              @qubes-dom0-cached
    Upgraded qubes-gui-dom0-4.1.21-1.fc32.x86_64              @@System
    Upgrade  qubes-manager-4.1.25-1.fc32.noarch               @qubes-dom0-cached
    Upgraded qubes-manager-4.1.23-1.fc32.noarch               @@System
    Removed  kernel-1000:5.10.90-1.fc32.qubes.x86_64          @@System
    Removed  kernel-qubes-vm-1000:5.10.90-1.fc32.qubes.x86_64 @@System

I first expirienced a freeze on Sep 18 and then a crash on Sep 28
As can be seen from my dnf history I had a big upgrade on Sep 6 with a bunch of xen related packages.
So I suspect this crash was caused by one of those packages, tho here

the grant tables xen issue seems to be diagnosed and resolved in a 5.19 kernel which is still in testing I guess (and as I write this I get a notification about new dom0 updates available )
We’ll see if it solves the issue.

host                   : dom0
release                : 5.15.64-1.fc32.qubes.x86_64
version                : #1 SMP Mon Sep 5 04:26:01 CEST 2022
machine                : x86_64
nr_cpus                : 6
max_cpu_id             : 5
nr_nodes               : 1
cores_per_socket       : 6
threads_per_core       : 1
cpu_mhz                : 3000.025
hw_caps                : bfebfbff:77faf3ff:2c100800:00000121:0000000f:009c6fbf:00000000:00000100
virt_caps              : pv hvm hvm_directio pv_directio hap iommu_hap_pt_share
total_memory           : 32424
free_memory            : 5716
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 14
xen_extra              : .5
xen_version            : 4.14.5
xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit2
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : 
xen_commandline        : placeholder console=none dom0_mem=min:1024M dom0_mem=max:4096M ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096
cc_compiler            : gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1)
cc_compile_by          : mockbuild
cc_compile_domain      : [unknown]
cc_compile_date        : Wed Aug 24 00:00:00 UTC 2022
build_id               : 78083462a31dbc218d043c59c52f9fa65f71bb04
xend_config_format     : 4

FudgingDonut · October 31, 2022, 6:44am

I just got a freeze, it wasn’t triggered by an update though.

Sven · November 3, 2022, 4:05pm

… and it happened again (freeze): this time when launching a dispVM very shortly after booting the machine (a few minutes maybe). I’ll keep posting when freezes happen to create a record of how frequently this occurs (I use my machine 10+ hours a day).

last few minutes of log before the freeze

Nov 03 10:51:55 dom0 kernel: Linux version 5.15.74-2.fc32.qubes.x86_64 (mockbuild@cdc3111f5463453da67fb7dece6473>
-- Reboot --
Nov 03 10:50:22 dom0 kernel: audit: type=1105 audit(1667490622.850:313): pid=7116 uid=0 auid=4294967295 ses=4294>
Nov 03 10:50:22 dom0 kernel: audit: type=1103 audit(1667490622.849:312): pid=7116 uid=0 auid=4294967295 ses=4294>
Nov 03 10:50:22 dom0 qubesd[1414]: vm.disp4574: Creating directory: /var/lib/qubes/appvms/disp4574
Nov 03 10:47:58 dom0 qrexec-policy-daemon[1498]: qrexec: qubes.Gpg+: mail -> vault: allowed to vault

enmus · November 3, 2022, 11:47pm

I had one freeze immediately after upgrading to kernel 6.02, 3 days ago and rebooting, but not after that.

Sven · November 4, 2022, 12:36am

Same when updating to 5.15.74-2. After update I rebooted and resigned my boot files in heads. Then the first boot attempt after that just froze. Power off/on and next reboot worked.

enmus · November 5, 2022, 8:15pm

Frequent kernel updates. Still no freezes/crashes after initial one when upgraded to kernel 6.02

Latest dom0 updates

Updating dom0

local:
----------
kernel-latest:
----------
new:
1000:5.19.14-1.fc32.qubes,1000:6.0.2-2.fc32.qubes,1000:6.0.7-1.fc32.qubes
old:
1000:5.19.14-1.fc32.qubes,1000:5.19.9-1.fc32.qubes,1000:6.0.2-2.fc32.qubes
kernel-latest-qubes-vm:
----------
new:
1000:5.19.14-1.fc32.qubes,1000:6.0.2-2.fc32.qubes,1000:6.0.7-1.fc32.qubes
old:
1000:5.19.14-1.fc32.qubes,1000:5.19.9-1.fc32.qubes,1000:6.0.2-2.fc32.qubes
libxdo:
----------
new:
1:3.20210804.2-3.fc32
old:
1:3.20210804.2-2.fc32
qubes-desktop-linux-common:
----------
new:
4.1.13-1.fc32
old:
4.1.12-1.fc32
qubes-menus:
----------
new:
4.1.13-1.fc32
old:
4.1.12-1.fc32
scrypt:
----------
new:
1.3.1-1.fc32
old:
1.2.1-3.fc32
xdotool:
----------
new:
1:3.20210804.2-3.fc32
old:
1:3.20210804.2-2.fc32

qub411 · November 6, 2022, 1:26am

Have not done any exhaustive testing, but just installed kernel latest 6.0.2-2. Ran my acid test in i3wm, which was to switch a window to floating and move it around: result was an instant crash and reboot!
Was pretty sure @marmarek said the xorg bug fix was backported to 6.0.2, but maybe I was mistaken…
Anyway, the pain continues…

Sven · November 14, 2022, 10:56pm

It’s been 11 days and several updates incl. dom0 updates without any freeze. Starting to think (at least for my use case) this might be over.

Xen:            4.14.5
Kernel:         5.15.74-2

enmus · November 14, 2022, 11:02pm

Yep. Me on 6.0.7 still stable, so common thing is (was) Xen… But who’d care anyway…

SteveC · November 14, 2022, 11:22pm

For what it’s worth I got the very occasional “freeze” a few weeks back, too. Not enough to really notice a pattern. But it has been a while now that you mention it. (I’ve managed to lose mouse and keyboard but that was something I did wrong.)

qub411 · November 14, 2022, 11:37pm

Still totally reproducible here, despite the recent Xen updates. I’m still on kernel 6.0.2 but I guess the difference with me is that I’m on i3wm and the system halts continue…

unman · November 15, 2022, 3:28pm

Regrettably, I gave up on testing 5.15: didn’t fix issues for me.

I’m now testing kernel-latest 6.0.7 - with poor results so far.
I’ve had one crash the other day that left it impossible to activate
lvs. (With live distros, unlocking the drive left the same lv activated.)
Numerous freezes, and lock ups.
Starting (and stopping) qubes seems a trigger in many cases.

Just to check, I dropped in a 4.0.4 drive, and everything was as I
expect. Solid performance with many qubes open.
This still leaves open the possibility that 4.1 is stressing components
that are on their way out, but I’ve tried dropping the 4.1 drive in to
otherwise working machines, only to see them tank.

I never presume to speak for the Qubes team.
When I comment in the Forum or in the mailing lists I speak for myself.

Sven · November 15, 2022, 4:35pm

@unman do you think there could be something to @enmus’ conclusion that it’s in fact XEN and not the kernel that caused/causes these issues? What version of XEN are your running?

HPOA909 · November 16, 2022, 4:07pm

I’ve started to think that Xen is needed to upgrade their project from current version of 4.16 to 5.0 to be more compatible to this current version of Linux—V6.0.

unman · November 18, 2022, 12:46pm

4.14.5

unman · November 18, 2022, 12:50pm

Well perhaps.
But while these problems trouble some users, they dont seem to trouble
all users,not even those using qubes-testing.
I assume from the fact that these packages are released to testing, and no
one else from the dev team has commented at all, that none of them
experience these problems.

enmus · November 19, 2022, 9:42am

This is not enough to state, since there are several upgrades of this version, and 4.14.5-12 resolved some other significant issues (can’t remember or search at the moment). Just updated to 4.14.5-14, though.
frequent Xen updates noticeable, anyway…

unman · November 20, 2022, 12:07am

I thought I said I am always running latest.
4.14.5-12 fixed no issues for me.

tripleh · November 20, 2022, 8:39am

Btw if anyone experiences freezes or reboots on VM starts with PCI devices attached, I recommend hiding those devices from dom0 via the kernel command-line option rd.qubes.hide_pci=[comma-separated list of PCI devices] (which should be the recommended default anyway).

However IIRC most people on this thread have freees on different occasions.