Whonix qubes randomly shutdown, especially gateway

scallyob · May 27, 2024, 8:49pm

I’ve had qubes randomly shutting down on me lately. So far it’s been just qubes based on whonix templates and usually it is a whonix gateway. Usually the whonix workstation will not run without the gateway booting first, but what has been happening is a service goes offline,the workstation is still running but the gateway has crashed.

I had to make a change to my Xen configuration to get my machine to work with recent versions: No Qubes/VMs boot after latest updates - #32 by scallyob

It’s possible this problem started after making that change.

Here’s the log from the last crash from /var/log/xen/console/guest-sys-whonix.log:

[2024-05-27 13:41:13] [20937.502042] #PF: supervisor read access in kernel mode^M
[2024-05-27 13:41:13] [20937.503416] #PF: error_code(0x0000) - not-present page^M
[2024-05-27 13:41:13] [20937.504752] PGD 0 P4D 0 ^M
[2024-05-27 13:41:13] [20937.505434] Oops: 0000 [#1] PREEMPT SMP NOPTI^M
[2024-05-27 13:41:13] [20937.506609] CPU: 0 PID: 56 Comm: xenbus Not tainted 6.6.29-1.qubes.fc37.x86_64 #1^M
[2024-05-27 13:41:13] [20937.508455] RIP: 0010:__wake_up_common+0x4c/0x180^M
[2024-05-27 13:41:13] [20937.509960] Code: 24 0c 89 4c 24 08 4d 85 c9 74 0a 41 f6 01 04 0f 85 a3 00 00 00 48 8b 43 08 4c 8d 40 e8 48 83 c3 08 49 8d 40 18 48 39 c3 74 5b <49> 8b 40 18 31 ed 4c 8d 70 e8 45 8b 28 41 f6 c5 04 75 5f 49 8b 40^M
[2024-05-27 13:41:13] [20937.514811] RSP: 0018:ffffc90000dabdf0 EFLAGS: 00010082^M
[2024-05-27 13:41:13] [20937.515678] RAX: 0000000000000000 RBX: ffff88802e9f9b98 RCX: 0000000000000000^M
[2024-05-27 13:41:13] [20937.517510] RDX: 0000000000000001 RSI: 0000000000000003 RDI: ffff88802e9f9b90^M
[2024-05-27 13:41:13] [20937.519184] RBP: 0000000000000246 R08: ffffffffffffffe8 R09: ffffc90000dabe48^M
[2024-05-27 13:41:13] [20937.520675] R10: ffff88800d3d6ea8 R11: ffffc9000002d000 R12: ffffc90000dabe48^M
[2024-05-27 13:41:13] [20937.521637] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000^M
[2024-05-27 13:41:13] [20937.522753] FS:  0000000000000000(0000) GS:ffff888018400000(0000) knlGS:0000000000000000^M
[2024-05-27 13:41:13] [20937.523543] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033^M
[2024-05-27 13:41:13] [20937.524155] CR2: 0000000000000000 CR3: 0000000006b7a000 CR4: 00000000000406f0^M
[2024-05-27 13:41:13] [20937.524767] Call Trace:^M
[2024-05-27 13:41:13] [20937.524946]  <TASK>^M
[2024-05-27 13:41:13] [20937.525127]  ? __die+0x23/0x70^M
[2024-05-27 13:41:13] [20937.525397]  ? page_fault_oops+0x98/0x190^M
[2024-05-27 13:41:13] [20937.525669]  ? exc_page_fault+0x77/0x170^M
[2024-05-27 13:41:13] [20937.525940]  ? asm_exc_page_fault+0x26/0x30^M
[2024-05-27 13:41:13] [20937.526215]  ? __wake_up_common+0x4c/0x180^M
[2024-05-27 13:41:13] [20937.526484]  __wake_up_common_lock+0x82/0xd0^M
[2024-05-27 13:41:13] [20937.526839]  ? __pfx_xenbus_thread+0x10/0x10^M
[2024-05-27 13:41:13] [20937.527196]  process_msg+0x18e/0x2f0^M
[2024-05-27 13:41:13] [20937.527464]  xenbus_thread+0x4a/0x1e0^M
[2024-05-27 13:41:13] [20937.527732]  ? __pfx_autoremove_wake_function+0x10/0x10^M
[2024-05-27 13:41:13] [20937.528090]  kthread+0xe8/0x120^M
[2024-05-27 13:41:13] [20937.528360]  ? __pfx_kthread+0x10/0x10^M
[2024-05-27 13:41:13] [20937.528629]  ret_from_fork+0x34/0x50^M
[2024-05-27 13:41:13] [20937.528904]  ? __pfx_kthread+0x10/0x10^M
[2024-05-27 13:41:13] [20937.529173]  ret_from_fork_asm+0x1b/0x30^M
[2024-05-27 13:41:13] [20937.529443]  </TASK>^M
[2024-05-27 13:41:13] [20937.529622] Modules linked in: nf_conntrack_netlink nft_flow_offload nf_flow_table_inet nf_flow_table xen_netback dummy ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xenfs xt_multiport xt_nat xt_owner xt_REDIRECT nft_chain_nat nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic binfmt_misc ghash_clmulni_intel nf_tables sha512_ssse3 sha256_ssse3 nfnetlink xen_netfront sha1_ssse3 xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop ip_tables overlay xen_blkfront^M
[2024-05-27 13:41:13] [20937.533019] CR2: 0000000000000000^M
[2024-05-27 13:41:13] [20937.533290] ---[ end trace 0000000000000000 ]---^M
[2024-05-27 13:41:13] [20937.533642] RIP: 0010:__wake_up_common+0x4c/0x180^M
[2024-05-27 13:41:13] [20937.533998] Code: 24 0c 89 4c 24 08 4d 85 c9 74 0a 41 f6 01 04 0f 85 a3 00 00 00 48 8b 43 08 4c 8d 40 e8 48 83 c3 08 49 8d 40 18 48 39 c3 74 5b <49> 8b 40 18 31 ed 4c 8d 70 e8 45 8b 28 41 f6 c5 04 75 5f 49 8b 40^M
[2024-05-27 13:41:13] [20937.704565] RSP: 0018:ffffc90000dabdf0 EFLAGS: 00010082^M
[2024-05-27 13:41:13] [20937.705264] RAX: 0000000000000000 RBX: ffff88802e9f9b98 RCX: 0000000000000000^M
[2024-05-27 13:41:13] [20937.706294] RDX: 0000000000000001 RSI: 0000000000000003 RDI: ffff88802e9f9b90^M
[2024-05-27 13:41:13] [20937.707311] RBP: 0000000000000246 R08: ffffffffffffffe8 R09: ffffc90000dabe48^M
[2024-05-27 13:41:13] [20937.708988] R10: ffff88800d3d6ea8 R11: ffffc9000002d000 R12: ffffc90000dabe48^M
[2024-05-27 13:41:13] [20937.709805] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000^M
[2024-05-27 13:41:13] [20937.710685] FS:  0000000000000000(0000) GS:ffff888018400000(0000) knlGS:0000000000000000^M
[2024-05-27 13:41:13] [20937.712434] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033^M
[2024-05-27 13:41:13] [20937.713674] CR2: 0000000000000000 CR3: 0000000006b7a000 CR4: 00000000000406f0^M
[2024-05-27 13:41:13] [20937.714301] Kernel panic - not syncing: Fatal exception^M
[2024-05-27 13:41:13] [20937.718046] Kernel Offset: disabled^M

apparatus · May 28, 2024, 4:41am

Seems related:

github.com/QubesOS/qubes-issues

kernel NULL pointer dereference in xenbus_thread, Linux 6.1.57 on openQA

opened 02:07PM - 22 Oct 23 UTC

marmarek

T: bug C: kernel P: default needs diagnosis waiting for upstream affects-4.1 affects-4.2

## Observation openQA test in scenario qubesos-4.1-release-upgrade-x86_64-ins…tall_default@64bit fails in [release_upgrade](https://openqa.qubes-os.org/tests/84586/modules/release_upgrade/steps/90) <details> <summary>kernel NULL pointer dereference, stack trace</summary> ``` [ 876.712812] BUG: kernel NULL pointer dereference, address: 0000000000000000 [ 876.715099] #PF: supervisor read access in kernel mode [ 876.717222] #PF: error_code(0x0000) - not-present page [ 876.718919] PGD 101f9f067 P4D 101f9f067 PUD 103eae067 PMD 0 [ 876.721633] Oops: 0000 [#1] PREEMPT SMP NOPTI [ 876.723184] CPU: 1 PID: 28 Comm: xenbus Not tainted 6.1.57-1.qubes.fc37.x86_64 #1 [ 876.725629] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552-rebuilt.opensuse.org 04/01/2014 [ 876.729399] RIP: e030:__wake_up_common+0x4c/0x180 [ 876.731221] Code: 24 0c 89 4c 24 08 4d 85 c9 74 0a 41 f6 01 04 0f 85 a3 00 00 00 48 8b 43 08 4c 8d 40 e8 48 83 c3 08 49 8d 40 18 48 39 c3 74 5b <49> 8b 40 18 31 ed 4c 8d 70 e8 45 8b 28 41 f6 c5 04 75 5f 49 8b 40 [ 876.737539] RSP: e02b:ffffc900400f7e10 EFLAGS: 00010082 [ 876.740443] RAX: 0000000000000000 RBX: ffff888066582f98 RCX: 0000000000000000 [ 876.742913] RDX: 0000000000000001 RSI: 0000000000000003 RDI: ffff888066582f90 [ 876.745239] RBP: ffffc900400f0280 R08: ffffffffffffffe8 R09: ffffc900400f7e68 [ 876.748484] R10: 0000000000007ff0 R11: ffff888100ad9000 R12: ffffc900400f7e68 [ 876.750837] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 876.753734] FS: 0000000000000000(0000) GS:ffff88813ff00000(0000) knlGS:0000000000000000 [ 876.756569] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 876.758422] CR2: 0000000000000000 CR3: 0000000104ac4000 CR4: 0000000000040660 [ 876.760599] Call Trace: [ 876.761359] <TASK> [ 876.762025] ? show_trace_log_lvl+0x1d3/0x2ef [ 876.763390] ? show_trace_log_lvl+0x1d3/0x2ef [ 876.764731] ? show_trace_log_lvl+0x1d3/0x2ef [ 876.766061] ? __wake_up_common_lock+0x82/0xd0 [ 876.767465] ? __die_body.cold+0x8/0xd [ 876.769374] ? page_fault_oops+0x163/0x1a0 [ 876.770706] ? exc_page_fault+0x70/0x170 [ 876.771922] ? asm_exc_page_fault+0x22/0x30 [ 876.773235] ? __wake_up_common+0x4c/0x180 [ 876.774502] __wake_up_common_lock+0x82/0xd0 [ 876.775835] ? process_writes+0x240/0x240 [ 876.777251] process_msg+0x18e/0x2f0 [ 876.778364] xenbus_thread+0x165/0x1c0 [ 876.779520] ? cpuusage_read+0x10/0x10 [ 876.780694] kthread+0xe9/0x110 [ 876.781680] ? kthread_complete_and_exit+0x20/0x20 [ 876.783168] ret_from_fork+0x22/0x30 [ 876.784287] </TASK> [ 876.784974] Modules linked in: joydev snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm ppdev intel_rapl_msr intel_rapl_common snd_timer e1000e snd pcspkr parport_pc soundcore parport i2c_piix4 fuse loop xenfs dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt crct10dif_pclmul xhci_pci crc32_pclmul crc32c_intel xhci_pci_renesas polyval_clmulni polyval_generic xhci_hcd ghash_clmulni_intel sha512_ssse3 virtio_console virtio_scsi serio_raw bochs drm_vram_helper drm_ttm_helper ttm ata_generic pata_acpi floppy qemu_fw_cfg xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn scsi_dh_rdac scsi_dh_emc scsi_dh_alua uinput dm_multipath [ 876.806036] CR2: 0000000000000000 [ 876.807126] ---[ end trace 0000000000000000 ]--- [ 876.808589] RIP: e030:__wake_up_common+0x4c/0x180 [ 876.810069] Code: 24 0c 89 4c 24 08 4d 85 c9 74 0a 41 f6 01 04 0f 85 a3 00 00 00 48 8b 43 08 4c 8d 40 e8 48 83 c3 08 49 8d 40 18 48 39 c3 74 5b <49> 8b 40 18 31 ed 4c 8d 70 e8 45 8b 28 41 f6 c5 04 75 5f 49 8b 40 [ 876.815813] RSP: e02b:ffffc900400f7e10 EFLAGS: 00010082 [ 876.817375] RAX: 0000000000000000 RBX: ffff888066582f98 RCX: 0000000000000000 [ 876.819549] RDX: 0000000000000001 RSI: 0000000000000003 RDI: ffff888066582f90 [ 876.821725] RBP: ffffc900400f0280 R08: ffffffffffffffe8 R09: ffffc900400f7e68 [ 876.823885] R10: 0000000000007ff0 R11: ffff888100ad9000 R12: ffffc900400f7e68 [ 876.826063] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 876.828252] FS: 0000000000000000(0000) GS:ffff88813ff00000(0000) knlGS:0000000000000000 [ 876.830647] CS: 10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 876.832502] CR2: 0000000000000000 CR3: 0000000104ac4000 CR4: 0000000000040660 [ 876.834667] Kernel panic - not syncing: Fatal exception [ 876.836257] Kernel Offset: disabled (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. ``` </details> ## Test suite description The test is about release upgrade R4.1->R4.2, but the crash looks unrelated to the specific workload. ## Reproducible Fails since (at least) Build [2023102123-4.1](https://openqa.qubes-os.org/tests/84586) (current job) Similar crash was observed also on a real hardware in a domU on an older kernel: 6.1.43. ## Expected result Last good: [2023101101-4.1](https://openqa.qubes-os.org/tests/84007) (or more recent) ## Further details Always latest result in this scenario: [latest](https://openqa.qubes-os.org/tests/latest?arch=x86_64&distri=qubesos&flavor=release-upgrade&machine=64bit&test=install_default&version=4.1)

scallyob · July 2, 2024, 2:04am

I posted a bit on the github issue.

Since turning off memory balancing and increasing RAM and CPU, i’m still getting crashes every 2 weeks. Still always Whonix-based VMs.

This is a pretty significant problem for me. Not sure if I should try new hardware or what to do.

scallyob · July 3, 2024, 3:07am

As this is an ongoing problem with unknown cause, I created the following bashscript and set it to run as a cron job every hour. This will meet my needs to keep services up regularly enough.

#!/bin/bash
# leading spaces for names that overlap
servers=(" vm1" " vm2" " vm3" " vm4")

statuslist=$(qvm-ls --fields=status,name,state | grep Running)
echo $statuslist
for i in "${servers[@]}"; do
	echo $i
	if [[ ! $statuslist =~ $i ]]; then
		d=$(date)
		echo "$d: $i not running" >> /home/user/servercheck.log
		qvm-start $i
	fi
done