Qubes is unexpectedly crashing after only several hours of up-time. I haven’t been able to make sense of any of this. Updating the kernel, switching kernels, seems to not change the results.
Pulled logs using journalctl --boot -1 for the most recent crash and have appended here, the final few entries for review.
This Qubes Widget service seems to crash and restart hundreds of times before the system eventually restarts.
Dec 15 06:37:02 dom0 systemd[1]: systemd-coredump@4855-55751-0.service: Succeeded.
Dec 15 06:37:02 dom0 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@4855-55751-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 15 06:37:02 dom0 widget-wrapper[55749]: /usr/bin/widget-wrapper: line 14: 55750 Illegal instruction (core dumped) "$@"
Dec 15 06:37:02 dom0 widget-wrapper[55749]: exiting with 1
Dec 15 06:37:02 dom0 systemd[5070]: qubes-widget@qui-domains.service: Main process exited, code=exited, status=132/n/a
Dec 15 06:37:02 dom0 systemd[5070]: qubes-widget@qui-domains.service: Failed with result 'exit-code'.
Dec 15 06:37:02 dom0 audit: BPF prog-id=0 op=UNLOAD
Dec 15 06:37:02 dom0 audit: BPF prog-id=0 op=UNLOAD
Dec 15 06:37:02 dom0 audit: BPF prog-id=0 op=UNLOAD
Dec 15 06:37:03 dom0 systemd[5070]: qubes-widget@qui-domains.service: Scheduled restart job, restart counter is at 4847.
Dec 15 06:37:03 dom0 systemd[5070]: Stopped Qubes Widget service.
Dec 15 06:37:03 dom0 systemd[5070]: Started Qubes Widget service.
Dec 15 06:37:04 dom0 audit[55762]: ANOM_ABEND auid=1000 uid=1000 gid=1001 ses=3 pid=55762 comm="qui-domains" exe="/usr/bin/python3.8" sig=4 res=1
Dec 15 06:37:04 dom0 audit: BPF prog-id=14608 op=LOAD
Dec 15 06:37:04 dom0 audit: BPF prog-id=14609 op=LOAD
Dec 15 06:37:04 dom0 audit: BPF prog-id=14610 op=LOAD
Dec 15 06:37:04 dom0 systemd[1]: Started Process Core Dump (PID 55763/UID 0).
Dec 15 06:37:04 dom0 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@4856-55763-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 15 06:37:04 dom0 systemd-coredump[55764]: Removed old coredump core.qui-domains.1000.48bd9585ab3648f1aab466d305abbc61.49735.1671085070000000.lz4.
Dec 15 06:37:04 dom0 systemd-coredump[55764]: Process 55762 (qui-domains) of user 1000 dumped core.
Stack trace of thread 55762:
#0 0x00007b1c35f9e7e0 gtk_css_node_real_update_style (libgtk-3.so.0 + 0x19e7e0)
#1 0x00007b1c35f9d4e7 gtk_css_node_ensure_style.part.0 (libgtk-3.so.0 + 0x19d4e7)
#2 0x00007b1c35f9d775 gtk_css_node_get_style (libgtk-3.so.0 + 0x19d775)
#3 0x00007b1c3610c015 gtk_style_context_save_named (libgtk-3.so.0 + 0x30c015)
#4 0x00007b1c35ee32de gtk_style_update_from_context (libgtk-3.so.0 + 0xe32de)
#5 0x00007b1c35ee3623 gtk_style_constructed (libgtk-3.so.0 + 0xe3623)
#6 0x00007b1c377432e7 g_object_new_internal (libgobject-2.0.so.0 + 0x192e7)
#7 0x00007b1c37744fac g_object_new_valist (libgobject-2.0.so.0 + 0x1afac)
#8 0x00007b1c3774530d g_object_new (libgobject-2.0.so.0 + 0x1b30d)
#9 0x00007b1c35ee20ad _gtk_style_new_for_path (libgtk-3.so.0 + 0xe20ad)
#10 0x00007b1c35ee2103 gtk_style_new (libgtk-3.so.0 + 0xe2103)
#11 0x00007b1c35ee5075 gtk_widget_get_default_style (libgtk-3.so.0 + 0xe5075)
#12 0x00007b1c361ab15d gtk_widget_init (libgtk-3.so.0 + 0x3ab15d)
#13 0x00007b1c3775d457 g_type_create_instance (libgobject-2.0.so.0 + 0x33457)
#14 0x00007b1c37743205 g_object_new_internal (libgobject-2.0.so.0 + 0x19205)
#15 0x00007b1c37744fac g_object_new_valist (libgobject-2.0.so.0 + 0x1afac)
#16 0x00007b1c3774530d g_object_new (libgobject-2.0.so.0 + 0x1b30d)
#17 0x00007b1c35edb86b gtk_status_icon_init (libgtk-3.so.0 + 0xdb86b)
#18 0x00007b1c3775d498 g_type_create_instance (libgobject-2.0.so.0 + 0x33498)
#19 0x00007b1c37743205 g_object_new_internal (libgobject-2.0.so.0 + 0x19205)
#20 0x00007b1c377446ad g_object_new_with_properties (libgobject-2.0.so.0 + 0x1a6ad)
#21 0x00007b1c37909b7c pygobject_constructv (_gi.cpython-38-x86_64-linux-gnu.so + 0x21b7c)
#22 0x00007b1c3790005f pygobject_init (_gi.cpython-38-x86_64-linux-gnu.so + 0x1805f)
#23 0x00007b1c38b145bb type_call (libpython3.8.so.1.0 + 0x1145bb)
#24 0x00007b1c38b142d1 _PyObject_MakeTpCall (libpython3.8.so.1.0 + 0x1142d1)
#25 0x00007b1c38b110ad _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x1110ad)
#26 0x00007b1c38b0abaf _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x10abaf)
#27 0x00007b1c38b19aa3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x119aa3)
#28 0x00007b1c38b13cea _PyObject_FastCallDict (libpython3.8.so.1.0 + 0x113cea)
#29 0x00007b1c38b1ff75 slot_tp_init (libpython3.8.so.1.0 + 0x11ff75)
#30 0x00007b1c38b145bb type_call (libpython3.8.so.1.0 + 0x1145bb)
#31 0x00007b1c38b142d1 _PyObject_MakeTpCall (libpython3.8.so.1.0 + 0x1142d1)
#32 0x00007b1c38b10b21 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x110b21)
#33 0x00007b1c38b19df7 function_code_fastcall (libpython3.8.so.1.0 + 0x119df7)
#34 0x00007b1c38b0bbf7 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10bbf7)
#35 0x00007b1c38b0a6e4 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x10a6e4)
#36 0x00007b1c38b86ddd PyEval_EvalCodeEx (libpython3.8.so.1.0 + 0x186ddd)
#37 0x00007b1c38b86d8f PyEval_EvalCode (libpython3.8.so.1.0 + 0x186d8f)
#38 0x00007b1c38ba82a8 run_eval_code_obj (libpython3.8.so.1.0 + 0x1a82a8)
#39 0x00007b1c38ba7533 run_mod (libpython3.8.so.1.0 + 0x1a7533)
#40 0x00007b1c38a90d18 pyrun_file (libpython3.8.so.1.0 + 0x90d18)
#41 0x00007b1c38a8ff91 PyRun_SimpleFileExFlags (libpython3.8.so.1.0 + 0x8ff91)
#42 0x00007b1c38a87238 Py_RunMain.cold (libpython3.8.so.1.0 + 0x87238)
#43 0x00007b1c38b7a66d Py_BytesMain (libpython3.8.so.1.0 + 0x17a66d)
#44 0x00007b1c38da5082 __libc_start_main (libc.so.6 + 0x27082)
#45 0x00005660e849009e _start (python3.8 + 0x109e)
Dec 15 06:37:04 dom0 systemd[1]: systemd-coredump@4856-55763-0.service: Succeeded.
Dec 15 06:37:04 dom0 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@4856-55763-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 15 06:37:04 dom0 widget-wrapper[55761]: /usr/bin/widget-wrapper: line 14: 55762 Illegal instruction (core dumped) "$@"
Dec 15 06:37:04 dom0 widget-wrapper[55761]: exiting with 1
Dec 15 06:37:04 dom0 systemd[5070]: qubes-widget@qui-domains.service: Main process exited, code=exited, status=132/n/a
Dec 15 06:37:04 dom0 systemd[5070]: qubes-widget@qui-domains.service: Failed with result 'exit-code'.
Dec 15 06:37:04 dom0 audit: BPF prog-id=0 op=UNLOAD
Dec 15 06:37:04 dom0 audit: BPF prog-id=0 op=UNLOAD
Dec 15 06:37:04 dom0 audit: BPF prog-id=0 op=UNLOAD
Dec 15 06:37:05 dom0 systemd[5070]: qubes-widget@qui-domains.service: Scheduled restart job, restart counter is at 4848.
Dec 15 06:37:05 dom0 systemd[5070]: Stopped Qubes Widget service.
Dec 15 06:37:05 dom0 systemd[5070]: Started Qubes Widget service.
Dec 15 06:37:06 dom0 audit[55772]: ANOM_ABEND auid=1000 uid=1000 gid=1001 ses=3 pid=55772 comm="qui-domains" exe="/usr/bin/python3.8" sig=4 res=1
Dec 15 06:37:06 dom0 kernel: show_signal: 20 callbacks suppressed
Dec 15 06:37:06 dom0 kernel: traps: qui-domains[55772] trap invalid opcode ip:7ed53d99e7e0 sp:7ffef9b7d380 error:0 in libgtk-3.so.0.2404.24[7ed53d886000+38d000]
Dec 15 06:37:06 dom0 kernel: audit: type=1701 audit(1671086226.160:43990): auid=1000 uid=1000 gid=1001 ses=3 pid=55772 comm="qui-domains" exe="/usr/bin/python3.8" sig=4 res=1
Dec 15 06:37:06 dom0 audit: BPF prog-id=14611 op=LOAD
Dec 15 06:37:06 dom0 audit: BPF prog-id=14612 op=LOAD
Dec 15 06:37:06 dom0 kernel: audit: type=1334 audit(1671086226.179:43991): prog-id=14611 op=LOAD
Dec 15 06:37:06 dom0 kernel: audit: type=1334 audit(1671086226.179:43992): prog-id=14612 op=LOAD
Dec 15 06:37:06 dom0 kernel: audit: type=1334 audit(1671086226.180:43993): prog-id=14613 op=LOAD
Dec 15 06:37:06 dom0 audit: BPF prog-id=14613 op=LOAD
Dec 15 06:37:06 dom0 systemd[1]: Started Process Core Dump (PID 55773/UID 0).
Dec 15 06:37:06 dom0 kernel: audit: type=1130 audit(1671086226.188:43994): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@4857-55773-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 15 06:37:06 dom0 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@4857-55773-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 15 06:37:06 dom0 systemd-coredump[55774]: Removed old coredump core.qui-domains.1000.48bd9585ab3648f1aab466d305abbc61.49745.1671085072000000.lz4.
Dec 15 06:37:06 dom0 systemd-coredump[55774]: Process 55772 (qui-domains) of user 1000 dumped core.
Stack trace of thread 55772:
#0 0x00007ed53d99e7e0 gtk_css_node_real_update_style (libgtk-3.so.0 + 0x19e7e0)
#1 0x00007ed53d99d4e7 gtk_css_node_ensure_style.part.0 (libgtk-3.so.0 + 0x19d4e7)
#2 0x00007ed53d99d775 gtk_css_node_get_style (libgtk-3.so.0 + 0x19d775)
#3 0x00007ed53db0c015 gtk_style_context_save_named (libgtk-3.so.0 + 0x30c015)
#4 0x00007ed53d8e32de gtk_style_update_from_context (libgtk-3.so.0 + 0xe32de)
#5 0x00007ed53d8e3623 gtk_style_constructed (libgtk-3.so.0 + 0xe3623)
#6 0x00007ed53f0832e7 g_object_new_internal (libgobject-2.0.so.0 + 0x192e7)
#7 0x00007ed53f084fac g_object_new_valist (libgobject-2.0.so.0 + 0x1afac)
#8 0x00007ed53f08530d g_object_new (libgobject-2.0.so.0 + 0x1b30d)
#9 0x00007ed53d8e20ad _gtk_style_new_for_path (libgtk-3.so.0 + 0xe20ad)
#10 0x00007ed53d8e2103 gtk_style_new (libgtk-3.so.0 + 0xe2103)
#11 0x00007ed53d8e5075 gtk_widget_get_default_style (libgtk-3.so.0 + 0xe5075)
#12 0x00007ed53dbab15d gtk_widget_init (libgtk-3.so.0 + 0x3ab15d)
#13 0x00007ed53f09d457 g_type_create_instance (libgobject-2.0.so.0 + 0x33457)
#14 0x00007ed53f083205 g_object_new_internal (libgobject-2.0.so.0 + 0x19205)
#15 0x00007ed53f084fac g_object_new_valist (libgobject-2.0.so.0 + 0x1afac)
#16 0x00007ed53f08530d g_object_new (libgobject-2.0.so.0 + 0x1b30d)
#17 0x00007ed53d8db86b gtk_status_icon_init (libgtk-3.so.0 + 0xdb86b)
#18 0x00007ed53f09d498 g_type_create_instance (libgobject-2.0.so.0 + 0x33498)
#19 0x00007ed53f083205 g_object_new_internal (libgobject-2.0.so.0 + 0x19205)
#20 0x00007ed53f0846ad g_object_new_with_properties (libgobject-2.0.so.0 + 0x1a6ad)
#21 0x00007ed53f249b7c pygobject_constructv (_gi.cpython-38-x86_64-linux-gnu.so + 0x21b7c)
#22 0x00007ed53f24005f pygobject_init (_gi.cpython-38-x86_64-linux-gnu.so + 0x1805f)
#23 0x00007ed5403145bb type_call (libpython3.8.so.1.0 + 0x1145bb)
#24 0x00007ed5403142d1 _PyObject_MakeTpCall (libpython3.8.so.1.0 + 0x1142d1)
#25 0x00007ed5403110ad _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x1110ad)
#26 0x00007ed54030abaf _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x10abaf)
#27 0x00007ed540319aa3 _PyFunction_Vectorcall (libpython3.8.so.1.0 + 0x119aa3)
#28 0x00007ed540313cea _PyObject_FastCallDict (libpython3.8.so.1.0 + 0x113cea)
#29 0x00007ed54031ff75 slot_tp_init (libpython3.8.so.1.0 + 0x11ff75)
#30 0x00007ed5403145bb type_call (libpython3.8.so.1.0 + 0x1145bb)
#31 0x00007ed5403142d1 _PyObject_MakeTpCall (libpython3.8.so.1.0 + 0x1142d1)
#32 0x00007ed540310b21 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x110b21)
#33 0x00007ed540319df7 function_code_fastcall (libpython3.8.so.1.0 + 0x119df7)
#34 0x00007ed54030bbf7 _PyEval_EvalFrameDefault (libpython3.8.so.1.0 + 0x10bbf7)
#35 0x00007ed54030a6e4 _PyEval_EvalCodeWithName (libpython3.8.so.1.0 + 0x10a6e4)
#36 0x00007ed540386ddd PyEval_EvalCodeEx (libpython3.8.so.1.0 + 0x186ddd)
#37 0x00007ed540386d8f PyEval_EvalCode (libpython3.8.so.1.0 + 0x186d8f)
#38 0x00007ed5403a82a8 run_eval_code_obj (libpython3.8.so.1.0 + 0x1a82a8)
#39 0x00007ed5403a7533 run_mod (libpython3.8.so.1.0 + 0x1a7533)
#40 0x00007ed540290d18 pyrun_file (libpython3.8.so.1.0 + 0x90d18)
#41 0x00007ed54028ff91 PyRun_SimpleFileExFlags (libpython3.8.so.1.0 + 0x8ff91)
#42 0x00007ed540287238 Py_RunMain.cold (libpython3.8.so.1.0 + 0x87238)
#43 0x00007ed54037a66d Py_BytesMain (libpython3.8.so.1.0 + 0x17a66d)
#44 0x00007ed540717082 __libc_start_main (libc.so.6 + 0x27082)
#45 0x000055b1c5deb09e _start (python3.8 + 0x109e)