Q4.3.1, dom0 kernel 7.0.6-1, whonix workstation 18 VM with 6.18.31-1.f41 kernel, boot mode - unrestricted session. I came back and my disposable qube is just no longer there. I was running it for days. Happened on Q4.3.1 before, with other ws-18 disp VMs - but not on Q4.2 ws-17 (before the upgrade). Preload disp qubes feature was enabled (I disabled it now).
You may think that “initial” app inside the disp VM crashed, therefore whole VM was shut down - but I start disp VMs with “Run Terminal”, and its seem unlikely that QTerminal would crash. Although it didn’t happen on Q4.2 & Whonix WS 17, which used different terminal app.
Am I the only one with that problem? journalctl --reverse logs (lots of lines like dom0 qrexec-policy-daemon[8989]: qrexec: sdwdate-gui.ConnectCheck+: disp2186 -> sys-whonix: allowed to sys-whonix are omitted)
Jun 30 18:16:13 dom0 kernel: audit: type=1105 audit(1782832573.897:1517): pid=1119914 uid=1000 auid=1000 ses=7 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=>
Jun 30 18:16:13 dom0 audit[1119914]: USER_START pid=1119914 uid=1000 auid=1000 ses=7 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/>
Jun 30 18:16:13 dom0 sudo[1119914]: pam_unix(sudo:session): session opened for user root(uid=0) by user(uid=1000)
Jun 30 18:16:13 dom0 kernel: audit: type=1110 audit(1782832573.889:1516): pid=1119914 uid=1000 auid=1000 ses=7 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/pts/7 res=success'
Jun 30 18:16:13 dom0 audit[1119914]: CRED_REFR pid=1119914 uid=1000 auid=1000 ses=7 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/pts/7 res=success'
Jun 30 18:16:13 dom0 kernel: audit: type=1123 audit(1782832573.888:1515): pid=1119914 uid=1000 auid=1000 ses=7 msg='cwd="/home/user" cmd=6A6F75726E616C63746C202D2D72657665727365 exe="/usr/bin/sudo" terminal=pts/7 res=success'
Jun 30 18:16:13 dom0 audit[1119914]: USER_CMD pid=1119914 uid=1000 auid=1000 ses=7 msg='cwd="/home/user" cmd=6A6F75726E616C63746C202D2D72657665727365 exe="/usr/bin/sudo" terminal=pts/7 res=success'
Jun 30 18:16:13 dom0 sudo[1119914]: user : TTY=pts/7 ; PWD=/home/user ; USER=root ; COMMAND=/usr/bin/journalctl --reverse
Jun 30 18:16:13 dom0 kernel: audit: type=1101 audit(1782832573.886:1514): pid=1119914 uid=1000 auid=1000 ses=7 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/pts/7 res=success'
Jun 30 18:16:13 dom0 audit[1119914]: USER_ACCT pid=1119914 uid=1000 auid=1000 ses=7 msg='op=PAM:accounting grantors=pam_unix acct="user" exe="/usr/bin/sudo" hostname=dom0 addr=? terminal=/dev/pts/7 res=success'
Jun 30 18:13:55 dom0 kernel: xen-backend console-5-0: xenbus: device forcefully removed from xenstore
Jun 30 18:14:29 dom0 qubesd[2961]: INFO: vm.disp24: Removing volume kernel: 6.18.31-1.fc41
Jun 30 18:14:29 dom0 qubesd[2961]: INFO: vm.disp24: Removing volume volatile: qubes_dom0/vm-disp24-volatile
Jun 30 18:14:29 dom0 qubesd[2961]: INFO: vm.disp24: Removing volume private: qubes_dom0/vm-disp24-private
Jun 30 18:14:29 dom0 qubesd[2961]: INFO: vm.disp24: Removing volume root: qubes_dom0/vm-disp24-root
Jun 30 18:14:29 dom0 kernel: audit: type=1104 audit(1782832469.315:1511): pid=1118508 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="user" exe="/usr/sbin/runuser" hostname=? addr=? terminal=? res=success'
Jun 30 18:14:29 dom0 kernel: audit: type=1106 audit(1782832469.315:1510): pid=1118508 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_unix acct="user" exe="/usr/sbin/runuser" hostname=? addr=? terminal=>
Jun 30 18:14:29 dom0 runuser[1118508]: pam_unix(runuser:session): session closed for user user
Jun 30 18:14:29 dom0 audit[1118508]: CRED_DISP pid=1118508 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="user" exe="/usr/sbin/runuser" hostname=? addr=? terminal=? res=success'
Jun 30 18:14:29 dom0 audit[1118508]: USER_END pid=1118508 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_unix acct="user" exe="/usr/sbin/runuser" hostname=? addr=? terminal=? res=success'
Jun 30 18:14:29 dom0 qrexec-policy-daemon[8989]: qrexec: sdwdate-gui.ConnectCheck+: disp2186 -> sys-whonix: allowed to sys-whonix
Jun 30 18:14:29 dom0 qrexec-policy-daemon[8989]: qrexec: sdwdate-gui.ConnectCheck+: wst-work-0-msg-toxic -> sys-whonix: allowed to sys-whonix
Jun 30 18:14:29 dom0 widget-wrapper[18681]: qubesadmin.exc.QubesVMNotFoundError: No such domain: 'disp24'
Jun 30 18:14:29 dom0 widget-wrapper[18681]: raise exc_class(format_string, *args)
Jun 30 18:14:29 dom0 widget-wrapper[18681]: File "/usr/lib/python3.13/site-packages/qubesadmin/base.py", line 130, in _parse_qubesd_response
Jun 30 18:14:29 dom0 widget-wrapper[18681]: ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Jun 30 18:14:29 dom0 widget-wrapper[18681]: return self._parse_qubesd_response(return_data)
Jun 30 18:14:29 dom0 widget-wrapper[18681]: File "/usr/lib/python3.13/site-packages/qubesadmin/app.py", line 876, in qubesd_call
Jun 30 18:14:29 dom0 widget-wrapper[18681]: ^^^^^^^^^^^^^^^
Jun 30 18:14:29 dom0 widget-wrapper[18681]: payload_stream)
Jun 30 18:14:29 dom0 widget-wrapper[18681]: ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 30 18:14:29 dom0 widget-wrapper[18681]: return self.app.qubesd_call(dest, method, arg, payload,
Jun 30 18:14:29 dom0 widget-wrapper[18681]: File "/usr/lib/python3.13/site-packages/qubesadmin/base.py", line 96, in qubesd_call
Jun 30 18:14:29 dom0 widget-wrapper[18681]: None)
Jun 30 18:14:29 dom0 widget-wrapper[18681]: None,
Jun 30 18:14:29 dom0 widget-wrapper[18681]: self._method_prefix + 'GetAll',
Jun 30 18:14:29 dom0 widget-wrapper[18681]: self._method_dest,
Jun 30 18:14:29 dom0 widget-wrapper[18681]: properties_str = self.qubesd_call(
Jun 30 18:14:29 dom0 widget-wrapper[18681]: File "/usr/lib/python3.13/site-packages/qubesadmin/base.py", line 345, in _fetch_all_properties
Jun 30 18:14:29 dom0 widget-wrapper[18681]: ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Jun 30 18:14:29 dom0 widget-wrapper[18681]: self._fetch_all_properties()
Jun 30 18:14:29 dom0 widget-wrapper[18681]: File "/usr/lib/python3.13/site-packages/qubesadmin/base.py", line 240, in __getattr__
Jun 30 18:14:29 dom0 widget-wrapper[18681]: self._vm.devices_denied
Jun 30 18:14:29 dom0 widget-wrapper[18681]: File "/usr/lib/python3.13/site-packages/qui/devices/backend.py", line 144, in update_denied_devices
Jun 30 18:14:29 dom0 widget-wrapper[18681]: ~~~~~~~~~~~~~~~~~~~~~~~~~~^^
Jun 30 18:14:29 dom0 widget-wrapper[18681]: self.update_denied_devices()
Jun 30 18:14:29 dom0 widget-wrapper[18681]: File "/usr/lib/python3.13/site-packages/qui/devices/backend.py", line 58, in __init__
Jun 30 18:14:29 dom0 widget-wrapper[18681]: wrapped_vm = backend.VM(vm)
Jun 30 18:14:29 dom0 widget-wrapper[18681]: File "/usr/lib/python3.13/site-packages/qui/devices/device_widget.py", line 651, in vm_shutdown
Jun 30 18:14:29 dom0 widget-wrapper[18681]: ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 30 18:14:29 dom0 widget-wrapper[18681]: handler(subject, event, **kwargs)
Jun 30 18:14:29 dom0 widget-wrapper[18681]: File "/usr/lib/python3.13/site-packages/qubesadmin/events/__init__.py", line 307, in handle
Jun 30 18:14:29 dom0 widget-wrapper[18681]: Traceback (most recent call last):
Jun 30 18:14:29 dom0 widget-wrapper[18681]: Failed to handle event: disp24, domain-shutdown, {}
Jun 30 18:14:29 dom0 qrexec-policy-daemon[8989]: qrexec: sdwdate-gui.ConnectCheck+: wst-work-0-crypto-monero -> sys-whonix: allowed to sys-whonix
Jun 30 18:14:29 dom0 qrexec-policy-daemon[8989]: qrexec: sdwdate-gui.ConnectCheck+: wst-work-0-crypto-monero -> sys-whonix: allowed to sys-whonix
Jun 30 18:14:29 dom0 qrexec-policy-daemon[8989]: qrexec: sdwdate-gui.ConnectCheck+: wst-work-0-msg-toxic -> sys-whonix: allowed to sys-whonix
Jun 30 18:14:29 dom0 qrexec-policy-daemon[8989]: qrexec: sdwdate-gui.ConnectCheck+: wst-work-0-crypto-electrum-4 -> sys-whonix: allowed to sys-whonix
Jun 30 18:14:29 dom0 kernel: audit: type=1130 audit(1782832469.010:1509): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 30 18:14:29 dom0 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 30 18:14:29 dom0 systemd[1]: Started systemd-hostnamed.service - Hostname Service.
Jun 30 18:14:28 dom0 qrexec-policy-daemon[8989]: qrexec: sdwdate-gui.ConnectCheck+: disp2186 -> sys-whonix: allowed to sys-whonix
Jun 30 18:14:28 dom0 qrexec-policy-daemon[8989]: qrexec: sdwdate-gui.ConnectCheck+: disp5388 -> sys-whonix: allowed to sys-whonix
Jun 30 18:14:28 dom0 systemd[1]: Starting systemd-hostnamed.service - Hostname Service...
Jun 30 18:14:28 dom0 kernel: audit: type=1334 audit(1782832468.958:1508): prog-id=120 op=LOAD
Jun 30 18:14:28 dom0 kernel: audit: type=1334 audit(1782832468.958:1507): prog-id=119 op=LOAD
Jun 30 18:14:28 dom0 kernel: audit: type=1334 audit(1782832468.958:1506): prog-id=118 op=LOAD
Jun 30 18:14:28 dom0 audit: BPF prog-id=120 op=LOAD
Jun 30 18:14:28 dom0 audit: BPF prog-id=119 op=LOAD
Jun 30 18:14:28 dom0 audit: BPF prog-id=118 op=LOAD
Jun 30 18:14:28 dom0 kernel: audit: type=1334 audit(1782832468.957:1505): prog-id=116 op=UNLOAD
Jun 30 18:14:28 dom0 kernel: audit: type=1334 audit(1782832468.957:1504): prog-id=117 op=UNLOAD
Jun 30 18:14:28 dom0 audit: BPF prog-id=116 op=UNLOAD
Jun 30 18:14:28 dom0 audit: BPF prog-id=117 op=UNLOAD
Jun 30 18:14:28 dom0 audit[1118508]: USER_START pid=1118508 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_unix acct="user" exe="/usr/sbin/runuser" hostname=? addr=? terminal=? res=success'
Jun 30 18:14:28 dom0 kernel: audit: type=1105 audit(1782832468.943:1503): pid=1118508 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_unix acct="user" exe="/usr/sbin/runuser" hostname=? addr=? terminal=?>
Jun 30 18:14:28 dom0 runuser[1118508]: pam_unix(runuser:session): session opened for user user(uid=1000) by (uid=0)
Jun 30 18:14:28 dom0 kernel: audit: type=1103 audit(1782832468.938:1502): pid=1118508 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="user" exe="/usr/sbin/runuser" hostname=? addr=? terminal=? res=success'
Jun 30 18:14:28 dom0 audit[1118508]: CRED_ACQ pid=1118508 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="user" exe="/usr/sbin/runuser" hostname=? addr=? terminal=? res=success'
Jun 30 18:14:28 dom0 qubesd[2961]: INFO: Removing appmenus for 'disp24' in 'dom0'
Jun 30 18:14:28 dom0 systemd-homed[2820]: block device /sys/devices/virtual/block/dm-151 has been removed.
Jun 30 18:14:28 dom0 systemd-homed[2820]: block device /sys/devices/virtual/block/dm-151 has been removed.
Jun 30 18:14:28 dom0 systemd-homed[2820]: block device /sys/devices/virtual/block/dm-181 has been removed.
Jun 30 18:14:28 dom0 systemd-homed[2820]: block device /sys/devices/virtual/block/dm-181 has been removed.
Jun 30 18:14:21 dom0 systemd-homed[2820]: block device /sys/devices/virtual/block/dm-179 has been removed.
Jun 30 18:14:21 dom0 systemd-homed[2820]: block device /sys/devices/virtual/block/dm-179 has been removed.
Jun 30 18:14:21 dom0 systemd-homed[2820]: block device /sys/devices/virtual/block/dm-179 has been removed.
Jun 30 18:14:21 dom0 systemd-homed[2820]: block device /sys/devices/virtual/block/dm-179 has been removed.
If suspend was not the case. Can you share more logs? Can you look at /var/log/xen/console/guest-disp24.log? Also, I remember someone reporting this issue before preloaded disposables existed:
Can you report more information on Github with the guidance from Marek’s posts there?
Started disp VMs with said teplates and swiotlb=2048 systemd.journald.forward_to_console=1 systemd.journald.max_level_console=debug, 1.5Gb ram each and 1G+ of it available, PVH, routed to whonix gateway, will be waiting for them to crash in next few days or weeks. Disp preload OFF.
Can you report more information on Github with the guidance from Marek’s posts there?
In my experience, Github shadowbans posting from Tor IPs and rejects teporary emails. I can get to that later, if I would know how to reliably reproduce the issue or if it simply would happen more often.
Catched. Whonix ws 18 disp vm was shut down, fedora and debian are still running, here are detailed logs: https://pastebin.com/raw/b876yD5k
By the way, there were no “heavy usage” and actually no usage at all, unlike the github issue.
I have shared this thread in the Qubes’ matrix channel and the Whonix’s developers have seen it. So far, there is not enough information in the kernel trace. I think they will introduce more debugging to get a better error message. I think it is appropriate for you to follow-up with them on the Whonix forum.