Is documentation for preloaded dispvms available?

kenosen · August 11, 2025, 9:35am

Is documentation for preloaded dispvms available? Or does the documentation come after the release candidate?

alimirjamali · August 11, 2025, 10:04am

There is a draft announcement for it

github.com/QubesOS/qubes-posts

Announcement: Preloaded disposables

main ← ben-grande:announce-preload

opened 04:59PM - 02 Jul 25 UTC

ben-grande

+163 -0

I marked as draft but consider it mostly done besides the TODOs that cannot be d…one at this stage (because of docs repos freeze related to RST conversion) or cannot be done by me (see `TODO: reviewer`). Depends: https://github.com/QubesOS/qubes-attachment/pull/75 Depends: https://github.com/QubesOS/qubes-core-admin/pull/708 Depends: https://github.com/QubesOS/qubes-desktop-linux-manager/pull/272

Spoilers: The draft has 133 posts and replies (so far). This should show how complicated and involved is the documentation. Usually end users do not need to bother too much about the details. There is easy to use settings for preloaded disposables in installer, global config and individual DispVM settings. I will post some more screenshots in today’s newsletter.

kenosen · August 11, 2025, 8:14pm

Documentation says it’s set and forget. I’ve set 5 preloaded vms for my browser DVM. When I start one, I get five notifications of vms starting then five notifications of those vms shutting down. The same process occurs each time I start a new disposable.

alimirjamali · August 11, 2025, 8:17pm

Interesting. Is the system fully updated with testing repositories (and is it restarted after update)?

If the error persists, I would recommend opening a new issue.

kenosen · August 11, 2025, 8:48pm

Everything is updated and I’ve restarted. Still persists.

alimirjamali · August 11, 2025, 9:05pm

Then I would highly recommend you to file a new issue. The entire preloaded disposable design is extremely complex. Some bugs might be present at this stage.

ben-grande · August 12, 2025, 9:40am

They are trying to preload but not being able to.

Which template are you using? Did you set via the global config or vm settings?
Start watching the journal: `sudo journalctl -fu qubesd -o cat | grep -v pam_unix
Set the max to 0, wait a bit and then set to 1

What message is printed on the qubesd logs?

kenosen · August 12, 2025, 11:38pm

A browser-dvm, based on fedora-42-xfce (upgraded dirty).
First set directly from VM settings. Then tried global override.
Set max to 0 then 1, same issue.
You’ll see in the output below, I set to 2, then receive an error re qubes.WaitForRunningSystem

Output of qubesd logs

INFO: vm.browser-dvm: Received preload event 'start' because global feature was set to 2
INFO: vm.browser-dvm: Preloading '2' qube(s)
INFO: vm.disp6589: Marking preloaded qube
INFO: vm.disp6589: Creating directory: /var/lib/qubes/appvms/disp6589
INFO: vm.disp5201: Marking preloaded qube
INFO: vm.disp5201: Creating directory: /var/lib/qubes/appvms/disp5201
INFO: Updating appmenus for 'disp6589' in 'dom0'
INFO: Removing appmenus for 'disp6589' in 'dom0'
INFO: Updating appmenus for 'disp5201' in 'dom0'
INFO: Removing appmenus for 'disp5201' in 'dom0'
INFO: Updating appmenus for 'disp6589' in 'dom0'
INFO: Updating appmenus for 'disp5201' in 'dom0'
INFO: vm.disp6589: Starting qube disp6589
INFO: Created sparse file: '/var/lib/qubes/appvms/disp6589/volatile-dirty.img~lvadsgml'
INFO: Hardlinked file: '/var/lib/qubes/vm-templates/fedora-41-xfce-OLD/root.img' -> '/var/lib/qubes/appvms/disp6589/root.img'
INFO: Hardlinked file: '/var/lib/qubes/appvms/browser-dvm/private.img' -> '/var/lib/qubes/appvms/disp6589/private.img'
INFO: Renamed file: '/var/lib/qubes/appvms/disp6589/volatile-dirty.img~lvadsgml' -> '/var/lib/qubes/appvms/disp6589/volatile-dirty.img'
INFO: Reflinked file: '/var/lib/qubes/appvms/disp6589/private.img' -> '/var/lib/qubes/appvms/disp6589/private-dirty.img~gdlnx_pw'
INFO: Renamed file: '/var/lib/qubes/appvms/disp6589/private-dirty.img~gdlnx_pw' -> '/var/lib/qubes/appvms/disp6589/private-dirty.img'
INFO: vm.disp5201: Starting qube disp5201
INFO: Reflinked file: '/var/lib/qubes/appvms/disp6589/root.img' -> '/var/lib/qubes/appvms/disp6589/root-dirty.img~hkhersk6'
INFO: Renamed file: '/var/lib/qubes/appvms/disp6589/root-dirty.img~hkhersk6' -> '/var/lib/qubes/appvms/disp6589/root-dirty.img'
INFO: vm.disp6589: Setting Qubes DB info for the qube
INFO: vm.disp6589: Starting Qubes DB
INFO: Created sparse file: '/var/lib/qubes/appvms/disp5201/volatile-dirty.img~0ct9lsqf'
INFO: vm.disp6589: Activating qube
INFO: Hardlinked file: '/var/lib/qubes/appvms/browser-dvm/private.img' -> '/var/lib/qubes/appvms/disp5201/private.img'
INFO: Hardlinked file: '/var/lib/qubes/vm-templates/fedora-41-xfce-OLD/root.img' -> '/var/lib/qubes/appvms/disp5201/root.img'
INFO: Reflinked file: '/var/lib/qubes/appvms/disp5201/private.img' -> '/var/lib/qubes/appvms/disp5201/private-dirty.img~7torg_a3'
INFO: Renamed file: '/var/lib/qubes/appvms/disp5201/private-dirty.img~7torg_a3' -> '/var/lib/qubes/appvms/disp5201/private-dirty.img'
INFO: Reflinked file: '/var/lib/qubes/appvms/disp5201/root.img' -> '/var/lib/qubes/appvms/disp5201/root-dirty.img~knkqnpma'
INFO: Renamed file: '/var/lib/qubes/appvms/disp5201/root-dirty.img~knkqnpma' -> '/var/lib/qubes/appvms/disp5201/root-dirty.img'
INFO: Renamed file: '/var/lib/qubes/appvms/disp5201/volatile-dirty.img~0ct9lsqf' -> '/var/lib/qubes/appvms/disp5201/volatile-dirty.img'
INFO: vm.disp5201: Setting Qubes DB info for the qube
INFO: vm.disp5201: Starting Qubes DB
INFO: vm.disp6589: Preload startup waiting 'qubes.WaitForRunningSystem' with '60' seconds timeout
INFO: vm.disp5201: Activating qube
ERROR: vm.disp6589: Start failed: Error on call to '$(PATH="/run/qubes-rpc:/usr/local/etc/qubes-rpc:/etc/qubes-rpc" command -v qubes.WaitForRunningSystem)' during preload startup
INFO: Removed file: '/var/lib/qubes/appvms/disp6589/private-dirty.img'
INFO: Removed file: '/var/lib/qubes/appvms/disp6589/volatile-dirty.img'
INFO: Removed file: '/var/lib/qubes/appvms/disp6589/private.img'
INFO: Removed file: '/var/lib/qubes/appvms/disp6589/root-dirty.img'
INFO: Removed file: '/var/lib/qubes/appvms/disp6589/root.img'
INFO: vm.disp6589: Automatic cleanup removes qube from preload list
INFO: vm.browser-dvm: Removing qube(s) from preloaded list: 'disp6589'
INFO: Removing appmenus for 'disp6589' in 'dom0'
INFO: vm.disp6589: Removing volume root: appvms/disp6589/root
INFO: vm.disp6589: Removing volume private: appvms/disp6589/private
INFO: vm.disp6589: Removing volume volatile: appvms/disp6589/volatile
INFO: vm.disp6589: Removing volume kernel: 6.15.7-1.fc41
INFO: Removed empty directory: '/var/lib/qubes/appvms/disp6589'
INFO: Removed file: '/var/lib/qubes/appvms/disp5201/private-dirty.img'
INFO: Removed file: '/var/lib/qubes/appvms/disp5201/volatile-dirty.img'
INFO: Removed file: '/var/lib/qubes/appvms/disp5201/private.img'
INFO: Removed file: '/var/lib/qubes/appvms/disp5201/root-dirty.img'
INFO: Removed file: '/var/lib/qubes/appvms/disp5201/root.img'
INFO: vm.disp5201: Automatic cleanup removes qube from preload list
INFO: vm.browser-dvm: Removing qube(s) from preloaded list: 'disp5201'
INFO: Removing appmenus for 'disp5201' in 'dom0'
INFO: vm.disp5201: Removing volume root: appvms/disp5201/root
INFO: vm.disp5201: Removing volume private: appvms/disp5201/private
INFO: vm.disp5201: Removing volume volatile: appvms/disp5201/volatile
INFO: vm.disp5201: Removing volume kernel: 6.15.7-1.fc41
INFO: Removed empty directory: '/var/lib/qubes/appvms/disp5201'
ERROR: Task exception was never retrieved
future: <Task finished name='Task-52553' coro=<Emitter.fire_event_async() done, defined at /usr/lib/python3.13/site-packages/qubes/events.py:211> exception=ExceptionGroup('unhandled errors in a TaskGroup', [QubesException('Error on call to \'$(PATH="/run/qubes-rpc:/usr/local/etc/qubes-rpc:/etc/qubes-rpc" command -v qubes.WaitForRunningSystem)\' during preload startup')])>
  + Exception Group Traceback (most recent call last):
  |   File "/usr/lib/python3.13/site-packages/qubes/events.py", line 243, in fire_event_async
  |     effect = task.result()
  |   File "/usr/lib/python3.13/site-packages/qubes/vm/mix/dvmtemplate.py", line 329, in on_domain_preload_dispvm_used
  |     async with asyncio.TaskGroup() as task_group:
  |                ~~~~~~~~~~~~~~~~~^^
  |   File "/usr/lib64/python3.13/asyncio/taskgroups.py", line 71, in __aexit__
  |     return await self._aexit(et, exc)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib64/python3.13/asyncio/taskgroups.py", line 173, in _aexit
  |     raise BaseExceptionGroup(
  |     ...<2 lines>...
  |     ) from None
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/lib/python3.13/site-packages/qubes/vm/dispvm.py", line 346, in on_domain_started_dispvm
    |     await asyncio.wait_for(
    |     ...<6 lines>...
    |     )
    |   File "/usr/lib64/python3.13/asyncio/tasks.py", line 507, in wait_for
    |     return await fut
    |            ^^^^^^^^^
    |   File "/usr/lib/python3.13/site-packages/qubes/vm/qubesvm.py", line 1929, in run_for_stdio
    |     raise subprocess.CalledProcessError(
    |         p.returncode, args[0], *stdouterr
    |     )
    | subprocess.CalledProcessError: Command '$(PATH="/run/qubes-rpc:/usr/local/etc/qubes-rpc:/etc/qubes-rpc" command -v qubes.WaitForRunningSystem)' returned non-zero exit status 1.
    |
    | During handling of the above exception, another exception occurred:
    |
    | Traceback (most recent call last):
    |   File "/usr/lib/python3.13/site-packages/qubes/vm/dispvm.py", line 563, in from_appvm
    |     await dispvm.start()
    |   File "/usr/lib/python3.13/site-packages/qubes/vm/dispvm.py", line 642, in start
    |     await super().start(**kwargs)
    |   File "/usr/lib/python3.13/site-packages/qubes/vm/qubesvm.py", line 1536, in start
    |     await self.fire_event_async(
    |         "domain-start", start_guid=start_guid
    |     )
    |   File "/usr/lib/python3.13/site-packages/qubes/events.py", line 243, in fire_event_async
    |     effect = task.result()
    |   File "/usr/lib/python3.13/site-packages/qubes/vm/dispvm.py", line 360, in on_domain_started_dispvm
    |     raise qubes.exc.QubesException(
    |         "Error on call to '%s' during preload startup" % service
    |     )
    | qubes.exc.QubesException: Error on call to '$(PATH="/run/qubes-rpc:/usr/local/etc/qubes-rpc:/etc/qubes-rpc" command -v qubes.WaitForRunningSystem)' during preload startup
    +------------------------------------

ben-grande · August 13, 2025, 7:02am

Some service is failing to start on browser-dvm or its template:

From dom0:

qvm-shutdown browser-dvm
qvm-run --service browser-dvm qubes.StartApp+qubes-run-terminal

From browser-dvm:

systemctl --wait is-system-running
systemctl --failed

kenosen · August 13, 2025, 9:03am

Seems you’ve found the cause: `efi.mount` unit fails · Issue #8954 · QubesOS/qubes-issues · GitHub

This is in F42, though again this was a dirty upgrade.
Works fine with freshly-downloaded F42-xfce template.