Preloaded dispoables in 4.3

I’ve had reports of issues with the preloaded disposables in 4.3, which
agree with my own experience.

I thought this would be a great new feature, but something seems off.
I find that the time saving is marginal, and if I start a number of
disposables quickly, the load time is actually slower than under 4.2.

I see that this is on GitHub as a possibility, but it’s a regular
feature for me and some other users. Does anyone else see this behavior?
I’d like to get a feeling for incidence before going to GitHub.

1 Like

I don’t see this behavior. For me preloaded disposables are “starting” in under 5 seconds, which is faster than regular boots, if feels. But I haven’t exactly benchmarked.

But I think @ben-grande showcased some benchmark during his summit talk this year.

1 Like

In the case of using a preloaded, this unlocks the pre-loaded VM then start a new qube in the background. In comparison to running a displosable from scratch, this requires more CPU and I/O and memory work.

It should be faster to get your program running in the disposable, but depending on hardware, it might be a bit slower or feel slower immediately after it started because of the preloading adding an extra load that was not there before.

Maybe this could be delayed by some time instead of rushing a start of a new preload.

3 Likes

This has been my experience is well.

I’ve also run into qrexec policy clashes between sys-audio and any predisposable that is either preloaded or started; but I suspect that’s a separate issue.

Mods: please merge this post with the mentioned thread.

@unman: continuation of Preloaded dispoables in 4.3

On 25-10-28 12:43:11, deeplow via Qubes OS Forum wrote:

[45_2] deeplow Volunteer Moderator
October 28

I don’t see this behavior. For me preloaded disposables are “starting” in under
5 seconds, which is faster than regular boots, if feels. But I haven’t exactly
benchmarked.

But I think @ben-grande showcased some benchmark during his summit talk this
year.

Previous Replies

[45] unman Qubes Team
October 28

I’ve had reports of issues with the preloaded disposables in 4.3, which
agree with my own experience.

This is my first time using the forum for a long message via e-mail, I
hope everything works. If this doesn’t work, we can try using the
qubes-users mailing list (which I am not subscribed to yet). And… it failed.

Can you please share the reports? So I could comment on that.

I thought this would be a great new feature, but something seems off.
I find that the time saving is marginal, and if I start a number of
disposables quickly, the load time is actually slower than under 4.2.

  • Can you measure the time the qube takes too boot and the time it takes
    to use it? I’ve had some templates taking a longer time to boot
    because some services were delaying the boot.
  • When you say quickly, is it concurrent or sequential calls?
  • I think that comparing against 4.2 is not a good metric in this case,
    you should compare against the same version and different modes, with
    preloaded disposables and without preloaded disposables. Comparing
    across versions may be impacted by the template being different, then
    you would need to compare no preloaded disposables on 4.2 and 4.3 to
    see the difference.

Measure startup time:

qvm-shutdown default-dvm
time qvm-run --no-gui --service -- default-dvm qubes.WaitForRunningSystem
qvm-shutdown default-dvm

If it takes too long to start:

systemd-analayze blame
systemd-analayze plot

Measure preload usage time:

qvm-features default-dvm preload-dispvm-max 1
disp="$(qvm-features default-dvm preload-dispvm)"
while ! qvm-check --paused -- "$disp"; do sleep 1; done
time qvm-run --no-gui --service -- default-dvm qubes.WaitForRunningSystem

Increase preload-dispvm-max and loop the qvm-run calls. I recommend
experimenting with different templates (Debian and Fedora).

Note that this test is barebones and checks total time of the qvm-run
call, not how fast the target replied. The performance benchmark gives
different results for different methods:

  • Admin API (qvm-run): dom0 to disposable
  • Admin API (raw): dom0 to disposable
  • Qrexec (qrexec-client-vm): qube to disposable

The Admin API (raw) results shows enough information on stages:

  • Execution
  • Cleanup
  • Total = Execution + Cleanup

But the raw method is a bit skewed because users normally use
qvm-run or qrexec-client-vm, which are a bit slower than the raw
variant. Anyway, it is the best method to see the per stage result.

The cleanup time can double on sequential calls because during this
process, other disposables are being preloaded.

I see that this is on GitHub as a possibility, but it’s a regular
feature for me and some other users. Does anyone else see this behavior?
I’d like to get a feeling for incidence before going to GitHub.

OpenQA reports preloaded disposables performance in
JSON
with some images generated from that results file. There is no alt text
for the images, but there is an idea to use matplotalt in the future.
If you have recommendations on how to render matplotlib graphs with alt
text, I’d be glad. Currently difficult to use matplotalt because it
requires pip, not available in distribution repositories. Note that
if you have the results files, you can generate the images from any
machine even outside of Qubes OS.

OpenQA @hw7 has a weak hardware with the following specs:

  • RAM: 16226 MiB
  • CPU 13th Gen Intel Core i3-1315U

And software:

  • Qubes R4.3-rc2
  • Kernel 6.12.47-1.fc41
  • Xen 4.19.3

And it still results on a great increase. Raw API calls from dom0 to
debian-13-xfce preloaded disposables, on 12 sequential calls, on calls
that don’t require a GUI, qubes.WaitForRunningSystem (results
in seconds):

Execution:

  • 0 preloads: start 6.2, lowest 5.8, highest 7.1, mean 6.3;
  • 3 preloads: start 0.4, lowest 0.4, highest 5.4 (outlier), mean 1.6;
  • 4 preloads: start 0.4, lowest 0.3, highest 5.6 (outlier), mean 1.1;
  • 5 preloads: start 0.4, lowest 0.4, highest 2.6 (outlier), mean 0.9;

Total:

  • 0 preloads: start 7.0, lowest 6.9, highest 8.0, mean 7.1;
  • 3 preloads: start 2.6, lowest 2.2, highest 7.3 (outlier), mean 3.9;
  • 4 preloads: start 2.4, lowest 1.0, highest 7.1 (outlier), mean 3.5;
  • 5 preloads: start 2.6, lowest 1.8, highest 5.6 (outlier), mean 3.4;

Images:

If you benchmark with qvm-run, you will see the total, while the
results above uses raw Admin API. You can also benchmark on your own
machine, the script is available at
qubes-core-admin/tests/dispvm_perf.py. The reader can be run outside
of Qubes and is available in the same directory at
dispvm_perf_reader.py.

The debian-13-xfce template with qvm-run and GUI call
(qubes.WaitForSession) had some weird results recently, they took longer
to be used than their no preload counterpart, 23% slower actually. No
idea why at this moment but on the same test, fedora is 80% faster against it’s no preload counterpart, which would be expected for debian also.

2 Likes

@solene

True, I’d need results from @unman to know when it is not worth it.

There is an idea for that:

But never a case there the mean for preloaded disposables was higher than normal/standard disposables, from my tests. If I see results from @unman or anyone else that shows something that could be improved with delays, I can consider that, but adding delays when doing various quick calls is worse, because there will be a time when there is no preloaded disposable and the run time will take as long as a normal preloaded disposable.

1 Like

You can also watch the journal:

sudo journalctl -fu qubesd

It will have timestamps and most related messages have preload in them to be easily searched (use case insensitive search).

1 Like

It is not because of lack of XP. It is because it is posted in Testing Team area of the forum. This is how to join:

1 Like

@alimirjamali: thanks, request sent.

1 Like

@kenosen

I cannot help based on the information provided. More information is needed. See the following:

https://forum.qubes-os.org/t/level-me-up-to-reply-to-any-thread/37035/2

XP awarded :stuck_out_tongue:

2 Likes

Hi @ben-grande

This is my first time using the forum for a long message via e-mail, I
hope everything works. If this doesn’t work, we can try using the
qubes-users mailing list (which I am not subscribed to yet). And… it failed.
This seems fine.

Can you please share the reports? So I could comment on that.
I doubt you want to see the reports - they rarely add anything to what
I have said, other than some fruity language.

If I understand you correctly you say that Debian based disposables show
some weird results, and do indeed take longer to load. I suspect this
is what we are seeing.
I’ll see if I can get some metrics for you. At the moment I have reports
and (some) timings to appearance of window. Testing calls that dont
require a GUI seems pointless to me - we’re here to test releases in
use, not to produce metrics that dont relate to user experience.

  • I think that comparing against 4.2 is not a good metric in this case,
    you should compare against the same version and different modes, with
    preloaded disposables and without preloaded disposables. Comparing
    across versions may be impacted by the template being different, then
    you would need to compare no preloaded disposables on 4.2 and 4.3 to
    see the difference.

I’m not clear on what you are saying. disposable use in 4.3 should be
the same as, or faster than 4.2. You seem to be suggesting that it may
not be. Is that right?

I never presume to speak for the Qubes team. When I comment in the Forum I speak for myself.

The outlier of debian-13-xfce being slower happend on Qubes OS openQA: qubesos-4.3-pull-requests-x86_64-Build2025102219-4.3-system_tests_dispvm_perf@hw7 test results, with “template-buildtime”: “2025-09-19 11:46:06”, “last-update”: “2025-10-22 20:14:35”.


{
...
    "template": "debian-13-xfce",
    "date": "2025-10-22T21:54:49",
    "memory": 400,
    "maxmem": 4000,
    "vcpus": 2,
    "qrexec_timeout": 60,
    "shutdown_timeout": 60,
    "kernel": "6.12.47-1.fc41",
    "kernelopts": "swiotlb=2048",
    "template-buildtime": "2025-09-19 11:46:06",
    "last-update": "2025-10-22 20:14:35",
    "hcl-qubes": "R4.3-rc2",
    "hcl-xen": "4.19.3",
    "hcl-kernel": "6.12.47-1",
    "hcl-memory": 16226,
    "hcl-certified": false,
    "hcl-brand": "Star Labs",
    "hcl-model": "StarBook",
    "hcl-bios": "8.97",
    "hcl-cpu": "13th Gen Intel(R) Core(TM) i3-1315U"
  },
}

On Qubes OS openQA: qubesos-4.3-pull-requests-x86_64-Build2025110304-4.3-system_tests_dispvm_perf@hw7 test results, it appears normal again:

{
...
    "template": "debian-13-xfce",
    "date": "2025-11-03T06:14:37",
    "memory": 400,
    "maxmem": 4000,
    "vcpus": 2,
    "qrexec_timeout": 60,
    "shutdown_timeout": 60,
    "kernel": "6.17.4-1.fc41",
    "kernelopts": "swiotlb=2048",
    "template-buildtime": "2025-10-23 14:56:15",
    "last-update": "2025-11-03 04:30:40",
    "hcl-qubes": "R4.3-rc3",
    "hcl-xen": "4.19.3",
    "hcl-kernel": "6.17.4-1",
    "hcl-memory": 16226,
    "hcl-certified": false,
    "hcl-brand": "Star Labs",
    "hcl-model": "StarBook",
    "hcl-bios": "8.97",
    "hcl-cpu": "13th Gen Intel(R) Core(TM) i3-1315U"
  },
}

Which kernel are your qubes using? It seems that kernel 6.12.47-1.fc41 with template from 2025-09-19 and updates till 2025-10-22 had the strange results. Kernel 6.17.2 with template 2025-10-23 and updates till 2025-11-03 had normal results.

Use qubes.WaitForSession instead of qubes.WaitForRunningSystem them, that is the only thing necessary to know how fast the gui agent can reply. It won’t account starting a GUI application but it is as close as we can go.

Nothing internally was changed that would making normal disposables slower on 4.3 than on 4.2, but different template versions can give different results.

Preloaded disposable should always be faster than normal disposable though.

The performance benchmark for GUI test variations with 0 and 4 preloaded disposables, but not in between. The following PR will fill this gap, from 0-6 preloaded disposables for the GUI variant.

@ben-grande -
I’ve gathered some data on this, generally taken on Qubes Certified
machines, (old school x230 based, 16G RAM).
All data was gathered on clean 4.3 installs, using debian-13-xfce
templates. The preloaded template was cloned to produce the second
debian-dvm template used in testing, no preloaded set.

  1. Simple call from dom0:
    “i use shortcuts A LOT, generally to open terminals. I checked the
    opening time for a terminal with 8 qubes running in the background.
    default-dvm: 15.2
    debian-dvm: 15.5
    Average result from 10 runs
    Median values- 14, 15”

  2. Loading simple web page in firefox in disposable:
    “i guess this is basic in my mutt - opening html mail in an offline
    dispVM.
    i tried my standard set up, running 12 vm with some background
    activity. i guess it’s about 10s as i scroll through email to see
    brain dead html.
    Average for load 2nd firefox window with preloaded - 89.52
    with normal dispVM - 74.61
    WTF?”

"ok, i kept background activity to a minimum:
2nd window - 59. (On one run the dispVM didnt load at all)
normal dispVM - 36.4

“ok, i kept background activity to a minimum AND stretched the time to 20s
2nd window - 52.
with normal dispVM - 32.8
That’s better, but still shit. its another Qubes “improvement” that
makes things worse for me. MAYBE, if I slow down my use of dispVM I
could see this improvement, but its not working for me as is.”

This is quite informal, but I made sure that these were default
debian-13-xfce installs, using kernel latest. Since they use the same
template, and the disposable templates are identical bar the preloaded
property, we should be able to rule out any other factors.
It does confirm my take in actual use - preloaded disposables do not
seem to make a significant impact, and in some cases actually slow
things down.

I could try to run similar tests with longer delays between the calls to
a second disposable - 30, 40, 60 seconds? What would you suggest?
I have not had any testing done with Fedora templates - they do not
feature in my workflow, but I could try that if necessary.

Of course, this may be absolutely expected, and this is a feature that
will not be of any use to users with limited resources, and a reasonable
number of qubes in use. If so, I do not think it should be enabled in
a standard install without significant warnings.

It would be extremely helpful if other testers could share their
experience of this feature in actual use.

unman

2 Likes

Is the average the total? Is that 89.52s the total of 10 runs? Or is it 8.9s? Anyway, the time is longer for preloaded disposables, just want to get a sense of scale.

Ok, understood that reducing load did not show better results for preloaded disposables.

Does it take 32.8s to open in Firefox in a fresh disposable while opening a terminal takes 15.5s? When there is a lot of load, it takes 36.4 seconds to open Firefox, this is what I understood.

Without load:

  • Normal disposable: Firefox: 32.8s, Terminal: 15.5s
  • Preloaded disposable: Firefox: 52s

With load:

  • Normal disposable: Firefox: 36.4
  • Preloaded disposable: Firefox: 59s

Slowing down shouldn’t be necessary at all. Bear in mind that you were the first to report such thing and I have not left any user complaints lingering. It will be solved eventually when I have enough information to be able to reproduce the issue. Although you may not perceive it as an improvement now because you found an issue, I encourage you to decide only after this issue is solved.

Thanks for the report, it helps narrow the path to solve the issue as I couldn’t reproduce it myself.

I still need some information.

Yes, I understand these are raw measures, that is intentional to not consider application startup time (browser, editor, viewer, whatever).

Change qubes.WaitForRunningSystem for qubes.WaitForSession for GUI calls.

From the issue 10365 you opened and I commented:

  1. As to why it is taking longer, please provide the qubesd logs, see my forum reply.

While you do your workflow, please collect logs qubesd in dom0:

sudo journalctl -fu qubesd

It will be timestamped, search case-insensitive the string preload.

When a disposable starts to be preloaded, it is logged “Marking preloaded qube”

  • When a disposable completes the preloading stage, it is logged “Preloading finished”.
  • When a preloaded disposable is requested, it is logged “Requesting preloaded qube”

It would be best if the qubesd log was fully provided from all your tests. You should not manage the state of other qubes else it may be logged and leak to qubesd and included in your post.

Shouldn’t be necessary, I saw you did 20s stretch and it didn’t help, that is enough.

Please report if that helps, it may trim down the path to understand if this issue is Debian specific or not.

Not expected at all. Haven’t seen this happen. It is tested on computers with limited resources, such as the Star Labs StarBook, CPU 13th Gen Intel(R) Core™ i3-1315U", RAM 16GB, which didn’t show the same results as yours, thus we haven’t noticed. And yes, it is intended to be used even with limited resources.

1 Like

@ben-grand thanks for the productive reply.
The quotes were taken from other users who are testing 4.3, and their
results match my own perception, (and that of @kenosen).
I think you have interpreted correctly.

Is the average the total? Is that 89.52s the total of 10 runs? Or is it 8.9s? Anyway, the time is longer for preloaded disposables, just want to get a sense of scale.
I understand the 89.52 to be the average of the runs - not the total.

Ok, understood that reducing load did not show better results for preloaded disposables.
Does it take 32.8s to open in Firefox in a fresh disposable while opening a terminal takes 15.5s? When there is a lot of load, it takes 36.4 seconds to open Firefox, this is what I understood.
Right.

Slowing down shouldn’t be necessary at all. Bear in mind that you were the first to report such thing and I have not left any user complaints lingering. It will be solved eventually when I have enough information to be able to reproduce the issue. Although you may not perceive it as an improvement now because you found an issue, I encourage you to decide only after this issue is solved.
I agree. To avoid doubt, the quotes were not mine but taken from email I
received.

I still need some information.
I will run the tests you have suggested myself. I have a vanilla 4.3
install on a certified x230 machine, but it will take me a little time
to gather results.

Cheers

I never presume to speak for the Qubes team.
When I comment in the Forum I speak for myself.

Not sure this is helpful:

user@dom0:~$ qvm-shutdown default-dvm && time qvm-run --service -- default-dvm qubes.WaitForRunningSystem && qvm-shutdown default-dvm
Running 'qubes.WaitForRunningSystem' on default-dvm

real	0m10.319s
user	0m0.151s
sys	0m0.059s
user@dom0:~$ qvm-shutdown browser-dvm && time qvm-run --service -- browser-dvm qubes.WaitForRunningSystem && qvm-shutdown browser-dvm
Running 'qubes.WaitForRunningSystem' on browser-dvm

real	0m10.074s
user	0m0.087s
sys	0m0.038s

But like @unman I have hotkeys to load a browser, or other application, and that takes much longer. So I wrote a little script (adding an xterm call) and got these results:

Testing: default-dvm
----------------------------
Run 1:
  Without GUI (qvm-start):

real	0m12.530s
user	0m0.223s
sys	0m0.076s
  With GUI (opening terminal):
Running 'xterm -e exit' on default-dvm

real	0m12.718s
user	0m0.109s
sys	0m0.057s

Run 2:
  Without GUI (qvm-start):

real	0m13.166s
user	0m0.191s
sys	0m0.079s
  With GUI (opening terminal):
Running 'xterm -e exit' on default-dvm

real	0m12.840s
user	0m0.104s
sys	0m0.054s

Run 3:
  Without GUI (qvm-start):

real	0m13.318s
user	0m0.200s
sys	0m0.078s
  With GUI (opening terminal):
Running 'xterm -e exit' on default-dvm

real	0m11.868s
user	0m0.101s
sys	0m0.042s


Testing: browser-dvm
----------------------------
Run 1:
  Without GUI (qvm-start):

real	0m12.673s
user	0m0.322s
sys	0m0.114s
  With GUI (opening terminal):
Running 'xterm -e exit' on browser-dvm

real	0m11.956s
user	0m0.106s
sys	0m0.049s

Run 2:
  Without GUI (qvm-start):

real	0m18.133s
user	0m0.281s
sys	0m0.084s
  With GUI (opening terminal):
Running 'xterm -e exit' on browser-dvm

real	0m12.359s
user	0m0.130s
sys	0m0.063s

Run 3:
  Without GUI (qvm-start):

real	0m13.317s
user	0m0.245s
sys	0m0.077s
  With GUI (opening terminal):
Running 'xterm -e exit' on browser-dvm

real	0m14.000s
user	0m0.149s
sys	0m0.061s


Test complete!
user@dom0:~$ 

@kenosen

Preload one or more disposables for each template and try the same.

Well, not the same…

--dispvm=default-dvm

For each template.