[qubes-users] dvm considerably slower on R4.1

Has the 4.1 as well as 4.1.1, because I did test on both, release added additional overhead to the creation of a dvm?

I have identical dvm's for Firefox on 4.0 and 4.1 and on 4.1 it takes almost, in the region of 0.5 - 0.75 seconds almost, **double** the time for a new dvm to be ready for usage, 18 seconds vs 36 (both rounded for convenience).

That is quite a hefty difference. Is there any explanation for this behavior?

I have the similar as yours, but the time of one of my qubes cannot be open because of light VM error that I got, which is error 135. The time doubling effect is just to place for the computer took way too long on some occasional reasons to go for it. Anyways, it’s just the distro issue from Debian.

Howard Chen wrote:

I have the similar as yours, but the time of one of my qubes cannot be open
because of light VM error that I got, which is error 135. The time doubling
effect is just to place for the computer took way too long on some
occasional reasons to go for it. Anyways, it's just the distro issue from
Debian.

How can it be a distro problem from Debian? Or Fedora? The same version of either on either 4.0 or 4.1 should be the same. Or not?

The problem lies within your computer processing speed and the numbers of qubes you has opened and active in one minute; This will significantly increases the opening time. As a result, it doubles the time of opening and activating the qubes to be online.

Howard Chen wrote:

The problem lies within your computer processing speed and the numbers of
qubes you has opened and active in one minute; This will significantly
increases the opening time. As a result, it doubles the time of opening and
activating the qubes to be online.

I didn't spell that out down to the letter but to make matters worse when i tested the creation time of the dvm on 4.0 the system was in fact far more busy compared to my test on 4.1. on 4.1 only the bare minimum was booted, sys-net, sys-firewall, sys-usb, sys-whonix, and that is it. On any regular day there are about 18 VMs running at any given time. So definitely the measured time was not influenced by how busy, or lack of, my system was.

It might be the speed in which to generate a new random number for this disposableVM and taking the file for a long time due to the size of that one file.

Qubes wrote:

Has the 4.1 as well as 4.1.1, because I did test on both, release added additional overhead to the creation of a dvm?

I have identical dvm's for Firefox on 4.0 and 4.1 and on 4.1 it takes almost, in the region of 0.5 - 0.75 seconds almost, **double** the time for a new dvm to be ready for usage, 18 seconds vs 36 (both rounded for convenience).

That is quite a hefty difference. Is there any explanation for this behavior?

This really is quite an annoyance, my system has slowed down quite a bit in the area of dvm's and i use them intensively. Can someone help me understand this problem? Where must i scratch under the bonnet to determine why dvm creation takes so much longer on my 4.1 install compared to my 4.0 install. I have two identical SSD's (make, model, etc) one installed with 4.0 and the other with 4.1. The SSD i used for the 4.1 install has much less mileage on so general wear of the SSD should effect my 4.1 install less.

Qubes wrote:

Qubes wrote:

Has the 4.1 as well as 4.1.1, because I did test on both, release added additional overhead to the creation of a dvm?

I have identical dvm's for Firefox on 4.0 and 4.1 and on 4.1 it takes almost, in the region of 0.5 - 0.75 seconds almost, **double** the time for a new dvm to be ready for usage, 18 seconds vs 36 (both rounded for convenience).

That is quite a hefty difference. Is there any explanation for this behavior?

This really is quite an annoyance, my system has slowed down quite a bit in the area of dvm's and i use them intensively. Can someone help me understand this problem? Where must i scratch under the bonnet to determine why dvm creation takes so much longer on my 4.1 install compared to my 4.0 install. I have two identical SSD's (make, model, etc) one installed with 4.0 and the other with 4.1. The SSD i used for the 4.1 install has much less mileage on so general wear of the SSD should effect my 4.1 install less.

It is not only dvm creation that is slow any VM boot time has increased significantly.

There can be multiple reasons for a slower 4.1 experience. Known ones are:

1. CPU runs at ~800 MHz or so [1]
2. You're a file pool user. File pools were serialized in 4.1, likely dropping their performance by ~30-50%. [2]
3. Possibly further issues in 4.1 [2].

[1] https://github.com/QubesOS/qubes-issues/issues/4604
[2] https://github.com/QubesOS/qubes-issues/issues/7075

Anyway it seems that just a few users are affected. Others report 8s VM startup performance or less (see forum).

David Hobach wrote:

There can be multiple reasons for a slower 4.1 experience. Known ones are:

1. CPU runs at ~800 MHz or so [1]
2. You're a file pool user. File pools were serialized in 4.1, likely dropping their performance by ~30-50%. [2]
3. Possibly further issues in 4.1 [2].

[1] Intel CPU Frequency Scaling Broken · Issue #4604 · QubesOS/qubes-issues · GitHub

If 4604 was my issue my understanding is that i would have experienced performance issues on 4.0 as well.

[2] 4.1 VM startup & qrexec performance issues · Issue #7075 · QubesOS/qubes-issues · GitHub

I have read through 7075 but i am still distilling the info. It does sound like i may be affected by the same issue.

What makes this debacle so uncomfortable is the fact that time spent waiting is cumulative and at the end of the day it adds up to a lot of time wasted.

Anyway it seems that just a few users are affected. Others report 8s VM startup performance or less (see forum).

I will try and make time to run some additional tests with the help of the scripts you provided in your detailed 7075 bug report. Would you mind sending me your updated scripts?

Thanks a lot for pointing these out to me.

You can see your pool driver by executing `qvm-pool`.

And the latest version of my scripts will always be available at [1].

Btw somehow one gets used to the worse performance after a while...

[1] https://github.com/3hhh/qubes-performance

Mine currently looks like this:

Qubes release 4.1 (R4.1)
dom0 kernel: 5.10.109-1.fc32.qubes.x86_64
dom0 Xen: 4.14.5
VM kernel: 5.15.52-1.fc32

Xen:
   qrexec startup: 240 ms
   Qubes DB: 8800 ms
   VM handover: 8901 ms
Linux:
   kernel: 4480 ms
   system: 4298 ms
   user: 178 ms
   system critical-chain:
     multi-user.target @4.298s
     └─qubes-qrexec-agent.service @4.164s +133ms
       └─systemd-user-sessions.service @4.102s +31ms
         └─network.target @4.092s
           └─qubes-network-uplink.service @2.638s +1.447s
             └─basic.target @2.544s
               └─sockets.target @2.544s
                 └─dbus.socket @2.542s
                   └─sysinit.target @2.531s
                     └─systemd-update-utmp.service @2.482s +47ms
                       └─systemd-tmpfiles-setup.service @2.349s +122ms
                         └─local-fs.target @2.317s
                           └─run-user-1000.mount @4.773s
                             └─local-fs-pre.target @1.427s
                               └─systemd-tmpfiles-setup-dev.service @1.150s +276ms
                                 └─systemd-sysusers.service @1.090s +58ms
                                   └─systemd-remount-fs.service @1.006s +80ms
                                     └─systemd-fsck-root.service @771ms +233ms
                                       └─systemd-journald.socket @572ms
                                         └─-.mount @519ms
                                           └─-.slice @519ms
   qrexec service critical-chain:
     qubes-qrexec-agent.service +133ms
     └─systemd-user-sessions.service @4.102s +31ms
       └─network.target @4.092s
         └─qubes-network-uplink.service @2.638s +1.447s
           └─basic.target @2.544s
             └─sockets.target @2.544s
               └─dbus.socket @2.542s
                 └─sysinit.target @2.531s
                   └─systemd-update-utmp.service @2.482s +47ms
                     └─systemd-tmpfiles-setup.service @2.349s +122ms
                       └─local-fs.target @2.317s
                         └─run-user-1000.mount @4.773s
                           └─local-fs-pre.target @1.427s
                             └─systemd-tmpfiles-setup-dev.service @1.150s +276ms
                               └─systemd-sysusers.service @1.090s +58ms
                                 └─systemd-remount-fs.service @1.006s +80ms
                                   └─systemd-fsck-root.service @771ms +233ms
                                     └─systemd-journald.socket @572ms
                                       └─-.mount @519ms
                                         └─-.slice @519ms
   qubes.GetDate: n/a
   qubes.WindowIconUpdater: n/a
   qrexec service: 4102 ms
qrexec (1st run):
   exec time: 4602 ms (depends on clock sync)
   return: 5479 ms
qrexec (2nd run):
   exec time: -105 ms (depends on clock sync)
   return: 782 ms
Overall: 23202 ms (excl. 2nd qrexec run)

David Hobach wrote:

You can see your pool driver by executing `qvm-pool`.

And the latest version of my scripts will always be available at [1].

Btw somehow one gets used to the worse performance after a while...

[1] GitHub - 3hhh/qubes-performance: Analyze Qubes OS VM startup performance.

Thank you very much.

And i really appreciate the efforts that you have already put into resolving this issue. You are really helping your community tremendously. High five.

This is the first time i am suffering from Qubes performance issues and i have to say it is quite frustrating. If 4.0 wasn't going EOL next month i would continue using 4.0 for now.