Xen scheduler weights

[KitsuneNoBaka]

I’ve made myself small python script to change xen scheduler weight of qubes domain.
First it was for Dom0 only but it evolved for others too.

#!/usr/bin/env python3
# changing weight and cap of domains

import subprocess

def captureos( cmd ):
	with subprocess.Popen( cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE ) as proc:
		output, _ = proc.communicate()
	return output.decode().strip()

def xenweight( xlinfo, domV ):
	for x in domV:
		if captureos( [ 'xl', 'list', x[0] ] ):
			captureos( [ 'xl', 'sched-'+xlinfo, '-d', x[0], '-w', x[1], '-c', x[2] ] )

domV = [
	[ 'Domain-0',		'2048',	'0' ],
	[ 'sys-net',		'64',	'0' ],
	[ 'sys-firewall',	'64',	'0' ],
	[ 'sys-usb',		'64',	'0' ],
	[ 'sys-audio',		'64',	'0' ],
	[ 'sys-gui',		'1024',	'0' ],
	[ 'sys-gui-gpu',	'1024',	'0' ],
]
xlinfo = captureos( [ 'xl', 'info', 'xen_scheduler' ] )
xenweight( xlinfo, domV )

I would like discussion what weights are best for service qubes.

As I tested with iperf3 in sys-net and sys-firewall with YouTube playing, weights 64 for them was enought. For now.

Anyone have some idea what those should be?

PS: didn’t tested usb yet

[KitsuneNoBaka]

YT 720p at 1.5 speed is playing
Download from NAS of 10GB file at ~100MiB/s
iperf3 to NAS from sys-firewall

$ iperf3 -c 0.0.0.0
Connecting to host 0.0.0.0, port 5201
[  5] local 1.2.3.4 port 38160 connected to 0.0.0.0 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   954 Mbits/sec    0   1.12 MBytes       
[  5]   1.00-2.00   sec   112 MBytes   937 Mbits/sec    0   1.12 MBytes       
[  5]   2.00-3.00   sec   112 MBytes   938 Mbits/sec    0   1.12 MBytes       
[  5]   3.00-4.00   sec   112 MBytes   944 Mbits/sec    0   1.12 MBytes       
[  5]   4.00-5.00   sec   111 MBytes   933 Mbits/sec    0   1.12 MBytes       
[  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec    0   1.12 MBytes       
[  5]   6.00-7.00   sec   111 MBytes   931 Mbits/sec    0   1.12 MBytes       
[  5]   7.00-8.00   sec   113 MBytes   947 Mbits/sec    0   1.12 MBytes       
[  5]   8.00-9.00   sec   112 MBytes   936 Mbits/sec    0   1.12 MBytes       
[  5]   9.00-10.00  sec   112 MBytes   942 Mbits/sec    0   1.12 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec    0            sender
[  5]   0.00-10.01  sec  1.09 GBytes   939 Mbits/sec                  receiver

iperf Done.

[Atrate]

You might find this interesting if you’d like to apply these weights automatically at every Qube startup:

Qubes Auto Credit2: automatically set xen credit2 scheduling priority

Hello everyone! Posting another one of my scripts that I use to make Qubes snappier and more usable interactively This one makes makes use of the credit2-weight feature to automatically set credit2 scheduler weight (scheduling priority) for VMs. Enjoy! And thanks to @noskb and @renehoj for providing the idea for the Admin Events approach.

[KitsuneNoBaka]

I know this thread and as of now I’m not interested in any dynamic weight changing.
I’m just setting it at start with service. One shot.

[solene]

@Atrate 's script is not dynamic, it’s just using the weight information you store in a qube with qvm-feature to apply it when the qube starts?

[KitsuneNoBaka]

USB testing.

Machine:

Lenovo ThinkPad T14 Gen 3 Intel integrated
CPU: i5 1245U
P-Core: 2
E-Core: 8
RAM: 32 GB (dual channel)
drive: 1TB NVME

testing drive:

Micron OEM MTFDKBA256TFK
Capacity: 256GB
Sequential READ: 3600 MB/s
Sequential WRITE: 1600 MB/s
READ Performance: 190K IOPS
WRITE Performance: 400K IOPS
Interface: 8 GB/s
TLC

https://www.harddrivebenchmark.net/hdd.php?hdd=Micron%20MTFDKBA256TFK&id=31564

USB enclosure:

Qoltec M2 NVME 50312
Producer: Qoltec
Standard: NVME PCI-E
Disk dimentions: 2230 , 2242, 2260, 2280
Transmission speed: to 10 Gb/s
Chipset: JMS583 chipset supports TRIM command for SSD, compatible with BOT and UASP

Connected by enclosure supplied cable USB-C to USB-A to ThinkPad USB 3.2 Gen 1 ( 5 Gbit/s ).

First disk benchmarks from Disks application with sys-net, sys-firewall and sys-audio weight=64.

Few qubes loaded but not doing much.

sys-usb weight=64

sys-usb weight=128

sys-usb weight=256

–
Not much difference

Now running YT windowed and coping 10GB file from NAS to same VM that test is done. CPU power throttle is observed. Max cpu frequency 2.4 GHz (normally can go to 3.9-4.2 GHz)

sys-usb weight=128

sys-usb weight=256

–
Also not much difference.

Now running YT windowed only. Moderate load.

sys-net, sys-firewall, sys-audio and sys-usb all have weight=64

sys-net, sys-firewall, sys-audio and sys-usb all have weight=128

sys-net, sys-firewall, sys-audio and sys-usb all have weight=256

–
Also not much difference.

Now running YT windowed and coping 10GB file from NAS to same VM that test is done. CPU power throttle is observed. Max cpu frequency 2.4 GHz (normally can go to 3.9-4.2 GHz)

sys-net, sys-firewall, sys-audio and sys-usb all have weight=128

sys-net, sys-firewall, sys-audio and sys-usb all have weight=256

PS: edited because I forgot to mention my machine

[KitsuneNoBaka]

Now filesystem testing.
Light load.

EXT2, sys-net, sys-firewall, sys-audio and sys-usb weight=64

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.91162 s, 369 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.74571 s, 391 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.9299 s, 63.4 MB/s

EXT2, sys-net, sys-firewall, sys-audio and sys-usb weight=128

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.89646 s, 371 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.73613 s, 392 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.7997 s, 63.9 MB/s

EXT2, sys-net, sys-firewall, sys-audio and sys-usb weight=256

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.89545 s, 371 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.73921 s, 392 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.8242 s, 63.8 MB/s

–

EXT4, sys-net, sys-firewall, sys-audio and sys-usb weight=64

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.97854 s, 360 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.83719 s, 378 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 17.4459 s, 61.5 MB/s

EXT4, sys-net, sys-firewall, sys-audio and sys-usb weight=128

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.88141 s, 373 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.70299 s, 397 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.7442 s, 64.1 MB/s

EXT4, sys-net, sys-firewall, sys-audio and sys-usb weight=256

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.86444 s, 375 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.70288 s, 397 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.8374 s, 63.8 MB/s

–

XFS, sys-net, sys-firewall, sys-audio and sys-usb weight=64

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.89796 s, 371 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.7232 s, 394 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.893 s, 63.6 MB/s

XFS, sys-net, sys-firewall, sys-audio and sys-usb weight=128

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.87682 s, 373 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.73493 s, 393 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.8835 s, 63.6 MB/s

XFS, sys-net, sys-firewall, sys-audio and sys-usb weight=256

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.86378 s, 375 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.6952 s, 398 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.8056 s, 63.9 MB/s

–

BTRFS, sys-net, sys-firewall, sys-audio and sys-usb weight=64

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.91057 s, 369 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.71791 s, 395 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.8906 s, 63.6 MB/s

BTRFS, sys-net, sys-firewall, sys-audio and sys-usb weight=128

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.93538 s, 366 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.71947 s, 395 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.9214 s, 63.5 MB/s

BTRFS, sys-net, sys-firewall, sys-audio and sys-usb weight=256

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.8832 s, 372 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.745 s, 391 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.8715 s, 63.6 MB/s

[KitsuneNoBaka]

Test on BTRFS, mild load by YT windowed playback.

sys-net , sys-firewall , sys-audio and sys-usb weight=64

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.00634 s, 357 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.76694 s, 388 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.9154 s, 63.5 MB/s

sys-net , sys-firewall , sys-audio and sys-usb weight=128

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.00596 s, 357 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.74541 s, 391 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.9009 s, 63.5 MB/s

sys-net , sys-firewall , sys-audio and sys-usb weight=256

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.98381 s, 360 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.77151 s, 387 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64k count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 17.0344 s, 63.0 MB/s

–

And last test, on BTRFS but this time on same drive that test is running is 4k x265 encoded movie that is played by mpv in full screen.

sys-net , sys-firewall , sys-audio and sys-usb weight=64

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.36557 s, 319 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.82114 s, 381 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64K count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 18.4094 s, 58.3 MB/s

sys-net , sys-firewall , sys-audio and sys-usb weight=128

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.22412 s, 333 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.82038 s, 381 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64K count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 18.4396 s, 58.2 MB/s

sys-net , sys-firewall , sys-audio and sys-usb weight=256

$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=1G count=1 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.10957 s, 345 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=256M count=4 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.79713 s, 384 MB/s
$ sudo dd if=/dev/zero of=/tmp/usbtest/test.bin bs=64K count=16384 oflag=direct
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 18.4544 s, 58.2 MB/s

–

There is no major difference between weights, also in cpu usage of VMs.
With weight=64 for all, sys-usb used ~14-15%, with weight=128 ~12% and with weight=256 ~11%.

I’m using credit xen scheduler and soft affinity. Every qubes except for dom0, app-personal (testing qube) and app-YouTube have soft affinity to e-cores and the rest have soft affinity to p-cores.
dom0 utilize all cores, app-YouTube uses 4 vcpu and 4GB RAM and app-personal uses 4 vcpu and 8 GB RAM.

[renehoj]

@KitsuneNoBaka

There is no major difference between weights, also in cpu usage of VMs.

What was the overall system load?

If you are running the tests in a mostly idle system, I don’t think the weights do anything, there will be enough resources for all the VMs. You probably need a very high system load, before you start seeing the effect of the weights.

[KitsuneNoBaka]

Test with cpu power throttle are under high load I imagine.
In conclusion, most service qubes can live with weight=64, maybe with exception to sys-audio where it might be better to have weight=128

dom0 as it is overall service qube and gui service qube might be better with weight=2048 i think