CPU Pinning Alder Lake

In fact, my goal is to find a better place for the e-core in the qubes so that they do their job, but without sacrificing overall performance.

You just have to pick qubes that can’t take advantage of the P cores and pin them to the E cores, freeing up more resources on the P cores for qubes that can use them.

I put whonix, Dom0, templates, sys-net, sys-firewall, sys-vpn, disp-mgmt on the E cores, I don’t think there is any meaningful way they can take advantage of running on faster cores.

It frees up the P cores for running my “user” qubes.

1 Like

Yes, you are right, using p cores for dom0 and test vm, I got the best time:

Min Max Mean Median pstdev
2.97 3.47 3.20 3.19 0.10

I have a few questions:

  1. how much dom0_max_vcpus are you using?
  2. Are you using any specific e cores or dynamically 16-23?
  3. Do you turn on governor performance for e cores?
2 Likes

I’m using 4 cores for Dom0, and 4 cores for the other system qubes.

Dom0 is 20-23, the rest is 16-19, it just let Xen do the placements it’s too much work micromanaging the pinning on core level.

I’m doing some test to see what the power consumption difference is between ondemand and performance, until I have an idea of the difference I don’t want to just run everything in performance.

I think running dom0 in performance mode makes sense, but then you might also want to reduce the cores to just 2, I don’t think it really benefits from 4 dedicated cores.

2 Likes

I have been looking at cpumask and cpupool to control on what cores a new vm is created, the idea being that you could make a small group of cores what are running in performance mode to allow fast start up of new VMs, and once they are running they are moved to ondemand cores.

You can’t use cpumask, it doesn’t allow you to pin/use cores outside the mask.

It seems to work with cpupools, all new VMs are created in pool0, the only limitation I have found is that also dom0 has to be in pool0.

I was thinking about doing something like this
0-3 pool0 always in performance mode, for dom0 and starting VMs
4-15 p-cores
16-23 e-cores

It requires a bit of configuration by the boot scripts, you need to set up the pools every time the system boots, and you need to migrate the vm to the pool you want it to be running in.

2 Likes

I’ve tested xl cpupool a bit, overall it’s not a bad solution. But I can’t figure out in which configuration file I can assign the default pool to vm, because you have to assign the pool before starting vm otherwise the whole thing would be pointless.

You can use xen-user.xml to assign the pool to the qubes, I haven’t tested it, but I think this should work.

I don’t think there is any other way to change the default pool, Pool-0 is the default pool used when no other pool is specified.

I want to configure pool0 to be just 4 p-cores in performance mode, and use that pool0 to start all new VMs, and when the VM is ready I want to migrate it to another pool with cores running ondemand.

I don’t know if this improves start up time, but ideally it gives all qubes the improved start up time, without the power consumption from running all P cores in performance mode.

Slightly offtopic: I haven’t had time to measure performance vs. ondemand power consumption yet but since I’m not always at my desk I’ve hacked together a python program that monitors logind’s IdleHint (via dbus) and sets the governor accordingly (performance when IdleHint=False, powersave when True). That should definitely save some Wh.

I remember that logind took a bit of time to set that flag but I usually turn off my monitor with a keyboard shortcut when I know I’ll be away for some time (but not enough time to suspend/power off the PC); in that case ‘dpms off’ immediately triggers a IdleHint=True dbus event and the python dbus listener immediately sets the governor to ‘powersave’.

I can paste the program and systemd unit if someone’s interested.

2 Likes

I’ve made the top post wiki, you can add it there is you want, could make it easier for people to find it in the future.

1 Like

By the way - more on the topic - while monitoring dbus events and inadvertently starting a qube I noticed a bunch of dbus events just after qvm-start. If that wasn’t a coincidence that means you may be able to program whatever pinning logic immediately after startup instead of using cpupool, xen-user.xml, etc. and then re-pin the qube eg. to E-cores once it’s started (or after a delay) by integrating @noskb’s functions.
(I can give you a hand if you’re not comfortable with python)

[edit - that means it should also be possible to set the governor to performance only when a qube is starting]
[edit2] - it is probably possible to do the same with qubes python bindings actually.

1 Like

I have no idea how to do that in Python, you would also need to get the vm object, or you are missing important data.

If you just want to pin as fast as possible, xen-user.xml seems like the best option, with that method you are telling xen which cores to use then the vm is created.

I personally prefer the method @noskb showed, because it happens after the vm has booted, which is advantages when you move from fast to slow cores. You can also add more commands to the same script if you want to change the governor when specific vms start, and expand the script to revert the state when the vm shutdown.

I’ve done some testing on the power consumption

6 hours ondemand: 0.735 kWh
6 hours performance: 0.985 kWh

This just from working on the PC for 6 hours, doing similar but not identical work.
The is not the most precise test, but the difference in consumption seems to somewhere around 25%.

2 Likes

How did you measure the consumption? For some reason I have p-core even in ondemand mode have 5.2GHz at idle. There is no overclocking as such in the bios. E-core is running at 800GHz at idle.

If you just want to pin as fast as possible, xen-user.xml seems like the best option, with that method you are telling xen which cores to use then the vm is created.

xen-user.xml is likely “as fast as possible” - but the difference with pinning just after qvm-start should be minimal (the “qube is starting” GUI notification is shown almost immediately after qvm-start so I expect the difference to be < 1s).

I personally prefer the method @noskb showed […]

It’s still used for pinning a qube once it’s started; the idea was to avoid having pinning configuration in several places. I see there’s a ‘domain-spawned’ ’ domain-pre-start’ qubes handler so dbus isn’t even needed. So:

  • the python program handles ‘domain-spawned’ ’ domain-pre-start’ and ‘domain-start’ events
  • qvm-start triggers a ‘domain-spawned’ ’ domain-pre-start’ event → the python program pins the qube to performance P-cores so that it starts faster.
  • a ‘domain-start’ event is triggered once qrexec is functional → the python program pins the qube to whatever it’s configured to do (eg. pin to E-cores), optionally with a delay to let the qube start additional stuff after qrexec.

I’ll try that a bit later today :slightly_smiling_face:

2 Likes

I used a wattmeter to measure the total consumption of the PC.

I do have some extra hardware 2xPCI USB controller, nvidia GPU, 2x HDD, 2xNVMe, so the total usage wouldn’t be the same on a system with different hardware, but the difference between performance and ondemand should be similar.

1 Like

If the to add the pool to xen-user.xml does not work, you can remove all CPUs from Pool-0 when the host boots. We leave only CPUs that run in performance mode. All vm will use these CPUs for the first run, and afterwards we can do migration or pinning with the @noskb script. This is a theory, I haven’t tested it, but should work as a workaround if there is no way to add a pool in xen-user.xml.

You can’t just remove the cores.

You need to make the P and E core pools, then remove all cores from pool0, except for the cores you want to use in pool0. Then you can add the P and E cores to their pools, and they are ready for use. You have to do this each time you reboot the system.

I easily removed with:

xl cpupool-cpu-remove Pool-0 16
xl cpupool-cpu-remove Pool-0 17
xl cpupool-cpu-remove Pool-0 18
xl cpupool-cpu-remove Pool-0 19
xl cpupool-cpu-remove Pool-0 20
xl cpupool-cpu-remove Pool-0 21
xl cpupool-cpu-remove Pool-0 22
xl cpupool-cpu-remove Pool-0 23

I then added them to a new pool:

xl cpupool-cpu-add testing 16
xl cpupool-cpu-add testing 17
xl cpupool-cpu-add testing 18
xl cpupool-cpu-add testing 19
xl cpupool-cpu-add testing 20
xl cpupool-cpu-add testing 21
xl cpupool-cpu-add testing 22
xl cpupool-cpu-add testing 23

By pre-creating a new pool:
xl cpupool-create name=\"testing\"

All this can be done before starting vm at host startup with systemd for example. All vm will initially use pool-0, you can leave the necessary CPUs there. Then the pinning script will do its job By assigning the right cpus by tag, by p-cores, e-cores, etc.

1 Like

FWIW the ‘domain-pre-start’ / ‘domain-start’ approach seems to work well, without needing to tweak xml files, cpu pools, etc.; PoC:

dom0 1:1 E-core pinning + dynamic qube vcpu->core pinning · GitHub

Alg: pin any starting qube to P-cores; then pin started qubes that don’t have the performance tag to E-cores, after an optional 20s timeout to allow apps to start. Obviously any other logic can be implemented.

Eg. start a dispMedium qube that has the ‘performance’ tag set, and then immediately start vault (that doesn’t have the tag set):

...:19 dom0 cpu_pinning.py[12309]: INFO:root:dispMedium: domain-pre-start
...:20 dom0 cpu_pinning.py[12309]: INFO:root:dispMedium: pinned to cores 0-5
...:22 dom0 cpu_pinning.py[12309]: INFO:root:vault: domain-pre-start
...:23 dom0 cpu_pinning.py[12309]: INFO:root:vault: pinned to cores 0-5
...:25 dom0 cpu_pinning.py[12309]: INFO:root:dispMedium: domain-start
...:30 dom0 cpu_pinning.py[12309]: INFO:root:vault: domain-start
...:50 dom0 cpu_pinning.py[12309]: INFO:root:vault: pinned to cores 6-13

btw here’s a gist of the dbus listener that sets performance/ondemand according to user presence/IdleHint:

Qubes OS: set governor to powersave when systemd logind's IdleHint is True ; set to performance otherwise · GitHub

(I’ll wait a little bit before updating the wiki page - looks like this whole thread will eventually morph into a full blown guide :slightly_smiling_face:)

2 Likes

I’m a little confused as to what you mean? Are you explaining the @noscb script?

Or are you already using a modified one? Just the @noscb script uses xl vcpu-pin, and with it you can’t assign vcpus before starting vm, which won’t actually make the moment vm starts on the correct vcpus performance, since it uses the domain-start event.