Maybe it’s just my system, but the default credit2 scheduler seems to perform significant worse than credit with my 13900K
With credit, the load seems to be spread evenly among all cores, with more active domains being on the P cores.
With credit2, Xen just wants to run everything on the E cores, which in my case make the CPU run a lot hotter.
I typically have ~20 VMs running, and my CPU temp average seems to be around 45C when using credit, with credit2 the same workload seems to average closer to 55C
1 Like
Here is the difference between booting my system with credit and credit2.
Credit: vcpus use 27/13 P/E cores
Credit2: vcpus use 7/33 P/E cores
This is with 4 P cores pinned to dom0, so with credit2 dumUs only use 3 P cores.
With credit the vcpus are moved between cores continuously, this doesn’t happen with credit2 the vcpus just seem to stay on the E cores.
Credit
Name ID VCPU CPU State Time(s) Affinity (Hard / Soft)
Domain-0 0 0 0 -b- 22.2 0 / all
Domain-0 0 1 1 r-- 22.0 1 / all
Domain-0 0 2 2 -b- 20.8 2 / all
Domain-0 0 3 3 -b- 19.1 3 / all
sys-net 1 0 10 -b- 3.7 all / all
sys-net 1 1 20 -b- 1.5 all / all
sys-net-dm 2 0 19 -b- 1.5 all / all
sys-firewall 3 0 27 -b- 2.3 all / all
sys-firewall 3 1 4 -b- 1.5 all / all
sys-usb 4 0 22 -b- 3.0 all / all
sys-usb 4 1 8 -b- 0.9 all / all
sys-usb-dm 5 0 4 -b- 1.4 all / all
sys-vpn 6 0 22 -b- 2.9 all / all
sys-vpn 6 1 28 r-- 1.2 all / all
browser-personal 7 0 8 -b- 3.3 all / all
browser-personal 7 1 7 -b- 2.9 all / all
browser-social 8 0 12 -b- 3.6 all / all
browser-social 8 1 14 -b- 2.8 all / all
browser-streaming 9 0 5 -b- 2.1 all / all
browser-streaming 9 1 3 -b- 1.3 all / all
browser-streaming 9 2 11 -b- 1.4 all / all
browser-streaming 9 3 13 -b- 1.6 all / all
user-home 10 0 31 -b- 1.0 all / all
user-home 10 1 11 -b- 1.3 all / all
user-home 10 2 28 -b- 1.0 all / all
user-home 10 3 12 -b- 1.4 all / all
browser-user-dvm 11 0 12 -b- 1.7 all / all
browser-user-dvm 11 1 7 -b- 1.6 all / all
user-work 12 0 24 -b- 2.1 all / all
user-work 12 1 7 -b- 1.7 all / all
sys-whonix 13 0 9 -b- 3.4 all / all
sys-whonix 13 1 29 -b- 2.8 all / all
user-office 14 0 29 -b- 1.8 all / all
user-office 14 1 7 -b- 1.0 all / all
user-office 14 2 15 -b- 2.7 all / all
user-office 14 3 9 -b- 0.6 all / all
user-media 15 0 15 -b- 2.2 all / all
user-media 15 1 7 -b- 1.6 all / all
user-devel 16 0 6 -b- 2.2 all / all
user-devel 16 1 10 -b- 2.1 all / all
Credit2
Name ID VCPU CPU State Time(s) Affinity (Hard / Soft)
Domain-0 0 0 0 r-- 21.2 0 / all
Domain-0 0 1 1 -b- 20.5 1 / all
Domain-0 0 2 2 -b- 18.7 2 / all
Domain-0 0 3 3 -b- 22.0 3 / all
sys-net 1 0 22 -b- 4.6 all / all
sys-net 1 1 24 -b- 2.8 all / all
sys-net-dm 2 0 23 -b- 1.8 all / all
sys-firewall 3 0 30 -b- 3.4 all / all
sys-firewall 3 1 23 -b- 2.2 all / all
sys-vpn 4 0 24 -b- 4.2 all / all
sys-vpn 4 1 16 -b- 2.1 all / all
sys-usb 5 0 18 -b- 4.3 all / all
sys-usb 5 1 26 -b- 1.7 all / all
sys-usb-dm 6 0 17 -b- 1.8 all / all
browser-personal 7 0 31 -b- 5.2 all / all
browser-personal 7 1 22 -b- 4.0 all / all
browser-streaming 8 0 16 -b- 2.8 all / all
browser-streaming 8 1 25 -b- 2.8 all / all
browser-streaming 8 2 17 -b- 2.0 all / all
browser-streaming 8 3 29 -b- 2.1 all / all
browser-social 9 0 28 -b- 5.1 all / all
browser-social 9 1 21 -b- 3.7 all / all
browser-user-dvm 10 0 25 -b- 2.5 all / all
browser-user-dvm 10 1 29 -b- 2.5 all / all
user-home 11 0 23 -b- 2.3 all / all
user-home 11 1 19 -b- 1.7 all / all
user-home 11 2 16 -b- 1.2 all / all
user-home 11 3 20 -b- 1.2 all / all
user-work 12 0 6 -b- 3.6 all / all
user-work 12 1 27 -b- 1.7 all / all
sys-whonix 13 0 20 -b- 4.5 all / all
sys-whonix 13 1 28 -b- 2.8 all / all
user-media 14 0 26 -b- 2.6 all / all
user-media 14 1 17 -b- 2.8 all / all
user-office 15 0 4 -b- 4.0 all / all
user-office 15 1 30 -b- 1.3 all / all
user-office 15 2 22 -b- 0.9 all / all
user-office 15 3 21 -b- 1.0 all / all
user-devel 16 0 27 -b- 3.2 all / all
user-devel 16 1 21 -b- 2.8 all / all
I think it is by design and to be expected. When you got pinned p-Cores as pCPUs for dom0 the probability of locking conditions on those CPUs is higher.
Those dom0 assigned pCPUs are generally more prone to burning time credits, too. So if the constant „load“ is assigned to a certain pCPU the distribution of vCPUs and the corresponding tasks is more likely to be assigned to some other pCPU. Furthermore: Even if a pCPU becomes free, it is a matter of load to reassign/migrate a task. Most of the time it‘s costly.
AFAIK credit was more like dumb „round robin“,
https://wiki.xenproject.org/wiki/Credit_Scheduler
while credit2 implemented a lot more conditions and metrics to get better response/latency under load.
https://wiki.xenproject.org/wiki/Credit2_Scheduler
Did you try credit2 without CPU pinning?
I don’t understand what you mean, pinning just makes dom0 run on the cores 0-3.
I don’t see how this would have any impact on how credit2 is placing the vcpus, and almost completely avoiding using the remaining P cores, my system has 16P and 16E cores.
It makes no sense for a Lake-S CPU not to use the P cores, typically the firmware will have an option to disable E cores and only use P cores, only using the E cores doesn’t make any sense.
Not using pinning would assign all 32 core to dom0, the Xen documentation says this isn’t recommended, and specifically mentions not assigning more than 4 cores unless you are running a lot of HVMs without stubs.
Limiting the number of vCPUs is one thing. But to pin dom0 to physical CPUs (pCPUs), is another.
The latter prevents dom0 tasks from being scheduled out from those pCPUs and changes the weight or probability (or even possibility) of whether tasks other than dom0 end up on the pCPUs by vCPU scheduling.
I haven’t read credit2‘s schedule() code, yet. But I‘ll try to do. Until then compare „dom0 vCPUs“ here:
https://wiki.xenproject.org/wiki/Tuning_Xen_for_Performance
How do You interprete that?