Greetings, all.
(My original install/config issues thread is here: Windows 10 Questions - #46 by disp6252 )
I was here back in May working on getting my Windows 10 Qube working on my system and I was largely successful. I have been using it for a good 3 months solid now, and the only real issues I have been having are performance-related. Stability is very good to excellent.
First, let me describe what I had to end up doing.
My system specs:
Xeon E5-2697A 2.6GHz 26C/32T
X99 motherboard (MSI X99A GG)
128MB PC28800 DDR4 Memory
Samsung Pro 4TB SSD
AMD Radeon RX550 4GB
AMD Radeon RX6950XT 16GB
Built-in dual E2400 1GB NICs and 1535 WiFi/bluetooth adapter
Additional hard drives, both magnetic and SSD.
Realtek ALC1150+ESS ES9018K2M sound
X99 USB 2.0/3.0 & Asmedia ASM1042AE/1142 USB 3.1 Controllers
OS setup:
QubesOS 4.1
Windows 10 Pro 22H2
Qubes is configured to use all threads on the Xeon, which I know is not the most secure possible, due to the known core/thread bleedover issues with this and many other CPUs, but I am aiming to use the hardware I have to the fullest, and the potential attack surface is very small in my usage.
The Windows Qube has a custom config that is managed manually which I have to tweak due to the passthrough and performance settings. Here are the resources it is assigned:
- 16 Threads (CPUs), with preference of CPUs 16-31. All other Qubes share CPUs 0-15.
- 64GB of the 128GB of system memory.
- The Radeon RX 6950XT is assigned directly via PCI passthrough to Windows. The RX550 is left for Qubes to run. There was some boot magic involved to get it to work because the Primary is the 6950XT, and I cannot find a way to change that.
- The audio codecs are also passed through directly to Windows.
- Originally I used sys-usb to pass the keyboard/mouse and some ports through to Windows, but this was very laggy, dropped/repeated keystrokes and caused mouse stuttering, so I opted to pass through the X99 chipset’s USB controllers to Windows, and tried to use an external KM switchbox to switch the K/M between the Windows passthrough ports and the Asmedia USB ports which Qubes still manages. Unfortunately, after 2 different KM switches had massive keystroke hangs and mouse stutter/freeze problems, among other things, I am left with moving cabled manually between ports. If anyone has a suggestion for a USB KM switch box that, you know, actually works that actually isn’t e-waste, please let me know. I have tried a Jancane and an Aimos, and they both suck. The Jancane was really bad, and the Aimos almost works, but not good enough.
- The 4TB SSD is partitioned into two LVM volumes (C&D on windows), and most of the other drives in my system I give to Windows directly to manage, since they had existing data on them from the previous Win8 install I had.
- I am using one of the E2400 ethernet ports through sys-net and sys-firewall using a VIF adapter to Windows. I am not presently using the other, or the wireless adapter, but it is configured in the sys-net qube.
Now, as far as stability is concerned, I have only had 3 hard locks/crashes of the Windows guest in 3 months, and they seem to have been due to problems with the Radeon driver. Since I last updated it, I have had no more issues. Thus, I think stability is very good to excellent. I have had no corruption issues, and no serious windows errors (outside of the occasional SysMain crashing, which is common; I should just disable it anyway). I have no app or game crash issues at all, unlike my previous Win8 install, which is why I had to upgrade finally. Was tired of the constant random app crashes with no explanation.
Now comes the real issue I would like some advice on: performance. While the system runs ok, and most apps/games run without too serious a problem, there are some which stutter and lag horribly. I originally did not have the CPU preferences set in Qubes, and after I set the preferences, it helped a bit. I also disabled CPU parking in Windows. However, I do notice a serious amount of issues related to the mouse movement causing stutters and lag in both just the Windows desktop and in games. Some games are impacted worse than others; some are almost smooth as glass, while others are barely playable with 500ms-2s stutters when there is significant mouse movement. I also get a lot of sound stutters watching YouTube or listening to MP3s, even without moving the mouse. Running LatencyMon shows the typical culprits of ndis.sys and dxgkrnl.sys, along with other parts of the device driver stack (Wdf01000.sys, HDAudBus.sys, etc). I also notice that just scrubbing the mouse around the screen with nothing running but the desktop, I get 7-10% CPU usage increase across most cores. The mouse is a Razer Naga 2014, and it is using just the driver, not the Synapse cloud bloatware crap. I’ve also set it at the lowest report rate of 125/s.
I installed the recent Windows Cumulative rollup update at the same time I did a Qubes update where it updated Xen. After restarting, the stuttering appears to have gotten significantly worse. I am expecting that the Windows update may have clobbered some of the registry performance tweaks I did, but it is also possible that the Qubes Xen update might have changed something for the worse along with it.
I also happened to recently run a benchmark suite from PassMark to see how well (or how badly) my system performance was compared to similar systems. To put it mildly, it was REALLY bad. While I expect some performance hit from running Windows as a Guest under a Linux-based Hypervisor, I expected it to be reasonable, like perhaps 10% at most. Based on the benchmark results, it is about 70-80% slower, which explains a few things.
Basically, at this point, I would like to involve the community here in a project to investigate and improve on Windows performance while running on Qubes. I plan on doing a lot more tweaking to try and get the performance up, because it is atrociously bad in my experience at the moment. I want to share what I’ve done and what I end up doing, but I would like to target getting 70-90+% of the performance of a native Windows 10 system. Given how much hardware I have passed through to Windows, which is completely dedicated to it, not managed by Qubes/Xen outside of the absolute minimum, I would expect better performance. Thus, I would think there is still plenty of room for tweaking and adjusting things to get to that performance goal.
I figure that a lot of it has to do with Windows. It doesn’t like not owning all the hardware, but I also expect that there are optimizations that can be made on Xen and qemu to improve things. It would seem also that some of my issues are driver/interrupt-related. ISR and DPC latency are regularly through the roof, especially when networking is involved. I expect that this may be due to the Xen PV drivers doing a lot of unnecessary blocking or other shenanigans, but there aren’t a lot of options with them. You either use them, or you don’t; they are largely black boxes config-wise.
As always, any advice or suggestions are welcome. I am sold on continuing to use Xen and Qubes as my HV setup. It largely works, and is dependable.
I’ll answer these individually.