Hey neo, sad to hear this motherboard is being a pain with qubes. Do you think using a usb-c or usb-a ethernet or wireless adapter might alleviate any of those issues? Instead of using the built-in networking interfaces? Just hope there is a workaround until the built in issues are ironed out.
afaik ASUS can do something about the iommu grouping, it may help to email them about this (and/or see if there’s a thread about it on l1t forum, that community is quite into virtualization)
@crowizz yes it would work.
And yup, reading a lot of things from l1t forum 
& yes, need to contact asus support again
Some progress about igpu passthrough.
Using some recommandation from l1t I extracted the zen4 igpu rom from the motherboard uefi using GitHub - LongSoft/UEFITool: UEFI firmware image viewer and editor .
Zen4 IGPU ROM: https://neowutran.ovh/igpu.bin
sha1sum 1fbac0cbde86cf7dbc9d1aa827263cf8f167c52e
( ethernet card use this specific rom: https://neowutran.ovh/ethernet.bin
sha1sum: ab85435aee2a978c06b1a67da9436a2832576748 )
Then I patched the guest kernel to force loading the gpu firmware from file:
#include <linux/pci.h>
#include <linux/slab.h>
#include <linux/acpi.h>
#include <linux/firmware.h>
/*
* BIOS.
*/
static bool amdgpu_bios_firmware(struct amdgpu_device *adev)
{
const struct firmware* vbios_firmware;
const char* vbios_name = "vbios/renoir.bin";
const size_t vbios_size = 32 << PAGE_SHIFT;
int ret;
DRM_INFO("amdgpu_bios_firmware()\n");
ret = request_firmware(&vbios_firmware, vbios_name, adev->dev);
if (ret) {
DRM_ERROR("failed to load '%s': %d\n", vbios_name, ret);
return false;
}
if (!AMD_IS_VALID_VBIOS(vbios_firmware->data)) {
DRM_ERROR("vbios file '%s' signature incorrect %x %x\n",
vbios_name, vbios_firmware->data[0], vbios_firmware->data[1]);
release_firmware(vbios_firmware);
return false;
}
if (vbios_firmware->size > vbios_size) {
DRM_ERROR("vbios file '%s' size too large, %zu > %zu\n",
vbios_name, vbios_firmware->size, vbios_size);
release_firmware(vbios_firmware);
return false;
}
adev->bios = kzalloc(vbios_size, GFP_KERNEL);
if (adev->bios == NULL) {
DRM_ERROR("no memory to allocate for BIOS\n");
release_firmware(vbios_firmware);
return false;
}
adev->bios_size = vbios_size;
memcpy_fromio(adev->bios, vbios_firmware->data, vbios_firmware->size);
release_firmware(vbios_firmware);
return true;
}
bool amdgpu_get_bios(struct amdgpu_device *adev)
{
if (amdgpu_bios_firmware(adev)) {
dev_info(adev->dev, "Fetched VBIOS from firmware file\n");
goto success;
}
Now the IGPU is initialized by the guest. GPU is recognized by system tools. But there is some issues with power management so for the moment the IGPU is unusable.
This kind of error:
[ 6.329420] [drm] Fence fallback timer expired on ring comp_1.0.0
[ 7.339193] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.1.0 (-110).
[ 8.379193] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.2.0 (-110).
[ 9.419196] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.3.0 (-110).
[ 10.459187] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.0.1 (-110).
[ 11.499207] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.1.1 (-110).
[ 12.539183] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.2.1 (-110).
[ 13.579192] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.3.1 (-110).
[ 14.089164] [drm] Fence fallback timer expired on ring kiq_2.1.0
[ 14.619186] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on kiq_2.1.0 (-110).
[ 15.659206] [drm:sdma_v5_2_ring_test_ib [amdgpu]] *ERROR* amdgpu: IB test timed out
[ 15.659356] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on sdma0 (-110).
[ 16.169166] [drm] Fence fallback timer expired on ring vcn_dec_0
[ 16.679412] [drm] Fence fallback timer expired on ring vcn_enc_0.0
[ 17.189172] [drm] Fence fallback timer expired on ring vcn_enc_0.1
[ 17.189305] [drm:process_one_work] *ERROR* ib ring test failed (-110).
neowutran…
I will not pretend I know exactly what you are working on here, but thank you!! I hope you can work out these bugs and have the IGPU and other functions working. Do you work with the team that does releases, or will we just use your fork or whatever when the time comes that you have it working? If you need help or for me to test anything on my system, let me know.
Here is a summary of my system:
Computer:
Computer Type ACPI x64-based PC
Motherboard:
CPU Type 16-Core AMD Ryzen 9 7950X, 5500 MHz (55 x 100)
Motherboard Name Asus TUF Gaming X670E-Plus WiFi (1 PCI-E x4, 2 PCI-E x16, 4 M.2, 4 DDR5 DIMM, Audio, Video, 2.5GbE LAN, WiFi)
Motherboard Chipset AMD X670E, AMD K19.6 FCH, AMD K19.6 IMC
System Memory 31894 MB
DIMM2: A-Data AX5U6000C4016G-B 16 GB DDR5-4800 DDR5 SDRAM (42-41-41-77 @ 2403 MHz) (40-40-40-77 @ 2400 MHz) (36-36-36-70 @ 2160 MHz) (32-32-32-62 @ 1920 MHz) (30-30-30-58 @ 1800 MHz) (28-28-28-54 @ 1680 MHz) (26-26-26-50 @ 1560 MHz) (22-22-22-43 @ 1320 MHz)
DIMM4: A-Data AX5U6000C4016G-B 16 GB DDR5-4800 DDR5 SDRAM (42-41-41-77 @ 2403 MHz) (40-40-40-77 @ 2400 MHz) (36-36-36-70 @ 2160 MHz) (32-32-32-62 @ 1920 MHz) (30-30-30-58 @ 1800 MHz) (28-28-28-54 @ 1680 MHz) (26-26-26-50 @ 1560 MHz) (22-22-22-43 @ 1320 MHz)
BIOS Type AMI (11/15/2022)
Display:
Video Adapter AMD Radeon RX 7900 XTX (24560 MB)
Video Adapter AMD Radeon RX 7900 XTX (24560 MB)
Video Adapter AMD Radeon RX 7900 XTX (24560 MB)
Video Adapter AMD Radeon RX 7900 XTX (24560 MB)
Video Adapter AMD Radeon(TM) Graphics (512 MB)
Video Adapter AMD Radeon(TM) Graphics (512 MB)
Video Adapter AMD Radeon(TM) Graphics (512 MB)
Video Adapter AMD Radeon(TM) Graphics (512 MB)
Video Adapter AMD Radeon(TM) Graphics (512 MB)
3D Accelerator AMD Radeon RX 7900 XTX (Navi 31)
3D Accelerator AMD Raphael
Monitor Acer Predator XB271H [27" TN LCD] (#ASNV2CQi+Hjd)
Monitor LG 27GN950 (DisplayPort) [27" LCD] (204NTMXDU283)
Storage:
IDE Controller Standard SATA AHCI Controller
IDE Controller Standard SATA AHCI Controller
Storage Controller Microsoft Storage Spaces Controller
Storage Controller Microsoft VHD Loopback Controller
Storage Controller Standard NVM Express Controller
Storage Controller Standard NVM Express Controller
Disk Drive HGST HDN726060ALE614 (6 TB, 7200 RPM, SATA-III)
Disk Drive Samsung SSD 980 PRO 2TB (2000 GB, PCI-E 4.0 x4)
Disk Drive WD Blue SN570 1TB (1000 GB, PCI-E 3.0 x4)
Disk Drive WD easystore 25FB USB Device
Yes, i did.
No change in behavoir with the 4.1.2-rc1 also, by the way. For this, i sadly gave up on the t16 (though being a fine machine built-wise).
This may not work for you, but it did work for my Ryzen 7 Pro 5875u Thinkpad L14:
I added dom0_max_vcpus=1 dom0_vcpu_pin to the Xen line and nomodeset to the kernel parameters. clocksource=tsc tsc=unstable hpetbroadcast=0 instead of the vpcu parameters may also work.
Hi neowutran, I also encountered this issue in my 22.04 ubuntu(kernel 6.0/Xen 5.18 unstable). Can you please provide a more detailed description of what you have done about the rpm spec? Thank you.
Hi @Jiqian, welcome to the forum!
There’s no need to paste an image of the post you want to reply to.
1 Like
Hi @neowutran ,
thanks for your amazing work! I am looking forward to buy a new computer and want to go with the new ryzen series. Can you recommend a mainboard for that? I didn’t find any on the hcl.
Also, I have read all your comments but am still a bit confused. You made qubes running with a 7000er amd cpu and compiled an iso. (I don’t care about a missing installation prozess). Would you say, that it is working stable? You mentioned some remaining issues, but I am not sure what you mean. May you explain it.
And are you in contact with the qubes team to merge this at some point in time?
Hi, some irl stuff taking lot of time, so I didn’t make any progress.
Some important things have been merged already.
If you use the latest official development build (R4.2), you will have thoses issues:
- Need to manually configure the x2apic. Some workaround exist (Improve X2APIC compatibility with modern AMD platform · Issue #8111 · QubesOS/qubes-issues · GitHub)
- Graphical driver are borked, the ISO I published doesn’t have this issue, but I don’t known what is the thing that broke it, no doubt it is Qubes specific (graphical instabilities, crashes)
For my specific motherboard:
- Wifi work most of the time. When it stop working, you need to unplug power for few seconds. This issue is not related to xen and not related to Qubes. I need to complain again to have some answer from asus.
- Ethernet will work time to time. Ethernet seems to not like VM/IOMMU, it will disconnect after some time of use. You need to reboot the computer to fix it.
For specific things:
- I was unable to passthrough the IGPU. It is bug in the amdgpu driver I think.
Overall: It can work, you can make it stable. But it don’t work out of the box, and it is not stable out of the box.
Some things still need to be fixed on Qubes side
Some things still need to be fixed on Xen hypervisor side
Some things still need to be fixed on Asus side
Update: the graphical driver issue was not related to Qubes OS, it was related to recent asus bios that incorrectly handle CPU voltage by default. Manually setting the CPU voltage fix this issue
3 Likes
Thank you for the update. I guess I will wait then. Is it reasonable to assume, that the issue is the am5 support and not the specific cpu? So by the time the 7000 series is supported one can assume that the 9000series is also doing ok?
@Rnd3sB3g13rng Some of the issues I had were specific to the cpu ( frequencies ), some specific to AMD ( the currently remaining issue with x2apic is in fact a multi-years old issue ).
Some update:
Ethernet seems to be stable if you disable ASPM in the bios.
Will confirm in few days
3 Likes
Thanks for that update neowutran. We all appreciate your work on this, irl stuff always gets in the way and I hope everything is going well. Keep up the great work and I am sure you will get it all worked out eventually. Do you have a place you post your ISOs to? Thanks again.
I stopped building my custom ISOs, enough things have been upstreamed to use the system. Now I use the official developpment build of Qubes OS R4.2.
And then I rebuild the xen hypervisor to integrate this patch Improve X2APIC compatibility with modern AMD platform · Issue #8111 · QubesOS/qubes-issues · GitHub so I don’t need to configure x2apic on boot command line
1 Like
Hi @neowutran ,
can you please give more info about the CPU voltage settings in bios? Or any other bios changes you made? I’m getting quite lost in all the settings asus has there.
thank you
PS: Just to be sure its the same, the issue I have is that the pc reboots each time after few seconds after I see the cursor on screen. booting with dom0_max_vcpus=1, dom0_vcpus_pin, x2apic=false. HW is ryzen 7950x with asus x670e-pro
not the same consequences ( for me, it was: graphical drivers crash / ui restart in loop right with lightdm start (cursor start the be on screen). No PC reboot), but maybe the same root cause.
I recommand you to try the development build of R4.2.
“dom0_max_vcpus=1, dom0_vcpus_pin” is not needed.
Some screenshot of my bios ( nothing new, already described all of it in previous posts ):
Since Ryzen 7000 serie - #101 by neowutran, didn’t had any wifi nor ethernet issue
Sorry, don’t remember exactly, every tries I did are here: History for xen.spec.in - neowutran/qubes-vmm-xen · GitHub
And I fixed the RPM spec between 21 nov and 22 nov.
But you can also just use the RPM spec of the official qubes repo, it have been upgraded for Xen 4.17
neo, what bios revision are you using? Just curious; with the whole AMD 7000 burnout debacle they have “beta” versions and change them almost daily right now. I am on 1409 because it is stable and working for now and am thinking about waiting until they have it all fully fleshed out.
are you on one of the beta bios?
Yes, I am using the beta bios version 1410, old bios must be avoided.
( Or you could manually set the soc voltage too )
Thx for the settings.
But I do think that the issue seem almost same. Or at least it seems to happen at the same spot. What I found out is that if I disable the Wifi/bluetooth/Lan in bios after installation, I can boot properly. It seems to continue with some part of the installation, as it apparently didnt complete before. I can even enable all the disabled devices and still continue to boot properly. Only that I cant get network to work.
I will try out your settings with a clean install and see how far I get.
I’m also using the new beta bios and nightly build of 4.2.