Ryzen 7000 serie

sindrex · January 29, 2023, 12:19am

Ryzen 7 pro 6850u (thinkpad t16, latest bios) user here… got a similar problem that the installer won’t go past the log entry “Xen is relinquishing VGA console” (shortly after dropping that “AMD processor family 25 is not supported”). Then the log disappears rapidly from top to bottom and the system either reboots or stays up with a black screen, backlight still on.

The problem persists when trying to boot using the working.iso, kindly posted above. I added the x2apic=false to the GRUB lines as instructed. The only difference here being that when using the working.iso, the system always and slightly faster reboots.

I also tried installing using latest and current builds from Index of /qubes/iso/, including adding above mentioned grub lines, with the same result and therefore no success. I have never gotten past the first couple of seconds after choosing the GRUB boot option.

I’m not sure this feedback is of any value, but i will follow this topic closely nonetheless. Any suggestions for further troubleshooting are welcome! Hope there will be further progress with this issue and thanks for the effort.

neowutran · January 31, 2023, 10:25am

In the grub options you are sure you selected the one with kernel 6.1 ?

On my side didn’t make progress for the IGPU passthrough and the occasional issues with ethernet/wifi.
Currently I am … testing my gpu passthrough with a lot of cyberpunk
But after, need to try that AMD iGPU passthrough attempt - #24 by yann , seems interesting. Also need to find a way to dump the ROM of the IGPU and ethernet card.

neowutran · February 6, 2023, 12:27pm

Some conclusions about the network card of the ASUS x670e strix F.

Wifi card:
It seems that the wifi card is never really offline. And it seems to have an issue in the hardware ( motherboard itself or firmware of the card ? ). Very rarely, the wifi card will stop working (the card enter in a non working state). Rebooting the computer is useless as the wifi card is never offline even when the computer is shutdown. Need to make sure no electricity can reach the motherboard during few seconds. This issue is not related to QubesOS.

Ethernet card:
There is some IOMMU grouping issues on my motherboard. I have WIFI + Ethernet + 1 usb controller + SATA controller + some other things connected to the same PCI bridge. Every elements of this list need to be in the same VM, or else some random issues will appear ( PCI link of the Ethernet card disconnecting, or the WIFI card randomly showing up as a USB device ).

( this motherboard is expensive and is quite shitty. I don’t recommend it. )

crowizz · February 6, 2023, 3:35pm

Hey neo, sad to hear this motherboard is being a pain with qubes. Do you think using a usb-c or usb-a ethernet or wireless adapter might alleviate any of those issues? Instead of using the built-in networking interfaces? Just hope there is a workaround until the built in issues are ironed out.

fjdh · February 8, 2023, 7:37am

afaik ASUS can do something about the iommu grouping, it may help to email them about this (and/or see if there’s a thread about it on l1t forum, that community is quite into virtualization)

neowutran · February 12, 2023, 9:38am

@crowizz yes it would work.

And yup, reading a lot of things from l1t forum
& yes, need to contact asus support again

Some progress about igpu passthrough.

Using some recommandation from l1t I extracted the zen4 igpu rom from the motherboard uefi using GitHub - LongSoft/UEFITool: UEFI firmware image viewer and editor .

Zen4 IGPU ROM: https://neowutran.ovh/igpu.bin
sha1sum 1fbac0cbde86cf7dbc9d1aa827263cf8f167c52e

( ethernet card use this specific rom: https://neowutran.ovh/ethernet.bin
sha1sum: ab85435aee2a978c06b1a67da9436a2832576748 )

Then I patched the guest kernel to force loading the gpu firmware from file:

#include <linux/pci.h>
#include <linux/slab.h>
#include <linux/acpi.h>
#include <linux/firmware.h>
/*
 * BIOS.
 */

static bool amdgpu_bios_firmware(struct amdgpu_device *adev)
{
	const struct firmware* vbios_firmware;
	const char* vbios_name = "vbios/renoir.bin";
	const size_t vbios_size = 32 << PAGE_SHIFT;
	int ret;
	DRM_INFO("amdgpu_bios_firmware()\n");
	ret = request_firmware(&vbios_firmware, vbios_name, adev->dev);
	if (ret) {
		DRM_ERROR("failed to load '%s': %d\n", vbios_name, ret);
		return false;
	}

	if (!AMD_IS_VALID_VBIOS(vbios_firmware->data)) {
		DRM_ERROR("vbios file '%s' signature incorrect %x %x\n",
			  vbios_name, vbios_firmware->data[0], vbios_firmware->data[1]);
		release_firmware(vbios_firmware);
		return false;
	}

	if (vbios_firmware->size > vbios_size) {
		DRM_ERROR("vbios file '%s' size too large, %zu > %zu\n",
			  vbios_name, vbios_firmware->size, vbios_size);
		release_firmware(vbios_firmware);
		return false;
	}

	adev->bios = kzalloc(vbios_size, GFP_KERNEL);
	if (adev->bios == NULL) {
		DRM_ERROR("no memory to allocate for BIOS\n");
		release_firmware(vbios_firmware);
		return false;
	}

	adev->bios_size = vbios_size;
	memcpy_fromio(adev->bios, vbios_firmware->data, vbios_firmware->size);
	release_firmware(vbios_firmware);
	return true;
}

bool amdgpu_get_bios(struct amdgpu_device *adev)
{
	if (amdgpu_bios_firmware(adev)) {
		dev_info(adev->dev, "Fetched VBIOS from firmware file\n");
		goto success;
	}

Now the IGPU is initialized by the guest. GPU is recognized by system tools. But there is some issues with power management so for the moment the IGPU is unusable.
This kind of error:

 [    6.329420] [drm] Fence fallback timer expired on ring comp_1.0.0            
  [    7.339193] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.1.0 (-110).
  [    8.379193] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.2.0 (-110).
  [    9.419196] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.3.0 (-110).
  [   10.459187] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.0.1 (-110).
  [   11.499207] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.1.1 (-110).
  [   12.539183] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.2.1 (-110).
  [   13.579192] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on comp_1.3.1 (-110).
  [   14.089164] [drm] Fence fallback timer expired on ring kiq_2.1.0             
  [   14.619186] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on kiq_2.1.0 (-110).
  [   15.659206] [drm:sdma_v5_2_ring_test_ib [amdgpu]] *ERROR* amdgpu: IB test timed out
  [   15.659356] amdgpu 0000:00:07.0: [drm:amdgpu_ib_ring_tests [amdgpu]] *ERROR* IB test failed on sdma0 (-110).
  [   16.169166] [drm] Fence fallback timer expired on ring vcn_dec_0             
  [   16.679412] [drm] Fence fallback timer expired on ring vcn_enc_0.0           
  [   17.189172] [drm] Fence fallback timer expired on ring vcn_enc_0.1           
  [   17.189305] [drm:process_one_work] *ERROR* ib ring test failed (-110).

crowizz · February 12, 2023, 9:19pm

neowutran…
I will not pretend I know exactly what you are working on here, but thank you!! I hope you can work out these bugs and have the IGPU and other functions working. Do you work with the team that does releases, or will we just use your fork or whatever when the time comes that you have it working? If you need help or for me to test anything on my system, let me know.

Here is a summary of my system:

Computer:
      Computer Type                                     ACPI x64-based PC


    Motherboard:
      CPU Type                                          16-Core AMD Ryzen 9 7950X, 5500 MHz (55 x 100)
      Motherboard Name                                  Asus TUF Gaming X670E-Plus WiFi  (1 PCI-E x4, 2 PCI-E x16, 4 M.2, 4 DDR5 DIMM, Audio, Video, 2.5GbE LAN, WiFi)
      Motherboard Chipset                               AMD X670E, AMD K19.6 FCH, AMD K19.6 IMC
      System Memory                                     31894 MB
      DIMM2: A-Data AX5U6000C4016G-B                    16 GB DDR5-4800 DDR5 SDRAM  (42-41-41-77 @ 2403 MHz)  (40-40-40-77 @ 2400 MHz)  (36-36-36-70 @ 2160 MHz)  (32-32-32-62 @ 1920 MHz)  (30-30-30-58 @ 1800 MHz)  (28-28-28-54 @ 1680 MHz)  (26-26-26-50 @ 1560 MHz)  (22-22-22-43 @ 1320 MHz)
      DIMM4: A-Data AX5U6000C4016G-B                    16 GB DDR5-4800 DDR5 SDRAM  (42-41-41-77 @ 2403 MHz)  (40-40-40-77 @ 2400 MHz)  (36-36-36-70 @ 2160 MHz)  (32-32-32-62 @ 1920 MHz)  (30-30-30-58 @ 1800 MHz)  (28-28-28-54 @ 1680 MHz)  (26-26-26-50 @ 1560 MHz)  (22-22-22-43 @ 1320 MHz)
      BIOS Type                                         AMI (11/15/2022)

    Display:
      Video Adapter                                     AMD Radeon RX 7900 XTX  (24560 MB)
      Video Adapter                                     AMD Radeon RX 7900 XTX  (24560 MB)
      Video Adapter                                     AMD Radeon RX 7900 XTX  (24560 MB)
      Video Adapter                                     AMD Radeon RX 7900 XTX  (24560 MB)
      Video Adapter                                     AMD Radeon(TM) Graphics  (512 MB)
      Video Adapter                                     AMD Radeon(TM) Graphics  (512 MB)
      Video Adapter                                     AMD Radeon(TM) Graphics  (512 MB)
      Video Adapter                                     AMD Radeon(TM) Graphics  (512 MB)
      Video Adapter                                     AMD Radeon(TM) Graphics  (512 MB)
      3D Accelerator                                    AMD Radeon RX 7900 XTX (Navi 31)
      3D Accelerator                                    AMD Raphael
      Monitor                                           Acer Predator XB271H  [27" TN LCD]  (#ASNV2CQi+Hjd)
      Monitor                                           LG 27GN950 (DisplayPort)  [27" LCD]  (204NTMXDU283)

    Storage:
      IDE Controller                                    Standard SATA AHCI Controller
      IDE Controller                                    Standard SATA AHCI Controller
      Storage Controller                                Microsoft Storage Spaces Controller
      Storage Controller                                Microsoft VHD Loopback Controller
      Storage Controller                                Standard NVM Express Controller
      Storage Controller                                Standard NVM Express Controller
      Disk Drive                                        HGST HDN726060ALE614  (6 TB, 7200 RPM, SATA-III)
      Disk Drive                                        Samsung SSD 980 PRO 2TB  (2000 GB, PCI-E 4.0 x4)
      Disk Drive                                        WD Blue SN570 1TB  (1000 GB, PCI-E 3.0 x4)
      Disk Drive                                        WD easystore 25FB USB Device

sindrex · February 22, 2023, 9:14pm

Yes, i did.
No change in behavoir with the 4.1.2-rc1 also, by the way. For this, i sadly gave up on the t16 (though being a fine machine built-wise).

Scumbag · February 26, 2023, 7:04pm

This may not work for you, but it did work for my Ryzen 7 Pro 5875u Thinkpad L14:
I added dom0_max_vcpus=1 dom0_vcpu_pin to the Xen line and nomodeset to the kernel parameters. clocksource=tsc tsc=unstable hpetbroadcast=0 instead of the vpcu parameters may also work.

Jiqian · April 6, 2023, 5:29am

Hi neowutran, I also encountered this issue in my 22.04 ubuntu(kernel 6.0/Xen 5.18 unstable). Can you please provide a more detailed description of what you have done about the rpm spec? Thank you.

deeplow · April 6, 2023, 7:56am

Hi @Jiqian, welcome to the forum!

There’s no need to paste an image of the post you want to reply to.

Rnd3sB3g13rng · April 11, 2023, 12:05pm

Hi @neowutran ,
thanks for your amazing work! I am looking forward to buy a new computer and want to go with the new ryzen series. Can you recommend a mainboard for that? I didn’t find any on the hcl.

Also, I have read all your comments but am still a bit confused. You made qubes running with a 7000er amd cpu and compiled an iso. (I don’t care about a missing installation prozess). Would you say, that it is working stable? You mentioned some remaining issues, but I am not sure what you mean. May you explain it.
And are you in contact with the qubes team to merge this at some point in time?

neowutran · April 12, 2023, 10:13am

Hi, some irl stuff taking lot of time, so I didn’t make any progress.

Some important things have been merged already.
If you use the latest official development build (R4.2), you will have thoses issues:

Need to manually configure the x2apic. Some workaround exist (Improve X2APIC compatibility with modern AMD platform · Issue #8111 · QubesOS/qubes-issues · GitHub)
Graphical driver are borked, the ISO I published doesn’t have this issue, but I don’t known what is the thing that broke it, no doubt it is Qubes specific (graphical instabilities, crashes)

For my specific motherboard:

Wifi work most of the time. When it stop working, you need to unplug power for few seconds. This issue is not related to xen and not related to Qubes. I need to complain again to have some answer from asus.
Ethernet will work time to time. Ethernet seems to not like VM/IOMMU, it will disconnect after some time of use. You need to reboot the computer to fix it.

For specific things:

I was unable to passthrough the IGPU. It is bug in the amdgpu driver I think.

Overall: It can work, you can make it stable. But it don’t work out of the box, and it is not stable out of the box.

Some things still need to be fixed on Qubes side
Some things still need to be fixed on Xen hypervisor side
Some things still need to be fixed on Asus side

Update: the graphical driver issue was not related to Qubes OS, it was related to recent asus bios that incorrectly handle CPU voltage by default. Manually setting the CPU voltage fix this issue

Rnd3sB3g13rng · April 12, 2023, 11:08am

Thank you for the update. I guess I will wait then. Is it reasonable to assume, that the issue is the am5 support and not the specific cpu? So by the time the 7000 series is supported one can assume that the 9000series is also doing ok?

neowutran · May 5, 2023, 7:03am

@Rnd3sB3g13rng Some of the issues I had were specific to the cpu ( frequencies ), some specific to AMD ( the currently remaining issue with x2apic is in fact a multi-years old issue ).

Some update:
Ethernet seems to be stable if you disable ASPM in the bios.

Will confirm in few days

crowizz · May 5, 2023, 9:38pm

Thanks for that update neowutran. We all appreciate your work on this, irl stuff always gets in the way and I hope everything is going well. Keep up the great work and I am sure you will get it all worked out eventually. Do you have a place you post your ISOs to? Thanks again.

neowutran · May 6, 2023, 6:28am

I stopped building my custom ISOs, enough things have been upstreamed to use the system. Now I use the official developpment build of Qubes OS R4.2.
And then I rebuild the xen hypervisor to integrate this patch Improve X2APIC compatibility with modern AMD platform · Issue #8111 · QubesOS/qubes-issues · GitHub so I don’t need to configure x2apic on boot command line

mirome · May 11, 2023, 8:28am

Hi @neowutran ,
can you please give more info about the CPU voltage settings in bios? Or any other bios changes you made? I’m getting quite lost in all the settings asus has there.

thank you
PS: Just to be sure its the same, the issue I have is that the pc reboots each time after few seconds after I see the cursor on screen. booting with dom0_max_vcpus=1, dom0_vcpus_pin, x2apic=false. HW is ryzen 7950x with asus x670e-pro

neowutran · May 11, 2023, 5:12pm

not the same consequences ( for me, it was: graphical drivers crash / ui restart in loop right with lightdm start (cursor start the be on screen). No PC reboot), but maybe the same root cause.
I recommand you to try the development build of R4.2.
“dom0_max_vcpus=1, dom0_vcpus_pin” is not needed.

Some screenshot of my bios ( nothing new, already described all of it in previous posts ):

Since Ryzen 7000 serie - #101 by neowutran, didn’t had any wifi nor ethernet issue

neowutran · May 11, 2023, 5:22pm

Sorry, don’t remember exactly, every tries I did are here: History for xen.spec.in - neowutran/qubes-vmm-xen · GitHub
And I fixed the RPM spec between 21 nov and 22 nov.
But you can also just use the RPM spec of the official qubes repo, it have been upgraded for Xen 4.17