HVM no bootable device after gpu passthrough

neowutran · January 10, 2024, 7:58am

the recent patch applied to xen that should fix the max-ram-below-4g thing modified the behavior of this loop (allow_memory_relocate is now defined to 0 instead of 1, it should never have been set to 1 for our usecase )

github.com

xen-project/xen/blob/master/tools/firmware/hvmloader/pci.c#L353


      
           * if we're running qemu-traditional.  Items that don't fit will be
           * relocated into the 64-bit address space.
           *
           * This loop now does the following:
           * - If allow_memory_relocate, increase the MMIO hole until it's
           *   big enough, or until it's 2GiB
           * - If !allow_memory_relocate, increase the MMIO hole until it's
           *   big enough, or until it's 2GiB, or until it overlaps guest
           *   memory
           */
          while ( (mmio_total > (pci_mem_end - pci_mem_start))
                  && ((pci_mem_start << 1) != 0)
                  && (allow_memory_relocate
                      || (((pci_mem_start << 1) >> PAGE_SHIFT)
                          >= hvm_info->low_mem_pgend)) )
              pci_mem_start <<= 1;
          
          /*
           * Try to accommodate RMRRs in our MMIO region on a best-effort basis.
           * If we have RMRRs in the range, then make pci_mem_start just after
           * hvm_info->low_mem_pgend.

You could try to downgrade the xen version to before 4.17.2-8 to check if it work with the manual max-ram-below-4g workaround

In case everything else fail, someone could create a patch to add log in this code to monitor the value of all the important variables ( pci_mem_start / hvm_info->low_mem_pgend / hvm_info->low_mem_pgend << PAGE_SHIFT / pci_mem_end ), recompile xen with this patch, activate all xen logs, start the HVM and publish the logs related to the startup of the HVM. And compare it with the logs of people that doesn’t have this issue

Could be interesting to check the influence of this patch qemu: fix TOLUD for PCI passthrough by mati7337 · Pull Request #44 · QubesOS/qubes-vmm-xen-stubdom-linux · GitHub ( is it still needed ? does it have an influence on the issue you are having ? )