4.3.1 - grub entries missing for older kernels

This is a new one for me. I needed to boot into an older kernel today but grub showed no entries in advanced → XEN 4.19.5 menu. There were no submenu entries in grub for the older kernels. I have checked and GRUB_DISABLE_SUBMENU is false /etc/default/grub and I did properly regenerate the grub.cfg with grub2-mkconfig -o /boot/grub2/boot.cfg. I do see all the submenu entries for the older kernels in /boot/grub2/boot.cfg yet grub not only fails to show them but when I hit enter on the menu item XEN 4.19.5 in the advanced menu, instead of showing me the list of older kernels, it just boots the default entry. What am I missing? I feel like I’m missing something obvious so any hints to solve this would help

the file is being generated in the proper place and the efi config is pointing to the proper grub.cfg. The problem is grub is treating the Advanced options for Qubes (R4.3) (with Xen hypervisor) entry as a bootable entry instead of a menu item. I can edit this entry with e which I don’t think is proper given that this should be a menu entry

Doesn’t grub just pick up what dracut left behind?!?

What’s on your /boot? initramfs+vmlinuz for the missing kernels?

I reran both dracut and grub2-mkconfig to be sure and it’s always the same thing, it doesn’t treat this submenu ‘Xen hypervisor, version 4.19.5’ as something that has other menu items in it. It seems like it treats it as a menuitem. Really weird

I just tried commenting out the Xen hypervisor, version 4.19.5 submenu entry to remove the nesting and this works, it shows me all the old kernels and I can pick exactly which kernel to boot. Confirmed that grub 2.06 is running everywhere and it should support nested submenu. Really weird

I simply disabled grubs submenu (GRUB_DISABLE_SUBMENU=“true”) and this flattened the config and fixed it for me. Just weird that the nesting of submenu fails