Xen debugging, the freeze into reboot

I recently had to help debug an issue with Xen, and I thought this might be useful to someone else who doesn’t have much experience with Xen.

Freeze into reboot

When Xen crashes, it will generate a crash report with stack dump and backtrace. This is what happens during the “freeze”, the crash report is printed to the terminal, and 5 second later Xen reboots the host.

Serial terminal

At this point, dom0 and any domUs are dead, and there is no easy way to get the crash report without having Xen output the report to a serial terminal. This is the most difficult part, because most modern systems don’t have easy accessible serial ports.

Most desktop systems should allow you to fairly easily connect a TTL/UART to USB cable to the motherboard using jumper pins, and the cable can be ordered on eBay or AliExpress for around $5.

It’s a different story with laptops, you can probably also connect a cable to a laptop, but you might not have standard jumper pins, and you would likely need to run the motherboard naked.

To enable the serial terminal, edit /etc/default/grub and add
loglvl=all guest_loglvl=all com1=115200,8n1 console=com1 to the GRUB_CMDLINE_XEN_DEFAULT line.

You are going to need a second device to read the serial console, connect usb cable to and second PC/laptop with picocom installed and run the command
picocom -b 115200 -r -l /dev/ttyUSB0

Check the manual before you try to connect anything to your hardware.

Building Xen

You might need to build Xen with debug options enabled, which works without using the Qubes builder.
You need to clone the specific version of Xen used by the version of QubesOS you are running, e.g. 4.2.0 would be Xen 4.17, and you need to use the config from qubes-vmm-xen e.g. https://github.com/QubesOS/qubes-vmm-xen/blob/main/config.

Then you just need to set up the build environment for Xen and compile it, the result being xen.gz.

Booting Xen

Now you can copy xen.gz into /boot and configure grub
grub2-mkconfig -o /boot/grub2/grub.cfg

You can now reboot the system and the new Xen should boot as default, if it doesn’t work you can select the previous version in the grub boot menu.


Thanks for this, it may be helpful, just a question about the last section. Qubes 4.2 introduced a unified grub.cfg location for both legacy and EFI systems, but I see that you are still using the EFI path there. Is it a mistake or is it due to an in-place upgrade that causes this to still exist somehow?

It never stopped working, so I just kept using it.

I’m using RC4 installed from ISO, and the old path works just fine. I haven’t looked at the content of /boot, maybe there are some symlinks that keeps it working.

I’ve edited the post and changed the path.

1 Like