Following this superb guide I intended to use a kernel with debugging symbols enabled.
Booting an AppVM with a debugging kernel works (which can be inspected with qvm-console-dispvm appvm), unfortunately qrexec services won’t get started resulting in a timeout and consecutive shutdown of the AppVM.
Basically, I tried two methods:
### pvh kernel
# inside dom0
sudo qubes-dom0-update qubes-kernel-vm-support kernel-debug
sudo qubes-prepare-vm-kernel 6.5.12.100.fc37.x86_64+debug
qvm-create -t template -l red appvm
qvm-prefs appvm kernel 6.5.12.100.fc37.x86_64+debug
qvm-prefs appvm qrexec_timeout 600
qvm-run -u root appvm xterm
and
### distro kernel
# inside dom0
qvm-run -u root template xterm
# inside the template
dnf update
dnf install kernel-debug
grub2-install /dev/xvda
grub2-mkconfig -o /boot/grub2/grub.cfg
shutdown now
# inside dom0
qvm-create -t template -l red appvm
qvm-prefs appvm virt_mode hvm
qvm-prefs appvm kernel ''
qvm-prefs appvm qrexec_timeout 600
qvm-service appvm memwriter-info off
qvm-run -u root appvm xterm
And as mentioned above
### for inspection what goes wrong during the boot process
# inside dom0
qvm-console-dispvm appvm
Here it says “Install whatever kernel you want.”
However, there is a warning in the log files saying:
[ 1.788450] *************************************************************
[ 1.789442] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
[ 1.790444] ** **
[ 1.791441] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL **
[ 1.792444] ** **
[ 1.793443] ** This means that this kernel is built to expose internal **
[ 1.794455] ** IOMMU data structures, which may compromise security on **
[ 1.795452] ** your system. **
[ 1.796441] ** **
[ 1.797438] ** If you see this message and you are not debugging the **
[ 1.798435] ** kernel, report this immediately to your vendor! **
[ 1.799435] ** **
[ 1.800435] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
[ 1.801435] *************************************************************
Are there tweaks which prevent the appvm from communicating via qrexec if a modified kernel is used?
If anyone likes to help or assist I can provide more of the log file’s content.
It looks like the kernel just crashes or maybe some of it’s modules crash the box. The following is using kernel-debug from the distro installed inside a fedora-39-minimal template.
[2024-02-12 08:05:55] [.[0;32m OK .[0m] Finished .[0;1;39mmodprobe@efi_pstore.service.[0m - Load Kernel Module efi_pstore.
[2024-02-12 08:05:55] [.[0;32m OK .[0m] Finished .[0;1;39mmodprobe@fuse.service.[0m - Load Kernel Module fuse.
[2024-02-12 08:05:55] [.[0;32m OK .[0m] Finished .[0;1;39mmodprobe@loop.service.[0m - Load Kernel Module loop.
[2024-02-12 08:05:55] [.[0;32m OK .[0m] Finished .[0;1;39msystemd-modules-load.service.[0m - Load Kernel Modules.
[2024-02-12 08:05:55] [.[0;32m OK .[0m] Finished .[0;1;39msystemd-network-generator….[0mork units from Kernel command line.
[2024-02-12 08:05:56] [.[0;32m OK .[0m] Finished .[0;1;39msystemd-remount-fs.servic….[0mmount Root and Kernel File Systems.
[2024-02-12 08:05:56] Mounting .[0;1;39msys-fs-fuse-connections.mount.[0m - FUSE Control File System...
[2024-02-12 08:05:56] Starting .[0;1;39mqubes-db.service.[0m - Qubes DB agent...
[2024-02-12 08:05:56] Starting .[0;1;39msystemd-journal-flush.ser….[0msh Journal to Persistent Storage...
[2024-02-12 08:05:56] Starting .[0;1;39msystemd-sysctl.service.[0m - Apply Kernel Variables...
[2024-02-12 08:05:56] Starting .[0;1;39msystemd-tmpfiles-setup-de….[0m Device Nodes in /dev gracefully...
[2024-02-12 08:05:57] [.[0;32m OK .[0m] Mounted .[0;1;39msys-fs-fuse-connections.mount.[0m - FUSE Control File System.
[2024-02-12 08:05:57] [ 26.551419] ------------[ cut here ]------------
[2024-02-12 08:05:57] [ 26.568250] kernel BUG at mm/page_table_check.c:53!
[2024-02-12 08:05:57] [ 26.584221] invalid opcode: 0000 [#1] PREEMPT SMP KASAN PTI
[2024-02-12 08:05:57] [ 26.595320] systemd-journald[532]: Received client request to flush runtime journal.
[2024-02-12 08:05:57] [ 26.601653] CPU: 1 PID: 560 Comm: qubesdb-daemon Not tainted 6.7.4-200.fc39.x86_64+debug #1
[2024-02-12 08:05:57] [ 26.601663] Hardware name: Xen HVM domU, BIOS 4.17.3 01/30/2024
[2024-02-12 08:05:57] [ 26.601667] RIP: 0010:page_table_check_set.part.0+0x23d/0x580
[2024-02-12 08:05:57] [ 26.601681] Code: 48 8b 04 24 80 38 00 0f 85 87 02 00 00 48 83 c5 01 48 03 1d 25 95 bf 08 4c 39 e5 0f 84 e8 01 00 00 48 85 db 0f 85 53 ff ff ff <0f> 0b be 04 00 00 00 4c 89 ef e8 d4 d6 f4 ff 4c 89 e8 48 c1 e8 03
[2024-02-12 08:05:57] [ 26.601685] RSP: 0018:ffffc90000eb78a8 EFLAGS: 00010246
[2024-02-12 08:05:57] [ 26.601692] RAX: dffffc0000000000 RBX: 0000000000000000 RCX: ffffffff94c64292
[2024-02-12 08:05:57] [ 26.601695] RDX: 1ffffd400043fffb RSI: 0000000000000008 RDI: ffffea00021fffd8
[2024-02-12 08:05:57] [ 26.601698] RBP: 0000000000000000 R08: 0000000000000000 R09: fffff9400043fff8
[2024-02-12 08:05:57] [ 26.601701] R10: ffffea00021fffc7 R11: 0000000000000001 R12: 0000000000000001
[2024-02-12 08:05:57] [ 26.601704] R13: ffffea00021fffc0 R14: 0000000000000000 R15: ffffea00021fffc0
[2024-02-12 08:05:57] [ 26.601707] FS: 00007f5133121200(0000) GS:ffff888061a00000(0000) knlGS:0000000000000000
[2024-02-12 08:05:57] [ 26.601711] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2024-02-12 08:05:57] [ 26.601715] CR2: 00007f1311501383 CR3: 0000000031d48001 CR4: 00000000003706f0
[2024-02-12 08:05:57] [ 26.601722] Call Trace:
[2024-02-12 08:05:57] [ 26.601725] <TASK>
[2024-02-12 08:05:57] [ 26.715050] systemd-journald[532]: /var/log/journal/33eb62d1064f437fb816de10a54149ec/system.journal: Journal file uses a different sequence number ID, rotating.
[2024-02-12 08:05:57] [ 26.715626] ? die+0x36/0x90
[2024-02-12 08:05:57] [ 26.715724] systemd-journald[532]: Rotating system journal.
[2024-02-12 08:05:57] [ 27.095421] ? do_trap+0x199/0x240
[2024-02-12 08:05:57] [ 27.123182] ? page_table_check_set.part.0+0x23d/0x580
[2024-02-12 08:05:57] [ 27.156335] ? page_table_check_set.part.0+0x23d/0x580
[2024-02-12 08:05:57] [ 27.177337] ? do_error_trap+0xa3/0x160
and so forth
Any guess or hint how debug this further or what I might have missed while installing a different kernel?