Greetings. I am a longtime user of Qubes and have always had great interest in the bones and inner workings of the platform. I understand there has been interest in removing fedora as a dependency from dom0 for some time now. We recently carried out some research into a micro-kernel adjacent construction using Alpine linux as a base for dom0 in Xen. We wanted to share our notes here, in case this is ever of use to the community for future Qubes developments.
TLDR: We were able to boot a fully RAM-resident Xen + Alpine linux for dom0 as the base. We believe this research is valuable to eventually migrate Qubes away from Fedora as the base for dom0.
Summary
We booted a custom Alpine Linux dom0 under Xen 4.17 entirely from RAM on a 128GB server with no disk dependency after initial GRUB loading. The final image is a 34MB gzipped cpio archive containing a complete Xen toolstack, OpenRC init system, and SSH server. This post documents the specific technical problems encountered and their solutions.
Motivation
The goal was a minimal, stateless Xen dom0 that runs entirely from tmpfs. No persistent storage after boot. Power cycle returns the system to a known-good state from the same image. The dom0 provides the Xen toolstack (xl, xenstored, xenconsoled), SSH access, and the ability to create and manage domU virtual machines.
Boot Chain
The final working boot chain is:
UEFI PXE -> pxelinux (LOCALBOOT 0) -> NVMe EFI partition -> GRUB 2.06
-> multiboot2 Xen 4.17.6 hypervisor
-> module2 Alpine Linux 6.18.7-lts kernel (PV dom0)
-> module2 --nounzip 34MB gzipped cpio rootfs
-> OpenRC -> xenstored, xenconsoled, sshd
-> Login prompt on hvc0
GRUB loads three files: the Xen hypervisor binary, the Linux kernel, and the rootfs archive. The kernel unpacks the cpio into tmpfs and runs /sbin/init. After GRUB finishes, the NVMe is never accessed again.
Building the Rootfs
The rootfs is built inside a Docker container running Alpine 3.23. Base packages (openrc, openssh-server, dhcpcd, busybox) come from Alpine 3.23 repos. The Xen toolstack comes from Alpine 3.18 repos to match the 4.17 hypervisor (more on why below). Kernel modules are extracted from Alpine’s modloop squashfs, trimmed to a allowlist of ~60 modules needed for the specific hardware, and bundled into the cpio along with all config files, SSH host keys, and an authorized_keys file. The entire build takes about two minutes.
The cpio is created with find | cpio -o -H newc | gzip -9 from the container’s root filesystem, excluding /proc, /sys, and /dev virtual filesystems.
Problem 1: Console Under Xen PV
Symptom: After the kernel printed “Run /sbin/init as init process”, the system appeared to freeze. No output, no response. This happened consistently.
Cause: Under Xen PV, dom0’s interactive console is /dev/hvc0, not /dev/ttyS0. The physical UART (ttyS0) is owned by Xen for kernel printk output only. Userspace programs that try to open ttyS0 as a terminal fail silently. The system was actually booting and running OpenRC the entire time, but all output and the login prompt were going to hvc0 which was not connected to the serial console.
Fix: Add console=hvc0 to the kernel command line (in addition to console=ttyS0 for kernel messages). Configure the inittab getty to use hvc0:
hvc0::respawn:/sbin/getty 115200 hvc0
Both console=ttyS0 (kernel messages visible through the serial port) and console=hvc0 (userspace getty visible through the same serial port via Xen’s console multiplexing) are needed.
Problem 2: Xen Toolstack Version Mismatch
Symptom: xenstored failed to start with “FATAL: Failed to open connection to gnttab: No such file or directory”. This happened even with all xen-gntdev, xen-gntalloc, and xen-privcmd kernel modules loaded and /dev/xen/ device nodes present.
Cause: Alpine 3.23 ships Xen 4.20 toolstack packages (xen-libs, xen, xl, xenstored). Our hypervisor was Xen 4.17.6. The Xen toolstack communicates with the hypervisor via domctl and sysctl hypercalls that include an interface version field. These version constants changed between 4.17 and 4.20. The 4.17 hypervisor rejected every hypercall from the 4.20 toolstack with a version mismatch error.
This is documented: libxenctrl (libxc) has an unstable ABI that must match the hypervisor version exactly.
Fix: Install the Xen toolstack from Alpine 3.18 repos, which ships Xen 4.17:
apk add --no-cache \
--repository=http://dl-cdn.alpinelinux.org/alpine/v3.18/main \
--repository=http://dl-cdn.alpinelinux.org/alpine/v3.18/community \
xen
After this change, xenstored started cleanly and xl list showed Domain-0.
Problem 3: Device Node Creation Without devtmpfs
Symptom: Even with kernel modules loaded successfully (confirmed via lsmod and /proc/misc), /dev/xen/ did not exist. xenstored could not open /dev/xen/gntdev.
Cause: Alpine’s OpenRC boot environment uses mdev (BusyBox’s lightweight device manager) instead of udev. devtmpfs was not mounted at /dev. The kernel registered all Xen misc devices in /proc/misc with correct minor numbers, but without devtmpfs or a working mdev scan, no device nodes were created in /dev.
Running mdev -s failed because /sys/dev did not exist in the Xen PV dom0 environment (PV dom0 has a stripped-down sysfs).
Fix: Create device nodes manually from /proc/misc in the boot init script. All Xen devices are misc devices with major number 10:
mkdir -p /dev/xen
awk '/xen\//{split($2,a,"/"); mknod="/dev/xen/"a[2]; system("mknod "mknod" c 10 "$1" 2>/dev/null")}' /proc/misc
chmod 600 /dev/xen/*
This reads /proc/misc, finds all entries under the xen/ namespace, extracts the minor number, and creates the device node. No udev, no mdev, no devtmpfs required.
The hvc0 console device also needed manual creation:
[ ! -e /dev/hvc0 ] && mknod /dev/hvc0 c 229 0 && chmod 666 /dev/hvc0
Problem 4: dom0_mem and the Struct Page Trap
Symptom: Intermittent freezes during early kernel boot at “Grant tables using version 1 layout”. The freeze was sensitive to rootfs size (34MB froze, 48MB sometimes worked). Removing dom0_mem entirely caused a different failure: “VFS: Unable to mount root fs on unknown-block(0,0)”.
Cause: On a 128GB system, the Xen dom0_mem parameter has a subtle and destructive interaction with the Linux kernel’s memory management.
With dom0_mem=2048M (no max: qualifier): Xen allocates 2GB to dom0 but sets the maximum reservation to the full 128GB. The Linux kernel allocates struct page metadata for all 128GB of potential pages, roughly 33.5 million pages at 64 bytes each, consuming approximately 2GB. This exhausts the entire dom0 allocation before the kernel reaches grant table initialization.
Without any dom0_mem: Xen gives dom0 nearly all 128GB. The resulting memory layout places the initramfs at a high physical address that becomes inaccessible due to e820 memory map conflicts in the PV dom0 builder. The kernel cannot extract the cpio, finds no rootfs, and panics trying to mount block device (0,0).
This is documented in the Xen Project blog post “Dom0 Memory, Where It Has Not Gone” and is the reason Qubes OS always specifies both min and max qualifiers.
Fix: Always specify both the allocation and the maximum:
dom0_mem=4096M,max:4096M
The max:4096M caps the maximum reservation, reducing struct page overhead to ~64MB and leaving 3.9GB for the kernel, initramfs extraction, and normal operation. 4GB provides comfortable headroom for a RAM-only rootfs.
Problem 5: Module Dependencies in Trimmed Rootfs
Symptom: NIC drivers (ice, ixgbe) and USB host controller (xhci-hcd) failed to load with “Unknown symbol” errors. The ice driver couldn’t find hwmon symbols, ixgbe couldn’t find mdio/libphy symbols, xhci-hcd couldn’t find usbcore symbols.
Cause: The module whitelist in the build script included the top-level drivers but not their dependency modules. When the modloop was trimmed from ~3700 modules to ~60, the dependency modules were stripped out. depmod regenerated modules.dep correctly for the remaining modules, but modprobe couldn’t resolve the dependency chain because the required .ko files were missing from the filesystem.
Fix: Add the dependency modules to the whitelist:
hwmon # needed by ice, ixgbe, nvme
libphy phylib # needed by ixgbe (mdio bus)
usbcore usb-common # needed by xhci-hcd
The OpenRC Init Script
All of the Xen-specific boot logic is handled by a single OpenRC init script that runs in the boot runlevel before xenstored:
#!/sbin/openrc-run
description="Mount xenfs and load Xen modules"
depend() {
need modules
before xenstored
}
start() {
ebegin "Loading Xen modules and mounting xenfs"
modprobe xen_gntdev
modprobe xen_gntalloc
modprobe xen_privcmd
modprobe xen_evtchn
mkdir -p /dev/xen
awk '/xen\//{split($2,a,"/"); m="/dev/xen/"a[2]; system("mknod "m" c 10 "$1" 2>/dev/null")}' /proc/misc
chmod 600 /dev/xen/*
[ ! -e /dev/hvc0 ] && mknod /dev/hvc0 c 229 0 && chmod 666 /dev/hvc0
mkdir -p /proc/xen
grep -q xenfs /proc/mounts || mount -t xenfs xenfs /proc/xen
eend $?
}
This loads the four required Xen kernel modules, creates device nodes by parsing /proc/misc, creates the hvc0 console device, and mounts xenfs. After this script completes, xenstored can start and xl commands work.
GRUB Configuration
The final working GRUB entry:
menuentry 'Alpine Xen 4.17 (dom0)' {
insmod part_gpt
insmod ext2
insmod multiboot2
search --no-floppy --fs-uuid --set=root <BOOT_PARTITION_UUID>
multiboot2 /xen-4.17.6.gz console=com1,vga com1=115200,8n1 dom0_mem=4096M,max:4096M gnttab_max_frames=256 gnttab_max_maptrack_frames=1024 smt=off
module2 /vmlinuz-alpine-lts rdinit=/sbin/init console=tty0 console=ttyS0,115200 console=hvc0
module2 --nounzip /alpine-dom0-rootfs.gz
boot
}
Key parameters:
dom0_mem=4096M,max:4096Mwith the max qualifier to prevent struct page exhaustiongnttab_max_frames=256increased from default 64 for headroom--nounzipon the rootfs module to prevent GRUB from decompressing before Xen passes it to the kernelrdinit=/sbin/initto run OpenRC directly from the cpio rootfsconsole=hvc0for userspace output through Xen’s console
Final State
The system boots to a login prompt in approximately 3 seconds after GRUB hands off to Xen. The running dom0 provides:
- Xen hypervisor with full HVM, IOMMU, and HAP support for creating isolated domU VMs
- xl toolstack (xl list, xl create, xl info all functional)
- xenstored and xenconsoled running
- SSH server listening on port 22 with key-based authentication
- 127GB of free memory available for domU allocation
- 34MB total footprint, running entirely from RAM
- Stateless operation: power cycle returns to identical known-good state
Lessons
-
When running under Xen PV, the userspace console is hvc0. The serial port (ttyS0) is for kernel messages only. This one fact explains an entire category of “system appears frozen” reports.
-
On systems with large RAM (64GB+),
dom0_memwithoutmax:is a trap. The kernel allocates struct page arrays for the maximum reservation, not the current allocation. Always specify both. -
The Xen toolstack version must exactly match the hypervisor. The domctl/sysctl hypercall interface is versioned and there is no backward compatibility.
-
Minimal Linux environments (Alpine with mdev/busybox) that lack devtmpfs and udev need manual device node creation for Xen interfaces. Reading /proc/misc and calling mknod is reliable and dependency-free.
-
A complete Xen dom0 with toolstack, SSH, and networking fits in 34MB. Most of that is kernel modules. The userspace toolstack and init system together are under 10MB.