Starting work on NixOS Template

hey there, building on the recent xen work in nixpkgs - I have created a NixOS template which you can find here:

this is my first stab at creating public nix packages / modules so apologies in advance :smiley:

many of the basics are working at this point, details in the README. no RPM yet for aid with install. if you want to try it out before I get that working you should be able to use the nixos installer under HVM mode, and then can switch over to a config using this with PVH.

4 Likes

Post a Qubes Backup of the template with the password “a” on github or someplace else and I will look into it. Really if I have to put hours just to install… you lost me. Don’t ask for mine!

Here is an idea based on my understanding of Nix history. Knoppix (Linux written in Assembly) is the precursor of Nix. The last act of Knopper was a Debian kernel.
I would take a ISO editor to a very good Debian eliminate a lot of things, add Qubes functions and add Nix. This has been done already and it is not easy. This is very hard because you have to configure a lot of things and make the Nix part retentive/persistent. I guess solicitation for Qubes template is the same as solicitation of drugs or beggin on a street corner on this forum. Let them shoot themselves in the foot.

That looks really promising. Please correct me if I’m wrong, but from my understanding I need to

I’m not sure about the process. May you explain it a bit further?

Just as a friendly reminder for anyone who might be reading, Qubes Backups will contain metadata about qubes which are not selected for inclusion in the backup (you can see this warning in the bold text at the bottom of the first screen that opens after selecting “Qubes Backup” from the “Qubes Tools” menu. This may or may not be relevant to your threat model, but posting a public backup is potentially dangerous even if none of the qubes you selected contain sensitive data.

3 Likes

Thanks @evq for your work on this. Like @Rnd3sB3g13rng I’m not 100% sure of the process.
I tried to install nixos via .iso in a hvm and couldn’t even get that far. When I try to boot it I get:

SeaBIOS (version 1.13.0-3.fc32)
Machine UUID 7a...
Booting from Hard Disk...
GRUB _

this is probably down to my HVM nativity, but any pointer would be much appreciated. Here is my process:
download nixos iso
create HVM
qvm-create nixos --class StandaloneVM --label red --prop virt_mod=hvm --prop kernel='' --prop size=40G --prop private=40G --prop memory=8000 --prop maxmem=16000 --prop default_dispvm=''
set system storage size to 30gb

launch with iso image
qvm-start nixos --cdrom=disp534:/home/user/Downloads/nixos-minimal.iso

setup networking

go to VM settings identifiy:
HVM IP addres: 10.137.0.49
gateway addres: 10.138.35.49
Virtual DNS: 10.139.1.1, 10.139.1.2

set up IP address

sudo ip a add 10.137.0.49/32 dev enX0
sudo ip l set dev enX0 up

configure connection to net qube

sudo ip r add 10.138.35.49 dev enX0
sudo ip r add default via 10.138.35.49

setup DNS

sudo nano /etc/resolv.conf
append add these lines
nameserver 10.139.1.1
nameserver 10.139.1.2

set up partitions

cfdisk /dev/xvda
creata a 0.5g partition and set type to ‘BIOS’
create a partition with the remainder and leave as linux filesystem

lsblk
sudo mkfs.ext4 /dev/xvda1 -L NIXBOOT
sudo mkfs.ext4 /dev/xvda2 -L NIXROOT
sudo mount /dev/disk/by-label/NIXROOT /mnt
sudo mkdir -p /mnt/boot
sudo mount /dev/disk/by-label/NIXBOOT /mnt/boot

generate config

sudo nixos-generate-config --root /mnt
sudo nano /mnt/etc/nixos/configuration.nix

uncomment the grub.device line and change to = “/dev/xvda”

cd /mnt
sudo nixos-install --no-root-passwd

grub entry appears in /mnt/boot/grub/grub.cfg.
reboot and it hangs with GRUB_ as mentioned.

The contents of grub.cfg would be helpful.

Also, it looks like you missed the third backtick in the “generate config” section, causing the code snippets and commentary to be reversed (happens to everyone sometimes :slightly_smiling_face:).

Thanks @skyvine. It isn’t obvious how to edit the post to correct the formatting.

Anyway here is the grub.cfg generated by nixos-install:

# Automatically generated. DO NOT EDIT THIS FILE!

    search --set=drive` --fs-uuid 6f2b...ee8
    if [ -s $prefix/grubenv ]; then
        load_env
    fi

    # grub-reboot sets a one-time saved entry, which we process here and
    # then delete.
    if [ "${next_entry}" ]; then
        set default="${next_entry}"
        set next_entry=
        save_env next_entry
        set timeout=1
        set boot_once=true
    else
        set default=0
        set timeout=5
    fi
    set timeout_style=menu

    function savedefault {
        if [ -z "${boot_once}"]; then
        saved_entry="${chosen}"
        save_env saved_entry
        fi
    }

    # Setup the graphics stack for bios and efi systems
    if [ "${grub_platform}" = "efi" ]; then
        insmod efi_gop
        insmod efi_uga
    else
        insmod vbe
    fi
    
        insmod font
        if loadfont ($drive1)//converted-font.pf2; then
            insmod gfxterm
            if [ "${grub_platform}" = "efi: ]; then
                set gfxmode=auto
                set gfxpayload=keep
            else
                set gfxmode=1024x768
                set gfxpayload=text
            fi
            terminal_output gfterm
        fi

        background_color `#2F302F`

        insmod png
        if background_image --mode `normal` ($drive1)//background.png; then
            set color_normal=white/black
            set color_highlight=black/white
        else
            set menu_color_normal=cyan/blue
            set menu_color_highlight=white/blue
        fi

menuentry "NixOS" --class nixos --unrestricted {
search --set=drive1 --fs-uuid 6f2b...ee8
    linux ($drive1)//kernels/1kp51...zm4-linux-6.6.52-bzImage init=/nix/store/i7z1q...sx2-nixos-system-nixos-24.05.5287.759537f06e69/init loglevel=4
    initrd ($drive1)//kernels/m1xly...kzh-initrd-linux-6.6.52-initrd
}

submenu "NixOS - All configurations" --class submenu {
menuentry "NixOS - Configuration 1 (2024-09-27 - 24.05.5287.759537f06e69)" --class nixos {
search --set=drive1 --fs-uuid 6f2b...ee8
    linux ($drive1)//kernels/1kp51...zm4-linux-6.6.52-bzImage init=/nix/store/i7z1q...sx2-nixos-system-nixos-24.05.5287.759537f06e69/init loglevel=4
    initrd ($drive1)//kernels/m1xly...kzh-initrd-linux-6.6.52-initrd
}

}

Does the error message you printed appear after selecting an entry in the boot menu or do you not even get to the boot menu?

Can you load an ISO and confirm that the filesystem with the UUID listed in:

search --set=drive1 --fs-uuid 6f2b...ee8

exists? You can use findfs UUID=6f2b...ee8 (with the full hash instead of the …). If it does, can you mount it and confirm that the files from the line following the search command exist? In particular, /kernels/...-bzimage and /nix/store/.../init.

I do not even get to the boot menu.

6f2b…ee8 is the filesystem UUID associated to label NIXBOOT. The filesystem contains 2 directories:

kernels/
  ...linux-6.6.52-bzImage
  ...initrd-linux-6.6.52-initrd
grub/
  ...

it doesnt contain /nix as NixOS is installed on the b465…402 filesystem on xvda2 UUID associated with label NIXROOT.

I suspect the problem is that NIXROOT is not getting mounted. In my grub.cfg’s menu entries the linux command contains root=GUIX_ROOT but there isn’t anything similar in yours. As a quick test you should be able to add root=NIXROOT manually to that line, anywhere after the *-bzImage should be fine but mine is immediately after it. The file will probably have read-only permissions because it was copied from the store but it shouldn’t be a symlink to the store so making it writable is safe. You might want to double-check that in case NixOS is different, but I don’t think symlinking would technically work for files on /boot. The manual edits will get overwritten the next time you reconfigure but this will prove/disprove the root of the problem. If this fixes it we can start looking at a long-term solution.

menuentry "NixOS" --class nixos --unrestricted {
search --set=drive1 --fs-uuid 6f2b...ee8
    linux ($drive1)//kernels/1kp51...zm4-linux-6.6.52-bzImage root=NIXROOT init=/nix/store/i7z1q...sx2-nixos-system-nixos-24.05.5287.759537f06e69/init loglevel=4
    initrd ($drive1)//kernels/m1xly...kzh-initrd-linux-6.6.52-initrd
}

I could write the file using sudo. I made the change you suggested but still get this:

SeaBIOS (version 1.13.0-3.fc32)
Machine UUID 7a88b9...ab5
Booting from Hard Disk...
GRUB _

I then commented out the selection section and replaced it with timeout=10 and I still didn’t get presented with a menu. I am not sure the grub.cfg file is being used?

Should that machine UUID correlate some UUID on xvda? I cannot find a reference to it anywhere else.

My assumption is that the UUID is what Xen assigned to the VM. You can double-check this by running xenstore-read -f /vm | grep 'name = "nixos"' in a dom0 terminal. The -f tells it to print the full path in every line, and the UUID appears in the path.

I’m thinking about why Grub might fail to find the config. You can mount the filesystem and it has the configuration file in it, so that should be fine. My machine also uses ext4 so unless your Grub has a strange configuration that shouldn’t be the problem. Personally, I don’t keep /boot on my BIOS boot partition. From what I understand from random things I’ve read in various places (so not the most reliable information, but it’s lined up with my experiences) the BIOS uses a very early part of the disk which the filesystem doesn’t use for historical reasons that are technically unrelated to the BIOS. So it’s supposed to be safe to do that, and the fact that you’re able to explore the filesystem and the output indicates that Grub is loading (but hitting some error condition) makes me think that this isn’t the source of the problem.

Also, not sure if you saw this already but: