Automated Qubes OS Installation using Kickstart and/or PXE Network Boot

qstateless · September 30, 2022, 9:28am

How’s resolving the development issues with sys-pxe coming?

Would you like to point out the specific parts of the code that aren’t working, so that specific coding solutions could potentially be identified by others?

Maybe a quick How To tutorial on running your salt files could encourage more of us to jump in with using, testing, & contributing?

alzer89 · September 30, 2022, 9:38am

The qvm.clone function to clone the fedora-36-minimal template and rename it is the only part of it not working. All other parts seem to work.

Any help would be massively appreciated.

qstateless · September 30, 2022, 9:55am

@alzer89 Ok, so is the following salt code within the sys-pxe-template.sls file what you are specifically using to attempt cloning and renaming the fedora-36-minimal template (but its not working for some unknown reason)?

I noticed that code is commented out…

sys-pxe-template.sls

alzer89 · September 30, 2022, 10:04am

Yes, because it failed (most likely because it’s not the appropriate way to issue a clone command).

In the testing scripts of Salt installed on every Qubes OS machine, there is a qvm.clone command, and this function would work perfectly…if I could get it to work.

That is the trouble I am having…

qstateless · September 30, 2022, 10:47am

@alzer89

How about something like this?

qvm-clone-id:
  qvm.clone:
    - name: sys-pxe-template
    - source: fedora-36-minimal

Reference:

github.com

unman/shaker/blob/main/multimedia/clone.sls

multimedia_precursor:
  qvm.template_installed:
    - name: debian-11-minimal

qvm-clone-id:
  qvm.clone:
    - name: template-multimedia
    - source: debian-11-minimal

alzer89 · September 30, 2022, 9:33pm

I tried that, but it threw an error. My guess is that I hadn’t configured something properly.

I can see references to those functions in /srv/formulas/test, but not in base.

But yes, that should work…

unman · September 30, 2022, 11:02pm

Where are the salt files to be found?
How are you calling salt?

enmus · October 1, 2022, 9:39am

I’m still learning Salt, but I’d try this

include:
- sys-pxe-template
qvm-clone-id:
qvm.clone:
- require:
- sls: sys-pxe-template
- name: sys-pxe
- source: fedora-36-minal

sys-pxe-template.sls should be in a parent directory to this as how I understand, meaning in the /srv/salt

alzer89 · October 1, 2022, 9:25pm

They’re all in /srv/salt. Should they be somewhere else?

qubesctl top.enable sys-pxe
qubesctl —all state.highstate

Thank you @unman!

unman · October 2, 2022, 1:29am

I mend, where is the source?
Are they on GitHub?

qstateless · October 7, 2022, 6:35am

@unman: to answer your question, @alzer89’s source salt files for sys-pxe are not on GitHub, but actually inside of this web forum thread at post #8.

There are 4 source salt files, which are apparently placed in dom0’s /srv/salt

sys-pxe.top
sys-pxe-template.sls
sys-pxe-vm.sls
sys-pxe.sls

Executed with:

qubesctl top.enable sys-pxe
qubesctl —all state.highstate

I believe the not working qvm.clone command is supposed to be in the sys-pxe-template.sls file, with an alternative tested version of the command presently commented out in that provided salt file.

Thank you @unman for your assistance in solving this salt puzzle for this very interesting sys-pxe community project.

Here is a link to and quote of @alzer89’s post with the 4 original sys-pxe salt files:

https://forum.qubes-os.org/t/automated-qubes-os-installation-using-kickstart-and-or-pxe-network-boot/12963/8

Full quote of source post:

alzer89:

I have written some salt files to:

Check to make sure that fedora-36-minimal is installed, and install it if it isn’t
Clone it into sys-pxe-template (this is the bit I’m having trouble with)
Change sys-pxe-template into a disposable VM template
Install the following packages into sys-pxe-template :

nfs-server
tftp-server
dhcp-server
syslinux

Put the following config files into sys-pxe-template :

/etc/dhcp/dhcpd.conf
- Configure DHCP server with subnet of 192.168.100.0/24, and TFTP boot server
/var/lib/tftpboot/*
- All the PXE files from syslinux
- The pxelinux.cfg and custom grub2 EFI files from Qubes OS /boot directory (needed to multiboot Xen)
/usr/local/bin/download-and-verify-latest-qubes-iso
- Bash script to download and verify the latest Qubes OS ISO
/etc/systemd/system/download-and-verify-latest-qubes-iso.service
- Systemd service to automatically run download-and-verify-latest-qubes-iso when the Qube starts
/etc/systemd/network/20-all-interfaces.network
- Set static IP address of sys-pxe to 192.168.100.1
- Disable Qubes virtual NICs because they aren’t needed

All of these config files are actually written in plain text in the *.sls files for transparency and to keep the install size down.

Create a disposable VM called sys-pxe , and base it on sys-pxe-template

All the user has to do is open this DispVM, and their ethernet NIC automatically becomes a Qubes OS PXE Network Boot Server!

These are the salt files:

`sys-pxe.top`

# -*- coding: utf-8 -*-
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

# Installs 'sys-pxe' Qubes OS ISO Network Boot Server Qube.
#
# Pillar data will also be merged if available within the ``qvm`` pillar key:
#   ``qvm:sys-pxe``
#
# located in ``/srv/pillar/dom0/qvm/init.sls``
#
# Execute:
#   qubesctl top.enable qvm.sys-pxe
#   qubesctl --all state.highstate

{% if salt['pillar.get']('qvm:sys-pxe:name', 'sys-pxe') != salt['pillar.get']('qvm:sys-gui:name', 'sys-gui') %}
{% set vmname = salt['pillar.get']('qvm:sys-pxe:name', 'sys-pxe') %}
{% else %}
{% set vmname = salt['pillar.get']('qvm:sys-gui:name', 'sys-gui') %}
{% endif %}

base:
  dom0:
    - match: nodegroup
    - qvm.sys-pxe
  {{ salt['pillar.get']('qvm:sys-pxe:template', 'fedora-36-minimal') }}:
    - qvm.sys-pxe-template
  {{ vmname }}:
    - qvm.sys-pxe-vm

`sys-pxe-template.sls`

# -*- coding: utf-8 -*-
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

##
# qvm.sys-pxe-template
# ====================
##

fedora-36-minimal:
  qvm.template_installed: []

#sys-pxe-template:
#  qvm.template_installed: []

#dom0:
#  cmd.run:
#    - qvm-clone fedora-36-minimal sys-pxe-template

sys-pxe-template:
  qvm.vm:
    - present:
      - label: black
    - prefs:
      - label: black
      - dispvm-allowed: True
    - features:
      - enable:
        - appmenus-dispvm

  pkg.installed:
    - pkgs:
      - qubes-core-admin-client
      - syslinux
      - tftp-server
      - nfs-server
      - dhcp-server
      - systemd-networkd

  /var/lib/tftpboot/pxelinux.cfg/default:
    file.managed:
      - user: root
      - mode: 644
      - makedirs: True
      - contents: |
          default vesamenu.c32
          timeout 100
          
          menu background splash.png
          
          label Qubes-Manual
          menu label ^Qubes OS 4.1.1 - Automated Install - Kickstart
          kernel mboot.c32
          append qubes/xen.gz console=none --- qubes/vmlinuz inst.stage2=nfs:192.168.100.1:/var/lib/tftpboot/qubes/iso/Qubes.iso plymouth.ignore-serial-consoles ip=dhcp inst.ks=nfs:192.168.100.1:/var/lib/tftpboot/qubes/iso/anaconda-ks.cfg i915.alpha_support=1 quiet rhgb --- qubes/initrd.img
          
          label Qubes-Manual 
          menu label ^Qubes OS 4.1.1 - Manual Install - Regular ISO Boot
          kernel mboot.c32
          append qubes/xen.gz console=none --- qubes/vmlinuz inst.stage2=nfs:192.168.100.1:/var/lib/tftpboot/qubes/iso/Qubes.iso plymouth.ignore-serial-consoles ip=dhcp i915.alpha_support=1 quiet rhgb --- qubes/initrd.img
          
          label local
          menu label Boot from ^Local Drive
          localboot 0xffff
  
  /var/lib/tftpboot/EFI/qubes/grub.cfg:
    file.managed:
      - user: root
      - mode: 644
      - makedirs: True
      - contents: |
          function load_video {
          	insmod vga
          	insmod vbe
          	insmod efi_gop
          	insmod efi_vga
          	insmod ieee1275_fb
          	insmod video_bochs
          	insmod video_cirrus
          	insmod all_video
          }
          
          load_env
          
          load_video
          set gfxpayload=keep
          insmod gzio
          
          insmod gfxterm
          insmod gfxtext


  /etc/systemd/network/20-all-interfaces.network:
    file.managed:
      - user: root
      - mode: 644
      - makedirs: True
      - contents: |
          # Match all ethernet NICs
          [Match]
          Type=ether
          # For some reason, this network device exists and causes TFTP to fail if it's active...
          Name=!enX0
          
          # Set Ethernet NIC static IP address of 192.168.100.1
          [Network]
          Address=192.168.100.1/24
          # May not be necessary, but including them just in case
          Gateway=192.168.100.1
          DNS=192.168.100.1

   /etc/dhcp/dhcpd.conf:
    file.managed:
      - user: root
      - mode: 644
      - makedirs: True
      - contents: |
          option arch code 93 = unsigned integer 16;
          # If you are planning to PXE install Qubes OS on more than 253 machines at the same time, you are absolutely insane, and I love it!
          # If you are, then you probably know how to edit this config file to expand the subnet...
          subnet 192.168.100.0 netmask 255.255.255.0 {
          authoritative;
          default-lease-time 600;
          max-lease-time 7200;
          ddns-update-style none;
          option domain-name-servers 192.168.100.1;
          option routers 192.168.100.1;
          option broadcast-address 192.168.100.255;
          option subnet-mask 255.255.255.0;
          range 192.168.100.2 192.168.100.254;
          if option arch = 00:07 {
          # amd64 UEFI
          filename "uefi/shimx64.efi";
          next-server 192.168.100.1;
          } else if option arch = 00:0b {
          # aarch64 UEFI
          filename "uefi/shimaa64.efi";
          server-name 192.168.100.1;
          } else {
          filename "pxelinux.0";
          next-server 192.168.100.1;
          }
          
          }


  /var/lib/tftpboot/EFI/qubes/themes/qubes/theme.txt:
    file.managed:
      - user: root
      - mode: 644
      - makedirs: True
      - contents: |
          # Copyright (C) 2016 Harald Sitter <sitter@kde.org>
          #
          # This program is free software; you can redistribute it and/or
          # modify it under the terms of the GNU General Public License as
          # published by the Free Software Foundation; either version 3 of
          # the License or any later version accepted by the membership of 
          # KDE e.V. (or its successor approved by the membership of KDE
          # e.V.), which shall act as a proxy defined in Section 14 of
          # version 3 of the license.
          #
          # This program is distributed in the hope that it will be useful,
          # but WITHOUT ANY WARRANTY; without even the implied warranty of
          # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
          # GNU General Public License for more details.
          #
          # You should have received a copy of the GNU General Public License
          # along with this program.  If not, see <http://www.gnu.org/licenses/>.
          
          # paperwhite - #fcfcfc
          # icongrey - #4d4d4d
          # plasmablue - #3daee9
          # black - #000000
          
          # Global Property
          # General settings
          title-text: ""
          title-font: "Unifont Regular 14"
          message-font: "Unifont Regular 14"
          message-color: "#7f8c8d"
          message-bg-color: "#4d4d4d" # TODO: whatever is this for?
          desktop-image: "qubes.png"
          
          # title
          # NOTE: can't put this in a vbox because GRUB is crap and item highlighting
          #   is broken if you put the boot_menu in a vbox...
          # TODO: file bug report
          + label {
              top = 50%-225 # (150+43+32) menu + height + spacer
              left = 0%
              width = 100%
              text = "Qubes OS Installer - Network Boot"
              align = "center"
              font = "Unifont Regular 32"
              color = "#ffffff"
          }
          
          # Show the boot menu
          + boot_menu {
              left = 50%-200
              width = 450
              # NB: this is scooped upwards from the middle.
              #     effectively 50px are below and the remaining 150 above
              top = 50%-150
              height = 200
              # Icon
              icon_width = 4
              icon_height = 0
              # Item
              item_height = 33
              item_padding = 1
              item_icon_space = 0
              item_spacing = 1
              item_font =  "Unifont Regular 16"
              item_color = "#4d4d4d"
              selected_item_font = "Unifont Bold 16"
              selected_item_color = "#ffffff"
          }
          
          + vbox {
              left = 50%-200 # same as menu
              top = 50%+113 # (50+16+19+28) half menu + spacer + progress + spacer
              width = 400 # same as menu
              + label { width = 400 align = "center" color = "#4d4d4d" font = "Unifont Regular 14" text = "[Enter] Boot the selected OS" }
              + label { width = 400 align = "center" color = "#4d4d4d" font = "Unifont Regular 14" text = "[Up and Down Key] navigation" }
              + label { width = 400 align = "center" color = "#4d4d4d" font = "Unifont Regular 14" text = "[E] Edit Selection" }
              + label { width = 400 align = "center" color = "#4d4d4d" font = "Unifont Regular 14" text = "[C] GRUB Command Line" }
          }
          
          # Show a styled horizontal progress bar
          + progress_bar {
              id = "__timeout__"
              left = 0
              top = 100%-32
              width = 100%
              height = 32
              show_text = false
              bar_style = "progress_bar_*.png"
              highlight_style = "progress_bar_hl_*.png"
          }
          
          # Show text progress bar
          + progress_bar {
              id = "__timeout__"
              left = 50%-200 # same as menu
              top = 50%+66 # (50+16) half menu + spacer
              width = 400 # same as menu
              height = 19 # 14pt
              show_text = true
              font = "Unifont Regular 14"
              text_color = "#4d4d4d"
              align = "center"
              text = "@TIMEOUT_NOTIFICATION_MIDDLE@"
              bar_style = "progress_bar2_*.png"
          }

  /etc/exports:
    file.managed:
      - user: root
      - mode: 644
      - makedirs: True
      - contents: |
          /var/lib/tftpboot/qubes/iso	*(rw,no_root_squash)






 
  cmd.run:
    - name: systemctl enable tftp.service
    - name: systemctl enable tftp.socket
    - name: systemctl enable systemd-networkd
    - name: exportfs -r
    - name: systemctl enable nfs-server
    - name: systemctl enable dhcpd
# TODO:  Add more stuff to put the syslinux files in /var/lib/tftpboot

`sys-pxe-vm.sls` (needs a bit of work)

# -*- coding: utf-8 -*-
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

##
# qvm.sys-pxe-vm
# ==============
##

# WIP: currently use default user 'user'
/var/lib/tftpboot/pxelinux.cfg/default:
  file.managed:
    - user: root
    - mode: 644
    - makedirs: True
    - contents: |
        default vesamenu.c32
        timeout 100
        
        menu background splash.png
        
        label Qubes-Manual
        menu label ^Qubes OS 4.1.1 - Automated Install - Kickstart
        kernel mboot.c32
        append qubes/xen.gz console=none --- qubes/vmlinuz inst.stage2=nfs:192.168.100.1:/var/lib/tftpboot/qubes/iso/Qubes.iso plymouth.ignore-serial-consoles ip=dhcp inst.ks=nfs:192.168.100.1:/var/lib/tftpboot/qubes/iso/anaconda-ks.cfg i915.alpha_support=1 quiet rhgb --- qubes/initrd.img
        
        label Qubes-Manual 
        menu label ^Qubes OS 4.1.1 - Manual Install - Regular ISO Boot
        kernel mboot.c32
        append qubes/xen.gz console=none --- qubes/vmlinuz inst.stage2=nfs:192.168.100.1:/var/lib/tftpboot/qubes/iso/Qubes.iso plymouth.ignore-serial-consoles ip=dhcp i915.alpha_support=1 quiet rhgb --- qubes/initrd.img
        
        label local
        menu label Boot from ^Local Drive
        localboot 0xffff

/var/lib/tftpboot/EFI/qubes/grub.cfg:
  file.managed:
    - user: root
    - mode: 644
    - makedirs: True
    - contents: |
        function load_video {
        	insmod vga
        	insmod vbe
        	insmod efi_gop
        	insmod efi_vga
        	insmod ieee1275_fb
        	insmod video_bochs
        	insmod video_cirrus
        	insmod all_video
        }
        
        load_env
        
        load_video
        set gfxpayload=keep
        insmod gzio
        
        insmod gfxterm
        insmod gfxtext
        
        terminal_output gfxterm
        insmod gfxmenu
        loadfont $prefix/themes/qubes/unifont-bold-16.pf2
        loadfont $prefix/themes/qubes/unifont-regular-14.pf2
        loadfont $prefix/themes/qubes/unifont-regular-16.pf2
        loadfont $prefix/themes/qubes/unifont-regular-32.pf2
        insmod png
        set theme=$prefix/themes/qubes/theme.txt
        export theme
        
        set timeout_style=menu
        set timeout=10
        
        menuentry 'Qubes OS 4.1.1 - Automated Install - Kickstart'  --class fedora --class gnu-linux --class gnu --class os {
        #	echo "To make a tasty Qubes OS..."
        #	echo "We start with the Xen Hypervisor..."
        	echo "Loading Xen Hypervisor..."
        	multiboot2 qubes/xen.gz console=none
        #	echo "...add a whole raw Linux Kernel..."
        	echo "Loading Linux Kernel..."
        	module2 qubes/vmlinuz ip=dhcp inst.stage2=nfs:192.168.100.1:/var/lib/tftpboot/qubes/iso/Qubes.iso plymouth.ignore-serial-consoles i915.alpha_support=1 quiet rhgb inst.ks=nfs:192.168.100.1:/var/lib/tftpboot/qubes/iso/anaconda-ks.cfg
        #	echo "...infuse it with your Kickstart Config, for extra flavoury goodness..."
        	echo "Adding Kickstart File...l"
        #	echo "...and finally, sprinkle it with a robust Initramfs..."
        	echo "Loading Initramfs..."
        	module2 qubes/initrd.img
        #	echo "...and we have a delicious Qubes OS.  Enjoy ;-)"
        }
        
         
        menuentry 'Qubes OS 4.1.1 - Manuak Install - Regular ISO Boot'  --class fedora --class gnu-linux --class gnu --class os {
        #	echo "To make a tasty Qubes OS..."
        #	echo "We start with the Xen Hypervisor..."
        	echo "Loading Xen Hypervisor..."
        	multiboot2 qubes/xen.gz console=none
        #	echo "...add a whole raw Linux Kernel..."
        	echo "Loading Linux Kernel..."
        	module2 qubes/vmlinuz ip=dhcp inst.stage2=nfs:192.168.100.1:/var/lib/tftpboot/qubes/iso/Qubes.iso plymouth.ignore-serial-consoles i915.alpha_support=1 quiet rhgb
        #	echo "...and finally, sprinkle it with a robust Initramfs..."
        	echo "Loading Initramfs..."
        	module2 qubes/initrd.img
        #	echo "...and we have a delicious Qubes OS.  Enjoy ;-)"
        }
        
        menuentry 'Exit this GRUB' {
        	exit
        }

/var/lib/tftpboot/uefi/BOOTX64.CSV:
  file.managed:
    - user: root
    - mode: 644
    - makedirs: True
    - contents: |
        shimx64.efi,Fedora,,This is the boot entry for Fedora
        grubx64.efi,Qubes,,This is the boot entry for Qubes OS
 
/usr/local/bin/download-and-verify-latest-qubes-iso:
  file.managed:
    - user: root
    - mode: 644
    - makedirs: True
    - contents: |
        #!/bin/bash
        # Download the Qubes OS 4.1.1 ISO
        gpg2 --import /etc/pki/rpm-gpg/RPM-GPG-KEY-qubes*
        gpg2 --import /usr/share/qubes/qubes-master-ket.asc
        wget https://ftp.qubes-os.org/iso/Qubes-R4.1.1-x86_64.iso.asc -O /var/lib/tftpboot/qubes/iso/Qubes-R4.1.1-x86_64.iso.asc &
        wget https://ftp.qubes-os.org/iso/Qubes-R4.1.1-x86_64.iso -O /var/lib/tftpboot/qubes/iso/Qubes-R4.1.1-x86_64.iso
        wget https://ftp.qubes-os.org/iso/Qubes-R4.1.1-x86_64.iso.DIGESTS -O /var/lib/tftpboot/qubes/iso/Qubes-R4.1.1-x86_64.iso.DIGESTS
        md5sum -c /var/lib/tftpboot/qubes/iso/Qubes-R4.1.1-x86_64.iso.DIGESTS
        sha1sum -c /var/lib/tftpboot/qubes/iso/Qubes-R4.1.1-x86_64.iso.DIGESTS
        sha256sum -c /var/lib/tftpboot/qubes/iso/Qubes-R4.1.1-x86_64.iso.DIGESTS
        sha512sum -c /var/lib/tftpboot/qubes/iso/Qubes-R4.1.1-x86_64.iso.DIGESTS
        gpg2 -v --verify /var/lib/tftpboot/qubes/iso/Qubes-R4.1.1-x86_64.iso.asc /var/lib/tftpboot/qubes/iso/Qubes-R4.1.1-x86_64.iso

/etc/systemd/system/download-and-verify-latest-qubes-iso:
  file.managed:
    - user: root
    - mode: 644
    - makedirs: True
    - contents: |
        [Unit]
        Description=Download & Verify Latest Qubes OS ISO from ftp.qubes-os.org
        [Service]
        Type=oneshot
        ExecStart=/usr/local/bin/download-and-verify-latest-qubes-iso
        RemainAfterExit=yes
        
        [Install]
        WantedBy=multi-user.target

`sys-pxe.sls`

# -*- coding: utf-8 -*-
# vim: set syntax=yaml ts=2 sw=2 sts=2 et :

##
# qvm.sys-pxe
# ===========
#

fedora-36-minimal:
  qvm.template_installed: []

#sys-pxe-template:
#  qvm.template_installed: []

{% from "qvm/template.jinja" import load -%}

{% if salt['pillar.get']('qvm:sys-pxe:name', 'sys-pxe') != salt['pillar.get']('qvm:sys-gui:name', 'sys-gui') %}

{% set vmname = salt['pillar.get']('qvm:sys-pxe:name', 'sys-pxe') %}

{% load_yaml as defaults -%}
name:          sys-pxe
qvm.present:
  - name: sys-pxe
  - label:     red
  - mem:       400
prefs:
  # Uncomment if you want additional repos via HTTP to be avaialble
  #  - netvm:     "sys-firewall"
  # Default - No access to the internet
  - netvm:     ""
  - virt_mode: hvm
  - autostart: true
  - pci_strictreset: false
  - pcidevs:   {{ salt['grains.get']('pci_net_devs', [])|yaml }}
  - class:     DispVM
  - template:  sys-pxe-template

{%- endload %}

{{ load(defaults) }}

{% else %}

{% set vmname = salt['pillar.get']('qvm:sys-gui:name', 'sys-gui') %}

{{ vmname }}-pxe:
  qvm.prefs:
    - name: {{ vmname }}
    - virt_mode: hvm
    - pcidevs:   {{ salt['grains.get']('pci_net_devs', [])|yaml }}
    - pci_strictreset: false

{% endif %}

Is there any chance anyone could have a look at them and help me fix them?

Thank you so much in advance

unman · October 7, 2022, 9:53am

Thank you for this.
I’ll try to take a look this evening.

alzer89 · October 7, 2022, 11:36pm

My bad. Been a busy time for me.

I can create a git repo if you like, but you’ll likely need to fork it and make a branch with the files in the right places, haha.

Yes, thank you @qstateless!

Thank you, too, @unman!