Portable QubesOS - dynamic vm pci assignment

Community Guides
1

o/

I’m using QubesOS on an USB stick as a portable solution.

Since portable goes hand in hand with differing systems and devices I ran into some issues.

The most annoying one, remapping attached devices to my network vm.
Therefore, I automated it.

code: script 
#!/bin/bash
set -eo pipefail

[[ $EUID -ne 0 ]] && { echo "Run as root" >&2; exit 1; }

SYS_UUID=$(dmidecode -s system-uuid)
BOARD_NAME=$(dmidecode -s baseboard-product-name)
BOARD_SERIAL=$(dmidecode -s baseboard-serial-number)
echo "Running on SYS_UUID: $SYS_UUID (BOARD_NAME: $BOARD_NAME - BOARD_SERIAL: $BOARD_SERIAL)"

declare -A PCI_ASSIGNMENT

if [[ "$SYS_UUID" == "9f0dc5f0-84dd-4c7d-ac1a-97c6f2e8be26" ]]; then
    echo "Config: System N"
    #PCI_ASSIGNMENT["first-vm"]="dom0:00_01.2 dom0:00_02.0"
elif [[ "$SYS_UUID" == "2f42f0d2-6771-459a-b921-ef1ffd55cb91" ]]; then
    echo "Config: System N+1"
    #PCI_ASSIGNMENT["first-vm"]="dom0:00_01.4"
    #PCI_ASSIGNMENT["a-second-vm"]="dom0:00_01.5"
else
    echo "Config: Fallback"
fi

check_vm() {
    local vm_name="$1"
    local state
    state=$(qvm-ls -O STATE "$vm_name" 2>/dev/null | tail -n +2)

    if [[ -z "$state" ]]; then
        echo "Error: VM $vm_name does not exist" >&2
        return 1
    fi

    if [[ "$state" == "Halted" ]]; then
        return 0
    fi

    echo "Error: $vm_name is running, cannot modify PCI assignments" >&2
    return 1
}

check_device() {
    local pci="$1"
    local -a available
    mapfile -t available < <(qvm-pci ls 2>/dev/null | tail -n +1 | awk '{print $1}')
    for entry in "${available[@]}"; do
        [[ "$entry" == "$pci" ]] && return 0
    done
    echo "Error: PCI device $pci not found in system" >&2
    return 1
}

get_unassignments_delta() {
    local -n _pci_assignment="$1"
    local vm_name="$2"

    local -a current
    mapfile -t current < <(qvm-pci ls --assignments "$vm_name" 2>/dev/null | tail -n +1 | grep "$vm_name" | awk '{print $1}')

    local -a intended
    read -ra intended <<< "${_pci_assignment[$vm_name]}"

    for pci in "${current[@]}"; do
        local found=0
        for intended_pci in "${intended[@]}"; do
            [[ "$pci" == "$intended_pci" ]] && found=1 && break
        done
        [[ $found -eq 0 ]] && echo "$pci"
    done
}

get_assignments_delta() {
    local -n _pci_assignment="$1"
    local vm_name="$2"

    local -a current
    mapfile -t current < <(qvm-pci ls --assignments "$vm_name" 2>/dev/null | tail -n +1 | grep "$vm_name" | awk '{print $1}')

    local -a intended
    read -ra intended <<< "${_pci_assignment[$vm_name]}"

    for pci in "${intended[@]}"; do
        local found=0
        for current_pci in "${current[@]}"; do
            [[ "$pci" == "$current_pci" ]] && found=1 && break
        done
        [[ $found -eq 0 ]] && echo "$pci"
    done
}

unassign() {
    local vm_name="$1"
    echo "Unassigning pci devices from $vm_name"
    qvm-pci unassign "$vm_name" 
}

assign() {
    local vm_name="$1"
    local pci="$2"
    echo "Assigning $pci to $vm_name"
    qvm-pci assign -r -o no-strict-reset=True "$vm_name" "$pci"
}

verify_assignments() {
    local error=0

    for vm in "${!PCI_ASSIGNMENT[@]}"; do
        check_vm "$vm" || error=1

        read -ra pci_ids <<< "${PCI_ASSIGNMENT[$vm]}"
        for pci in "${pci_ids[@]}"; do
            check_device "$pci" || error=1
        done
    done

    return $error
}

apply_assignments() {
    for vm in "${!PCI_ASSIGNMENT[@]}"; do
        local -a to_unassign
        mapfile -t to_unassign < <(get_unassignments_delta PCI_ASSIGNMENT "$vm")
        if [[ ${#to_unassign[@]} -gt 0 ]]; then
            unassign "$vm"
        fi

        local -a to_assign
        mapfile -t to_assign < <(get_assignments_delta PCI_ASSIGNMENT "$vm")
        for pci in "${to_assign[@]}"; do
            assign "$vm" "$pci"
        done
    done
}

verify_assignments || { echo "not applying changes - exiting script">&2; exit 1; }
apply_assignments
code: service 
[Unit]
Description=dynamic assignment of pci devices to vms
After=qubesd.service
Before=qubes-vm@.service

[Service]
Type=oneshot
ExecStart=/where/did/i/put/my-script.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
code: qubesd-config 
[Unit]
After=my-custom.service
Requires=my-custom.service

If you want to adapt the solution:

  1. run the script and it will output your system uuid
    1.1. at the top of the script you can with that uuid distinguish between system configuration
  2. add the per-system config as: PCI_ASSIGNMENT["first-vm"]="dom0:00_01.2 dom0:00_02.0"
  3. that line would remove all other pci device of the vm and attach the specified ones

I recommend to only follow until here and just run+verify the config manually on your systems

  1. add the systemd service
  2. put qubesd-config on /etc/systemd/system/qubes-vm@.service.d/after-dyn-assign.conf

With that systemd integration we run the script each time QubesOS boots and chain it between the necessary qubesd.service and the startup of the configured vms.

2 Likes