QubesOS on the VM -> Unsuported Hardware

Hi there,

I am running MX Linux as my main OS,
and in the VM I want to run QubesOS, for which I am allocating 40GB of the SDD.

Unfortunately, I am encountering a problem at the beginning of the installation process.
Picture below

Quick System Info of the Hardware and MX OS

System: Kernel: 5.10.0-21-amd64 [5.10.162-1] x86_64 bits: 64 compiler: gcc v: 10.2.1 parameters: BOOT_IMAGE=/vmlinuz-5.10.0-21-amd64 root=UUID=<filter> ro quiet splash Desktop: Xfce 4.18.1 tk: Gtk 3.24.24 info: xfce4-panel wm: xfwm 4.18.0 vt: 7 dm: LightDM 1.26.0 Distro: MX-21.3_x64 Wildflower January 15 2023 base: Debian GNU/Linux 11 (bullseye) Machine: Type: Laptop System: Dell product: Latitude 7480 v: N/A serial: <filter> Chassis: type: 10 serial: <filter> Mobo: Dell model: N/A serial: <filter> UEFI-[Legacy]: Dell v: 1.19.0 date: 07/26/2020 Battery: ID-1: BAT0 charge: 16.4 Wh (52.7%) condition: 31.1/60.0 Wh (51.9%) volts: 7.2 min: 7.6 model: SMP DELL DM3WC64 type: Li-poly serial: <filter> status: Discharging CPU: Info: Dual Core model: Intel Core i7-7600U bits: 64 type: MT MCP arch: Amber/Kaby Lake note: check family: 6 model-id: 8E (142) stepping: 9 microcode: F0 cache: L2: 4 MiB flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 23199 Speed: 1420 MHz min/max: 400/3900 MHz Core speeds (MHz): 1: 1420 2: 1421 3: 1405 4: 1424 Vulnerabilities: Type: itlb_multihit status: KVM: VMX disabled Type: l1tf mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable Type: mds mitigation: Clear CPU buffers; SMT vulnerable Type: meltdown mitigation: PTI Type: mmio_stale_data mitigation: Clear CPU buffers; SMT vulnerable Type: retbleed mitigation: IBRS Type: spec_store_bypass mitigation: Speculative Store Bypass disabled via prctl and seccomp Type: spectre_v1 mitigation: usercopy/swapgs barriers and __user pointer sanitization Type: spectre_v2 mitigation: IBRS, IBPB: conditional, RSB filling, PBRSB-eIBRS: Not affected Type: srbds mitigation: Microcode Type: tsx_async_abort mitigation: Clear CPU buffers; SMT vulnerable Graphics: Device-1: Intel HD Graphics 620 vendor: Dell driver: i915 v: kernel bus-ID: 00:02.0 chip-ID: 8086:5916 class-ID: 0300 Device-2: Microdia Integrated_Webcam_HD type: USB driver: uvcvideo bus-ID: 1-5:2 chip-ID: 0c45:6715 class-ID: 0e02 Display: x11 server: X.Org 1.20.11 compositor: xfwm4 v: 4.18.0 driver: loaded: modesetting unloaded: fbdev,vesa display-ID: :0.0 screens: 1 Screen-1: 0 s-res: 2560x1440 s-dpi: 96 s-size: 677x381mm (26.7x15.0") s-diag: 777mm (30.6") Monitor-1: eDP-1 res: 2560x1440 hz: 60 dpi: 210 size: 309x174mm (12.2x6.9") diag: 355mm (14") OpenGL: renderer: Mesa Intel HD Graphics 620 (KBL GT2) v: 4.6 Mesa 20.3.5 direct render: Yes Audio: Device-1: Intel Sunrise Point-LP HD Audio vendor: Dell driver: snd_hda_intel v: kernel alternate: snd_soc_skl bus-ID: 00:1f.3 chip-ID: 8086:9d71 class-ID: 0403 Sound Server-1: ALSA v: k5.10.0-21-amd64 running: yes Sound Server-2: PulseAudio v: 14.2 running: yes Network: Device-1: Intel Ethernet I219-LM vendor: Dell driver: e1000e v: kernel port: f040 bus-ID: 00:1f.6 chip-ID: 8086:15d7 class-ID: 0200 IF: eth0 state: down mac: <filter> Device-2: Intel Wireless 8265 / 8275 driver: iwlwifi v: kernel modules: wl port: f040 bus-ID: 02:00.0 chip-ID: 8086:24fd class-ID: 0280 IF: wlan0 state: up mac: <filter> Bluetooth: Device-1: Intel Bluetooth wireless interface type: USB driver: btusb v: 0.8 bus-ID: 1-7:3 chip-ID: 8087:0a2b class-ID: e001 Report: hciconfig ID: hci0 rfk-id: 3 state: up address: <filter> bt-v: 2.1 lmp-v: 4.2 sub-v: 100 hci-v: 4.2 rev: 100 Info: acl-mtu: 1021:4 sco-mtu: 96:6 link-policy: rswitch hold sniff link-mode: slave accept service-classes: rendering, capturing, object transfer, audio Drives: Local Storage: total: 476.94 GiB used: 7.63 GiB (1.6%) SMART Message: Unable to run smartctl. Root privileges required. ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: A-Data model: SX6000PNP size: 476.94 GiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s lanes: 4 type: SSD serial: <filter> rev: V9002s16 temp: 29.9 C scheme: MBR Partition: ID-1: / raw-size: 23.38 GiB size: 22.85 GiB (97.71%) used: 7.34 GiB (32.1%) fs: ext4 dev: /dev/dm-0 maj-min: 253:0 mapped: root.fsm ID-2: /boot raw-size: 1024 MiB size: 973.4 MiB (95.06%) used: 187.1 MiB (19.2%) fs: ext4 dev: /dev/nvme0n1p1 maj-min: 259:1 ID-3: /home raw-size: 444.52 GiB size: 436.47 GiB (98.19%) used: 112.2 MiB (0.0%) fs: ext4 dev: /dev/dm-2 maj-min: 253:2 mapped: 1.home.fsm Swap: Kernel: swappiness: 15 (default 60) cache-pressure: 100 (default) ID-1: swap-1 type: partition size: 7.98 GiB used: 0 KiB (0.0%) priority: -2 dev: /dev/dm-1 maj-min: 253:1 mapped: swap Sensors: System Temperatures: cpu: 35.5 C mobo: N/A Fan Speeds (RPM): cpu: 0 Repos: Packages: note: see --pkg apt: 2041 lib: 1032 flatpak: 0 No active apt repos in: /etc/apt/sources.list Active apt repos in: /etc/apt/sources.list.d/brave-browser-release.list 1: deb [arch=amd64] https://brave-browser-apt-release.s3.brave.com/ bullseye main Active apt repos in: /etc/apt/sources.list.d/debian-stable-updates.list 1: deb http://deb.debian.org/debian bullseye-updates main contrib non-free Active apt repos in: /etc/apt/sources.list.d/debian.list 1: deb http://deb.debian.org/debian bullseye main contrib non-free 2: deb http://security.debian.org/debian-security bullseye-security main contrib non-free Active apt repos in: /etc/apt/sources.list.d/mx.list 1: deb http://nl.mxrepo.com/mx/repo/ bullseye main non-free Active apt repos in: /etc/apt/sources.list.d/nordvpn.list 1: deb https://repo.nordvpn.com/deb/nordvpn/debian stable main Active apt repos in: /etc/apt/sources.list.d/signal-xenial-added-by-mxpi.list 1: deb [arch=amd64] https://updates.signal.org/desktop/apt xenial main Active apt repos in: /etc/apt/sources.list.d/vivaldi.list 1: deb http://repo.vivaldi.com/stable/deb/ stable main Info: Processes: 251 Uptime: 21m wakeups: 2 Memory: 31 GiB used: 2.41 GiB (7.8%) Init: SysVinit v: 2.96 runlevel: 5 default: 5 tool: systemctl Compilers: gcc: N/A alt: 10 Client: shell wrapper v: 5.1.4-release inxi: 3.3.06 Boot Mode: BIOS (legacy, CSM, MBR)

Did you read the documentation? Installation guide | Qubes OS

There’s a big note in yellow:

Note: Qubes OS is not meant to be installed inside a virtual machine as a guest hypervisor. In other words, nested virtualization is not supported. In order for a strict compartmentalization to be enforced, Qubes OS needs to be able to manage the hardware directly.

1 Like

VMware should work, it’s the only hypervisor I’ve been able to use to run Qubes OS, but you need to enable nested virtualization.

Keep in mind, it’s mostly useful for testing/trying Qubes, it’s not a secure way to use the OS.

I have other Linux operation system which I want to use + the GRUB from that installation
I want to keep MX GRUB, and be able to load QUBES which Is installed on dedicated partition

before installation
|nvmeon1p1||ext4|1024|MiB|boot|
|nvmeon1p2||luks|24.4|GiB|MX|
|nvmeon1p3||swap|10|GiB|SwapMX|
|nvmeon1p4
||nvmeon1p5|luks|128|GiB|Qubes|
||nvmeon1p6|luks|313.54||HomeMX|

After installation
|nvmeon1p1||ext4|1024|MiB|boot|
|nvmeon1p2||luks|24.4|GiB|MX|
|nvmeon1p3||swap|10|GiB|SwapMX|
|nvmeon1p4|
||nvmeon1p5|luks|313.54||HomeMX|
||nvmeon1p6|luks|1|GiB|boot|
||nvmeon1p7|luks|127|GiB|Qubes|

I used ‘automatic’ installation, and choose which partition I want to preserve, and on which one system supposed to be installed.

Before automatic installation happened, I used custom installation option, but without success. Message that I saw was following.

You have not defined a root partition (/), which is required for installation of Qubes OS to continue.
You have not created a bootable partition

|How to install Qubes, so MX is installed, Grub from MX is installed, and I have option to choose between MX and Qubes?

Many Thanks

I’ve been able to get Qubes up and running in a VM using the 4.2 weekly build ISO (haven’t tried 4.1 yet). This is with a nested vIOMMU and full HVM support.

I used kvm/qemu configured via VMM (Virtual Machine Manager) running on bare metal fedora-37, with some config changes for compatibility and to support vIOMMU usage.

I am using an AMD Zen 2 Family CPU and obviously the bare metal has virtualization and iommu enabled in the bios. I installed fedora-37 using the standard iso+Rufus+usb key, and fully updated it. I don’t recall if VMM was already installed or not…if not pretty easy to find and download from the fedora repos using the Software application.

I created the VM using the Generic Linux 2022 option, set memory/storage/etc. and then chose the easy-to-miss “Customize Configuration before install” checkbox on the page that had the finish button on it. Note that sometimes when I do that the iso I chose in the wizard ends up not set and I have to set it again under the sata cdrom 1 node.

VMM needed some additional manual configs (via UI clicky or UI XML tab, depending):
_Verify Generic Linux 2022 baseline. This should default to a virtual Q35 chipset which is the only way intel/amd vIOMMU can be supported, at least from what I have read.
_Change Firmware from BIOS to UEFI, as this may be necessary for vIOMMU support.
_For now I set CPU to one socket and eight cores (1 thread each)…I can later experiment up to 32 with the current hardware. Depending on your processor layout you may want to experiment with using sockets instead of cores. From what I’ve read vIOMMU can be sensitive to NUMA or quasi-NUMA issues (though perhaps only if actually passing in real hardware(?), but I’m not there yet). One way may have better performance but less stability than the other, depending on hardware.
_Changed NIC to e1000 from default of virtio (the default which did not work…it locked up sys-net).
_Initial setup had both usb tablet as input0 and ps2 mouse as input1. When I finally got viommu working the pointer stopped working (presumably due to sys-usb starting up). To fix that I removed the tablet device and set the ps2 mouse as input0.
_Changed video from Virtio (the default which did not work) to VGA. Upped vram to 65536 from 16384 in xml for larger screen options.
_Added RNG /dev/urandom
_In VMM Overview, where you can edit the entire XML file, you also need to add a manual clause in the devices section (closer to the bottom of the file).

<iommu model=“intel”/>

Lastly in the VM bios on first boot, disable secure boot so that the iso can actually be booted!

Qubes config….

Qubes’s grub needed an addl line (what i have below works, though includes some ignored bits) and then it has to be rebuilt:

sudo bash
vi /etc/default/grub
#add next line at end
GRUB_CMD_LINE=“$GRUB_CMD_LINE iommu=on iommu=pt amd_iommu=on intel_iommu=on”
#exit vi
:wq
#rebuild grub
grub2-mkconfig > /etc/grub2.cfg

I also created the file /etc/modprobe.d/vfio.conf but am not sure if it was necessary. Contents are:
options vfio_iommu_type1 allow_unsafe_interrupts=1

B

PS The above solution integrated some random things I came across with some of the advice here:
https://www.berrange.com/posts/2017/02/16/setting-up-a-nested-kvm-guest-for-developing-testing-pci-device-assignment-with-numa/

PPS-if you haven’t used VMM before it’s really easy to miss the View/Details menu option, which is how you can continue to modify the configuration between VM restarts. View/Console brings you back to the UI of the VM.

1 Like

I did try to get HVM working inside a QubesOS VM using VMWare Workstation 17 under Windows 10 on AMD Zen 2 last week (and yes all hyper-v and related MS virtualization software components disabled) but got nowhere and gave up after about 10-15 hours. Got it working under Linux/kvm (see my previous post) tonight.

B

Someone on the forum said VMware can’t run Qubes on AMD CPUs, it’s some issues with the nested IOMMU that only works with Intel.

Good to know it works with KVM, I had tried running Qubes in Gnome Boxes, but I couldn’t get the nested virtualization to work.

1 Like

Yeah Boxes and VMM are just, basically, 90% configurators for the libvirt xml, esp. now that remote server connectivity stuff has moved from Boxes to Connections.

I suspect I could just port over the xml I now have in VMM to boxes and it’ll just work, perhaps with some minor boxes metadata added.

B

Made an edit in the longer reply above to be sure that the iommu xml tag isn’t hidden by the message parser.

B