In an attempt to get the sys-net VM running, I get the following message:
“Failed to start an HVM qube with PCI devices assigned - hardware does not support IOMMU/VT-d/AMD-Vi”
There is a documented way of running Qubes without such support. But how do I do so?
I am so frustrated - please help this poor guy.
Hello, not sure if you already figured it out. I was having the same problem but only just figured it out. I don’t know if this is the right way to do it - it just works for me. Probably not the secure way to do it.
Go to the “Q” icon > Service: sys-firewall > sys-firewall: Qube Settings > Advanced > Virtualization Mode: PVH.
Do the same/similar for the other services. Some of them can only use “PV.” Then check that your domains are set to the same (they should be already on PVH by default). That’s pretty much it.
[jasperjonze1] did it worked?
Hi, I have a similar problem, my hardware does also not support the IOMMU… but I am still willing to try this OS to see if I am capable of learning at least a bit of it. I am running Qubes 4.2.0-rc3. I can see dom0 and debian-12-xfce started but if I try to start sys-net or sys-whonix I get that error of the title of the original post. The weird thing for me is that in the [Dom0] Qube manager I see that all qubes are “default (default-dvm)”. Virtualization is enabled in my BIOS. I am asking as I couldn’t find the solution after a long time search. Thanks
https://www.qubes-os.org/doc/installation-troubleshooting/#unsupported-hardware-detected-error
For the step 1 (Change the virtualization mode), the option is in the Advanced tab of the qube settings.
For R4.2, the step 3 is now:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
https://www.qubes-os.org/doc/releases/4.2/release-notes/
because R4.2 use a
Unified grub.cfg location for both UEFI and legacy boot (#7985)
This is the default disposable qube (Default DispVM column), not the virtualization mode of the qube (Virt Mode column).
Thanks for your prompt reply. I’ve done step 1, now I am not sure where to do steps 2 and 3. I suppose I need to open a console but where and how? I’ve tried in the debian-12-xfce but after rebooting the PC I get this error if I try to start sys-net:
Start failed: internal error: libxenlight failed to create new domain ‘sys-usb’, see /var/log/libvirt/libxl/libxl-driver.log for details
Attempting to start sys-net results in: Start failed: internal error: Unable to reset PCI device 0000:02:00.2: internal error: Active 0000:02:00.0 devices on bus with 0000:02:00.2, not doing bus reset, see /var/log/libvirt/libxl/libxl-driver.log for details
Please also tell me if you are sure this line is correct as I have a grub directory in /boot but not a grub2 directory there: sudo grub2-mkconfig -o /boot/grub2/grub.cfg
The commands must be done in a dom0 terminal.
Qubes Menu > Settings > Other > Xfce Terminal
or
right click on desktop > Open Terminal Here
or
right click on desktop > Applications > Terminal Emulator
It’s correct if you do it in dom0.
- Add
qubes.enable_insecure_pv_passthroughtoGRUB_CMDLINE_LINUXin/etc/default/grub
The default file could looks like this.
[user@dom0 ~]$ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="gfxterm"
GRUB_CMDLINE_LINUX="rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles rd.driver.pre=btrfs rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_THEME="/boot/grub2/themes/qubes/theme.txt"
GRUB_CMDLINE_XEN_DEFAULT="console=none dom0_mem=min:1024M dom0_mem=max:4096M ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096"
GRUB_DISABLE_OS_PROBER="true"
. /etc/default/grub.qubes-kernel-vm-support
Add qubes.enable_insecure_pv_passthrough at the end of this line:
(doesn’t need to be at the end, but every arguments must be space separated)
GRUB_CMDLINE_LINUX="rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap plymouth.ignore-serial-consoles rd.driver.pre=btrfs rhgb quiet qubes.enable_insecure_pv_passthrough"
This part was because you don’t have IOMMU capabilities.
This error says you also probably need to add no-strict-reset=true option to your pci device.
https://www.qubes-os.org/doc/how-to-use-pci-devices/#additional-attach-options
This doc show how to do it via command line, still in dom0.
You can also do it via the GUI in the Devices tab of your sys-net qube.
I executed that grub2-mkconfig command and added the indicated argument to grub. I was not sure of how to add the option to my pci device so using the GUI, I passed the complete list to the right. Then I restarted, it asked for the encryption password, it shows a few lines and… the damn screen goes black again!
In your sys-net qube, you should have only 2 pci devices.
Ethernet Controller and Network controller, respectivly for ethernet and wifi.
In the Devices tab, there is a bar at the bottom: Configure strict reset for PCI devices.
Select your device (Ethernet Controller), click on that button and enable no-strict-reset.
Do the same for the wireless device.
Well, the black screen is expected, you have given to your sys-net all of your pci devices …
therefore dom0 doesn’t have any pci devices left.
Pci devices are your hardware.
your cpu, your SSD, your audio, etc … everything now belongs to your sys-net qube and then dom0 can’t use them.
You need to disable the autostarting of your sys-net qube.
Follow these steps: https://www.qubes-os.org/doc/autostart-troubleshooting/
Once Qubes OS has started again, in the Devices tab of your sys-net, remove everything except the ethernet and wireless devices.
Add the no-strict-reset to the wireless and/or the ethernet.
If sys-net start with no-strict-reset only for wireless, you don’t need it for ethernet.
Maybe you need it for both, you need to try.
If my explanation lacks details and/or you’re not sure about something, just ask.
Ok, I’ve done that for the wireless, then I started the debian qube (I don’t have the Fedora installed) and the sys-net qube manually: computer hangs (mouse and keyboard arrows are not responsive). I forced a turn off and on the computer by pressing the on/off button, I went back to the beginning of your instructions but this time I also added the no-strict-reset to the ethernet as well and computer hangs again when I started the mentioned qubes.
Again I forced a turn off and on of the computer by pressing the on/off button, this time I did not boot disabling the autostart, it asked me for the passphrase, some lines after that it stays forever in “Starting plymouth-quiet-wait.service - Hold until boot process finishes up…”
Maybe I was wrong about the no-strict-reset, I’ve read the eror message too quickly.
Your error is the same as this one in the troubleshooting doc:
https://www.qubes-os.org/doc/pci-troubleshooting/#unable-to-reset-pci-device-errors
I would try this fix without no-strict-reset and see if it works or if you got another error.
Well, I don’t know.
Try rebooting with autostarting off and fix your sys-net qube.
The only modification was with pci devices and sys-net, I guess that’s related.
I’ve booted with autostarting off and then started the sys-net which resulted in the pc freezing. Using the GUI (if possible): How to know if the no-strict-reset is on or off for each device? or how to turn it off?
It’s the same (large) button at the bottom: Configure strict reset for PCI devices.
In fact you don’t need to pre-select a device when clicking on it.
When you click on this button, the no-strict-reset option is enabled for every highlighted devices.
To see which option are assigned to each devices, you can run in dom0 the command qvm-pci.
e.g.
[user@dom0 ~]$ qvm-pci
BACKEND:DEVID DESCRIPTION USED BY
[...]
dom0:00_14.0 USB controller: [...] sys-usb-dvm (no-strict-reset=True)
See the link in my previous post for your issue.
There are 2 sections:
I guess you don’t have anymore any of these warning? just the freeze.
When you added qubes.enable_insecure_pv_passthrough, you also change the qube to PV virtualization mode ?
You could try to assign only the ethernet device and see if that works (probably yes).
Then assign only the wireless device to confirm that it’s the one causing the freeze.
You can run sudo dmesg in your sys-net to see the error.
If you cannot start it, remove the pci device, and run the command once the qube is started.
(the previous log message should still be there, hopefully).
Answering your question: I can see only 2 ‘pv’ at the ‘Virt Mode’ column: ‘sys-net’ and ‘sys-usb’
It looks like there is no way to remove the no-strict-reset with the GUI but to transfer that device to the left hand list, click ok and then put it back in the right hand list. As I used the command you told me to check it: sudo dmesg
I don’t know if I did something or what happened but now starting the PC without the qubes.skip_autostart results in the desktop loading, but before that it complains it cannot start some of the qubes (is it any way to pause that so I can read it before it continues?)
Regarding ‘sys-net’:
I removed the offending device by:
sudo su
echo -n "1" > /sys/bus/pci/devices/0000:02:00.0/remove
That device is a card reader that I don’t normally use so it is fine for me to have it removed.
Then I started the ‘sys-net’ and it does not complain but it freezes the PC and shows ‘Qube Status: disp5661 Qube disp5661 is starting’.
So, I rebooted and took all the steps but to start the ‘sys-net’, then:
You suggested running sudo dmesg in my ‘sys-net’, I suppose I have to click on ‘Qube Manager’, ‘sys-net’, ‘run a command’. I tried that after all the rest I mentioned and it freezes the PC.
On this notebook I am not using the ethernet LAN card. I am using only the wireless, so thinking that it may help, I passed the ethernet to the left hand list and then having only the wireless on the right hand list I restarted the PC and the desktop does not load anymore!
I hope I did not miss anything, I am a bit tired so I can’t guarantee it.
Just to be sure.
in dom0, when you run cat /etc/default/grub, you do have the qubes.enable_insecure_pv_passthrough?
When loading your laptop, the qube it cannot start is probably the sys-net qube.
And therefore, as sys-net don’t start, it doesn’t prevent Qubes OS to start.
I guess you have done this in dom0. This was meant to be done in the sys-net template.
Forget about it, as you’ve done it via the GUI.
In dom0, remove that file.
You need to open a terminal in your sys-net.
In the top-right corner, in the panel tray, click on the blue qube, under sys-net you have Run Terminal.
But before doing that, you need to remove the wireless pci device.
Otherwise you will not be able to start your sys-net qube.
Before running sudo dmesg, you can also try this (as mentionned in the doc):
Remove your wireless device from sys-net.
in dom0 terminal run:
qvm-pci attach --persistent --option permissive=true --option no-strict-reset=true sys-net dom0:02_00.2
Verify with qvm-pci that 02_00.2 is your wireless device.
If that doesn’t work, remove all pci devices, start sys-net, open a Terminal and run sudo dmesg > sysnet_dmesg.log.
You will have a file (sysnet_dmesg.log), use a usb drive or anything, and upload that file here.
Well, you know how to make it start now (with qubes.skip_autostart)
At this point, just open the setting of sys-net and sys-firewall and disable autostarting on boot for both of them until it works.
An other thing you could try.
If your ethernet is working, use it to fully update Qubes OS.
And after the update, try again the wireless.
When you installed Qubes, you also have an option to use the latest kernel.
It may help (not sure, but worth trying).
When running cat /etc/default/grub I have the qubes.enable_insecure_pv_passthrough
I checked using qvm-pci and my wireless card is now 03:00.0 (I think that is different as before). Then I ran qvm-pci attach --persistent --option permissive=true --option no-strict-reset=true sys-net dom0:03_00.0, I started the sys-net qube and I connected to the wireless network 
Then I rebooted and I can see that it still works. Thanks a lot.
Now I need to investigate other things not working, hopefully I don’t need to open a new thread for them.