Unable to passthrough audio device to sys-audio

enmus · February 22, 2022, 3:15am

I am trying to create functional sys-audio according to this post since there are no official manuals so far.

running qvm-pci:

dom0:00_1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller sys-audio (no-strict-reset=True)

I didn’t change automatically created /etc/qubes/policy.d/50-sys-audio.policy file, for the same reason and since I am not familiar with it’s meaning, so it’s content is

Summary

admin.Events * sys-audio sys-audio allow target=dom0
admin.Events * sys-audio @adminvm allow target=dom0
admin.Events * sys-audio @tag:audiovm-sys-audio allow target=dom0
admin.vm.CurrentState * sys-audio sys-audio allow target=dom0
admin.vm.CurrentState * sys-audio @adminvm allow target=dom0
admin.vm.CurrentState * sys-audio @tag:audiovm-sys-audio allow target=dom0
admin.vm.List * sys-audio sys-audio allow target=dom0
admin.vm.List * sys-audio @adminvm allow target=dom0
admin.vm.List * sys-audio @tag:audiovm-sys-audio allow target=dom0
admin.vm.property.Get +audiovm sys-audio @tag:audiovm-sys-audio allow target=dom0
admin.vm.property.Get +xid sys-audio @tag:audiovm-sys-audio allow target=dom0
admin.vm.feature.CheckWithTemplate +audio sys-audio @tag:audiovm-sys-audio allow target=dom0

I have also set sys-audio to manual start.

When I try to start it manually, the system crashes and no entries about crash can be found both in sudo journalctl, nor in sudo dmesg against “sys-audio”, last boot after crash, or “00:1b.0”, probably since it hasn’t time to record anything.

If I set sys-audio to autostart, this happens.

Considering no error could be traced I’m not sure documentation can be used to try to resolve the issue

Is there any advice what I could try in order to successfully passthrough audio device to sys-audio?

logoerthiner · February 22, 2022, 10:06am

Ah it just look like when I attempt to pass through the monitor device to a VM named “screen” - the time the VM starts, the machine power off immediately. I believe that was early in R4.0 or R3.2.

Anyway maybe your BIOS or your machine believe that the sound device is very important and it is just as important as a CPU, a SRAM, or a primary monitor.

I wonder if any Qubes OS user has tested to PCI passthrough any other devices? Is every device PCI passthroughable?

OK serious (though I am nowhere an expert)

Do you have only one audio device in your PCI device list?
What is the neighbor devices of your audio devices? Are they important devices?

enmus · February 23, 2022, 1:54am

Yes.
English is not my first language, can you please describe what you mean by “neighbor”? If it’s physically, I have no idea, if in qvm-pci, then precedent is USB Enhanced Controller (1a.0) and decedent is PCI Express Root Bridge (1c.0)?

Thanks for taking time to respond.

zithro · February 26, 2022, 11:38pm

As you know enmus, I run Qubes as a domU for now, so PCI passthrough is not working, still working on that (well I never tried to double passthrough a real PCIe device, but that’s off-topic).
My experience on vanilla Xen is that audio devices are not considered as important ones (like a GPU who has initialized a VGA BIOS).

I guess what he means is, when you have IOMMU groups, you cannot pass a single device to a VM, you have to passthrough a group of devices, all the ones that are in the same IOMMU group. But IIRC, there has been progress on that, it’s called the “ACS Override Kernel patch”. I don’t know if Qubes has this patch by default.
This post may help you a bit to understand and troubleshoot the problem (it’s about KVM but it applies to Xen the same way) :

Isolate single device into separate IOMMU group for PCI passthrough?

enmus · February 28, 2022, 1:02am

Thanks @zithro for your response.

After banging head to a wall for a whole day why all the info tell me that my iommu is enabled

Summary

$ sudo dmesg | grep -ie IOMMU|DMAR
[ 1.282178] Kernel command line: … plymouth.ignore-serial-consoles i915.alpha_support=1 rd.driver.pre=btrfs rhgb quiet iommu=1 iommu=pt intel_iommu=on usbcore.authorized_default=0 rd.qubes.hide_all_usb rd.qubes.hide_all_usb
[ 1.282440] DMAR: IOMMU enabled
[ 2.329612] iommu: Default domain type: Passthrough (set via kernel command line)
[ 8.089291] i915 0000:00:02.0: [drm] DMAR active, disabling use of stolen memory
$ cat /proc/cmdline
iommu=1 iommu=pt intel_iommu=on usbcore.authorized_default=0 rd.qubes.hide_all_usb rd.qubes.hide_all_usb
$ qubes-hcl-report
HVM: Active
I/O MMU: Active
HAP/SLAT: Yes
TPM: Device present
Remapping: yes
$ xl dmesg | grep -i vmx
(XEN) VMX: Supported advanced features:
(XEN) HVM: VMX enabled

while I still had my /sys/kernel/iommu_groups/ dir empty, I discovered at the end that “… you can’t see your IOMMU Group when you are using Xen (the information is hidden from dom0).” Off topic, Is this maybe related to what you wrote here?

So I ran Fedora Live USB, put intel_iommu=on in a grub at boot, and after running
[liveuser@localhost-live ~]$ for g in /sys/kernel/iommu_groups/*; do echo "IOMMU Group ${g##*/}:"; for d in $g/devices/*; do echo -e "\t$(lspci -nns ${d##*/})"; done; done
I finally saw my devices there. Luckily, my audio device is alone in it’s group

IOMMU Group 13:
00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)

But, it’s also not, because iommu_group obviously is not the cause of my issue.

So do you have any other idea what I could check?

Is the output of

$ xl -v dmesg

(XEN) Intel VT-d iommu 0 supported page sizes: 4kB
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
…
(XEN) [VT-D] It’s risky to assign 0000:00:1a.0 with shared RMRR at b9430000 for Dom5.
(XEN) [VT-D] It’s risky to assign 0000:00:1d.0 with shared RMRR at b9430000 for Dom7.
(XEN) [VT-D] It’s risky to assign 0000:00:14.0 with shared RMRR at b9430000 for Dom9.

of an any help to detect further steps?

enmus · February 28, 2022, 1:21am

And also this

$ sudo lspci -s 1b -v

Summary

00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller (rev 04)
Subsystem: Hewlett-Packard Company Device 179b
Flags: bus master, fast devsel, latency 0, IRQ 46
Memory at d4730000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [50] Power Management version 2
Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [130] Root Complex Link
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

zithro · February 28, 2022, 3:42am

@enmus is nice, my post only guided him, the real solution is in his answer.

I can’t really help you diagnose the xl dmesg output in bold, as my buffer ring exploded, I’m not using an Intel CPU, and those architectural-related things are chinese to me, still learning what they mean exactly !

This may tell me you’re trying to assign devices from the same IOMMU to different domains.
I found this old github issue about RMRR, didn’t read through the end but stopped at this comment : fail to assign pci device at start of usbvm · Issue #1544 · QubesOS/qubes-issues · GitHub, maybe that’ll help to pinpoint the issue.

Another advice, maybe useless on recent versions

Try the PCI “permissive” option, but IIRC that must be avoided for security reasons. Plus I have no idea how to set that up from Qubes (I only just discovered how to manually edit individualVM config files). On vanilla Xen, in the domU config file, you would have to set the permissive option on the pci assignments lines for individual devices, or using a pci_permissive=1 option for applying it globally to all PCI-PT devices of the domU.
But check below before trying that, or try using it via the manual way at the end of the post.

Again, don’t know how it’s handled by Qubes, but considering vanilla Xen and my usage/experience, one of the lines below seems in error to me.

For a device that would be PT’ed, the kernel driver should be pciback, or the module xen_pciback, so like this :

Kernel driver in use: pciback
Kernel modules: snd_hda_intel

OR

Kernel driver in use: snd_hda_intel
Kernel modules: xen_pciback

Maybe check the Xen PCI passtrough guide for known PCI-PT errors, and how it’s done, and you could compare with how it’s done on Qubes ? There are static or dynamic assignments, using the kernel driver or the module. I’ll gladly read your findings about the differences btw, that would enhance my Qubes knowledge ! So much to learn ^^

Another strange thing :

Dunno what that means, but maybe that’s an “important” device after all, on Intel platforms ? Here, nothing like that.

Anyways, good luck and thanks, you make me learn a lot with your problem ^^

Manual assignments

Last thing, but I don’t know if it’s recommended AT ALL on Qubes, here are the Xen commands useful to manage PCI-PT devices, use at your own discretion ! Except the 1st command which is totally harmless. Don’t worry though, nothing survives a dom0 reboot.
If trying this, first remove the device from the Qubes manager before launching the domU, as Qubes scripts handle that usually.

# list all devices available for PT
xl pci-assignable-list

# 1. make a device assignable
xl pci-assignable-add [OPTION] DEVICE_BDF
# 2. start the domU via Qubes manager
# 3. hotplug the device to the domU
xl pci-attach [OPTION] DOMID DEVICE_BDF
# or if wanna try permissive mode
xl pci-attach [OPTION] DOMID 'DEVICE_BDF,permissive=1'
# hotUNplug the device from the domU (care with -f, your domU may crash)
xl pci-detach [OPTION] [-f] DOMID DEVICE_BDF
# 4. remove assignable status (care with -r, it reassigns the device to dom0)
xl pci-assignable-remove [OPTION] [-r] DEVICE_BDF

Square brackets used for optional switches, caps used for replacements.
To get a DOMID, do xl list or xl domid DOMAIN_NAME
[OPTION] means you can use xl global options like -v for verbose, and -N for a dry-run (dont do anything, just output what would be done, if possible).
You shouldn’t have to use -f and -r, especially on Qubes for -r, but then you may have to reboot to do more tests. If you want more info, man xl explains what the switches do, and more about the above commands.

Offtopic

That’s freaking good to know, thanks !!! It’s also a nice pointer to help me solve the remarks you quoted ! More work to do ^^ In fact, that may be of GREAT help to make nested Qubes working, as I just discovered about virtual IOMMUs, but as you said, off topic

enmus · February 28, 2022, 9:14am

Actually between last 2 posts (mine and yours), I tried too to dynamically prepare the device following Xen passthrough guide. but with the same effect - boom happened with unbind, hahah. Command about listing assignable devices works and it shows only wifi is assignable?
I couldn’t do it manually since hide commands doesn’t seem to exist in Qubes, but will thoroughly read your respond and will be back with the details. Just wanted to say thanks for your time and good will to help.

zithro · February 28, 2022, 6:08pm

I’ll wait for your tests, but quickly, “need more info” ^^

Which method ?

Dunno what you mean exactly (but boom not fun ^^).

Yeah I don’t know if it’s supported, and my passthrough tests are quite old as I never had problems since, but IIRC I’ve found it easier (read less error prone) to bind to pciback early. I had more problems using the usual recommended method to bind devices via modprobe, and use the kernel way since (well in fact I use a rather unusual method since, but I don’t know how to make it work on Qubes/fedora yet).
But in your case it may render your system unbootable, don’t know.

enmus · February 28, 2022, 6:56pm

Both methods crash my Qubes (“boom”). No detected consequences after 3 crashes so far, but am worried to damage Qubes irrecoverably.

Testing the rest and be back.

It is indicative that beside simple salt script that creates sys-audio qube (which could be done manually anyway like for any other PCI-device-to-assign-to-a-HVM or any other kind of qube, if I’m not wrong) we don’t hear or read a word from developers. It’s just like they left it to community to resolve it and then to contrib it. I’d at least expect some feedback, just like info about hardware requirements for Qubes exists (“you need Vt-x, Vt-d, etc, etc in order to get a fully functional Qubes”)…

zithro · February 28, 2022, 8:06pm

You never know, disks are picky (SSD/HDD), and SATA cables too. I know because one of my Xen setup failed on me recently. That’s why I prefer ZFS than ext4/UFS/LVM, it’s more reliable in my experience.
Anyways, time for a backup ! From what I read, Qubes made that quite easy, and since at least 2015 ^^
PS: when I said unbootable I rather meant a boot error, like during kernel/system init, not a disk/system failure, dunno if I’m clear.

Maybe it works for them and it’s just your (hardware) config which makes it hard on you. Also, isn’t that a good collaboration model ? ^^ They provide things and in exchange you provide feedback !

Slightly off topic about documentation

TBH, I’ve not read ALL the Qubes docs, and I quickly read HW requirements, but I find it quite dense already, and very well written ! Maintaining up-to-date documentation WHILE developping and debugging new features is a time-consuming thing.
By the way, you can contribute to it ! I started. Not only you will make the project better, but you will alleviate some burden from the Qubes team, allowing them to focus on other things. Also, you may learn things just by contributing.
For example, when I started Xen (4+ years ago), I only had the kind of outdated Xen wiki, and had to do WAY MORE external research than i had to do to make Qubes work. Except for the problems related to the way I’m testing it (nested), I don’t really need to do external searches to the net to find answers, the docs and the forum are enough. But I admit I have MANY (new) features to try yet.

enmus · February 28, 2022, 10:33pm

Well it looks we have a winner!

God knows why (maybe I forgot to sudo grub2-mkconfig -o /boot/grub2/grub.cfg?), but this time adding

... quiet xen-pciback.hide=(00:1b.0)

to GRUB_CMDLINE_LINUX worked, so

$ sudo xl pci-assignable-list

finally produced

0000:00_1b.0

I started sys-audio then and it was up! Still, I was notified with some error messages, yet to discover what they are, and to check if audio actually works.

For now, @zithro I wanted to thank you for supporting me not to give up on Xen PCI passtrough guide, which was crucial, regardless the fact that I used it before you wrote about it. Your message inspired me doing it again and again to check both static and dynamic methods. So, I am indeed happy to flag your first non-self solution at the forum!

If you are interested to investigate further to a fully functional sys-audio, please let me know.

zithro · March 1, 2022, 12:52am

Glad that it worked !

Ahah, and thanks ^^

Well, before getting audio to work, the path will be long for me ! As you know my Qubes is a domU, which implies I’d have to passthrough an emulated virtual device to sys-audio, and I tried with NICs it would not lemme …
First I need networking ! I found a hack-ish solution to make it work, but have yet to test it.
In the meantime, I lost time trying to install the latest Kali as a domU, because of a bug in the kernel for which Debian proposed a patch 8 months ago, really ffs … Hopefully there’s a workaround. I’m wondering how people can install it on Qubes w/o this error.

Anyways, the error you got is pure Qubes, and for that you’ll need to ask someone else, I’m too noob for now !

enmus · March 1, 2022, 1:09am

I wish I could help you as you helped me.
Let us know about the progress.

Enable qubes-templates-community-testing repository, and run qvm-template-gui. Kali template provided there, installing and working out of the box.

zithro · March 1, 2022, 1:45am

Well, install a Qubes HVM from running Qubes ^^

I will, still have to split the topic and repost the “why nested” thing.
But to progress in Qubes and learn more, I’m first reading+converting the “Architecture” document to markdown.

Ah, good thing ! I’ll keep that info thanks.
Cause users shouldn’t try to install it from provided ISO on a blank HVM w/o the workaround, but off-topic >.< ^^

zithro · March 1, 2022, 1:50am

I may have an idea ^^

May I ask you if that’s the way it’s done in Qubes ? I mean, on which Qubes document(s) would I find this information ?

enmus · March 1, 2022, 1:59am

None, as far as I know.

A minute ago I discovered that mds=full,nosmt had to be added to grub for those having mds vulnerability which could cause data leak. Also, nowhere documented as far as I know.

enmus · March 1, 2022, 2:01am

The only place I am aware this is documented is

as well as on Github, but not familiar with it.

deeplow · March 4, 2022, 2:48pm

6 posts were split to a new topic: Smt isn’t off by default although smt=off is set in the grub

enmus · March 3, 2022, 6:27pm

Back to the topic.

At the end, after adjusting /etc/qubes/policy.d/50-sys-audio.policy for the errors, I got fully functional sys-audio qube.

Some things are (still) questionable, but at least it’s working flawlessly. I am so happy!

For the Linux based qubes, though. Cannot make it at the moment to work with Win7 qube(s) - templates and standalones.