Debian-minimal template for sys-audio

augsch · January 14, 2023, 2:42am

When you remove pulseaudio, there are some packages depending on it that will also be removed. I didn’t mention all of them, because I’m not sure which package is the vital one. It’s easy for you to know and copy the list of packages to be removed when running apt remove xxx, then key-in apt install and paste that list.

It depends on how you run this command. If you manage the qube through Qubes Manager’s open a console in the qube, you will be able to login as root and ignore that sudo. There are certainly other ways that require a sudo to work. You have to distill the crucial information, and adapt those insignificant part to your own use case.

SteveC · January 21, 2023, 8:24am

Ok…so I went into my template, did a sudo apt remove on pulseaudio. I then did a sudo apt install on all of the packages.

I shut down the template and started pavucontrol in sys-audio. It showed the output device as “Dummy output.”

OK, so that didn’t work.

I repeated the above process, only in the template xterm I did “su” to become root and did not do sudo; I just did apt remove and apt install. After shutting down the template and restarting pavucontrol in sys-audio…same result, output is “Dummy Output.”

Trying to “open a console in the qube” in Qube Manager simply starts up a disposable which shows me a blank terminal with no prompt. It’s not running on the template. (So what is it good for?)

In any case, I’m stuck. Please instead of expecting me to play a guessing game as to how to reinstall the thing, just tell me the command(s) you used that would install the thing not as root.

enmus · January 21, 2023, 5:26pm

Just to clarify. You did this in sys-audio’s template and sys-audio was shut down while template has been manipulating with?

This means actually, restarting sys-audio after template shutdown? Rebooting Qubes doesn’t help?

SteveC · January 22, 2023, 9:21am

In both cases I shut down sys audio, I did not reboot the entire system. And of course I shut down the template before restarting sys audio.

Since sys-audio is what is interfacing with the device (albeit doing so poorly), that should be sufficient.

Of course I had to reboot to listen to audio again, since the audio device won’t go back to being controlled by dom0 once it has been pulled away from dom0.

enmus · January 22, 2023, 10:14am

So strange… I’d suggest you to open separate topic for your issue and to start over from the very beginning, with the system rebooted without sys-audio existing.

augsch · February 17, 2023, 3:39pm

Just for clarification, I’m to update the guide posted in post #2. And because I cannot edit that post, I’ll post it here.
Creation of sys-audio:
In dom0:

Copy your debian-11-minimal to name deb11-audio.

In deb11-audio:

Install qubes-usb-proxy qubes-input-proxy-sender alsa-utils alsa-oss alsa-ucm-conf alsa-firmware-loaders without --no-install-recommends.
Install pavucontrol pulseaudio qubes-core-admin-client qubes-gui-daemon-pulseaudio pulseaudio-qubes with --no-install-recommends.
3.1. Remove pulseaudio through apt. Note that some other packages are also removed. Copy the list of removed packages in the terminal.
3.2 Install those removed packages whose names were copied, including pulseaudio.
Create a file at /etc/modprobe.d/, name it 50-snd.conf, and put options snd_hda_intel power_save=0 in it. Shutdown the template.

In dom0:

Create a appvm named sys-audio based on deb11-audio. Assign your PCI audio devices to it. Configure them to be no-strict-reset.
Create 50-sys-audio.policy in dom0 at /etc/qubes/policy.d/, enter those policy lines at #2.
In Qubes-Settings of sys-audio, turn to “Services” page, Add a (custom…) service called “audiovm”.
Run qubes-prefs default_audiovm sys-audio. Boot up sys-audio.

At this point, you have a working sys-audio. Any newly started qube is using sys-audio as its audio sink. You can start a new disposable and play music in it.

To adjust volume:
Run pavucontrol in a terminal in sys-audio. You’ll have a graphical interface to do adjust the volume.

To enable bluetooth:
See @dom0’s detailed guide.
Please note that if you are using a wireless card that also provides bluetooth, then the bluetooth device might appear in the Device Widget as a strange string of numbers, for example, 8087_2345.

To enable shortcut adjusting volume:
Add a shortcut based on your desktop environment (Xfce, KDE, etc.) to run the command qvm-run sys-audio 'amixer sset Master 2%+', which will turn up the volume.
Add a shortcut based on your desktop environment (Xfce, KDE, etc.) to run the command qvm-run sys-audio 'amixer sset Master 2%-', which will turn down the volume.
Add a shortcut based on your desktop environment (Xfce, KDE, etc.) to run the command qvm-run sys-audio 'amixer sset Master toggle', which will toggle the audio between muted and unmuted.

The shortcut can be the default Fn+F1/2/3 combinations. For KDE, no further config is required. For Xfce4, you might have to disable some audio related services in dom0, in order that your shortcut is correctly passed to sys-audio. That part is beyond this post and will not be discussed.

SteveC · February 17, 2023, 5:35pm

Your instructions read to copy debian-11-minimal to deb11-audio, then install to deb11-minimal. Typo?

On a related note; I’ll be able to try this on a different system (as well as the one I had trouble with) now as I am converting my Librem-13v3 to Qubes.

augsch · February 18, 2023, 1:37am

Yes, thank you.

dom0 · February 18, 2023, 6:33am

If you configure the shortcuts to run a script instead of a single command, you can create scripts that will adjust the volume and also display the current setting as a notification.

Here is a script for increasing the volume and displaying the current volume after the adjustment has been made:

#!/bin/bash

qvm-run --pass-io sys-audio "amixer sset Master 2%+"

VOLUME=$(qvm-run --pass-io sys-audio "amixer sget Master | awk -F 'Left:|[][]' '{ print \$3 }'")

notify-send -t 1 ${VOLUME}

To make the script for decreasing the volume, just copy the script above and change 2%+ to 2%-. You can of course change the 2 to the increment of your choice.

Here is the script for toggling between audio off (mute) and audio on (unmute) and then displaying the current state:

#!/bin/bash

qvm-run --pass-io sys-audio "amixer sset Master toggle"

STATE=$(qvm-run --pass-io sys-audio "amixer sget Master | awk -F 'Left:|[][]' '{ print \$5 }'")

notify-send -t 1 ${STATE}

Note that it will only display off or on to show whether audio is off (muted) or on (unmuted).

SteveC · February 19, 2023, 2:26am

@dom0 notify-send doesn’t appear to be part of the minimal template, by the way.

EDIT. Never mind, your scripts are meant to be in dom0.

SteveC · February 19, 2023, 2:31am

Step 3: You’re installing pulseaudio (plus an assortment of other packages).
Step 3.1: You’re removing pulseaudio [which removes a group of things].
Step 3.2: You’re reinstalling everything removed in step 3.1

How does this make sense? Is there something different about the way one does the (re) install in step 3.2?

augsch · February 19, 2023, 3:02am

I’m not skilled enough to tell between. I guess it’s some user group problems, because without this step, it seems that pulseaudio and alsa are running as root, so no userspace programs (qubes-daemon, pavucontrol) will be able to communicate with them.

SteveC · February 19, 2023, 7:44am

What I’m still struggling with here is that the only way I can install is with sudo apt. Plain old apt will simply fail. Thus I have to install as root. So I’m back to wondering what difference it could make, and if it somehow does, it sounds like a bug in apt to me, to be honest.

augsch · February 19, 2023, 8:06am

Telling people how to install software via apt is also beyond this thread.

Edit: I fail to get what are you saying. I give up.

SteveC · February 20, 2023, 2:30am

plain apt?

Is there some other kind of apt?

I’m not struggling to run apt, I’m struggling to understand YOU. I keep getting hints that there’s something different about this second install, and you keep just laughing at me when I ask what it is. Now you’re hinting I can’t run apt?

Honestly…I wonder if I even have any understanding of what you’re trying to tell me here.

I haven’t even tried to do this yet, because until now I was under the impression you were doing something “special” on the second install that you weren’t telling people.

So you’re literally running apt the same way both times?

dom0 · February 25, 2023, 9:21am

I was able to get my disposable sys-audio to work with Bluetooth earbuds using the USB Bluetooth 4.0 adapter available at ThinkPenguin:

https://thinkpenguin.com/gnu-linux/penguin-usb-bluetooth-40-micro-adapter-tpe-usbbluv4

This USB adapter does not require any non-free firmware.

The necessary packages for Bluetooth:

blueman
pulseaudio-module-bluetooth

The install commands for my Debian-minimal sys-audio template:

apt-get install alsa-firmware-loaders alsa-oss alsa-ucm-conf alsa-utils qubes-input-proxy-sender qubes-usb-proxy -y

apt-get install --no-install-recommends blueman pavucontrol pulseaudio pulseaudio-module-bluetooth pulseaudio-qubes qubes-audio-daemon qubes-core-admin-client qubes-gui-daemon-pulseaudio socat -y

Here are the steps to connect a Bluetooth wireless device to sys-audio using GUI:

Make sure that sys-audio and sys-usb are running by using the Qui-domains application in the Notification Area of the Panel, and if not, start them by using Launcher items in the Panel or by using the Qube Manager.
Attach the Bluetooth adapter to sys-audio using the Qui-devices application in the Notification Area of the Panel.
Start Blueman Bluetooth Manager in sys-audio using a Launcher item in the Panel.
On the window that appears with the prompt Shall bluetooth get enabled automatically?, select Yes. If the window does not appear after several seconds, you may need to click the Launcher to start the Blueman Bluetooth Manager again. If after several seconds, the window still does not start, you may need to detach the Bluetooth adapter and reattach it to sys-audio and then start the Blueman Bluetooth Manager again.
Select the Search button in the toolbar of the Blueman Bluetooth Manager window in order to scan for Bluetooth wireless devices.
Select your Bluetooth wireless device in the list of available devices, and then select the yellow four-pointed star button in the toolbar in order to mark the device as trusted.
Select the key button next to the four-pointed star button in the toolbar in order to create pairing with the device. Once the initial pairing has completed, the device will connect for a moment and then disconnect. When connected, there will be three colorful icons on the right side of the device item (orange, green, and blue), and when disconnected, the icons will disappear.
Select the Device button in the menu and select the option to Connect, or you can right-click (secondary-click) the Bluetooth device and select the option to Connect.
Once connected, you can select the Blueman-Applet application in the Notification Area of the Panel, move the cursor to the Recent Connections... option, and then select the available option Auto connect profiles on <DEVICE> so that the Bluetooth wireless device will automatically connect in the future. If this feature does not work to auto connect, you can simply click the Auto connect profiles on <DEVICE> option to manually connect, which is quicker and more convenient than connecting manually as described in step 8. You can close the Blueman Bluetooth Manager window now. The Blueman-Applet application can be used to turn Bluetooth on/off and can be used to connect manually. If you want to manually disconnect without turning Bluetooth off, you can select the Devices... option to open the Blueman Bluetooth Manager window again.

With a disposable sys-audio, these steps must be done for each new session, but this is a relatively quick and easy process.

I may make some more attempts to get the Librem 14 built-in Qualcomm Atheros AR9462 Bluetooth to work in the future, but this setup that I have outlined above is preferable to me, since there is no requirement for non-free firmware.

I wish everyone luck with your efforts to configure sys-audio for your needs.

dom0 · March 2, 2023, 6:52pm

I would like to confirm that Bluetooth audio is unaffected by the issue that I documented here and here of this post.

This makes me think that the issue has something to do with the hardware and possibly firmware of the Librem 14 built-in speakers and headphone jack. But the issue only presents itself when using sys-audio, which I do not understand.

The good news is that I only need to start my sys-audio qube once if I only intend to use Bluetooth audio, while allowing me the option to enable the built-in speakers later by restarting sys-audio twice.

dom0 · March 30, 2023, 10:06pm

The issue affecting my sys-audio qube has been fixed by Purism developer Jonathon Hall with the releases of Librem-EC 1.13 and coreboot 4.19-Purism-1/PureBoot 25:

Isolator · June 19, 2023, 6:39pm

Gods, this thread is such an absolute useless mess it’s not even funny - with such “guides” it is better to just throw stuff at random and hope that it sticks lol. It’s not like authors of such “guides” do anything different anyway…

Ok.
As I need to somewhat structure my sys-audio odyssey I will split my post into three parts:

My torturous journey of wading through the threads here in search for the crumbs of useful information. And my problems with the so-called “solutions” given in this particular thread.
The steps I took to set up a disposable vm that can use both debian-minimal and fedora-minimal templates.
The bugs I have encountered in the process or the story of how I ultimately failed.

First, I need to mention that even a cursory look at all the threads here would tell you that you should keep from this “sys-audio” thing as far away as possible for a foreseeable future. And this is exactly what I had intended to keep doing… until I found out that the random freezes I keep experiencing might be somehow related to the audio stack
So the obvious solution would be to isolate the audio stack in a vm, right? Right, if only things were that simple…

The threads.
Absolute mess. Don’t even bother.
This thread exemplifies it more than any other. We have two wonderful “solutions” here:

remove and reinstall the same package with the same deps and expect a different result …and
the kernel initializes the sound card only on the n-th reboot, so… just reboot n times! here’s even a helpful script! (genius lol)
This is the level we are talking about here…

The ONLY coherent guide I could find is that of neowutran’s, but he uses fedora-minimal for sys-audio, while I strongly prefer debian for my service vms. But this soon got proven to be irrelevant as the same issues got reproduced on both templates.
Nevertheless I used neowutran’s guide as my baseline for a disposable debian-minimal sys-audio vm. Here are my own notes documenting the process:

Create dvm disposable template based on debian-xx-min-clone-audio by enabling Disposable template option
Add restricted tag to the dvm disposable template

qvm-tags <disposable-Template-Name> add restricted

Create a named disposable qube sys-audio based on dvm disposable template

qvm-create --template <disposable-Template-Name> --class DispVM --label red sys-audio

In sys-audio qube settings Services tab, add the audiovm service
Change sys-audio virtualization mode to HVM
Change sys-audio memory to 1000 MB and uncheck Include in memory balancing
Change sys-audio Default disposable template to none
Passthrough the PCI device 00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS to sys-audio and Configure strict reset for PCI devices
In dom0 execute the following command (change sys-audio to dom0 to revert back)

qubes-prefs default_audiovm sys-audio

Configure policy in dom0 by creating the file /etc/qubes/policy.d/50-sys-audio.policy with the following content

admin.Events   *                         sys-audio   @adminvm                 allow   target=dom0
admin.Events   +domain-stopped           sys-audio   @tag:audiovm-sys-audio   allow   target=dom0
admin.Events   +domain-shutdown          sys-audio   @tag:audiovm-sys-audio   allow   target=dom0
admin.Events   +domain-start             sys-audio   @tag:audiovm-sys-audio   allow   target=dom0
admin.Events   +connection-established   sys-audio   @tag:audiovm-sys-audio   allow   target=dom0

admin.vm.CurrentState   *   sys-audio   @adminvm                 allow   target=dom0
admin.vm.CurrentState   *   sys-audio   @tag:audiovm-sys-audio   allow   target=dom0
admin.vm.List           *   sys-audio   @adminvm                 allow   target=dom0
admin.vm.List           *   sys-audio   @tag:audiovm-sys-audio   allow   target=dom0

admin.vm.property.Get                +audiovm       sys-audio   @tag:audiovm-sys-audio   allow   target=dom0
admin.vm.property.Get                +xid           sys-audio   @tag:audiovm-sys-audio   allow   target=dom0
admin.vm.feature.CheckWithTemplate   +audio         sys-audio   @tag:audiovm-sys-audio   allow   target=dom0
admin.vm.feature.CheckWithTemplate   +audio-model   sys-audio   @tag:audiovm-sys-audio   allow   target=dom0
admin.vm.property.Get                +stubdom_xid   sys-audio   @tag:audiovm-sys-audio   allow   target=dom0

admin.vm.property.Get      +keyboard_layout   sys-audio   @tag:audiovm-sys-audio   allow   target=dom0
admin.vm.property.GetAll   *                  sys-audio   @tag:audiovm-sys-audio   allow   target=dom0

Further elaboration:
Point 2, ie adding a restricted tag is irrelevant for this discussion - it is used in my rpc policies.

As explained in neowutran’s guide, adding audiovm service will launch at startup the following command: qvm-start-daemon --all --watch

PCI device 00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS is machine specific

debian-xx-min-clone-audio is an updated debian-minimal tempalte with the following installed packages:

Utils
- alsa-utils
- pciutils // for lspci command
- ~~pulseaudio-utils~~ // gets installed by the other package
Qubes specific
- pulseaudio-qubes
- ~~qubes-audio-daemon~~ // gets installed by the qubes-gui-daemon-pulseaudio
- qubes-gui-daemon-pulseaudio
- qubes-core-admin-client
Misc
- pasystray // volume tray icon
- pavucontrol
- alsa-oss
- alsa-firmware-loaders
- ~~alsa-ucm-conf~~ // gets installed by the other package
For mic
- socat

In the debian-xx-min-clone-audio template also:

Create the file /etc/modprobe.d/50-snd.conf with the following content

options snd_hda_intel enable=1 index=0 power_save=0

enable=1 needed to automatically switch to your Built-in Audio Analog Stereo output device when (or more like “if”) your soundcard gets initialized, otherwise it will use Qubes VCHAN sink, which is relevant in a disposable environment
power_save=0 is apparently there to help with a scratchy sound

Regarding policies, neowutran has the most concise list which makes me believe that he somehow understood what he was doing. I only needed to add two last lines because I was getting errors.
The policies provided in this thread are excessive - every vm with a default audiovm value gets assigned an audiovm-sys-audio tag when you execute qubes-prefs default_audiovm sys-audio

For a fedora-minimal template I installed packages as outlined in the neowutran’s guide.

This pretty much covers the setup. In the next post I will cover my experiences of actually trying to run it.

Isolator · June 19, 2023, 7:51pm

Ok. So I will structure this post by the issues I have expirienced. But first I think it would be better to start on a good note as a baseline - I actually got debian based sys-audio running and working. Once.
Here are the screenshots of what you should see in your working sys-audio pavucontrol panel with misc here being another vm.

Playback
Recording

If your soundcard doesn’t get initialized - you wouldn’t see anything related to your Built-in Audio here. You can test your audio from inside the sys-audio by running the following command aplay /usr/share/sounds/alsa/Noise.wav

So that’s all for good news. The issues are as follows:

Issue 1 - Inability to shutdown sys-audio when you make it a default_audiovm as a vm can’t be shutdown when it is an audiovm of the other running vms. And in the default config at minimum sys-audio becomes an audiovm of sys-audio, ie you can’t shutdown it without first changing its audiovm. Which you can do with the command qvm-prefs sys-audio audiovm dom0. Also you can’t leave audiovm field empty because then pulseaudio server wouldn’t be starting.
So I dunno of what use is that restarting script itt when the author forgot to mention this CRUCIAL detail.

Issue 2 - Random hung ups at startup with the error Qube sys-audio has failed to start: Cannot connect to qrexec agent for 60 seconds, see /var/log/xen/console/guest-sys-audio.log for details

/var/log/xen/console/guest-sys-audio.log has the following messages related to audio before it shuts down due to timeout

[2023-06-19 18:41:05] [   56.275679] watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kworker/1:3:251]
[2023-06-19 18:41:05] [   56.275703] Modules linked in: snd_soc_sst_dsp snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core crct10dif_pclmul crc32_pclmul snd_compress crc32c_intel ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_hwdep ghash_clmulni_intel snd_seq drm_vram_helper drm_ttm_helper snd_seq_device snd_pcm ttm serio_raw pcspkr drm_kms_helper i2c_piix4 floppy ehci_pci snd_timer cec snd ehci_hcd soundcore ata_generic pata_acpi xen_scsiback target_core_mod xen_netback xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop drm bpf_preload overlay xen_blkfront
[2023-06-19 18:41:05] [   56.275889] CPU: 1 PID: 251 Comm: kworker/1:3 Tainted: G             L    5.15.103-1.qubes.fc32.x86_64 #1
[2023-06-19 18:41:05] [   56.275913] Hardware name: Xen HVM domU, BIOS 4.14.5 03/21/2023
[2023-06-19 18:41:05] [   56.275930] Workqueue: events azx_probe_work [snd_hda_intel]
[2023-06-19 18:41:05] [   56.275950] RIP: 0010:snd_hdac_bus_send_cmd+0xac/0xd0 [snd_hda_core]
[2023-06-19 18:41:05] [   56.275973] Code: 28 a8 ec 66 90 fb 66 0f 1f 44 00 00 31 c0 5b 5d 41 5c 41 5d c3 cc cc cc cc 4c 89 ef e8 a1 28 a8 ec 66 90 fb 66 0f 1f 44 00 00 <b8> f5 ff ff ff eb dd 4c 89 ef e8 89 28 a8 ec 66 90 fb 66 0f 1f 44
[2023-06-19 18:41:05] [   56.276018] RSP: 0018:ffff9b01004abc18 EFLAGS: 00000246
[2023-06-19 18:41:05] [   56.276033] RAX: 0000000000000001 RBX: ffff8d9b09c80028 RCX: ffff8d9b09c80038
[2023-06-19 18:41:05] [   56.276053] RDX: 000000000000001f RSI: 00000000405f0005 RDI: ffff8d9b09c804c0
[2023-06-19 18:41:05] [   56.276072] RBP: 00000000405f0005 R08: 0000000000000000 R09: ffff8d9b09c80428
[2023-06-19 18:41:05] [   56.276092] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000004
[2023-06-19 18:41:05] [   56.276112] R13: ffff8d9b09c804c0 R14: ffff8d9b09c80028 R15: ffff8d9b09c804c8
[2023-06-19 18:41:05] [   56.276131] FS:  0000000000000000(0000) GS:ffff8d9b79700000(0000) knlGS:0000000000000000
[2023-06-19 18:41:05] [   56.276152] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2023-06-19 18:41:05] [   56.276168] CR2: 00007c8f3983bd37 CR3: 00000000013bc003 CR4: 00000000003706e0
[2023-06-19 18:41:05] [   56.276189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2023-06-19 18:41:05] [   56.276208] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2023-06-19 18:41:05] [   56.276228] Call Trace:
[2023-06-19 18:41:05] [   56.276236]  <TASK>
[2023-06-19 18:41:05] [   56.276244]  snd_hdac_bus_exec_verb_unlocked+0x7d/0x170 [snd_hda_core]
[2023-06-19 18:41:05] [   56.276264]  snd_hdac_bus_exec_verb+0x3a/0x60 [snd_hda_core]
[2023-06-19 18:41:05] [   56.276283]  hda_reg_read+0x1c8/0x260 [snd_hda_core]
[2023-06-19 18:41:05] [   56.276301]  __snd_hdac_regmap_read_raw+0x86/0xf0 [snd_hda_core]
[2023-06-19 18:41:05] [   56.276321]  setup_fg_nodes+0xc5/0x130 [snd_hda_core]
[2023-06-19 18:41:05] [   56.276338]  snd_hdac_device_init+0x18c/0x360 [snd_hda_core]
[2023-06-19 18:41:05] [   56.276358]  snd_hda_codec_device_init+0xc8/0x150 [snd_hda_codec]
[2023-06-19 18:41:05] [   56.276381]  snd_hda_codec_new+0x1c/0x50 [snd_hda_codec]
[2023-06-19 18:41:05] [   56.276399]  azx_probe_codecs+0xe3/0x150 [snd_hda_codec]
[2023-06-19 18:41:05] [   56.276419]  azx_probe_continue+0x358/0x3c0 [snd_hda_intel]
[2023-06-19 18:41:05] [   56.276435]  process_one_work+0x1ee/0x390
[2023-06-19 18:41:05] [   56.276447]  worker_thread+0x4c/0x320
[2023-06-19 18:41:05] [   56.276458]  ? process_one_work+0x390/0x390
[2023-06-19 18:41:05] [   56.276470]  kthread+0x124/0x150
[2023-06-19 18:41:05] [   56.276481]  ? set_kthread_struct+0x50/0x50
[2023-06-19 18:41:05] [   56.276493]  ret_from_fork+0x1f/0x30
[2023-06-19 18:41:05] [   56.276506]  </TASK>
[2023-06-19 18:41:06] .M
.[K[ .[0;31m*.[0;1;31m*.[0m.[0;31m*  .[0m] Job qubes-mount-dirs.service/start running (53s / no limit)
[2023-06-19 18:41:06] .M
.[K[  .[0;31m*.[0;1;31m*.[0m.[0;31m* .[0m] Job qubes-mount-dirs.service/start running (54s / no limit)
[2023-06-19 18:41:07] .M
.[K[   .[0;31m*.[0;1;31m*.[0m.[0;31m*.[0m] Job qubes-mount-dirs.service/start running (54s / no limit)
[2023-06-19 18:41:07] .M
.[K[    .[0;31m*.[0;1;31m*.[0m] Job qubes-mount-dirs.service/start running (55s / no limit)
[2023-06-19 18:41:08] .M
.[K[     .[0;31m*.[0m] Job qubes-mount-dirs.service/start running (55s / no limit)
[2023-06-19 18:41:08] .M
.[K[    .[0;31m*.[0;1;31m*.[0m] Job qubes-mount-dirs.service/start running (56s / no limit)

And /var/log/xen/console/guest-sys-audio-dm.log has the following:

[2023-06-19 18:40:08] wrote 34 bytes to vchan
[2023-06-19 18:40:08] from-vchan: {"execute":"device_add","id":2020372736,"arguments":{"driver":"xen-pci-passthrough","id":"pci-pt-00_1f.3","hostaddr":"0000:00:1f.3"}}

[2023-06-19 18:40:08] 
[2023-06-19 18:40:08] [00:06.0] xen_pt_realize: Assigning real physical device 00:1f.3 to devfn 0x30
[2023-06-19 18:40:08] [00:06.0] xen_pt_register_regions: IO region 0 registered (size=0x00004000 base_addr=0xa1330000 type: 0x4)
[2023-06-19 18:40:08] [00:06.0] xen_pt_register_regions: IO region 4 registered (size=0x00100000 base_addr=0xa1000000 type: 0x4)
[2023-06-19 18:40:08] [00:06.0] xen_pt_config_reg_init: Offset 0x000e mismatch! Emulated=0x0080, host=0x0000, syncing to 0x0000.
[2023-06-19 18:40:08] [00:06.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! Emulated=0x0000, host=0xa1330004, syncing to 0xa1330004.
[2023-06-19 18:40:08] [00:06.0] xen_pt_config_reg_init: Offset 0x0020 mismatch! Emulated=0x0000, host=0xa1000004, syncing to 0xa1000004.
[2023-06-19 18:40:08] [00:06.0] xen_pt_config_reg_init: Offset 0x0052 mismatch! Emulated=0x0000, host=0x0043, syncing to 0x0003.
[2023-06-19 18:40:08] [00:06.0] xen_pt_config_reg_init: Offset 0x0062 mismatch! Emulated=0x0000, host=0x0080, syncing to 0x0080.
[2023-06-19 18:40:08] [00:06.0] xen_pt_pci_intx: intx=1
[2023-06-19 18:40:08] [00:06.0] xen_pt_realize: Real physical device 00:1f.3 registered successfully
[2023-06-19 18:40:08] from-unix: {"return": {}, "id": 2020372736}

[2023-06-19 18:40:08] 
[2023-06-19 18:40:08] wrote 34 bytes to vchan
[2023-06-19 18:40:08] from-vchan: {"execute":"query-pci","id":2020372738}

[2023-06-19 18:40:08] 
[2023-06-19 18:40:08] from-unix: {"return": [{"bus": 0, "devices": [{"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 0, "class_info": {"class": 1536, "desc": "Host bridge"}, "id": {"device": 4663, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 1537, "desc": "ISA bridge"}, "id": {"device": 28672, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 257, "desc": "IDE controller"}, "id": {"device": 28688, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 1, "regions": [{"bar": 4, "size": 16, "address": -1, "type": "io"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 1, "class_info": {"class": 1664, "desc": "Bridge"}, "id": {"device": 28947, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 3, "regions": []}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 2, "class_info": {"class": 65408}, "id": {"device": 1, "subsystem-vendor": 22611, "vendor": 22611, "subsystem": 1}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": true, "mem_type_64": false, "bar": 1, "size": 16777216, "address": -1, "type": "memory"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 3, "class_info": {"class": 256, "desc": "SCSI controller"}, "id": {"device": 18, "subsystem-vendor": 0, "vendor": 4096, "subsystem": 4096}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": false, "mem_type_64": false, "bar": 1, "size": 1024, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 2, "size": 8192, "address": -1, "type": "memory"}]}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 4, "class_info": {"class": 768, "desc": "VGA controller"}, "id": {"device": 4369, "subsystem-vendor": 6900, "vendor": 4660, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": true, "mem_type_64": false, "bar": 0, "size": 16777216, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 2, "size": 4096, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 6, "size": 65536, "address": -1, "type": "memory"}]}, {"irq_pin": 4, "bus": 0, "qdev_id": "ehci", "irq": 0, "slot": 5, "class_info": {"class": 3075, "desc": "USB controller"}, "id": {"device": 9421, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": false, "bar": 0, "size": 4096, "address": -1, "type": "memory"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "pci-pt-00_1f.3", "irq": 0, "slot": 6, "class_info": {"class": 0}, "id": {"device": 41800, "subsystem-vendor": 0, "vendor": 32902, "subsystem": 0}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": true, "bar": 0, "size": 16384, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": true, "bar": 4, "size": 1048576, "address": -1, "type": "memory"}]}]}], "id": 2020372738}

[2023-06-19 18:40:08] 
[2023-06-19 18:40:08] wrote 2048 bytes to vchan
[2023-06-19 18:40:08] wrote 1075 bytes to vchan
[2023-06-19 18:40:08] vchan client disconnected
[2023-06-19 18:40:09] {"timestamp": {"seconds": 1687189209, "microseconds": 524449}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral-anon/device[3]"}}

[2023-06-19 18:40:09] {"timestamp": {"seconds": 1687189209, "microseconds": 524520}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral-anon/device[0]"}}

[2023-06-19 18:40:09] random: crng init done
[2023-06-19 18:40:13] [00:06.0] xen_pt_msgctrl_reg_write: setup MSI (register: 81).
[2023-06-19 18:40:13] [00:06.0] msi_msix_setup: requested pirq 151 for MSI (vec: 0x0, entry: 0x0)
[2023-06-19 18:40:13] [00:06.0] xen_pt_msi_setup: MSI mapped with pirq 151.
[2023-06-19 18:40:13] [00:06.0] msi_msix_update: Updating MSI with pirq 151 gvec 0x0 gflags 0x7097 (entry: 0x0)
[2023-06-19 18:40:13] [00:06.0] Write-back to unknown field 0x44 (partially) inhibited (0x00)
[2023-06-19 18:40:13] [00:06.0] If the device doesn't work, try enabling permissive mode
[2023-06-19 18:40:13] [00:06.0] (unsafe) and if it helps report the problem to xen-devel
[2023-06-19 18:41:08] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00

Note these lines:

[2023-06-19 18:40:13] [00:06.0] Write-back to unknown field 0x44 (partially) inhibited (0x00)
[2023-06-19 18:40:13] [00:06.0] If the device doesn't work, try enabling permissive mode
[2023-06-19 18:40:13] [00:06.0] (unsafe) and if it helps report the problem to xen-devel

Possible Xen bug?

Tho it is difficult to determine if it is related to this specific sys-audio issue or not as I’ve seen similar hang ups on ordinary vms (with not pci devices attached). Maybe it is just an irrelevant Qubes “feature” (ie bug) that just gets in there because of the sheer number of restarts. And speaking of restarts…

Issue 3 - audio card gets initialized 1 out of random number of restarts. And no, it is not 1 out of 3. More like 1 out of 10. Reproduced on both debian and fedora.
Relevant logs:

Not initialized

[    3.779104] hdaudio hdaudioC0D0: no AFG or MFG node found
[    3.779108] snd_hda_intel 0000:00:06.0: no codecs initialized

Initialized

[    3.956895] snd_hda_codec_realtek hdaudioC0D0: autoconfig for ALC892: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:line
[    3.956929] snd_hda_codec_realtek hdaudioC0D0:    speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[    3.956950] snd_hda_codec_realtek hdaudioC0D0:    hp_outs=1 (0x1b/0x0/0x0/0x0/0x0)
[    3.956973] snd_hda_codec_realtek hdaudioC0D0:    mono: mono_out=0x0
[    3.956990] snd_hda_codec_realtek hdaudioC0D0:    inputs:
[    3.957005] snd_hda_codec_realtek hdaudioC0D0:      Front Mic=0x19
[    3.957022] snd_hda_codec_realtek hdaudioC0D0:      Rear Mic=0x18
[    3.957040] snd_hda_codec_realtek hdaudioC0D0:      Line=0x1a

[    4.070581] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:06.0/sound/card0/input6
[    4.071734] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:06.0/sound/card0/input7
[    4.073122] input: HDA Intel PCH Line as /devices/pci0000:00/0000:00:06.0/sound/card0/input8
[    4.074564] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:06.0/sound/card0/input9
[    4.075419] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:06.0/sound/card0/input10

Issue 4 - Even when the card gets initialized sometimes there are no streams from the client vms. But sometimes there are. Hard to test in detail because of the Issue 3 and Issue 1.

So that’s about it. My guess is that it’s some xen bug related to pci passthrough that produces this erratic behavior. Maybe I will test this sys-audio thing again in Qubes 6.0 or something.

Debian-minimal template for sys-audio

Utils

Qubes specific

Misc

For mic