I had a similar issue and disabling power saving for PulseAudio (in dom0) worked for me.
Comment out the following in /etc/pulse/default.pa:
# load-module module-suspend-on-idle
Restart PulseAudio (by killing it; should restart automatically):
pulseaudio --kill