After some more trial and error I was able to figure out how it works. For me, the setup works with proprietary nvidia drivers.
- After your guest is starting and detecting dGPU, you still need to configure Xorg. To do that, run
X :1 -configureas root. - Place generated xorg config into
/etc/X11/xorg.conf.d/. You may or may not need to tweak the configuration. - Restart display manager or just reboot.