Using dGPU without complete "Gaming" HVM?

How can I passtrough my GPU to an AppVM without having to run an xorg session over that GPU? Tried some stuff but unfortunately to no success, the GPU is never usable mostly tested with amdgpu_top. I do have a working HVM (runs over an extra monitor tho).

I want to keep the qubes gui session thing package ( bc of clipboard support)