I have used a different approach.
First, blacklisted nvidia in the template in /etc/modprobe.d
- blacklist nouveau
- blacklist nvidia_drm
- install nvidia_drm /bin/true
- blacklist nvidia_modeset
- install nvidia_modeset /bin/true
also, permanently disabled nvidia-persistenced service in the template.
this way the module is not used by any services other than us.
Added nvidia to /rw/config/suspend-module-blacklist of the appvm.
Now I can suspend and resume (given that the nvidia driver is idle, i.e., we are not running a job on it, at the time of suspend)