After some research, I have found a “solution”… I hesitate to call it a solution because I don’t understand what it does, honestly. If anybody can shed some light I’d be grateful!
Solution
Disable nvidia-drm modeset, by edting /etc/modprobe.d/nvidia-options.conf
to
options nvidia-drm modeset=0
After this, X works well. I have also installed Ollama and the performance of the GPU is the expected one.
EDIT: for reference, I have found this option in https://wiki.debian.org/NvidiaGraphicsDrivers#bookworm-535, where in fact they suggest the opposite, to set modeset=1 when possibile.