You’re certain you’re not using the intel driver?
I tried using this driver for a time instead of modesetting but I found it would cause memory spiking in the dom0 Xorg process and perceptible lag for some window manager actions. Rapidly switching among workspaces could cause Xorg to spike from ~5% of %MEM to >70% of %MEM as reported by top. With the modesetting driver it’s a pretty constant ~5% regardless of what I do.