The correct output from dom0 was actually this
Screen 0: minimum 320 x 200, current 5120 x 2640, maximum 16384 x 16384
so the correct number is actually 52800 and even with inputting
qvm-features dom0 gui-videoram-min 52800
qvm-features dom0 gui-videoram-overhead 0
It still doesnt work at all no matter what orientation the screen is in
^ the above is a lie. Just needed to reboot apparently. When I plugged the third monitor in this morning it worked as inteded in both orientations. thank you so much @ChrisA