Great to hear. I never saw anyone except you confirming it works. How even to track this down, any idea?
What is more peculiar, after patching xen.xml even my old Win7 HVM without any device can’t be assigned with more than 2GB RAM now and it worked before with 6144MB.
check the name of your qube, and try to only use the patching stubdom method.
https://neowutran.ovh/qubes/articles/gaming_windows_hvm.html#patching-stubdom-linux-rootfs.gz
If your qube is named “gpu_test”, without any GPU passed through, after applying the stubdom-linux-rootfs.gz you can expect issues.
gpu_xxx → must have a gpu assigned
not starting with “gpu_” → work like before the patch
you could also need to patch “qemu-stubdom-linux-full-rootfs” the same way
Thanks. Yes, I have no naming misuse, was strict with that.
As far as I can remember I tried what you suggested, but eventually came up that only xen.xml patching had the effect, while patching sttubdoms (only, and altogether with xen.xml) didn’t change anything.
But I will give it another try and will give feedback here.
Best regards
This issue still exists on the latest version. I can also confirm that dmesg lists NVIDIA as “fallen off the bus and is not responding to commands” when using > 2G of memory.
It works! I can’t believe how simple and obvious your solution is. Thank you very much for this, I have spent so much time trying to figure it out.
@neowutran perhaps you can include a note in your guide for people who have a 2GB RAM limit rather than 3.5GB.
Thanks everyone for your help!
2 Likes
Can somebody explain what exactly max-ram-below-4g does? How exactly it should affect virtualization in Qubes OS?
To avoid setting this 2G, 3.5G and other numbers without understanding what is happening.
1 Like
I added a note in the guide here Create a Gaming HVM .
Does anyone known/understand/or have an idea on why some GPU require this value to exactly be “3.5G”, and some other seems to require it to be exactly “2G” ?
( https://github.com/qemu/qemu/blob/master/hw/i386/pc_piix.c#L131 ; )
2 Likes
@neowutran In your doc with the updated information, please link to my post. It was hell to figure out and I would appreciate the attribution 
1 Like
Exactly 3.5GB?? I thought it would be a range of acceptable values! 
done (& added you to the “contributor” list of this doc )
3 Likes
Some time ago, I tried different values for this parameter, it required to be exactly 3.5G to work, any other values I tried resulted in non working setup
What a mess, too complicated.
So, if I have spare 32GiB that I want to give some VM:
- how can I provide it all to the VM?
- why “2G” should work, and “3.5G” or any other number should not?
- what this 2G will mean out of this 32GiB amount that I decided to give to VM?
- How much memory will VM actually get, 32GiB? 30GiB (32-2)? Something else?
Completely unclear to me.
1 Like
I know basically nothing about these things but one thing that jumps out to me is the size of the individual RAM chips the GPU uses. For modern high end GPUs this is typically 2gb, which is exactly the size that was needed to pass through effectively. Something weird might happen when u pass through only a portion of GPU ram because its more integrated than a CPU and its RAM.
As far as I understand:
- how can I provide it all to the VM? : Just assign the memory you want through the Qubes gui
- what this 2G will mean out of this 32GiB amount that I decided to give to VM? : Irrelevant, ram is not “lost” it is just how the address space is reserved.
- How much memory will VM actually get, 32GiB? 30GiB (32-2)? Something else? Irrelevant, ram is not “lost”.
- why “2G” should work, and “3.5G” or any other number should not? : For sure, it is hardware specific. From the source code “Traditional split is 3.5G (lowmem = 0xe0000000)”, and it is what I use for my hardware. On some VFIO forum the recommanded value is “1G”. For some user here it need to be “2G”
1 Like
Sorry, still very unclear to me. If you can elaborate a bit more:
- You say “reserved”. Reserved for what? For whom and by who?
- If user used
max-ram-below-4g=3.5Gthan is 3.5GiB “reserved”, or “0.5GiB” is reserved? - Why is it
max-ram-below-4g, why 4G? Is it about int32 limit size of pointers or what? - Why this workaround with a parameter is needed at all on x64 systems?
- Do users still need to patch stubdom-linux-rootfs or downgrade xen-hvm-stubdom-linux? Or completely not and providing
max-ram-below-4gkernel parameter is enough nowadays? - If you understand the matter and architecture of this workaround, how do you think, will it be possible to make it just work (without any experiments with hardware specific values of some kernel parameter) or it is something like a legacy design flaw for us to live with for a long time?
So many questions, because I still do not understand what is happening and why is it still not properly fixed upstream in Xen, QEMU and Qubes OS to make it just work without tricks.
Out of curiosity, in your bios settings, what is the value of the parameter “Above 4G decoding” ?
If it was to “false”, could you set it to “true”, then modify the max-ram-below-4g to 3.5G again, and test if it work ?
(probably not interesting, tried to set “Above 4G decoding” to false and true. Same behavior, it work for me with 3.5G)
Unfortunately my BIOS settings are very minimal and I don’t have that parameter available to adjust.
still trying to understand the difference between our systems;
In Qubes Manager, if you go to the logs of your HVM with the gpu passthrough, in the “-dm.log” file, do you have a line that look like this one ?
from-unix: {"return": [{"bus": 0, "devices": [{"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 0, "class_info": {"class": 1536, "desc": "Host bridge"}, "id": {"device": 4663, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 1537, "desc": "ISA bridge"}, "id": {"device": 28672, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 257, "desc": "IDE controller"}, "id": {"device": 28688, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 1, "regions": [{"bar": 4, "size": 16, "address": -1, "type": "io"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 1, "class_info": {"class": 1664, "desc": "Bridge"}, "id": {"device": 28947, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 3, "regions": []}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 2, "class_info": {"class": 65408}, "id": {"device": 1, "subsystem-vendor": 22611, "vendor": 22611, "subsystem": 1}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": true, "mem_type_64": false, "bar": 1, "size": 16777216, "address": -1, "type": "memory"}]}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 3, "class_info": {"class": 768, "desc": "VGA controller"}, "id": {"device": 4369, "subsystem-vendor": 6900, "vendor": 4660, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": true, "mem_type_64": false, "bar": 0, "size": 16777216, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 2, "size": 4096, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 6, "size": 65536, "address": -1, "type": "memory"}]}, {"irq_pin": 4, "bus": 0, "qdev_id": "ehci", "irq": 0, "slot": 4, "class_info": {"class": 3075, "desc": "USB controller"}, "id": {"device": 9421, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": false, "bar": 0, "size": 4096, "address": -1, "type": "memory"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "nic0", "irq": 0, "slot": 5, "class_info": {"class": 512, "desc": "Ethernet controller"}, "id": {"device": 33081, "subsystem-vendor": 6900, "vendor": 4332, "subsystem": 4352}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": false, "mem_type_64": false, "bar": 1, "size": 256, "address": -1, "type": "memory"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "pci-pt-01_00.0", "irq": 0, "slot": 6, "class_info": {"class": 0}, "id": {"device": 9988, "subsystem-vendor": 0, "vendor": 4318, "subsystem": 0}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": false, "bar": 0, "size": 16777216, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": true, "bar": 1, "size": 268435456, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": true, "bar": 3, "size": 33554432, "address": -1, "type": "memory"}, {"bar": 5, "size": 128, "address": -1, "type": "io"}, {"prefetch": true, "mem_type_64": false, "bar": 6, "size": 524288, "address": -1, "type": "memory"}]}]}], "id": 2020372738}
And if you addition all the “size” values, what value do you get ?