Can somebody explain what exactly max-ram-below-4g does? How exactly it should affect virtualization in Qubes OS?
To avoid setting this 2G, 3.5G and other numbers without understanding what is happening.
1 Like
I added a note in the guide here Create a Gaming HVM .
Does anyone known/understand/or have an idea on why some GPU require this value to exactly be “3.5G”, and some other seems to require it to be exactly “2G” ?
( https://github.com/qemu/qemu/blob/master/hw/i386/pc_piix.c#L131 ; )
2 Likes
@neowutran In your doc with the updated information, please link to my post. It was hell to figure out and I would appreciate the attribution 
1 Like
Exactly 3.5GB?? I thought it would be a range of acceptable values! 
done (& added you to the “contributor” list of this doc )
3 Likes
Some time ago, I tried different values for this parameter, it required to be exactly 3.5G to work, any other values I tried resulted in non working setup
What a mess, too complicated.
So, if I have spare 32GiB that I want to give some VM:
- how can I provide it all to the VM?
- why “2G” should work, and “3.5G” or any other number should not?
- what this 2G will mean out of this 32GiB amount that I decided to give to VM?
- How much memory will VM actually get, 32GiB? 30GiB (32-2)? Something else?
Completely unclear to me.
1 Like
I know basically nothing about these things but one thing that jumps out to me is the size of the individual RAM chips the GPU uses. For modern high end GPUs this is typically 2gb, which is exactly the size that was needed to pass through effectively. Something weird might happen when u pass through only a portion of GPU ram because its more integrated than a CPU and its RAM.
As far as I understand:
- how can I provide it all to the VM? : Just assign the memory you want through the Qubes gui
- what this 2G will mean out of this 32GiB amount that I decided to give to VM? : Irrelevant, ram is not “lost” it is just how the address space is reserved.
- How much memory will VM actually get, 32GiB? 30GiB (32-2)? Something else? Irrelevant, ram is not “lost”.
- why “2G” should work, and “3.5G” or any other number should not? : For sure, it is hardware specific. From the source code “Traditional split is 3.5G (lowmem = 0xe0000000)”, and it is what I use for my hardware. On some VFIO forum the recommanded value is “1G”. For some user here it need to be “2G”
1 Like
Sorry, still very unclear to me. If you can elaborate a bit more:
- You say “reserved”. Reserved for what? For whom and by who?
- If user used
max-ram-below-4g=3.5Gthan is 3.5GiB “reserved”, or “0.5GiB” is reserved? - Why is it
max-ram-below-4g, why 4G? Is it about int32 limit size of pointers or what? - Why this workaround with a parameter is needed at all on x64 systems?
- Do users still need to patch stubdom-linux-rootfs or downgrade xen-hvm-stubdom-linux? Or completely not and providing
max-ram-below-4gkernel parameter is enough nowadays? - If you understand the matter and architecture of this workaround, how do you think, will it be possible to make it just work (without any experiments with hardware specific values of some kernel parameter) or it is something like a legacy design flaw for us to live with for a long time?
So many questions, because I still do not understand what is happening and why is it still not properly fixed upstream in Xen, QEMU and Qubes OS to make it just work without tricks.
Out of curiosity, in your bios settings, what is the value of the parameter “Above 4G decoding” ?
If it was to “false”, could you set it to “true”, then modify the max-ram-below-4g to 3.5G again, and test if it work ?
(probably not interesting, tried to set “Above 4G decoding” to false and true. Same behavior, it work for me with 3.5G)
Unfortunately my BIOS settings are very minimal and I don’t have that parameter available to adjust.
still trying to understand the difference between our systems;
In Qubes Manager, if you go to the logs of your HVM with the gpu passthrough, in the “-dm.log” file, do you have a line that look like this one ?
from-unix: {"return": [{"bus": 0, "devices": [{"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 0, "class_info": {"class": 1536, "desc": "Host bridge"}, "id": {"device": 4663, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 1537, "desc": "ISA bridge"}, "id": {"device": 28672, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 257, "desc": "IDE controller"}, "id": {"device": 28688, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 1, "regions": [{"bar": 4, "size": 16, "address": -1, "type": "io"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 1, "class_info": {"class": 1664, "desc": "Bridge"}, "id": {"device": 28947, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 3, "regions": []}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 2, "class_info": {"class": 65408}, "id": {"device": 1, "subsystem-vendor": 22611, "vendor": 22611, "subsystem": 1}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": true, "mem_type_64": false, "bar": 1, "size": 16777216, "address": -1, "type": "memory"}]}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 3, "class_info": {"class": 768, "desc": "VGA controller"}, "id": {"device": 4369, "subsystem-vendor": 6900, "vendor": 4660, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": true, "mem_type_64": false, "bar": 0, "size": 16777216, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 2, "size": 4096, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 6, "size": 65536, "address": -1, "type": "memory"}]}, {"irq_pin": 4, "bus": 0, "qdev_id": "ehci", "irq": 0, "slot": 4, "class_info": {"class": 3075, "desc": "USB controller"}, "id": {"device": 9421, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": false, "bar": 0, "size": 4096, "address": -1, "type": "memory"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "nic0", "irq": 0, "slot": 5, "class_info": {"class": 512, "desc": "Ethernet controller"}, "id": {"device": 33081, "subsystem-vendor": 6900, "vendor": 4332, "subsystem": 4352}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": false, "mem_type_64": false, "bar": 1, "size": 256, "address": -1, "type": "memory"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "pci-pt-01_00.0", "irq": 0, "slot": 6, "class_info": {"class": 0}, "id": {"device": 9988, "subsystem-vendor": 0, "vendor": 4318, "subsystem": 0}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": false, "bar": 0, "size": 16777216, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": true, "bar": 1, "size": 268435456, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": true, "bar": 3, "size": 33554432, "address": -1, "type": "memory"}, {"bar": 5, "size": 128, "address": -1, "type": "io"}, {"prefetch": true, "mem_type_64": false, "bar": 6, "size": 524288, "address": -1, "type": "memory"}]}]}], "id": 2020372738}
And if you addition all the “size” values, what value do you get ?
Yes, I have that line. Here it is:
from-unix: {"return": [{"bus": 0, "devices": [
{"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 0, "class_info": {"class": 1536, "desc": "Host bridge"}, "id": {"device": 4663, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []},
{"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 1537, "desc": "ISA bridge"}, "id": {"device": 28672, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []},
{"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 257, "desc": "IDE controller"}, "id": {"device": 28688, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 1, "regions": [{"bar": 4, "size": 16, "address": -1, "type": "io"}]},
{"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 1, "class_info": {"class": 1664, "desc": "Bridge"}, "id": {"device": 28947, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 3, "regions": []},
{"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 2, "class_info": {"class": 65408}, "id": {"device": 1, "subsystem-vendor": 22611, "vendor": 22611, "subsystem": 1}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": true, "mem_type_64": false, "bar": 1, "size": 16777216, "address": -1, "type": "memory"}]},
{"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 3, "class_info": {"class": 768, "desc": "VGA controller"}, "id": {"device": 4369, "subsystem-vendor": 6900, "vendor": 4660, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": true, "mem_type_64": false, "bar": 0, "size": 16777216, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 2, "size": 4096, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 6, "size": 65536, "address": -1, "type": "memory"}]},
{"irq_pin": 4, "bus": 0, "qdev_id": "ehci", "irq": 0, "slot": 4, "class_info": {"class": 3075, "desc": "USB controller"}, "id": {"device": 9421, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": false, "bar": 0, "size": 4096, "address": -1, "type": "memory"}]},
{"irq_pin": 1, "bus": 0, "qdev_id": "nic0", "irq": 0, "slot": 5, "class_info": {"class": 512, "desc": "Ethernet controller"}, "id": {"device": 33081, "subsystem-vendor": 6900, "vendor": 4332, "subsystem": 4352}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": false, "mem_type_64": false, "bar": 1, "size": 256, "address": -1, "type": "memory"}]},
{"irq_pin": 2, "bus": 0, "qdev_id": "pci-pt-01_00.1", "irq": 0, "slot": 6, "class_info": {"class": 0}, "id": {"device": 8846, "subsystem-vendor": 0, "vendor": 4318, "subsystem": 0}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": false, "bar": 0, "size": 16384, "address": -1, "type": "memory"}]},
{"irq_pin": 1, "bus": 0, "qdev_id": "pci-pt-01_00.0", "irq": 0, "slot": 7, "class_info": {"class": 0}, "id": {"device": 9568, "subsystem-vendor": 0, "vendor": 4318, "subsystem": 0}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": false, "bar": 0, "size": 16777216, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": true, "bar": 1, "size": 8589934592, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": true, "bar": 3, "size": 33554432, "address": -1, "type": "memory"}, {"bar": 5, "size": 128, "address": -1, "type": "io"}, {"prefetch": true, "mem_type_64": false, "bar": 6, "size": 524288, "address": -1, "type": "memory"}]}
]}], "id": 2020372738}
The “size” for the graphics card (“pci-pt-01_00.0”) totals 8640790656 and the total “size” parameters for all the devices is 8674435984.
Some more questions if you have time:
- Do you have any kind of “resizable bar” option activated in the bios ?
- can you post the result of “sudo lspci -vvvnns YOUR_GPU_PCI_IDENTIFIER” from your gpu vm ?
Mine is:
[gpu_gaming]$ sudo lspci -vvvnns 00:06.0
00:06.0 VGA compatible controller [0300]: NVIDIA Corporation AD103 [GeForce RTX 4080] [10de:2704] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation AD103 [GeForce RTX 4080] [10de:167a]
Physical Slot: 6
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 248
Interrupt: pin A routed to IRQ 91
Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at f0000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at c200 [size=128]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee57000 Data: 0700
Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L1, Exit Latency L1 <16us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x16
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp+ 10BitTagReq+ OBFF Via message, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
I don’t have a setting for resizable bar, but I am not sure if it is enabled or disabled by default and I couldn’t find out from the Lenovo website or a search.
Here is the output of lspci:
00:07.0 VGA compatible controller [0300]: NVIDIA Corporation GA106M [GeForce RTX 3060 Mobile / Max-Q] [10de:2560] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Lenovo GA106M [GeForce RTX 3060 Mobile / Max-Q] [17aa:3803]
Physical Slot: 7
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 248
Interrupt: pin A routed to IRQ 71
Region 0: Memory at 84000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at 600000000 (64-bit, prefetchable) [size=8G]
Region 3: Memory at 80000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at c200 [size=128]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00000 Data: 0024
Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x8 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp+ 10BitTagReq+ OBFF Via message, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b4] Vendor Specific Information: Len=14 <?>
Kernel driver in use: nvidia
Kernel modules: nvidia
Hi, thanks you !
I think I have an idea on what is different between my setup and your setup.
In the output of your lspci command, this line is interesting:
Region 1: Memory at 600000000 (64-bit, prefetchable) [size=8G]
8G is not the usual default BAR size. The standard BAR size is 256M (what I have on my config):
Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
So your motherboard seems to not use the standard BAR size (maybe “resizable bar” option is activated on your motherboard and you can’t change it).
I did some tests on my setup.
I enabled “Resizable bar” in my bios, and tried to launch my gpu hvm. And it crashed the same way it crashed for you before. I notice that the BAR size changed to:
Region 1: Memory at c00000000 (64-bit, prefetchable) [size=16G]
Then I switched to 2G instead of my 3.5G, and it work with “resizable bar” activated.
I will test other values instead of 2G and 3.5G, and I will try to find someone with a more advanced motherboard to check if the difference of value depend on the option “resizable bar” or not.
4 Likes