Cannot increase Qube memory above 1971 MB

Renevith · January 2, 2024, 7:05pm

I installed QubesOS successfully and it runs fine, but I noticed that one of my qubes was getting very little memory and quickly becoming sluggish. It was being assigned its minimum memory (400 MB), and some other Qubes were being assigned their maximum memory (4000 MB) so I thought maybe my system just didn’t have enough to allocate.

I first tried to reduce the maximum of two other Qubes to 2000 MB and fully restarted. That didn’t help; the other Qubes indeed only took 2000 MB but the Qube I’m using still got only 400MB. So I tried increasing its initial memory. I was able to increase it to 1971 MB successfully, but if I increase the initial memory to 1972 MB or more, the Qube fails to start. I get “Connection to the VM failed.”

I tried disabling “include in memory balancing” with the same result.

xentop shows I have plenty of available physical memory:

xentop - 12:54:34   Xen 4.17.2
8 domains: 1 running, 7 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
Mem: 33255292k total, 15447880k used, 17807412k free    CPUs: 12 @ 2188MHz

I didn’t see anything that I understood in the logs, but I’ve included them here. This is /var/log/xen/console/guest-anon-whonix-dm.log:

[2024-01-02 12:07:52] + printf '==== Press enter for shell ====\n'
[2024-01-02 12:07:52] ==== Press enter for shell ====
[2024-01-02 12:07:52] + read
[2024-01-02 12:07:52] + vchan-socket-proxy '--mode=server' '--reconnect-marker=1' --verbose 0 device-model/11/qmp-vchan /tmp/qemu.qmp
[2024-01-02 12:07:52] from-unix: {"QMP": {"version": {"qemu": {"micro": 2, "minor": 1, "major": 8}, "package": ""}, "capabilities": ["oob"]}}

[2024-01-02 12:07:52] 
[2024-01-02 12:07:52] wrote 110 bytes to vchan
[2024-01-02 12:07:52] from-vchan: 
[2024-01-02 12:07:52] from-vchan: {"execute":"qmp_capabilities","id":2020372737}

[2024-01-02 12:07:52] 
[2024-01-02 12:07:52] from-unix: {"return": {}, "id": 2020372737}

[2024-01-02 12:07:52] 
[2024-01-02 12:07:52] wrote 34 bytes to vchan
[2024-01-02 12:07:52] from-vchan: {"execute":"device_add","id":2020372736,"arguments":{"driver":"xen-pci-passthrough","id":"pci-pt-01_00.0","hostaddr":"0000:01:00.0"}}

[2024-01-02 12:07:52] 
[2024-01-02 12:07:52] [00:07.0] xen_pt_realize: Assigning real physical device 01:00.0 to devfn 0x38
[2024-01-02 12:07:52] [00:07.0] xen_pt_register_regions: IO region 0 registered (size=0x00004000 base_addr=0x7a300000 type: 0x4)
[2024-01-02 12:07:52] [00:07.0] xen_pt_config_reg_init: Offset 0x000e mismatch! Emulated=0x0080, host=0x0000, syncing to 0x0000.
[2024-01-02 12:07:52] [00:07.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! Emulated=0x0000, host=0x7a300004, syncing to 0x7a300004.
[2024-01-02 12:07:52] [00:07.0] xen_pt_config_reg_init: Offset 0x0082 mismatch! Emulated=0x0000, host=0x0003, syncing to 0x0003.
[2024-01-02 12:07:52] [00:07.0] xen_pt_config_reg_init: Offset 0x0092 mismatch! Emulated=0x0000, host=0x008a, syncing to 0x0080.
[2024-01-02 12:07:52] [00:07.0] xen_pt_config_reg_init: Offset 0x00c4 mismatch! Emulated=0x0000, host=0x10008f02, syncing to 0x8f02.
[2024-01-02 12:07:52] [00:07.0] xen_pt_config_reg_init: Offset 0x00d2 mismatch! Emulated=0x0000, host=0x1044, syncing to 0x1044.
[2024-01-02 12:07:52] [00:07.0] xen_pt_msix_init: get MSI-X table BAR base 0x7a300000
[2024-01-02 12:07:52] [00:07.0] xen_pt_config_reg_init: Offset 0x00b2 mismatch! Emulated=0x0000, host=0x0040, syncing to 0x0040.
[2024-01-02 12:07:52] [00:07.0] xen_pt_pci_intx: intx=1
[2024-01-02 12:07:52] [00:07.0] xen_pt_realize: Real physical device 01:00.0 registered successfully
[2024-01-02 12:07:52] from-unix: {"return": {}, "id": 2020372736}

[2024-01-02 12:07:52] 
[2024-01-02 12:07:52] wrote 34 bytes to vchan
[2024-01-02 12:07:52] from-vchan: {"execute":"query-pci","id":2020372738}

[2024-01-02 12:07:52] 
[2024-01-02 12:07:52] from-unix: {"return": [{"bus": 0, "devices": [{"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 0, "class_info": {"class": 1536, "desc": "Host bridge"}, "id": {"device": 4663, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 1537, "desc": "ISA bridge"}, "id": {"device": 28672, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": []}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 1, "class_info": {"class": 257, "desc": "IDE controller"}, "id": {"device": 28688, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 1, "regions": [{"bar": 4, "size": 16, "address": -1, "type": "io"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 1, "class_info": {"class": 1664, "desc": "Bridge"}, "id": {"device": 28947, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 3, "regions": []}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 2, "class_info": {"class": 65408}, "id": {"device": 1, "subsystem-vendor": 22611, "vendor": 22611, "subsystem": 1}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": true, "mem_type_64": false, "bar": 1, "size": 16777216, "address": -1, "type": "memory"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "", "irq": 0, "slot": 3, "class_info": {"class": 256, "desc": "SCSI controller"}, "id": {"device": 18, "subsystem-vendor": 0, "vendor": 4096, "subsystem": 4096}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": false, "mem_type_64": false, "bar": 1, "size": 1024, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 2, "size": 8192, "address": -1, "type": "memory"}]}, {"irq_pin": 0, "bus": 0, "qdev_id": "", "slot": 4, "class_info": {"class": 768, "desc": "VGA controller"}, "id": {"device": 4369, "subsystem-vendor": 6900, "vendor": 4660, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": true, "mem_type_64": false, "bar": 0, "size": 16777216, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 2, "size": 4096, "address": -1, "type": "memory"}, {"prefetch": false, "mem_type_64": false, "bar": 6, "size": 65536, "address": -1, "type": "memory"}]}, {"irq_pin": 4, "bus": 0, "qdev_id": "ehci", "irq": 0, "slot": 5, "class_info": {"class": 3075, "desc": "USB controller"}, "id": {"device": 9421, "subsystem-vendor": 6900, "vendor": 32902, "subsystem": 4352}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": false, "bar": 0, "size": 4096, "address": -1, "type": "memory"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "nic0", "irq": 0, "slot": 6, "class_info": {"class": 512, "desc": "Ethernet controller"}, "id": {"device": 33081, "subsystem-vendor": 6900, "vendor": 4332, "subsystem": 4352}, "function": 0, "regions": [{"bar": 0, "size": 256, "address": -1, "type": "io"}, {"prefetch": false, "mem_type_64": false, "bar": 1, "size": 256, "address": -1, "type": "memory"}]}, {"irq_pin": 1, "bus": 0, "qdev_id": "pci-pt-01_00.0", "irq": 0, "slot": 7, "class_info": {"class": 0}, "id": {"device": 20528, "subsystem-vendor": 0, "vendor": 5559, "subsystem": 0}, "function": 0, "regions": [{"prefetch": false, "mem_type_64": true, "bar": 0, "size": 16384, "address": -1, "type": "memory"}]}]}], "id": 2020372738}

[2024-01-02 12:07:52] 
[2024-01-02 12:07:52] wrote 2048 bytes to vchan
[2024-01-02 12:07:52] wrote 1366 bytes to vchan
[2024-01-02 12:07:53] vchan client disconnected
[2024-01-02 12:07:53] processing error - resetting ehci HC
[2024-01-02 12:07:53] random: crng init done
[2024-01-02 12:07:55] br0: port 2(vif11.0-emu) entered disabled state
[2024-01-02 12:07:55] device vif11.0-emu left promiscuous mode
[2024-01-02 12:07:55] br0: port 2(vif11.0-emu) entered disabled state
[2024-01-02 12:07:55] {"timestamp": {"seconds": 1704218875, "microseconds": 654751}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral-anon/device[3]"}}

[2024-01-02 12:07:55] {"timestamp": {"seconds": 1704218875, "microseconds": 654857}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral-anon/device[0]"}}

[2024-01-02 12:07:55] {"timestamp": {"seconds": 1704218875, "microseconds": 657008}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}

[2024-01-02 12:07:55] {"timestamp": {"seconds": 1704218875, "microseconds": 657543}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}}

[2024-01-02 12:07:55] {"timestamp": {"seconds": 1704218875, "microseconds": 714556}, "event": "DEVICE_DELETED", "data": {"device": "nic0", "path": "/machine/peripheral/nic0"}}

[2024-01-02 12:07:55] /etc/qemu-ifdown 
[2024-01-02 12:07:55] br0: port 1(eth0) entered disabled state
[2024-01-02 12:07:55] device eth0 left promiscuous mode
[2024-01-02 12:07:55] br0: port 1(eth0) entered disabled state
[2024-01-02 12:07:56] pcifront pci-0: Rescanning PCI Frontend Bus 0000:00
[2024-01-02 12:07:56] pci_bus 0000:00: busn_res: [bus 00-ff] is released
[2024-01-02 12:07:56] ------------[ cut here ]------------
[2024-01-02 12:07:56] sysfs group 'power' not found for kobject '0000:00'
[2024-01-02 12:07:56] WARNING: CPU: 0 PID: 10 at 0xffffffff810e9710
[2024-01-02 12:07:56] CPU: 0 PID: 10 Comm: xenwatch Tainted: G                T 5.10.200-xen-stubdom #1
[2024-01-02 12:07:56] RIP: e030:0xffffffff810e9710
[2024-01-02 12:07:56] Code: f6 74 2e 49 89 fc 48 89 df e8 ab fc ff ff 48 89 c3 48 85 c0 75 23 49 8b 14 24 48 8b 75 00 48 c7 c7 ff dd 84 81 e8 87 82 16 00 <0f> 0b 5b 5d 41 5c c3 48 89 df e8 30 d0 ff ff 48 89 ee 48 89 df e8
[2024-01-02 12:07:56] RSP: e02b:ffffc90000253d20 EFLAGS: 00010282
[2024-01-02 12:07:56] RAX: 0000000000000033 RBX: 0000000000000000 RCX: 0000000000000003
[2024-01-02 12:07:56] RDX: 000000000000009d RSI: 00000000ffffefff RDI: 0000000000000200
[2024-01-02 12:07:56] RBP: ffffffff81822800 R08: 0000000000000000 R09: ffffc90000253b68
[2024-01-02 12:07:56] R10: ffffffff81a3efe0 R11: ffffc90000253b60 R12:

apparatus · January 2, 2024, 7:16pm

github.com/QubesOS/qubes-issues

AppVM with GPU pass-through crashes when more than 3.5 GB (3584MB) of RAM is assigned to it

opened 06:48PM - 19 Sep 18 UTC

closed 06:46AM - 09 Jan 24 UTC

pqyptixa

T: bug C: core C: Xen P: default r4.1-dom0-stable diagnosed pr submitted backport pending affects-4.1 affects-4.2

### Qubes OS version: R4.0 ### Affected component(s): Debian 10 AppVM -…-- ### Steps to reproduce the behavior: So, I finally could pass a GPU to a VM with PCI passthrough some weeks ago, but for some reason, I can't get it to run with more RAM. I asked #qubes_os @ freenode about this issue, and ended up making a guide on how to get GPU-passthrough done. I'm not going to post the complete guide here, so here's a link to it: https://paste.debian.net/1043341/ . I don't mind if someone else posts it here or elsewhere, but hopefully people will discuss it and improve it. My system: Qubes 4.0 running on a AMD Ryzen 1600 CPU + MSI B350 Mortar motherboard (BIOS 1E0) + ASUS RX560 2GB GPU for the VMs + an old NVIDIA 610 GPU for dom0 (as secondary GPU). ### Expected behavior: AppVM runs just fine with > 3.5GB of RAM. ### Actual behavior: AppVM crashes with more than 3.5GB of RAM. The way it crashes depends on how much memory is assigned to it, but I usually see almost-instant kernel panics, systemd-init crashing, fsck failures and/or other random processes crashing, too. ### General notes: Other than removing "nomodeset" from the kernel command line, everything is set as default. PS: link to the guide expired. Here a a couple of similar guides: https://github.com/Qubes-Community/Contents/blob/master/docs/customization/windows-gaming-hvm.md and https://neowutran.ovh/qubeos.html

Renevith · January 2, 2024, 8:28pm

Thanks! That definitely looks related. I am using HVM with a PCI passthrough, although of a storage device rather than a GPU. I don’t encounter this issue if I unattach the PCI device

I was able to set -machine xenfv,max-ram-below-4g=3.5G using one of the workarounds on that ticket (editing the xen.xml template), which allowed me to get my initial memory up to 3400MB with the PCI device attached. Much appreciated.