What does "maptrack limit reached, can't map all guest pages" mean?

ddevz · January 8, 2023, 8:07pm

Was using qubes when I got auto-logged out. A while after logging back in I found several lines in dom0 dmesg that said “maptrack limit reached, can’t map all guest pages”.

Linux kernel source shows:

	case GNTST_no_device_space:
			pr_warn_ratelimited("maptrack limit reached, can't map all guest pages\n");
			break;

Does this mean I ran too many VMs and exceeded my system memory?
Or maybe it means that I used too much memory in dom0 and need to raise it above 8 gigs for dom0?

Any ideas are appreciated.

ChrisA · January 8, 2023, 8:25pm

Some random searching, landed me on:

github.com/QubesOS/qubes-issues

FS errors/corruption in Qubes R4.1 on low gnttab_max_frames / gnttab_max_maptrack_frames

opened 02:08PM - 20 Feb 20 UTC

arno01

T: bug C: Xen P: default r4.1-buster-stable r4.1-stretch-stable r4.1-bullseye-stable r4.1-dom0-stable

**Qubes OS version** R4.1 xen 4.13 Linux 4.19.94-1 **Affected component(s)… or functionality** domU (VMs) FS (probably LVM too?) **Brief summary**p When running many VMs, some VMs randomly might get I/O error on their disks (backed by the dom0's LV) when the xen values: gnttab_max_frames / gnttab_max_maptrack_frames are low. **Observations** - dom0's ``xen:grant_table: maptrack limit reached, can't map all guest pages`` messages are coming shortly (or at the same time) as the I/O errors in domU. - setting ``gnttab_max_frames=2048`` does not alleviate the issue as I have been monitoring the values, they were not higher than ``300``, e.g.: ``` [arno@dom0 ~]$ for i in $(xl list | tail -n+2 | awk '{print $2}'); do xen-diag gnttab_query_size $i; done; echo $(( $(for i in $(xl list | tail -n+2 | awk '{print $2}'); do xen-diag gnttab_query_size $i; done | awk '{print $2}' | cut -d\= -f2 | sed 's/,//g' | xargs | tr ' ' '+') )); domid=0: nr_frames=1, max_nr_frames=1024 domid=1: nr_frames=18, max_nr_frames=1024 domid=2: nr_frames=2, max_nr_frames=1024 domid=3: nr_frames=19, max_nr_frames=1024 domid=4: nr_frames=53, max_nr_frames=1024 domid=6: nr_frames=58, max_nr_frames=1024 151 ``` **Logs** ### sys-net domU ``` Feb 20 14:24:12 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9500856 Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9500856 Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9394232 Feb 20 14:24:13 sys-net kernel: EXT4-fs warning (device xvda3): ext4_end_bio:317: I/O error 10 writing to inode 130442 (offset 0 size 0 starting block 1174280) Feb 20 14:24:13 sys-net kernel: Buffer I/O error on device xvda3, logical block 1122311 Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9500856 Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 514168 Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9500856 Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 514248 Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9070560 Feb 20 14:24:13 sys-net kernel: Aborting journal on device xvda3-8. Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9066496 Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9066496 Feb 20 14:24:13 sys-net kernel: Buffer I/O error on dev xvda3, logical block 1081344, lost sync page write Feb 20 14:24:13 sys-net kernel: JBD2: Error -5 detected when updating journal superblock for xvda3-8. Feb 20 14:24:13 sys-net kernel: Buffer I/O error on dev xvda3, logical block 0, lost sync page write Feb 20 14:24:13 sys-net kernel: EXT4-fs (xvda3): I/O error while writing superblock Feb 20 14:24:13 sys-net kernel: EXT4-fs error (device xvda3): ext4_journal_check_start:61: Detected aborted journal Feb 20 14:24:13 sys-net kernel: EXT4-fs (xvda3): Remounting filesystem read-only Feb 20 14:24:13 sys-net kernel: Buffer I/O error on dev xvda3, logical block 0, lost sync page write Feb 20 14:24:13 sys-net kernel: EXT4-fs (xvda3): I/O error while writing superblock Feb 20 14:24:12 sys-net qubes.UpdatesProxy+-fedora-30-custom[1393]: /usr/lib/qubes/qubes-rpc-multiplexer: line 48: /etc/qubes-rpc/qubes.UpdatesProxy: Input/output error ``` ### dom0 1 second before ``sys-net`` got I/O errors. ``` Feb 20 14:24:10 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages ``` **Related, [non-duplicate](https://www.qubes-os.org/doc/reporting-bugs/#new-issues-should-not-be-duplicates-of-existing-issues) issues** https://github.com/QubesOS/qubes-issues/issues/5519#issuecomment-586800991 I think I haven't observed the same issue with Qubes R4.0 (xen 4.8).

ddevz · January 8, 2023, 9:19pm

That looks like my problem. Thanks for the help.

Vael_S · November 11, 2024, 12:07pm

what count for gnttab frames options should I have per guest?
i have currently 1gb ram and frames set to 2048
no log files have any io errors at all
i generally run 15 to 20 guests in general and 10 are always running

how do i find out if it is causing corruption and what log files do i need to watch for what these days?