ddevz
January 8, 2023, 8:07pm
1
Was using qubes when I got auto-logged out. A while after logging back in I found several lines in dom0 dmesg that said “maptrack limit reached, can’t map all guest pages”.
Linux kernel source shows:
case GNTST_no_device_space:
pr_warn_ratelimited("maptrack limit reached, can't map all guest pages\n");
break;
Does this mean I ran too many VMs and exceeded my system memory?
Or maybe it means that I used too much memory in dom0 and need to raise it above 8 gigs for dom0?
Any ideas are appreciated.
ChrisA
January 8, 2023, 8:25pm
2
Some random searching, landed me on:
opened 02:08PM - 20 Feb 20 UTC
T: bug
C: Xen
P: default
r4.1-buster-stable
r4.1-stretch-stable
r4.1-bullseye-stable
r4.1-dom0-stable
**Qubes OS version**
R4.1
xen 4.13
Linux 4.19.94-1
**Affected component(s)… or functionality**
domU (VMs) FS (probably LVM too?)
**Brief summary**p
When running many VMs, some VMs randomly might get I/O error on their disks (backed by the dom0's LV) when the xen values: gnttab_max_frames / gnttab_max_maptrack_frames are low.
**Observations**
- dom0's ``xen:grant_table: maptrack limit reached, can't map all guest pages`` messages are coming shortly (or at the same time) as the I/O errors in domU.
- setting ``gnttab_max_frames=2048`` does not alleviate the issue as I have been monitoring the values, they were not higher than ``300``, e.g.:
```
[arno@dom0 ~]$ for i in $(xl list | tail -n+2 | awk '{print $2}'); do xen-diag gnttab_query_size $i; done; echo $(( $(for i in $(xl list | tail -n+2 | awk '{print $2}'); do xen-diag gnttab_query_size $i; done | awk '{print $2}' | cut -d\= -f2 | sed 's/,//g' | xargs | tr ' ' '+') ));
domid=0: nr_frames=1, max_nr_frames=1024
domid=1: nr_frames=18, max_nr_frames=1024
domid=2: nr_frames=2, max_nr_frames=1024
domid=3: nr_frames=19, max_nr_frames=1024
domid=4: nr_frames=53, max_nr_frames=1024
domid=6: nr_frames=58, max_nr_frames=1024
151
```
**Logs**
### sys-net domU
```
Feb 20 14:24:12 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9500856
Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9500856
Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9394232
Feb 20 14:24:13 sys-net kernel: EXT4-fs warning (device xvda3): ext4_end_bio:317: I/O error 10 writing to inode 130442 (offset 0 size 0 starting block 1174280)
Feb 20 14:24:13 sys-net kernel: Buffer I/O error on device xvda3, logical block 1122311
Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9500856
Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 514168
Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9500856
Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 514248
Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9070560
Feb 20 14:24:13 sys-net kernel: Aborting journal on device xvda3-8.
Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9066496
Feb 20 14:24:13 sys-net kernel: print_req_error: I/O error, dev xvda, sector 9066496
Feb 20 14:24:13 sys-net kernel: Buffer I/O error on dev xvda3, logical block 1081344, lost sync page write
Feb 20 14:24:13 sys-net kernel: JBD2: Error -5 detected when updating journal superblock for xvda3-8.
Feb 20 14:24:13 sys-net kernel: Buffer I/O error on dev xvda3, logical block 0, lost sync page write
Feb 20 14:24:13 sys-net kernel: EXT4-fs (xvda3): I/O error while writing superblock
Feb 20 14:24:13 sys-net kernel: EXT4-fs error (device xvda3): ext4_journal_check_start:61: Detected aborted journal
Feb 20 14:24:13 sys-net kernel: EXT4-fs (xvda3): Remounting filesystem read-only
Feb 20 14:24:13 sys-net kernel: Buffer I/O error on dev xvda3, logical block 0, lost sync page write
Feb 20 14:24:13 sys-net kernel: EXT4-fs (xvda3): I/O error while writing superblock
Feb 20 14:24:12 sys-net qubes.UpdatesProxy+-fedora-30-custom[1393]: /usr/lib/qubes/qubes-rpc-multiplexer: line 48: /etc/qubes-rpc/qubes.UpdatesProxy: Input/output error
```
### dom0
1 second before ``sys-net`` got I/O errors.
```
Feb 20 14:24:10 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
Feb 20 14:24:12 dom0 kernel: xen:grant_table: maptrack limit reached, can't map all guest pages
```
**Related, [non-duplicate](https://www.qubes-os.org/doc/reporting-bugs/#new-issues-should-not-be-duplicates-of-existing-issues) issues**
https://github.com/QubesOS/qubes-issues/issues/5519#issuecomment-586800991
I think I haven't observed the same issue with Qubes R4.0 (xen 4.8).
1 Like
ddevz
January 8, 2023, 9:19pm
3
That looks like my problem. Thanks for the help.