[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: grant table issues mapping a ring order 10
On 20.05.20 08:00, Jürgen Groß wrote:
On 19.05.20 23:21, Stefano Stabellini wrote:
Hi Juergen, Boris,
I am trying to increase the size of the rings used for Xen 9pfs
connections for performance reasons and also to reduce the likehood of
the backend having to wait on the frontend to free up space from the
ring.
FYI I realized that we cannot choose order 11 or greater in Linux
because then we incur into the hard limit CONFIG_FORCE_MAX_ZONEORDER=11.
But that is not the reason why I am writing to you :-)
The reason why I am writing is that even order 10 fails for some
grant-table related reason I cannot explain. There are two rings, each
of them order 10. Mapping the first ring results into an error. (Order 9
works fine, resulting in both rings being mapped correctly.)
QEMU tries to map the refs but gets an error:
gnttab: error: mmap failed: Invalid argument
xen be: 9pfs-0: xen be: 9pfs-0: xengnttab_map_domain_grant_refs
failed: Invalid argument
xengnttab_map_domain_grant_refs failed: Invalid argument
The error comes from Xen. The hypervisor returns GNTST_bad_gntref to
Linux (drivers/xen/grant-table.c:gnttab_map_refs). Then:
if (map->map_ops[i].status) {
err = -EINVAL;
continue;
}
So Linux returns -EINVAL to QEMU. The ref seem to be garbage. The
following printks are in Xen in the implemenation of map_grant_ref:
(XEN) DEBUG map_grant_ref 1017 ref=998 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=999 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=2050669706 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x7a3abc8a for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=19 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=56423797 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x35cf575 for d1
(XEN) DEBUG map_grant_ref 1013 ref=348793 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x55279 for d1
(XEN) DEBUG map_grant_ref 1013 ref=1589921828 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x5ec44824 for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=2070386184 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x7b679608 for d1
(XEN) DEBUG map_grant_ref 1013 ref=3421871 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x3436af for d1
(XEN) DEBUG map_grant_ref 1013 ref=1589921828 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x5ec44824 for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=875999099 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x3436af7b for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=2705045486 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xa13bb7ee for d1
(XEN) DEBUG map_grant_ref 1013 ref=4294967295 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xffffffff for d1
(XEN) DEBUG map_grant_ref 1013 ref=213291910 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xcb69386 for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=4912 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0x1330 for d1
(XEN) DEBUG map_grant_ref 1013 ref=167788925 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xa00417d for d1
(XEN) DEBUG map_grant_ref 1017 ref=24 nr=2560
(XEN) DEBUG map_grant_ref 1013 ref=167788925 nr=2560
(XEN) grant_table.c:1015:d0v0 Bad ref 0xa00417d for d1
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
(XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
Full logs https://pastebin.com/QLTUaUGJ
It is worth mentioning that no limits are being reached: we are below
2500 entries per domain and below the 64 pages of grant refs per domain.
What it seems to happen is that after ref 999, the next refs are garbage.
Do you have any ideas why?
I don't think there is enough space for all the needed grant refs in the
initial interface page passed via Xenstore. So how do you pass the refs
to the backend?
Looking into the full log this seems to be the problem: The processing
is starting with ref=9 and the last successful ref is 999, so 991 refs
have been processed. Each ref needs 4 bytes, so a page could hold 1024
refs, but the first 132 bytes of the page are used for other information
resulting in 1024-33 == 991 refs possible.
Juergen
|
