Re: grant table issues mapping a ring order 10

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Wed, 20 May 2020, Jürgen Groß wrote:
> On 20.05.20 08:00, Jürgen Groß wrote:
> > On 19.05.20 23:21, Stefano Stabellini wrote:
> > > Hi Juergen, Boris,
> > > 
> > > I am trying to increase the size of the rings used for Xen 9pfs
> > > connections for performance reasons and also to reduce the likehood of
> > > the backend having to wait on the frontend to free up space from the
> > > ring.
> > > 
> > > FYI I realized that we cannot choose order 11 or greater in Linux
> > > because then we incur into the hard limit CONFIG_FORCE_MAX_ZONEORDER=11.
> > > But that is not the reason why I am writing to you :-)
> > > 
> > > 
> > > The reason why I am writing is that even order 10 fails for some
> > > grant-table related reason I cannot explain. There are two rings, each
> > > of them order 10. Mapping the first ring results into an error. (Order 9
> > > works fine, resulting in both rings being mapped correctly.)
> > > 
> > > QEMU tries to map the refs but gets an error:
> > > 
> > >    gnttab: error: mmap failed: Invalid argument
> > >    xen be: 9pfs-0: xen be: 9pfs-0: xengnttab_map_domain_grant_refs failed:
> > > Invalid argument
> > >    xengnttab_map_domain_grant_refs failed: Invalid argument
> > > 
> > > The error comes from Xen. The hypervisor returns GNTST_bad_gntref to
> > > Linux (drivers/xen/grant-table.c:gnttab_map_refs). Then:
> > > 
> > >          if (map->map_ops[i].status) {
> > >             err = -EINVAL;
> > >             continue;
> > >         }
> > > 
> > > So Linux returns -EINVAL to QEMU. The ref seem to be garbage. The
> > > following printks are in Xen in the implemenation of map_grant_ref:
> > > 
> > > (XEN) DEBUG map_grant_ref 1017 ref=998 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=999 nr=2560
> > > (XEN) DEBUG map_grant_ref 1013 ref=2050669706 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0x7a3abc8a for d1
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=19 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1013 ref=56423797 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0x35cf575 for d1
> > > (XEN) DEBUG map_grant_ref 1013 ref=348793 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0x55279 for d1
> > > (XEN) DEBUG map_grant_ref 1013 ref=1589921828 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0x5ec44824 for d1
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1013 ref=2070386184 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0x7b679608 for d1
> > > (XEN) DEBUG map_grant_ref 1013 ref=3421871 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0x3436af for d1
> > > (XEN) DEBUG map_grant_ref 1013 ref=1589921828 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0x5ec44824 for d1
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1013 ref=875999099 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0x3436af7b for d1
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1013 ref=2705045486 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0xa13bb7ee for d1
> > > (XEN) DEBUG map_grant_ref 1013 ref=4294967295 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0xffffffff for d1
> > > (XEN) DEBUG map_grant_ref 1013 ref=213291910 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0xcb69386 for d1
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1013 ref=4912 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0x1330 for d1
> > > (XEN) DEBUG map_grant_ref 1013 ref=167788925 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0xa00417d for d1
> > > (XEN) DEBUG map_grant_ref 1017 ref=24 nr=2560
> > > (XEN) DEBUG map_grant_ref 1013 ref=167788925 nr=2560
> > > (XEN) grant_table.c:1015:d0v0 Bad ref 0xa00417d for d1
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> > > 
> > > 
> > > Full logs https://pastebin.com/QLTUaUGJ
> > > It is worth mentioning that no limits are being reached: we are below
> > > 2500 entries per domain and below the 64 pages of grant refs per domain.
> > > 
> > > What it seems to happen is that after ref 999, the next refs are garbage.
> > > Do you have any ideas why?
> > 
> > I don't think there is enough space for all the needed grant refs in the
> > initial interface page passed via Xenstore. So how do you pass the refs
> > to the backend?
> 
> Looking into the full log this seems to be the problem: The processing
> is starting with ref=9 and the last successful ref is 999, so 991 refs
> have been processed. Each ref needs 4 bytes, so a page could hold 1024
> refs, but the first 132 bytes of the page are used for other information
> resulting in 1024-33 == 991 refs possible.

  O_O

Dho! That is definitely the issue. Thank you Jurgen!

I added all sorts of checks to the grant table pages and forgot about
the initial shared page used to pass the ring refs themselves.