Re: grant table issues mapping a ring order 10

[Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>]

On 5/19/20 5:21 PM, Stefano Stabellini wrote:
> Hi Juergen, Boris,
>
> I am trying to increase the size of the rings used for Xen 9pfs
> connections for performance reasons and also to reduce the likehood of
> the backend having to wait on the frontend to free up space from the
> ring.
>
> FYI I realized that we cannot choose order 11 or greater in Linux
> because then we incur into the hard limit CONFIG_FORCE_MAX_ZONEORDER=11.
> But that is not the reason why I am writing to you :-)
>
>
> The reason why I am writing is that even order 10 fails for some
> grant-table related reason I cannot explain. There are two rings, each
> of them order 10. Mapping the first ring results into an error. (Order 9
> works fine, resulting in both rings being mapped correctly.)
>
> QEMU tries to map the refs but gets an error:
>
>   gnttab: error: mmap failed: Invalid argument
>   xen be: 9pfs-0: xen be: 9pfs-0: xengnttab_map_domain_grant_refs failed: 
> Invalid argument
>   xengnttab_map_domain_grant_refs failed: Invalid argument
>
> The error comes from Xen. The hypervisor returns GNTST_bad_gntref to
> Linux (drivers/xen/grant-table.c:gnttab_map_refs). Then:
>
>       if (map->map_ops[i].status) {
>                       err = -EINVAL;
>                       continue;
>               }
>
> So Linux returns -EINVAL to QEMU. The ref seem to be garbage. The
> following printks are in Xen in the implemenation of map_grant_ref:
>
> (XEN) DEBUG map_grant_ref 1017 ref=998 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=999 nr=2560
> (XEN) DEBUG map_grant_ref 1013 ref=2050669706 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0x7a3abc8a for d1
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=19 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1013 ref=56423797 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0x35cf575 for d1
> (XEN) DEBUG map_grant_ref 1013 ref=348793 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0x55279 for d1
> (XEN) DEBUG map_grant_ref 1013 ref=1589921828 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0x5ec44824 for d1
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1013 ref=2070386184 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0x7b679608 for d1
> (XEN) DEBUG map_grant_ref 1013 ref=3421871 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0x3436af for d1
> (XEN) DEBUG map_grant_ref 1013 ref=1589921828 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0x5ec44824 for d1
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1013 ref=875999099 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0x3436af7b for d1
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1013 ref=2705045486 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0xa13bb7ee for d1
> (XEN) DEBUG map_grant_ref 1013 ref=4294967295 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0xffffffff for d1
> (XEN) DEBUG map_grant_ref 1013 ref=213291910 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0xcb69386 for d1
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1013 ref=4912 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0x1330 for d1
> (XEN) DEBUG map_grant_ref 1013 ref=167788925 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0xa00417d for d1
> (XEN) DEBUG map_grant_ref 1017 ref=24 nr=2560
> (XEN) DEBUG map_grant_ref 1013 ref=167788925 nr=2560
> (XEN) grant_table.c:1015:d0v0 Bad ref 0xa00417d for d1
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
> (XEN) DEBUG map_grant_ref 1017 ref=0 nr=2560
>
>
> Full logs https://pastebin.com/QLTUaUGJ
> It is worth mentioning that no limits are being reached: we are below
> 2500 entries per domain and below the 64 pages of grant refs per domain.
>
> What it seems to happen is that after ref 999, the next refs are garbage.
> Do you have any ideas why?


Y1K? ;-)


Have you tried verifying that entry #1000 is properly initialized?


-boris


>
>
> I tracked the gnttab_expand calls in Dom0 and they seemed to be done
> correctly. We need 5 grant table pages:
>
> - order 10 -> 1024 refs
> - 2 rings -> 2048 refs
> - 512 refs per grant table page -> 4 pages
> - plus few others refs by default -> 5 pages
>
> [    3.896558] DEBUG gnttab_expand 1287 cur=1 extra=1 max=64 rc=0
> [    5.115189] DEBUG gnttab_expand 1287 cur=2 extra=1 max=64 rc=0
> [    6.334027] DEBUG gnttab_expand 1287 cur=3 extra=1 max=64 rc=0
> [    7.350523] DEBUG gnttab_expand 1287 cur=4 extra=1 max=64 rc=0
>
> As expected gnttab_expand gets called 4 times to add 4 more pages to the
> initial page.
>
>
> Thanks,
>
> Stefano