Re: [PATCH v2 1/3] xen-blkback: add a parameter for disabling of persistent grants

[SeongJae Park <sjpark@xxxxxxxxxx>]

On Tue, 22 Sep 2020 13:35:11 +0200 "Roger Pau Monné" <roger.pau@xxxxxxxxxx> 
wrote:

> On Tue, Sep 22, 2020 at 01:26:38PM +0200, SeongJae Park wrote:
> > On Tue, 22 Sep 2020 13:12:59 +0200 "Roger Pau Monné" <roger.pau@xxxxxxxxxx> 
> > wrote:
> > 
> > > On Tue, Sep 22, 2020 at 12:52:07PM +0200, SeongJae Park wrote:
> > > > From: SeongJae Park <sjpark@xxxxxxxxx>
> > > > 
> > > > Persistent grants feature provides high scalability.  On some small
> > > > systems, however, it could incur data copy overheads[1] and thus it is
> > > > required to be disabled.  But, there is no option to disable it.  For
> > > > the reason, this commit adds a module parameter for disabling of the
> > > > feature.
> > > > 
> > > > [1] https://wiki.xen.org/wiki/Xen_4.3_Block_Protocol_Scalability
> > > > 
> > > > Signed-off-by: Anthony Liguori <aliguori@xxxxxxxxxx>
> > > > Signed-off-by: SeongJae Park <sjpark@xxxxxxxxx>
> > > > ---
> > > >  .../ABI/testing/sysfs-driver-xen-blkback      |  9 ++++++
> > > >  drivers/block/xen-blkback/xenbus.c            | 28 ++++++++++++++-----
> > > >  2 files changed, 30 insertions(+), 7 deletions(-)
> > > > 
> > > > diff --git a/Documentation/ABI/testing/sysfs-driver-xen-blkback 
> > > > b/Documentation/ABI/testing/sysfs-driver-xen-blkback
> > > > index ecb7942ff146..ac2947b98950 100644
> > > > --- a/Documentation/ABI/testing/sysfs-driver-xen-blkback
> > > > +++ b/Documentation/ABI/testing/sysfs-driver-xen-blkback
> > > > @@ -35,3 +35,12 @@ Description:
> > > >                  controls the duration in milliseconds that blkback 
> > > > will not
> > > >                  cache any page not backed by a grant mapping.
> > > >                  The default is 10ms.
> > > > +
> > > > +What:           /sys/module/xen_blkback/parameters/feature_persistent
> > > > +Date:           September 2020
> > > > +KernelVersion:  5.10
> > > > +Contact:        SeongJae Park <sjpark@xxxxxxxxx>
> > > > +Description:
> > > > +                Whether to enable the persistent grants feature or 
> > > > not.  Note
> > > > +                that this option only takes effect on newly created 
> > > > backends.
> > > > +                The default is Y (enable).
> > > > diff --git a/drivers/block/xen-blkback/xenbus.c 
> > > > b/drivers/block/xen-blkback/xenbus.c
> > > > index b9aa5d1ac10b..8a95ddd08b13 100644
> > > > --- a/drivers/block/xen-blkback/xenbus.c
> > > > +++ b/drivers/block/xen-blkback/xenbus.c
> > > > @@ -879,6 +879,12 @@ static void reclaim_memory(struct xenbus_device 
> > > > *dev)
> > > >  
> > > >  /* ** Connection ** */
> > > >  
> > > > +/* Enable the persistent grants feature. */
> > > > +static bool feature_persistent = true;
> > > > +module_param(feature_persistent, bool, 0644);
> > > > +MODULE_PARM_DESC(feature_persistent,
> > > > +               "Enables the persistent grants feature");
> > > > +
> > > >  /*
> > > >   * Write the physical details regarding the block device to the store, 
> > > > and
> > > >   * switch to Connected state.
> > > > @@ -906,11 +912,15 @@ static void connect(struct backend_info *be)
> > > >  
> > > >         xen_blkbk_barrier(xbt, be, be->blkif->vbd.flush_support);
> > > >  
> > > > -       err = xenbus_printf(xbt, dev->nodename, "feature-persistent", 
> > > > "%u", 1);
> > > > -       if (err) {
> > > > -               xenbus_dev_fatal(dev, err, "writing 
> > > > %s/feature-persistent",
> > > > -                                dev->nodename);
> > > > -               goto abort;
> > > > +       if (feature_persistent) {
> > > > +               err = xenbus_printf(xbt, dev->nodename, 
> > > > "feature-persistent",
> > > > +                               "%u", feature_persistent);
> > > > +               if (err) {
> > > > +                       xenbus_dev_fatal(dev, err,
> > > > +                                       "writing %s/feature-persistent",
> > > > +                                       dev->nodename);
> > > > +                       goto abort;
> > > > +               }
> > > >         }
> > > >  
> > > >         err = xenbus_printf(xbt, dev->nodename, "sectors", "%llu",
> > > > @@ -1093,8 +1103,12 @@ static int connect_ring(struct backend_info *be)
> > > >                 xenbus_dev_fatal(dev, err, "unknown fe protocol %s", 
> > > > protocol);
> > > >                 return -ENOSYS;
> > > >         }
> > > > -       pers_grants = xenbus_read_unsigned(dev->otherend, 
> > > > "feature-persistent",
> > > > -                                          0);
> > > > +       if (feature_persistent)
> > > > +               pers_grants = xenbus_read_unsigned(dev->otherend,
> > > > +                               "feature-persistent", 0);
> > > > +       else
> > > > +               pers_grants = 0;
> > > > +
> > > 
> > > Sorry for not realizing earlier, but looking at it again I think you
> > > need to cache the value of feature_persistent when it's first used in
> > > the blkback state data, so that it's consistent.
> > > 
> > > What would happen for example with the following flow (assume a
> > > persistent grants enabled frontend):
> > > 
> > > feature_persistent = false
> > > 
> > > connect(...)
> > > feature-persistent is not written to xenstore
> > > 
> > > User changes feature_persistent = true
> > > 
> > > connect_ring(...)
> > > pers_grants = true, because feature-persistent is set unconditionally
> > > by the frontend and feature_persistent variable is now true.
> > > 
> > > Then blkback will try to use persistent grants and the whole
> > > connection will malfunction because the frontend won't.
> > 
> > Ah, you're right.  I should also catch this before but didn't, sorry.
> > 
> > > 
> > > The other option is to prevent changing the variable when there are
> > > blkback instances already running.
> > 
> > I think storing the option value in xenstore would be simpler.  That said, 
> > if
> > you prefer this way, please let me know.
> 
> If possible I prefer to avoid reading from xenstore because it's slow,
> very slow compared to storing this somewhere in the blkback
> structure.

Agreed.

> 
> Could you use the feature_gnt_persistent field in xen_vbd maybe as a
> place to cache the value before it's set with the final negotiated
> value between the frontend and the backend?

Ok, I will do so in the next version.


Thanks,
SeongJae Park