Re: [Xen-devel] [RFC 16/16] xen/arm: Track page accessed between batch of Set/Way operations

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 6 Nov 2018, Julien Grall wrote:
> On 06/11/2018 17:43, Stefano Stabellini wrote:
> > On Mon, 5 Nov 2018, Julien Grall wrote:
> > > Hi Stefano,
> > > 
> > > On 11/5/18 9:35 PM, Stefano Stabellini wrote:
> > > > On Mon, 8 Oct 2018, Julien Grall wrote:
> > > > > At the moment, the implementation of Set/Way operations will go
> > > > > through
> > > > > all the entries of the guest P2M and flush them. However, this is very
> > > > > expensive and may render unusable a guest OS using them.
> > > > > 
> > > > > For instance, Linux 32-bit will use Set/Way operations during
> > > > > secondary
> > > > > CPU bring-up. As the implementation is really expensive, it may be
> > > > > possible
> > > > > to hit the CPU bring-up timeout.
> > > > > 
> > > > > To limit the Set/Way impact, we track what pages has been of the guest
> > > > > has been accessed between batch of Set/Way operations. This is done
> > > > > using bit[0] (aka valid bit) of the P2M entry.
> > > > 
> > > > This is going to improve performance of ill-mannered guests at the cost
> > > > of hurting performance of well-mannered guests. Is it really a good
> > > > trade-off? Should this behavior at least be configurable with a Xen
> > > > command line?
> > > 
> > > Well, we have the choice between not been able to boot Linux 32-bit
> > > anymore or
> > > have a slight impact at the boot time for all guests.
> > 
> > Wait -- I thought that with the set/way emulation introduced by patch
> > #15 we would be able to boot Linux 32-bit already. This patch is a
> > performance improvement. Or is it actually needed to boot Linux 32-bit?
> 
> The problem is Linux 32-bit calls a few time set/way during secondary CPU
> bring up. It also has a timeout of 1s to fully boot that CPU. In my testing, I
> can easily hit the timeout even with a small amount of memory.
 
Damn! It is worst than I thought.


> If we don't start tracking the page from the beginning, then you would need to
> clean the full RAM the first time. If you start to track from the beginning,
> you may just have to clean a couple of MB.
>
> > 
> > > As you may have noticed the command line is been suggested below. I didn't
> > > yet
> > > implemented as we agreed at Connect it would be good to start getting
> > > feedback
> > > on it.
> > 
> > Sure. I was thinking about this -- does it make sense to have different
> > defaults for 32bit and 64bit guests?
> > 
> > 32bit -> default p2m_invalidate_root
> > 64bit -> default not
> 
> The decision is not that easy. While Linux arm64 does not contain set/way
> anymore, UEFI still use them (I checked it a couple of months ago).
> 
> I haven't done any benchmark yet. That's my next step.

OK, makes sense


> > > I am not entirely to understand your last sentence, this feature is turned
> > > off
> > > when an IOMMU is present. So what is your use case here?
> >   My sentence was badly written, sorry. I meant to say that even when an
> > IOMMU is NOT present, there are cases where we might not want to call
> > p2m_invalidate_root.
> 
> Before implementing a command line option (or even plumbing that to the guest
> configuration), I want to understand what is the real impact on the boot time
> for "normal" guest.

yes, makes sense

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel