Re: [Xen-devel] [PATCH 1/3] x86: avoid flush IPI when possible

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 10.02.16 at 18:51, <andrew.cooper3@xxxxxxxxxx> wrote:
> On 10/02/16 15:37, Jan Beulich wrote:
>>>>> On 10.02.16 at 16:00, <andrew.cooper3@xxxxxxxxxx> wrote:
>>> On 10/02/16 12:56, Jan Beulich wrote:
>>>> Since CLFLUSH, other than WBINVD, is a coherency domain wide flush,
>>> I can't parse this sentence.
>> Should have been "..., is a cache coherency domain wide flush, ..." -
>> does it read any better then?
> 
> I believe, given the code in the patch, your intent is "if we WBINVD, we
> don't need to IPI other cores cache flushing reasons".

I don't see how this can be read from the sentence. The primary
statement is makes if "CLFLUSH is a cache coherency domain wide
flush". A secondary statement is that this is different from WBINVD.

> However, given your comment below...
> 
>>
>>> CLFUSH states "Invalidates from every level of the cache hierarchy in
>>> the cache coherence domain"
>>>
>>> WBINVD however states "The instruction then issues a special-function
>>> bus cycle that directs external caches to also write back modified data
>>> and another bus cycle to indicate that the external caches should be
>>> invalidated."
>>>
>>> I think we need input from Intel and AMD here as to the behaviour and
>>> terminology here, and in particular, where the coherency domain
>>> boundaries are.  All CPUs, even across multiple sockets, see coherent
>>> caching, but it is unclear whether this qualifies them to be in the same
>>> cache coherency domain per the instruction spec.
>> Linux already doing what this patch switches us to, I'm not sure
>> we need much extra input.
>>
>>> In particular, given the architecture of 8-socket systems and 45MB of
>>> RAM in L3 caches, does wbinvd seriously drain all caches everywhere? 
>> Not everywhere, just on the local socket (assuming there's no external
>> cache).
> 
> If this is true, then it is clearly not safe to omit the IPIs.

When using CLFLUSH it is safe, while when using WBINVD it's not.

>>> Causing 45MB of data to move to remote memory controllers all at once
>>> would cause a massive system stall.
>> That's why it takes (as we know) so long. See the figure in SDM Vol 3
>> section "Invalidating Caches and TLBs".
> 
> I presume you mean Figure 2-10. WBINVD Invalidation of Shared and
> Non-Shared Cache Hierarchy?
> 
> This quite clearly shows that WBINVD will not invalidate or write back
> the L1 caches for other cores in the same processor.
> 
> Have I misunderstood the logic for choosing when to omit the IPIs?

I'm afraid you did, or else I must have introduced a (latent,
because I didn't notice any issues so far) bug.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel