[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [PATCH v1 0/6] mm: online/offline_pages called w.o. mem_hotplug_lock


  • To: Balbir Singh <bsingharora@xxxxxxxxx>
  • From: David Hildenbrand <david@xxxxxxxxxx>
  • Date: Wed, 19 Sep 2018 09:35:07 +0200
  • Autocrypt: addr=david@xxxxxxxxxx; prefer-encrypt=mutual; keydata= xsFNBFXLn5EBEAC+zYvAFJxCBY9Tr1xZgcESmxVNI/0ffzE/ZQOiHJl6mGkmA1R7/uUpiCjJ dBrn+lhhOYjjNefFQou6478faXE6o2AhmebqT4KiQoUQFV4R7y1KMEKoSyy8hQaK1umALTdL QZLQMzNE74ap+GDK0wnacPQFpcG1AE9RMq3aeErY5tujekBS32jfC/7AnH7I0v1v1TbbK3Gp XNeiN4QroO+5qaSr0ID2sz5jtBLRb15RMre27E1ImpaIv2Jw8NJgW0k/D1RyKCwaTsgRdwuK Kx/Y91XuSBdz0uOyU/S8kM1+ag0wvsGlpBVxRR/xw/E8M7TEwuCZQArqqTCmkG6HGcXFT0V9 PXFNNgV5jXMQRwU0O/ztJIQqsE5LsUomE//bLwzj9IVsaQpKDqW6TAPjcdBDPLHvriq7kGjt WhVhdl0qEYB8lkBEU7V2Yb+SYhmhpDrti9Fq1EsmhiHSkxJcGREoMK/63r9WLZYI3+4W2rAc UucZa4OT27U5ZISjNg3Ev0rxU5UH2/pT4wJCfxwocmqaRr6UYmrtZmND89X0KigoFD/XSeVv jwBRNjPAubK9/k5NoRrYqztM9W6sJqrH8+UWZ1Idd/DdmogJh0gNC0+N42Za9yBRURfIdKSb B3JfpUqcWwE7vUaYrHG1nw54pLUoPG6sAA7Mehl3nd4pZUALHwARAQABzSREYXZpZCBIaWxk ZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT7CwX4EEwECACgFAljj9eoCGwMFCQlmAYAGCwkI BwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEE3eEPcA/4Na5IIP/3T/FIQMxIfNzZshIq687qgG 8UbspuE/YSUDdv7r5szYTK6KPTlqN8NAcSfheywbuYD9A4ZeSBWD3/NAVUdrCaRP2IvFyELj xoMvfJccbq45BxzgEspg/bVahNbyuBpLBVjVWwRtFCUEXkyazksSv8pdTMAs9IucChvFmmq3 jJ2vlaz9lYt/lxN246fIVceckPMiUveimngvXZw21VOAhfQ+/sofXF8JCFv2mFcBDoa7eYob s0FLpmqFaeNRHAlzMWgSsP80qx5nWWEvRLdKWi533N2vC/EyunN3HcBwVrXH4hxRBMco3jvM m8VKLKao9wKj82qSivUnkPIwsAGNPdFoPbgghCQiBjBe6A75Z2xHFrzo7t1jg7nQfIyNC7ez MZBJ59sqA9EDMEJPlLNIeJmqslXPjmMFnE7Mby/+335WJYDulsRybN+W5rLT5aMvhC6x6POK z55fMNKrMASCzBJum2Fwjf/VnuGRYkhKCqqZ8gJ3OvmR50tInDV2jZ1DQgc3i550T5JDpToh dPBxZocIhzg+MBSRDXcJmHOx/7nQm3iQ6iLuwmXsRC6f5FbFefk9EjuTKcLMvBsEx+2DEx0E UnmJ4hVg7u1PQ+2Oy+Lh/opK/BDiqlQ8Pz2jiXv5xkECvr/3Sv59hlOCZMOaiLTTjtOIU7Tq 7ut6OL64oAq+zsFNBFXLn5EBEADn1959INH2cwYJv0tsxf5MUCghCj/CA/lc/LMthqQ773ga uB9mN+F1rE9cyyXb6jyOGn+GUjMbnq1o121Vm0+neKHUCBtHyseBfDXHA6m4B3mUTWo13nid 0e4AM71r0DS8+KYh6zvweLX/LL5kQS9GQeT+QNroXcC1NzWbitts6TZ+IrPOwT1hfB4WNC+X 2n4AzDqp3+ILiVST2DT4VBc11Gz6jijpC/KI5Al8ZDhRwG47LUiuQmt3yqrmN63V9wzaPhC+ xbwIsNZlLUvuRnmBPkTJwwrFRZvwu5GPHNndBjVpAfaSTOfppyKBTccu2AXJXWAE1Xjh6GOC 8mlFjZwLxWFqdPHR1n2aPVgoiTLk34LR/bXO+e0GpzFXT7enwyvFFFyAS0Nk1q/7EChPcbRb hJqEBpRNZemxmg55zC3GLvgLKd5A09MOM2BrMea+l0FUR+PuTenh2YmnmLRTro6eZ/qYwWkC u8FFIw4pT0OUDMyLgi+GI1aMpVogTZJ70FgV0pUAlpmrzk/bLbRkF3TwgucpyPtcpmQtTkWS gDS50QG9DR/1As3LLLcNkwJBZzBG6PWbvcOyrwMQUF1nl4SSPV0LLH63+BrrHasfJzxKXzqg rW28CTAE2x8qi7e/6M/+XXhrsMYG+uaViM7n2je3qKe7ofum3s4vq7oFCPsOgwARAQABwsFl BBgBAgAPBQJVy5+RAhsMBQkJZgGAAAoJEE3eEPcA/4NagOsP/jPoIBb/iXVbM+fmSHOjEshl KMwEl/m5iLj3iHnHPVLBUWrXPdS7iQijJA/VLxjnFknhaS60hkUNWexDMxVVP/6lbOrs4bDZ NEWDMktAeqJaFtxackPszlcpRVkAs6Msn9tu8hlvB517pyUgvuD7ZS9gGOMmYwFQDyytpepo YApVV00P0u3AaE0Cj/o71STqGJKZxcVhPaZ+LR+UCBZOyKfEyq+ZN311VpOJZ1IvTExf+S/5 lqnciDtbO3I4Wq0ArLX1gs1q1XlXLaVaA3yVqeC8E7kOchDNinD3hJS4OX0e1gdsx/e6COvy qNg5aL5n0Kl4fcVqM0LdIhsubVs4eiNCa5XMSYpXmVi3HAuFyg9dN+x8thSwI836FoMASwOl C7tHsTjnSGufB+D7F7ZBT61BffNBBIm1KdMxcxqLUVXpBQHHlGkbwI+3Ye+nE6HmZH7IwLwV W+Ajl7oYF+jeKaH4DZFtgLYGLtZ1LDwKPjX7VAsa4Yx7S5+EBAaZGxK510MjIx6SGrZWBrrV TEvdV00F2MnQoeXKzD7O4WFbL55hhyGgfWTHwZ457iN9SgYi1JLPqWkZB0JRXIEtjd4JEQcx +8Umfre0Xt4713VxMygW0PnQt5aSQdMD58jHFxTk092mU+yIHj5LeYgvwSgZN4airXk5yRXl SE+xAvmumFBY
  • Cc: Kate Stewart <kstewart@xxxxxxxxxxxxxxxxxxx>, Michal Hocko <mhocko@xxxxxxxx>, linux-doc@xxxxxxxxxxxxxxx, Benjamin Herrenschmidt <benh@xxxxxxxxxxxxxxxxxxx>, Heiko Carstens <heiko.carstens@xxxxxxxxxx>, linux-mm@xxxxxxxxx, Paul Mackerras <paulus@xxxxxxxxx>, Rashmica Gupta <rashmica.g@xxxxxxxxx>, "K. Y. Srinivasan" <kys@xxxxxxxxxxxxx>, Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>, Michael Neuling <mikey@xxxxxxxxxxx>, Stephen Hemminger <sthemmin@xxxxxxxxxxxxx>, Jonathan Corbet <corbet@xxxxxxx>, Michael Ellerman <mpe@xxxxxxxxxxxxxx>, Pavel Tatashin <pasha.tatashin@xxxxxxxxxx>, linux-acpi@xxxxxxxxxxxxxxx, xen-devel@xxxxxxxxxxxxxxxxxxxx, Len Brown <lenb@xxxxxxxxxx>, Pavel Tatashin <pavel.tatashin@xxxxxxxxxxxxx>, Haiyang Zhang <haiyangz@xxxxxxxxxxxxx>, YASUAKI ISHIMATSU <yasu.isimatu@xxxxxxxxx>, Nathan Fontenot <nfont@xxxxxxxxxxxxxxxxxx>, Dan Williams <dan.j.williams@xxxxxxxxx>, Joonsoo Kim <iamjoonsoo.kim@xxxxxxx>, Vlastimil Babka <vbabka@xxxxxxx>, Oscar Salvador <osalvador@xxxxxxx>, Juergen Gross <jgross@xxxxxxxx>, Mathieu Malaterre <malat@xxxxxxxxxx>, Greg Kroah-Hartman <gregkh@xxxxxxxxxxxxxxxxxxx>, "Rafael J. Wysocki" <rjw@xxxxxxxxxxxxx>, linux-kernel@xxxxxxxxxxxxxxx, John Allen <jallen@xxxxxxxxxxxxxxxxxx>, Thomas Gleixner <tglx@xxxxxxxxxxxxx>, Philippe Ombredanne <pombredanne@xxxxxxxx>, Martin Schwidefsky <schwidefsky@xxxxxxxxxx>, devel@xxxxxxxxxxxxxxxxxxxxxx, Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>, linuxppc-dev@xxxxxxxxxxxxxxxx
  • Delivery-date: Wed, 19 Sep 2018 07:35:41 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Openpgp: preference=signencrypt

Am 19.09.18 um 03:22 schrieb Balbir Singh:
> On Tue, Sep 18, 2018 at 01:48:16PM +0200, David Hildenbrand wrote:
>> Reading through the code and studying how mem_hotplug_lock is to be used,
>> I noticed that there are two places where we can end up calling
>> device_online()/device_offline() - online_pages()/offline_pages() without
>> the mem_hotplug_lock. And there are other places where we call
>> device_online()/device_offline() without the device_hotplug_lock.
>>
>> While e.g.
>>      echo "online" > /sys/devices/system/memory/memory9/state
>> is fine, e.g.
>>      echo 1 > /sys/devices/system/memory/memory9/online
>> Will not take the mem_hotplug_lock. However the device_lock() and
>> device_hotplug_lock.
>>
>> E.g. via memory_probe_store(), we can end up calling
>> add_memory()->online_pages() without the device_hotplug_lock. So we can
>> have concurrent callers in online_pages(). We e.g. touch in online_pages()
>> basically unprotected zone->present_pages then.
>>
>> Looks like there is a longer history to that (see Patch #2 for details),
>> and fixing it to work the way it was intended is not really possible. We
>> would e.g. have to take the mem_hotplug_lock in device/base/core.c, which
>> sounds wrong.
>>
>> Summary: We had a lock inversion on mem_hotplug_lock and device_lock().
>> More details can be found in patch 3 and patch 6.
>>
>> I propose the general rules (documentation added in patch 6):
>>
>> 1. add_memory/add_memory_resource() must only be called with
>>    device_hotplug_lock.
>> 2. remove_memory() must only be called with device_hotplug_lock. This is
>>    already documented and holds for all callers.
>> 3. device_online()/device_offline() must only be called with
>>    device_hotplug_lock. This is already documented and true for now in core
>>    code. Other callers (related to memory hotplug) have to be fixed up.
>> 4. mem_hotplug_lock is taken inside of add_memory/remove_memory/
>>    online_pages/offline_pages.
>>
>> To me, this looks way cleaner than what we have right now (and easier to
>> verify). And looking at the documentation of remove_memory, using
>> lock_device_hotplug also for add_memory() feels natural.
>>
> 
> That seems reasonable, but also implies that device_online() would hold
> back add/remove memory, could you please also document what mode
> read/write the locks need to be held? For example can the device_hotplug_lock
> be held in read mode while add/remove memory via (mem_hotplug_lock) is held
> in write mode?

device_hotplug_lock is an ordinary mutex. So no option there.

Only mem_hotplug_lock is a per CPU RW mutex. And as of now it only
exists to not require get_online_mems()/put_online_mems() to take the
device_hotplug_lock. Which is perfectly valid, because these users only
care about memory (not any other devices) not suddenly vanish. And that
RW lock makes things fast.

Any modifications (online/offline/add/remove) require the
mem_hotplug_lock in write.

I can add some more details to documentation in patch #6.

"... we should always hold the mem_hotplug_lock (via
mem_hotplug_begin/mem_hotplug_done) in write mode to serialize memory
hotplug" ..."

"In addition, mem_hotplug_lock (in contrast to device_hotplug_lock) in
read mode allows for a quite efficient get_online_mems/put_online_mems
implementation, so code accessing memory can protect from that memory
vanishing."

Would that work for you?

Thanks!

> 
> Balbir Singh.
>  
> 


-- 

Thanks,

David / dhildenb

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.