[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] xen/arm: Domain not fully destroyed when using credit2



Hi Jan,

On 24/01/2017 08:20, Jan Beulich wrote:
On 23.01.17 at 20:42, <julien.grall@xxxxxxx> wrote:
Whilst testing other patches today, I have noticed that some part of the
resources allocated to a guest were not released during the destruction.

The configuration of the test is:
        - ARM platform with 6 cores
        - staging Xen with credit2 enabled by default
        - DOM0 using 2 pinned vCPUs

The test is creating a guest vCPUs and then destroyed. After the test,
some resourced are not released (or could be released a long time
after).

Looking at the code, domain resources are released in 2 phases:
        - domain_destroy: called when there is no more reference on the domain
(see put_domain)
        - complete_domain_destroy: called when the RCU is quiescent

The function domain_destroy will setup the RCU callback
(complete_domain_destroy) by calling call_rcu. call_rcu will add the
callback into the RCU list and then will may send an IPI (see
force_quiescent_state) if the threshold reached. This IPI is here to
make sure all CPUs are quiescent before calling the callbacks (e.g
complete_domain_destroy). In my case, the threshold has not reached and
therefore an IPI is not sent.

But wait - isn't it the nature of RCU that it may take arbitrary time
until the actual call(s) happen(s)?

Today this arbitrary time could be infinite if an idle pCPU does not receive an interrupt. So some part of domain resource will never be freed.

If I am power-cycling a domain in loop, after some time the toolstack will fail to allocate memory because of exhausted resources. Previous instance of the domain was not yet fully destroyed (e.g complete_domain_destroy was not called).

If an upper limit is required by
a user of RCU, I think it would need to be that entity to arrange
for early expiry.

This is happening with all the user and not only a domain. Looking at the code, there are already some upper limit: - call_rcu will call force_quiescent_state if the number of element in the RCU queue is > 10000 - the RCU has a grace period (not sure how long), but no timer to ensure the RCU will be called

Reducing the threshold in call_rcu (see qhimark) will not help as you may still face memory exhaustion (see above). So I think the only best solution is to actually implement properly the grace period.

I notice in this context that we don't even have
synchronize_rcu() in our sources.

I don't think this is a problem here if we handle properly the grace period.

Regards,

--
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.