Re: [Xen-devel] xen/arm: Domain not fully destroyed when using credit2

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 25/01/17 12:38, Julien Grall wrote:
> Hi Dario,
>
> On 25/01/17 11:10, Dario Faggioli wrote:
>> On Tue, 2017-01-24 at 15:06 +0000, Julien Grall wrote:
>>> On 24/01/17 14:16, Dario Faggioli wrote:
>>>> There, we have tracing (BTW, did that made it to ARM eventually?)
>>>> and
>>>> there's TRC_PM_IDLE_ENTRY/EXIT which do pretty much the same of
>>>> your
>>>> printk-s.
>>>
>>> There is patch on the ML for xentrace support (see [1]) but nothing
>>> has
>>> been upstreamed yet. Waiting for a new version from the contributor.
>>>
>> Yep, that was I was remembering, and referring to. Thanks for the
>> update.
>>
>>>> And if I look at it, I do see even totally idle (from the scheduler
>>>> point of view) pCPUs, I indeed see them going back and forth from
>>>> and
>>>> to C3.
>>>
>>> My knowledge on x86 is limited. When does a CPU decides to leave the
>>> idle mode?
>>>
>> I'm not an expert of that part either. Jan and Andrew for sure know
>> best how monitor/mwait works (both in general, and our own
>> implementation).
>>
>> What I know (and can quickly infer from glancing at the code), is that
>> timers are certainly involved.
>>
>> In fact, we wake up when the most imminent timer would expire (see
>> mwait_idle_with_hints()), and a timer set by the scheduler fully
>> qualifies as being the one (if it's the most imminent).
>>
>> My point was that, still from scheduling perspective, neither Credit1
>> nor Credit2 sets a wakeup timer for idle pCPUs.
>>
>> Well, in Credit1, the master_ticker timer is never stopped (while,
>> e.g., the per-pCPU tick is stopped before entering deep sleep,
>> via sched_tick_suspend(), see commit 964fae8ac), but that's only 1
>> pCPU.
>
> The function sched_tick_suspend is never called on ARM. The power
> saving in Xen ARM is still very limited and this would need to be
> updated in the future.
>
> So I guess that's why I still see interrupt coming on the idle pCPU
> when credit1 is used. Looking at credit2, the callback tick_suspend is
> not called. Does it mean there is no per-pCPU timer?
>
> Now, from my understanding, if we decide to call sched_tick_suspend on
> ARM before idling. We will likely have the same problem with credit1
> because there is no more interrupt to wake-up the pCPU.
>
> But I don't think this is an issue in the scheduler. IHMO, the problem
> is in the RCU. Indeed a CPU in lower power mode (i.e  wfi on ARM or
> pm_idle on x86 is been executed) will never get out to tell to the RCU
> : "I am quiet, go ahead". So the RCU will never be able to reclaim the
> memory and will result on a memory exhaustion if the pCPU never
> receive an interrupt (this could happen if pCPU has never ran a guest).

Yes.  This is a core problem, not ARM specific.

x86 is saved by the time calibration rendezvous which IPIs all cores
every 1s.

>
> The question now, is how to fix it?


This is going to involve a better understanding of how RCU is supposed
to work.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel