Re: [Xen-devel] Live-Patch application failure in core-scheduling mode

[Jan Beulich <jbeulich@xxxxxxxx>]

On 07.02.2020 10:25, Jürgen Groß wrote:
> On 07.02.20 09:49, Jan Beulich wrote:
>> On 07.02.2020 09:42, Jürgen Groß wrote:
>>> On 07.02.20 09:23, Jan Beulich wrote:
>>>> On 07.02.2020 09:04, Jürgen Groß wrote:
>>>>> On 06.02.20 15:02, Sergey Dyasli wrote:
>>>>>> On 06/02/2020 11:05, Sergey Dyasli wrote:
>>>>>>> On 06/02/2020 09:57, Jürgen Groß wrote:
>>>>>>>> On 05.02.20 17:03, Sergey Dyasli wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I'm currently investigating a Live-Patch application failure in core-
>>>>>>>>> scheduling mode and this is an example of what I usually get:
>>>>>>>>> (it's easily reproducible)
>>>>>>>>>
>>>>>>>>>         (XEN) [  342.528305] livepatch: lp: CPU8 - IPIing the other 
>>>>>>>>> 15 CPUs
>>>>>>>>>         (XEN) [  342.558340] livepatch: lp: Timed out on semaphore in 
>>>>>>>>> CPU quiesce phase 13/15
>>>>>>>>>         (XEN) [  342.558343] bad cpus: 6 9
>>>>>>>>>
>>>>>>>>>         (XEN) [  342.559293] CPU:    6
>>>>>>>>>         (XEN) [  342.559562] Xen call trace:
>>>>>>>>>         (XEN) [  342.559565]    [<ffff82d08023f304>] R 
>>>>>>>>> common/schedule.c#sched_wait_rendezvous_in+0xa4/0x270
>>>>>>>>>         (XEN) [  342.559568]    [<ffff82d08023f8aa>] F 
>>>>>>>>> common/schedule.c#schedule+0x17a/0x260
>>>>>>>>>         (XEN) [  342.559571]    [<ffff82d080240d5a>] F 
>>>>>>>>> common/softirq.c#__do_softirq+0x5a/0x90
>>>>>>>>>         (XEN) [  342.559574]    [<ffff82d080278ec5>] F 
>>>>>>>>> arch/x86/domain.c#guest_idle_loop+0x35/0x60
>>>>>>>>>
>>>>>>>>>         (XEN) [  342.559761] CPU:    9
>>>>>>>>>         (XEN) [  342.560026] Xen call trace:
>>>>>>>>>         (XEN) [  342.560029]    [<ffff82d080241661>] R 
>>>>>>>>> _spin_lock_irq+0x11/0x40
>>>>>>>>>         (XEN) [  342.560032]    [<ffff82d08023f323>] F 
>>>>>>>>> common/schedule.c#sched_wait_rendezvous_in+0xc3/0x270
>>>>>>>>>         (XEN) [  342.560036]    [<ffff82d08023f8aa>] F 
>>>>>>>>> common/schedule.c#schedule+0x17a/0x260
>>>>>>>>>         (XEN) [  342.560039]    [<ffff82d080240d5a>] F 
>>>>>>>>> common/softirq.c#__do_softirq+0x5a/0x90
>>>>>>>>>         (XEN) [  342.560042]    [<ffff82d080279db5>] F 
>>>>>>>>> arch/x86/domain.c#idle_loop+0x55/0xb0
>>>>>>>>>
>>>>>>>>> The first HT sibling is waiting for the second in the LP-application
>>>>>>>>> context while the second waits for the first in the scheduler context.
>>>>>>>>>
>>>>>>>>> Any suggestions on how to improve this situation are welcome.
>>>>>>>>
>>>>>>>> Can you test the attached patch, please? It is only tested to boot, so
>>>>>>>> I did no livepatch tests with it.
>>>>>>>
>>>>>>> Thank you for the patch! It seems to fix the issue in my manual testing.
>>>>>>> I'm going to submit automatic LP testing for both thread/core modes.
>>>>>>
>>>>>> Andrew suggested to test late ucode loading as well and so I did.
>>>>>> It uses stop_machine() to rendezvous cpus and it failed with a similar
>>>>>> backtrace for a problematic CPU. But in this case the system crashed
>>>>>> since there is no timeout involved:
>>>>>>
>>>>>>        (XEN) [  155.025168] Xen call trace:
>>>>>>        (XEN) [  155.040095]    [<ffff82d0802417f2>] R 
>>>>>> _spin_unlock_irq+0x22/0x30
>>>>>>        (XEN) [  155.069549]    [<ffff82d08023f3c2>] S 
>>>>>> common/schedule.c#sched_wait_rendezvous_in+0xa2/0x270
>>>>>>        (XEN) [  155.109696]    [<ffff82d08023f728>] F 
>>>>>> common/schedule.c#sched_slave+0x198/0x260
>>>>>>        (XEN) [  155.145521]    [<ffff82d080240e1a>] F 
>>>>>> common/softirq.c#__do_softirq+0x5a/0x90
>>>>>>        (XEN) [  155.180223]    [<ffff82d0803716f6>] F 
>>>>>> x86_64/entry.S#process_softirqs+0x6/0x20
>>>>>>
>>>>>> It looks like your patch provides a workaround for LP case, but other
>>>>>> cases like stop_machine() remain broken since the underlying issue with
>>>>>> the scheduler is still there.
>>>>>
>>>>> And here is the fix for ucode loading (that was in fact the only case
>>>>> where stop_machine_run() wasn't already called in a tasklet).
>>>>
>>>> This is a rather odd restriction, and hence will need explaining.
>>>
>>> stop_machine_run() is using a tasklet on each online cpu (excluding the
>>> one it was called one) for doing a rendezvous of all cpus. With tasklets
>>> always being executed on idle vcpus it is mandatory for
>>> stop_machine_run() to be called on an idle vcpu as well when core
>>> scheduling is active, as otherwise a deadlock will occur. This is being
>>> accomplished by the use of continue_hypercall_on_cpu().
>>
>> Well, it's this "a deadlock" which is too vague for me. What exactly is
>> it that deadlocks, and where (if not obvious from the description of
>> that case) is the connection to core scheduling? Fundamentally such an
>> issue would seem to call for an adjustment to core scheduling logic,
>> not placing of new restrictions on other pre-existing code.
> 
> This is the main objective of core scheduling: on all siblings of a
> core only vcpus of exactly one domain are allowed to be active.
> 
> As tasklets are only running on idle vcpus and stop_machine_run()
> is activating tasklets on all cpus but the one it has been called on
> to rendezvous, it is mandatory for stop_machine_run() to be called on
> an idle vcpu, too, as otherwise there is no way for scheduling to
> activate the idle vcpu for the tasklet on the sibling of the cpu
> stop_machine_run() has been called on.

I can follow all this, but it needs spelling out in the description
of the patch, I think. "only running on idle vcpus" isn't very
precise though, as this ignores softirq tasklets. Which got me to
think of an alternative (faod: without having thought through at
all whether this would indeed be viable): What if stop-machine used
softirq tasklets instead of "ordinary" ones?

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel