Re: [Xen-devel] Guest soft lockups with "xen: make xen_qlock_wait() nestable"

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

On 08/11/18 09:18, Juergen Gross wrote:
> On 08/11/2018 09:14, Sander Eikelenboom wrote:
>> On 08/11/18 08:08, Juergen Gross wrote:
>>> On 07/11/2018 10:30, Sander Eikelenboom wrote:
>>>> Hi Juergen / Boris,
>>>>
>>>> Last week i tested Linux kernel 4.19.0 stable with the Xen 
>>>> "for-linus-4.20" branch pulled on top.
>>>> Unfortunately i was seeing guests lockup after some time, see below for 
>>>> the logging from one of the guest
>>>> which i was able to capture.
>>>> Reverting "xen: make xen_qlock_wait() nestable" 
>>>> 7250f6d35681dfc44749d90598a2d51a118ce2b8,
>>>> made the lockups disappear.
>>>>
>>>> These guests are stressed quite hard in both CPU and networking, 
>>>> so they are probably more susceptible to locking issues.
>>>>
>>>> System is a AMD phenom x6, running Xen-unstable.
>>>>
>>>> Any ideas ?
>>>
>>> Just checked the hypervisor again: it seems a pending interrupt for a
>>> HVM/PVH vcpu won't let SCHEDOP_poll return in case interrupts are
>>> disabled.
>>>
>>> I need to rework the patch for that scenario. Until then I'll revert
>>> it.
>>
>> Thanks for looking into it.
> 
> Could you try the attached patch (on top of 7250f6d35681df)?

That blows up while booting the guest:

[    1.792870] installing Xen timer for CPU 1
[    1.796171] x86: Booting SMP configuration:
[    1.799410] .... node  #0, CPUs:      #1
[    1.882922] cpu 1 spinlock event irq 59
[    1.899446] installing Xen timer for CPU 2
[    1.902864]  #2
[    1.986248] cpu 2 spinlock event irq 65
[    1.996200] installing Xen timer for CPU 3
[    1.999522]  #3
[    2.082921] cpu 3 spinlock event irq 71
[    2.092749] smp: Brought up 1 node, 4 CPUs
[    2.096079] smpboot: Max logical packages: 1
[    2.099410] smpboot: Total of 4 processors activated (25688.36 BogoMIPS)
[    2.102893] BUG: unable to handle kernel paging request at 0000000000014f90
[    2.106063] PGD 0 P4D 0 
[    2.106063] Oops: 0002 [#1] SMP NOPTI
[    2.106063] CPU: 1 PID: 16 Comm: migration/1 Not tainted 
4.19.0-20181108-doflr-xennext-vlan-ppp-blkmq-qlockpatch+ #1
[    2.106063] Hardware name: Xen HVM domU, BIOS 4.12-unstable 10/30/2018
[    2.106063] RIP: 0010:xen_qlock_wait+0x23/0x70
[    2.106063] Code: 1f 84 00 00 00 00 00 55 53 48 83 ec 08 65 8b 2d 63 33 ff 
7e 83 fd ff 74 32 65 8b 05 47 3f ff 7e a9 00 00 10 00 75 24 48 89 fb <f0> ff 05 
36 33 ff 7e 8b 05 30 33 ff 7e 83 f8 01 74 16 0f b6 03 40
[    2.106063] RSP: 0018:ffffc900006d3dc0 EFLAGS: 00010046
[    2.106063] RAX: 0000000080000001 RBX: ffffffff831a5a68 RCX: 0000000000000008
[    2.106063] RDX: ffff88010f7ef700 RSI: 0000000000000003 RDI: ffffffff831a5a68
[    2.106063] RBP: 000000000000003b R08: 0000000000000008 R09: 000000000000006c
[    2.106063] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[    2.106063] R13: 0000000000000100 R14: 0000000000000000 R15: 0000000000080000
[    2.106063] FS:  0000000000000000(0000) GS:ffff88010b280000(0000) 
knlGS:0000000000000000
[    2.106063] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    2.106063] CR2: 0000000000014f90 CR3: 0000000002a24000 CR4: 00000000000006e0
[    2.106063] Call Trace:
[    2.106063]  ? __switch_to_asm+0x40/0x70
[    2.106063]  __pv_queued_spin_lock_slowpath+0x248/0x280
[    2.106063]  _raw_spin_lock+0x18/0x20
[    2.106063]  prepare_set+0xc/0x90
[    2.106063]  generic_set_all+0x26/0x2e0
[    2.106063]  ? __switch_to_asm+0x40/0x70
[    2.106063]  mtrr_rendezvous_handler+0x34/0x60
[    2.106063]  multi_cpu_stop+0xb6/0xe0
[    2.106063]  ? cpu_stop_queue_work+0xd0/0xd0
[    2.106063]  cpu_stopper_thread+0x86/0x100
[    2.106063]  smpboot_thread_fn+0x109/0x160
[    2.106063]  kthread+0xee/0x120
[    2.106063]  ? sort_range+0x20/0x20
[    2.106063]  ? kthread_park+0x80/0x80
[    2.106063]  ret_from_fork+0x22/0x40
[    2.106063] Modules linked in:
[    2.106063] CR2: 0000000000014f90
[    2.106063] BUG: unable to handle kernel paging request at 0000000000014f90
[    2.106063] ---[ end trace e5be82cfc3e40a5e ]---
[    2.106063] PGD 0 
[    2.106063] RIP: 0010:xen_qlock_wait+0x23/0x70
[    2.106063] P4D 0 
[    2.106063] Code: 1f 84 00 00 00 00 00 55 53 48 83 ec 08 65 8b 2d 63 33 ff 
7e 83 fd ff 74 32 65 8b 05 47 3f ff 7e a9 00 00 10 00 75 24 48 89 fb <f0> ff 05 
36 33 ff 7e 8b 05 30 33 ff 7e 83 f8 01 74 16 0f b6 03 40
[    2.106063] Oops: 0002 [#2] SMP NOPTI
[    2.106063] RSP: 0018:ffffc900006d3dc0 EFLAGS: 00010046


> 
> Juergen
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel