[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Core Scheduling "lock == schedule_lock" assertion failure

  • To: Xen-devel <xen-devel@xxxxxxxxxxxxx>, Juergen Gross <jgross@xxxxxxxx>
  • From: Sergey Dyasli <sergey.dyasli@xxxxxxxxxx>
  • Date: Wed, 12 Feb 2020 11:21:54 +0000
  • Authentication-results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=sergey.dyasli@xxxxxxxxxx; spf=Pass smtp.mailfrom=sergey.dyasli@xxxxxxxxxx; spf=None smtp.helo=postmaster@xxxxxxxxxxxxxxx
  • Autocrypt: addr=sergey.dyasli@xxxxxxxxxx; keydata= xsFNBFtMVHEBEADc/hZcLexrB6vGTdGqEUsYZkFGQh6Z1OO7bCtM1go1RugSMeq9tkFHQSOc 9c7W9NVQqLgn8eefikIHxgic6tGgKoIQKcPuSsnqGao2YabsTSSoeatvmO5HkR0xGaUd+M6j iqv3cD7/WL602NhphT4ucKXCz93w0TeoJ3gleLuILxmzg1gDhKtMdkZv6TngWpKgIMRfoyHQ jsVzPbTTjJl/a9Cw99vuhFuEJfzbLA80hCwhoPM+ZQGFDcG4c25GQGQFFatpbQUhNirWW5b1 r2yVOziSJsvfTLnyzEizCvU+r/Ek2Kh0eAsRFr35m2X+X3CfxKrZcePxzAf273p4nc3YIK9h cwa4ZpDksun0E2l0pIxg/pPBXTNbH+OX1I+BfWDZWlPiPxgkiKdgYPS2qv53dJ+k9x6HkuCy i61IcjXRtVgL5nPGakyOFQ+07S4HIJlw98a6NrptWOFkxDt38x87mSM7aSWp1kjyGqQTGoKB VEx5BdRS5gFdYGCQFc8KVGEWPPGdeYx9Pj2wTaweKV0qZT69lmf/P5149Pc81SRhuc0hUX9K DnYBa1iSHaDjifMsNXKzj8Y8zVm+J6DZo/D10IUxMuExvbPa/8nsertWxoDSbWcF1cyvZp9X tUEukuPoTKO4Vzg7xVNj9pbK9GPxSYcafJUgDeKEIlkn3iVIPwARAQABzShTZXJnZXkgRHlh c2xpIDxzZXJnZXkuZHlhc2xpQGNpdHJpeC5jb20+wsGlBBMBCgA4FiEEkI7HMI5EbM2FLA1L Aa+w5JvbyusFAltMVHECGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AAIQkQAa+w5JvbyusW IQSQjscwjkRszYUsDUsBr7Dkm9vK65AkEACvL+hErqbQj5yTVNqvP1rVGsXvevViglSTkHD4 9LGwEk4+ne8N4DPcqrDnyqYFd42UxTjVyoDEXEIIoy0RHWCmaspYEDX8fVmgFG3OFoeA9NAv JHssHU6B2mDAQ6M3VDmAwTw+TbXL/c1wblgGAP9kdurydZL8bevTTUh7edfnm5pwaT9HLXvl xLjz5qyt6tKEowM0xPVzCKaj3Mf/cuZFOlaWiHZ0biOPC0JeoHuz4UQTnBBUKk+n2nnn72k9 37cNeaxARwn/bxcej9QlbrrdaNGVFzjCA/CIL0KjUepowpLN0+lmYjkPgeLNYfyMXumlSNag 9qnCTh0QDsCXS/HUHPeBskAvwNpGBCkfiP/XqJ+V618ZQ1sclHa9aWNnlIR/a8xVx25t/14V R8EX/045HUpyPU8hI/yw+Fw/ugJ8W0dFzFeHU5K2tEW2W0m3ZWWWgpcBSCB17DDLIPjGX1Qc J8jiVJ7E4rfvA1JBg9BxVw5LVuXg2FB6bqnDYALfY2ydATk+ZzMUAMMilaE7/5a2RMV4TYcd 8Cf77LdgO0pB3vF6z1QmNA2IbOICtJOXpmvHj+dKFUt5hFVbvqXbuAjlrwFktbAFVGxaeIYz nQ44lQu9JqDuSH5yOytdek24Dit8SgEHGvumyj17liCG6kNzxd+2xh3uaUCA5MIALy5mZ87B TQRbTFRxARAAwqL3u/cPDA+BhU9ghtAkC+gyC5smWUL1FwTQ9CwTqcQpKt85PoaHn8sc5ctt Aj2fNT/F2vqQx/BthVOdkhj9LCwuslqBIqbri3XUyMLVV/Tf+ydzHW2AjufCowwgBguxedD1 f9Snkv+As7ZgMg/GtDqDiCWBFg9PneKvr+FPPd2WmrI8Kium4X5Zjs/a6OGUWVcIBoPpu088 z/0tlKYjTFLhoIEsf6ll4KvRQZIyGxclg3RBEuN+wgMbKppdUf2DBXYeCyrrPx809CUFzcik O99drWti2CV1gF8bnbUvfCewxwqgVKtHl2kfsm2+/lgG4CTyvnvWqUyHICZUqISdz5GidaXn TcPlsAeo2YU2NXbjwnmxzJEP/4FxgsjYIUbbxdmsK+PGre7HmGmaDZ8K77L3yHr/K7AH8mFs WUM5KiW4SnKyIQvdHkZMpvE4XrrirlZ+JI5vE043GzzpS2CGo0NFQmDJLRbpN/KQY6dkNVgA L0aDxJtAO1rXKYDSrvpL80bYyskQ4ivUa06v9SM2/bHi9bnp3Nf/fK6ErWKWmDOHWrnTgRML oQpcxoVPxw2CwyWT1069Y/CWwgnbj34+LMwMUYhPEZMitABpQE74dEtIFh0c2scm3K2QGhOP KQK3szqmXuX6MViMZLDh/B7FXLQyqwMBnZygfzZFM9vpDskAEQEAAcLBjQQYAQoAIBYhBJCO xzCORGzNhSwNSwGvsOSb28rrBQJbTFRxAhsMACEJEAGvsOSb28rrFiEEkI7HMI5EbM2FLA1L Aa+w5Jvbyuvvbg//S3d1+XL568K5BTHXaYxSqCeMqYbV9rPhEHyk+rzKtwNXSbSO8x0xZutL gYV+nkW0KMPH5Bz3I1xiRKAkiX/JLcMfx2HAXJ1Cv2rpR6bxyCGBJmuwR68uMS/gKe6AWwTY q2kt1rtZPjGl9OwVoWGJKbu2pFBLWmLAnHlXOL6WDSE1Mz2Ah3jMHOaSyAgPu1XSNa600gMJ QrSxgbe7bW72gCjeHcrIjfv+uh5cZ5/J/edpWXRuE4Tz82nxudBIHE2vnQEoJrXOh2kAJiYs G+IllDqFKDPrnS0R3DenBNG0Ir8h9W6heETnhQUc9NDFCSr81Mp0fROdBfYZnQzgSZMjN2eY pkNEWshJER4ZYY+7hAmqI51HnsKuM46QINh00jJHRMykW3TBMlwnUFxZ0gplAecjCFC7g2zj g1qNxLnxMS4wCsyEVhCkPyYnS8zuoa4ZUH37CezD01Ph4O1saln5+M4blHCEAUpZIkTGpUoi SEwtoxu6EEUYfbcjWgzJCs023hbRykZlFALoRNCwVz/FnPuVu291jn9kjvCTEeE6g2dCtOrO ukuXzk1tIeeoggsU7AJ0bzP7QOEhEckaBbP4k6ic26LJGWNMinllePyEMXzsgmMHVN//8wDT NWaanhP/JZ1v5Mfn8s1chIqC0sJIw73RvvuBkOa+jx0OwW3RFoQ=
  • Cc: George Dunlap <George.Dunlap@xxxxxxxxxxxxx>, "sergey.dyasli@xxxxxxxxxx >> Sergey Dyasli" <sergey.dyasli@xxxxxxxxxx>, Dario Faggioli <dfaggioli@xxxxxxxx>
  • Delivery-date: Wed, 12 Feb 2020 11:22:08 +0000
  • Ironport-sdr: a/MhqKzNNVEAoCSCeqrexKV8baEHgBtkImm5ULX4+MH2GXKOs4ODEOyTgjTaNSj4x7HT3DMmVm AnnMtChjVbObC557NV+TUlMT0WOHiXas4fl6UlA7kSgobR7pOVCV7pCR/A73o0n/Zb3VDfHPbm 6xtwXXhDdT5KnYJgNiyOnlwtBdLMk7aMiT0oJh32bPvsejVuJrpS3OKpS1QwUrbcsPnCFZcYdk c51qIAPvq57TUHToq23x/tFbgYpYHSt2JdmeDfTT8bzdL0jhLUIYHwT+U2g8ILlDyPwAfVmMsu Kec=
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
Hi Juergen,

Recently our testing has found a host crash which is reproducible.
Do you have any idea what might be going on here?

(XEN) [175654.165126] Assertion 'lock == 
get_sched_res(i->res->master_cpu)->schedule_lock' failed at 
...ild/BUILD/xen-4.13.1/xen/include/xen/sched-if.h:269
(XEN) [175654.165133] ----[ Xen-4.13.1-9.0.3-d  x86_64  debug=y   Not tainted 
]----
(XEN) [175654.165136] CPU:    28
(XEN) [175654.165138] RIP:    e008:[<ffff82d08023d2d2>] 
vcpu_runstate_get+0x11e/0x14f
(XEN) [175654.165146] RFLAGS: 0000000000010083   CONTEXT: hypervisor (d0v4)
(XEN) [175654.165151] rax: ffff83403ff0d340   rbx: ffff83807cc97ac8   rcx: 
0000000000000006
(XEN) [175654.165154] rdx: 0000006fbf942000   rsi: ffff83400f8e1cd8   rdi: 
00000000107898e2
(XEN) [175654.165158] rbp: ffff83807cc97ab8   rsp: ffff83807cc97a88   r8:  
deadbeefdeadf00d
(XEN) [175654.165160] r9:  deadbeefdeadf00d   r10: 0000000000000000   r11: 
0000000000000000
(XEN) [175654.165164] r12: ffff83400fa6f000   r13: ffff83400f8c9778   r14: 
ffff82d0805c8008
(XEN) [175654.165167] r15: ffff832e30854ae0   cr0: 0000000080050033   cr4: 
0000000000362660
(XEN) [175654.165170] cr3: 0000002130811000   cr2: ffff88817f50b728
(XEN) [175654.165172] fsb: 00007f40a40da740   gsb: ffff88831d300000   gss: 
0000000000000000
(XEN) [175654.165175] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   
cs: e008
(XEN) [175654.165179] Xen code around <ffff82d08023d2d2> 
(vcpu_runstate_get+0x11e/0x14f):
(XEN) [175654.165181]  04 10 4c 3b 68 10 74 02 <0f> 0b 4c 89 ef e8 7e 5d 00 00 
48 8d 05 41 9d 38
(XEN) [175654.165192] Xen stack trace from rsp=ffff83807cc97a88:
(XEN) [175654.165194]    ffff83807cc97aa8 ffff83400fa75a60 0000000000000000 
ffff83807cc97da0
(XEN) [175654.165199]    0000000000000230 ffff83807cc97fff ffff83807cc97af8 
ffff82d08023d41f
(XEN) [175654.165204]    0000000000000001 00009fc1ac1cb2f4 00004840c423acdc 
00005780e7f9735a
(XEN) [175654.165207]    0000000000000000 0000000000000000 ffff83807cc97c98 
ffff82d0802ea9f7
(XEN) [175654.165211]    0000000000000000 00009fc1ac1c6b99 0000000500000007 
ffff83807cc97c10
(XEN) [175654.165215]    ffff83807cc97bb0 0000000000000020 0000000000000000 
0000000000000000
(XEN) [175654.165251]    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) [175654.165254]    0000000000000000 0000000000000000 0000000000000000 
aaaaaaaaaaaaaaaa
(XEN) [175654.165258]    ffff82d0805c8038 ffff82d0805c74a0 aaaaaaaa00000000 
aaaaaaaaaaaaaa00
(XEN) [175654.165263]    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) [175654.165266]    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) [175654.165269]    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) [175654.165273]    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) [175654.165276]    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) [175654.165279]    0000000000000000 0000000000000000 0000000000000000 
0000000000000000
(XEN) [175654.165283]    ffff83400f813000 ffff83807cc97d98 0000000000000000 
ffff82d0805cda80
(XEN) [175654.165287]    0000000000000230 ffff83807cc97fff ffff83807cc97cc8 
ffff82d08026d99b
(XEN) [175654.165291]    ffff83807cc97ef8 ffff83400f813000 ffff82d0805cda80 
0000000000000230
(XEN) [175654.165295]    ffff83807cc97e48 ffff82d080244573 00007f40a40e6000 
0000000000000206
(XEN) [175654.165300]    ffff82004006c000 0000000000000000 0000000000000000 
ffff82e08a815e80
(XEN) [175654.165304] Xen call trace:
(XEN) [175654.165306]    [<ffff82d08023d2d2>] R vcpu_runstate_get+0x11e/0x14f
(XEN) [175654.165310]    [<ffff82d08023d41f>] F get_cpu_idle_time+0x4d/0x53
(XEN) [175654.165315]    [<ffff82d0802ea9f7>] F pmstat_get_cx_stat+0x82/0x8e7
(XEN) [175654.165319]    [<ffff82d08026d99b>] F do_get_pm_info+0x27b/0x2d4
(XEN) [175654.165322]    [<ffff82d080244573>] F do_sysctl+0x633/0x14e0
(XEN) [175654.165327]    [<ffff82d080382335>] F pv_hypercall+0x1f5/0x567
(XEN) [175654.165330]    [<ffff82d080389432>] F lstar_enter+0x112/0x120
(XEN) [175654.165332]
(XEN) [175654.550916]
(XEN) [175654.553243] ****************************************
(XEN) [175654.559449] Panic on CPU 28:
(XEN) [175654.563328] Assertion 'lock == 
get_sched_res(i->res->master_cpu)->schedule_lock' failed at 
...ild/BUILD/xen-4.13.1/xen/include/xen/sched-if****************************************
(XEN) [175654.581847]
(XEN) [175654.584173] Reboot in five seconds...
(XEN) [175654.588925] Executing kexec image on cpu28
(XEN) [175654.594987] Shot down all CPUs


The state of the sibling was:


  PCPU 29 Host state:
        RIP:    e008:[<ffff82d080219fb0>] Ring 0
        RFLAGS: 0000000000040002  AC IOPL0

        rax: ffff83400f8c91e4   rbx: 000000000000001d   rcx: ffff83400f8c91f4
        rdx: ffff83400f8c9104   rsi: ffff83400f8c9094   rdi: 0000000000000004
        rbp: ffff83807cc89f28   rsp: ffff83807cc89f28   r8:  0000000000000000
        r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
        r12: 0000000000000000   r13: 0000000000000000   r14: ffff83807cc8ffff
        r15: 0000000000000000

        cr0: 0000000080050033   PG AM WP NE ET MP PE
        cr3: 000000406e5ff000   cr2: 0000000000900030
        cr4: 0000000000362660   SMEP OSXSAVE PCIDE VMXE OSXMMEXCPT OSFXSR MCE 
PAE

        ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008

        stack current VCPU  ffff83400f80f000 DOM0 VCPU5
        percpu current VCPU ffff83400f80f000 DOM0 VCPU5
        VCPU was RUNNING

        Stack at ffff83807cc89f28:
          ffff83807cc89f20:                  ffff83807cc89f48 ffff82d0802758bb 
ffff82d080389d84
          ffff83807cc89f40: 0000000000000000 00007c7f83376087 ffff82d080389e21 
ffff83400f861060
          ffff83807cc89f60: 000000000000001d ffff82d0805ec5a0 ffff83400f8f09ae 
ffff83807cc8fd78
          ffff83807cc89f80: ffff83400f8f09a8 0000000000000000 0000000000000000 
ffff83400f8e1c20
          ffff83807cc89fa0: 0000000000000000 0000000000008326 0000000000000000 
0000000000000001
          ffff83807cc89fc0: ffff82d0805c8326 ffff83400f8f09ae 0000000200000000 
ffff82d080242e50
          ffff83807cc89fe0: 000000000000e008 0000000000000046 ffff83807cc8fd60 
000000000000e010

        Code:
           5b 41 5c 5d c3 66 2e 0f 1f 84 00 00 00 00 00 <55> 48 89 e5 4c 89 3f 
4c 89 77 08 4c 89 6f 10 4c 89

        Call Trace:
         [ffff82d080219fb0] elf_core_save_regs+0/0xae
          ffff82d0802758bb  do_nmi_crash+0x8b/0xf4
          ffff82d080389d84  handle_ist_exception+0xaa/0x1b6
          ffff82d080389e21  handle_ist_exception+0x147/0x1b6

              NMI interrupted Code at e008:ffff82d080242e50 and Stack at 
e010:ffff83807cc8fd60

         [ffff82d080242e50] got_lock+0/0x23
          ffff82d080242fcb  _spin_lock+0x41/0x5e
          ffff82d080242ffb  _spin_lock_irq+0x13/0x15
          ffff82d080240bc5  sched_wait_rendezvous_in+0x25a/0x2cc
          ffff82d08024109b  schedule+0x1bc/0x2b4
          ffff82d0803893d4  lstar_enter+0xb4/0x120
          ffff82d080382335  pv_hypercall+0x1f5/0x567
          ffff82d0803893d4  lstar_enter+0xb4/0x120
          ffff82d0802425f5  __do_softirq+0x85/0x90
          ffff82d08024264a  do_softirq+0x13/0x15
          ffff82d080386c76  process_softirqs+0x6/0x20

--
Thanks,
Sergey

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.