[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] High CPU temp, suspend problem - xen 4.1.5-pre, linux 3.7.x



On 26/03/2013 18:21, Marek Marczykowski wrote:
> On 26.03.2013 18:54, Andrew Cooper wrote:
>>>> Can you replace the ASSERT() with code similar to that in
>>>>
>>>> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/irq.c;h=5e0f463c381750090373dabd8967635bc297d457;hb=refs/heads/staging#l668
>>>>
>>>> Which should call dump_irqs() in before dying because of the ASSERT. 
>>>> You might need to also take the latest version of dump_irqs() from
>>>> unstable, as I seem to remember there was another assertion failure due
>>>> to xfree()'ing in IRQ context.
>>> Full log here:
>>> http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs.log
>>> Interesting part:
> (...)
>> Even more curious.  vector e9 does not appear to be programmed in.  Can
>> you extend the debugging to also call __print_IO_APIC().
>>
>> The i debug key and z debug key list IO-APIC entries from different
>> sources of information.
> As you wish, full log:
> http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs2.log
>
> Final part:
> (XEN) *** IRQ BUG found ***
> (XEN) CPU0 -Testing vector 233 from bitmap
> 43,49,64,72,80,87-88,95-96,103,112,119-121,127,135,143-144,151-152,159-160,168,192,197,200,211,216,218
> (XEN) Guest interrupt information:
> (XEN)    IRQ:   0 affinity:00000000,00000000,00000000,00000001 vec:f0
> type=IO-APIC-edge    status=00000000 mapped, unbound
> (XEN)    IRQ:   1 affinity:00000000,00000000,00000000,00000001 vec:7f
> type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  1(-S--),
> (XEN)    IRQ:   2 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:e2
> type=XT-PIC          status=00000000 mapped, unbound
> (XEN)    IRQ:   3 affinity:00000000,00000000,00000000,00000001 vec:40
> type=IO-APIC-edge    status=00000002 mapped, unbound
> (XEN)    IRQ:   4 affinity:00000000,00000000,00000000,00000001 vec:f1
> type=IO-APIC-edge    status=00000000 mapped, unbound
> (XEN)    IRQ:   5 affinity:00000000,00000000,00000000,00000001 vec:48
> type=IO-APIC-edge    status=00000002 mapped, unbound
> (XEN)    IRQ:   6 affinity:00000000,00000000,00000000,00000001 vec:50
> type=IO-APIC-edge    status=00000002 mapped, unbound
> (XEN)    IRQ:   7 affinity:00000000,00000000,00000000,00000008 vec:da
> type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  7(-S--),
> (XEN)    IRQ:   8 affinity:00000000,00000000,00000000,00000004 vec:d8
> type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  8(-S--),
> (XEN)    IRQ:   9 affinity:00000000,00000000,00000000,00000001 vec:87
> type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0:  9(-S--),
> (XEN)    IRQ:  10 affinity:00000000,00000000,00000000,00000001 vec:70
> type=IO-APIC-edge    status=00000002 mapped, unbound
> (XEN)    IRQ:  11 affinity:00000000,00000000,00000000,00000001 vec:78
> type=IO-APIC-edge    status=00000002 mapped, unbound
> (XEN)    IRQ:  12 affinity:00000000,00000000,00000000,00000001 vec:8f
> type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0: 12(-S--),
> (XEN)    IRQ:  13 affinity:00000000,00000000,00000000,0000000f vec:90
> type=IO-APIC-edge    status=00000002 mapped, unbound
> (XEN)    IRQ:  14 affinity:00000000,00000000,00000000,00000001 vec:98
> type=IO-APIC-edge    status=00000002 mapped, unbound
> (XEN)    IRQ:  15 affinity:00000000,00000000,00000000,00000001 vec:a0
> type=IO-APIC-edge    status=00000002 mapped, unbound
> (XEN)    IRQ:  16 affinity:00000000,00000000,00000000,00000001 vec:97
> type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 16(-S--),
> (XEN)    IRQ:  17 affinity:00000000,00000000,00000000,00000001 vec:9f
> type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 17(-S--),
> (XEN)    IRQ:  18 affinity:00000000,00000000,00000000,00000004 vec:79
> type=IO-APIC-level   status=00000002 mapped, unbound
> (XEN)    IRQ:  19 affinity:00000000,00000000,00000000,0000000f vec:c8
> type=IO-APIC-level   status=00000002 mapped, unbound
> (XEN)    IRQ:  20 affinity:00000000,00000000,00000000,00000002 vec:d3
> type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 20(-S--),
> (XEN)    IRQ:  22 affinity:00000000,00000000,00000000,0000000f vec:2b
> type=IO-APIC-level   status=00000002 mapped, unbound
> (XEN)    IRQ:  23 affinity:00000000,00000000,00000000,0000000f vec:a8
> type=IO-APIC-edge    status=00000002 mapped, unbound
> (XEN)    IRQ:  24 affinity:00000000,00000000,00000000,00000001 vec:28
> type=DMA_MSI         status=00000000 mapped, unbound
> (XEN)    IRQ:  25 affinity:00000000,00000000,00000000,00000001 vec:30
> type=DMA_MSI         status=00000000 mapped, unbound
> (XEN)    IRQ:  26 affinity:00000000,00000000,00000000,00000001 vec:c7
> type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:279(-S--),
> (XEN)    IRQ:  27 affinity:00000000,00000000,00000000,00000001 vec:cf
> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:278(-S--),
> (XEN)    IRQ:  28 affinity:00000000,00000000,00000000,00000001 vec:d7
> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:277(-S--),
> (XEN)    IRQ:  29 affinity:00000000,00000000,00000000,00000001 vec:df
> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:276(-S--),
> (XEN)    IRQ:  30 affinity:00000000,00000000,00000000,00000001 vec:38
> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:275(-S--),
> (XEN)    IRQ:  31 affinity:00000000,00000000,00000000,00000004 vec:47
> type=PCI-MSI         status=00000002 mapped, unbound
> (XEN)    IRQ:  32 affinity:00000000,00000000,00000000,00000001 vec:a7
> type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:273(-S--),
> (XEN)    IRQ:  33 affinity:00000000,00000000,00000000,00000001 vec:b7
> type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:272(-S--),
> (XEN)    IRQ:  34 affinity:00000000,00000000,00000000,00000004 vec:40
> type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:271(PS--),
> (XEN)    IRQ:  35 affinity:00000000,00000000,00000000,00000001 vec:af
> type=PCI-MSI         status=00000050 in-flight=0 domain-list=1: 55(-S--),
> (XEN) IO-APIC interrupt information:
> (XEN)     IRQ  0 Vec240:
> (XEN)       Apic 0x00, Pin  2: vec=f0 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ  1 Vec127:
> (XEN)       Apic 0x00, Pin  1: vec=7f delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ  3 Vec 64:
> (XEN)       Apic 0x00, Pin  3: vec=40 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ  4 Vec241:
> (XEN)       Apic 0x00, Pin  4: vec=f1 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ  5 Vec 72:
> (XEN)       Apic 0x00, Pin  5: vec=48 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ  6 Vec 80:
> (XEN)       Apic 0x00, Pin  6: vec=50 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ  7 Vec218:
> (XEN)       Apic 0x00, Pin  7: vec=da delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ  8 Vec216:
> (XEN)       Apic 0x00, Pin  8: vec=d8 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ  9 Vec135:
> (XEN)       Apic 0x00, Pin  9: vec=87 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=L mask=0 dest_id:0
> (XEN)     IRQ 10 Vec112:
> (XEN)       Apic 0x00, Pin 10: vec=70 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ 11 Vec120:
> (XEN)       Apic 0x00, Pin 11: vec=78 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ 12 Vec143:
> (XEN)       Apic 0x00, Pin 12: vec=8f delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ 13 Vec144:
> (XEN)       Apic 0x00, Pin 13: vec=90 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=1 dest_id:0
> (XEN)     IRQ 14 Vec152:
> (XEN)       Apic 0x00, Pin 14: vec=98 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ 15 Vec160:
> (XEN)       Apic 0x00, Pin 15: vec=a0 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=0 dest_id:0
> (XEN)     IRQ 16 Vec151:
> (XEN)       Apic 0x00, Pin 16: vec=97 delivery=LoPri dest=L status=0
> polarity=1 irr=0 trig=L mask=0 dest_id:0
> (XEN)     IRQ 17 Vec159:
> (XEN)       Apic 0x00, Pin 17: vec=9f delivery=LoPri dest=L status=0
> polarity=1 irr=0 trig=L mask=0 dest_id:0
> (XEN)     IRQ 18 Vec121:
> (XEN)       Apic 0x00, Pin 18: vec=79 delivery=LoPri dest=L status=0
> polarity=1 irr=0 trig=L mask=1 dest_id:0
> (XEN)     IRQ 19 Vec200:
> (XEN)       Apic 0x00, Pin 19: vec=c8 delivery=LoPri dest=L status=0
> polarity=1 irr=0 trig=L mask=1 dest_id:0
> (XEN)     IRQ 20 Vec211:
> (XEN)       Apic 0x00, Pin 20: vec=d3 delivery=LoPri dest=L status=0
> polarity=1 irr=0 trig=L mask=0 dest_id:0
> (XEN)     IRQ 22 Vec 43:
> (XEN)       Apic 0x00, Pin 22: vec=2b delivery=LoPri dest=L status=0
> polarity=1 irr=0 trig=L mask=1 dest_id:0
> (XEN)     IRQ 23 Vec168:
> (XEN)       Apic 0x00, Pin 23: vec=a8 delivery=LoPri dest=L status=0
> polarity=0 irr=0 trig=E mask=1 dest_id:0
> (XEN) number of MP IRQ sources: 15.
> (XEN) number of IO-APIC #2 registers: 24.
> (XEN) testing the IO APIC.......................
> (XEN) IO APIC #2......
> (XEN) .... register #00: 02000000
> (XEN) .......    : physical APIC id: 02
> (XEN) .......    : Delivery Type: 0
> (XEN) .......    : LTS          : 0
> (XEN) .... register #01: 00170020
> (XEN) .......     : max redirection entries: 0017
> (XEN) .......     : PRQ implemented: 0
> (XEN) .......     : IO APIC version: 0020
> (XEN) .... IRQ redirection table:
> (XEN)  NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
> (XEN)  00 0DC 0C  1    0    0   0   0    1    2    87
> (XEN)  01 000 00  0    0    0   0   0    1    1    7F
> (XEN)  02 000 00  0    0    0   0   0    1    1    F0
> (XEN)  03 000 00  0    0    0   0   0    1    1    40
> (XEN)  04 000 00  0    0    0   0   0    1    1    F1
> (XEN)  05 000 00  0    0    0   0   0    1    1    48
> (XEN)  06 000 00  0    0    0   0   0    1    1    50
> (XEN)  07 000 00  0    0    0   0   0    1    1    DA
> (XEN)  08 000 00  0    0    0   0   0    1    1    D8
> (XEN)  09 000 00  0    1    0   0   0    1    1    87
> (XEN)  0a 000 00  0    0    0   0   0    1    1    70
> (XEN)  0b 000 00  0    0    0   0   0    1    1    78
> (XEN)  0c 000 00  0    0    0   0   0    1    1    8F
> (XEN)  0d 000 00  1    0    0   0   0    1    1    90
> (XEN)  0e 000 00  0    0    0   0   0    1    1    98
> (XEN)  0f 000 00  0    0    0   0   0    1    1    A0
> (XEN)  10 000 00  0    1    0   1   0    1    1    97
> (XEN)  11 000 00  0    1    0   1   0    1    1    9F
> (XEN)  12 000 00  1    1    0   1   0    1    1    79
> (XEN)  13 000 00  1    1    0   1   0    1    1    C8
> (XEN)  14 000 00  0    1    0   1   0    1    1    D3
> (XEN)  15 000 00  1    0    0   0   0    0    0    00
> (XEN)  16 000 00  1    1    0   1   0    1    1    2B
> (XEN)  17 000 00  1    0    0   0   0    1    1    A8
> (XEN) Using vector-based indexing
> (XEN) IRQ to pin mappings:
> (XEN) IRQ240 -> 0:2
> (XEN) IRQ127 -> 0:1
> (XEN) IRQ64 -> 0:3
> (XEN) IRQ241 -> 0:4
> (XEN) IRQ72 -> 0:5
> (XEN) IRQ80 -> 0:6
> (XEN) IRQ218 -> 0:7
> (XEN) IRQ216 -> 0:8
> (XEN) IRQ135 -> 0:9
> (XEN) IRQ112 -> 0:10
> (XEN) IRQ120 -> 0:11
> (XEN) IRQ143 -> 0:12
> (XEN) IRQ144 -> 0:13
> (XEN) IRQ152 -> 0:14
> (XEN) IRQ160 -> 0:15
> (XEN) IRQ151 -> 0:16
> (XEN) IRQ159 -> 0:17
> (XEN) IRQ121 -> 0:18
> (XEN) IRQ200 -> 0:19
> (XEN) IRQ211 -> 0:20
> (XEN) IRQ43 -> 0:22
> (XEN) IRQ168 -> 0:23
> (XEN) .................................... done.
> (XEN) Xen BUG at io_apic.c:556
> (XEN) ----[ Xen-4.1.5-rc1  x86_64  debug=y  Tainted:    C ]----
> (XEN) CPU:    0
> (XEN) RIP:    e008:[<ffff82c48015e2db>] 
> smp_irq_move_cleanup_interrupt+0x216/0x28e
> (XEN) RFLAGS: 0000000000010092   CONTEXT: hypervisor
> (XEN) rax: 0000000000000000   rbx: 00000000000000e9   rcx: 0000000000000000
> (XEN) rdx: 0000000000000000   rsi: 000000000000000a   rdi: ffff82c4802592e0
> (XEN) rbp: ffff82c48029ff08   rsp: ffff82c48029feb8   r8:  0000000000000004
> (XEN) r9:  0000000000000004   r10: 0000000000000004   r11: 0000000000000002
> (XEN) r12: ffff830421080250   r13: ffff830421060534   r14: ffff82c48029ff18
> (XEN) r15: ffff82c4802dd9e0   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) cr3: 000000026582c000   cr2: ffff8804020701d8
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
> (XEN) Xen stack trace from rsp=ffff82c48029feb8:
> (XEN)    0000000000000000 ffff82c48029ff18 ffff82c4802dd9e0 000000e900000000
> (XEN)    000000000000e02b 0000000000000000 000000004bf51982 00000000000060a9
> (XEN)    0000000000000000 0000000000000000 00007d3b7fd600c7 ffff82c48014de60
> (XEN)    0000000000000000 0000000000000000 00000000000060a9 000000004bf51982
> (XEN)    ffff8802d2665b28 0000000000000000 0000000000000000 0000000000007ff0
> (XEN)    0000000000000022 0000000000000000 000000024bf57322 0000000001307da0
> (XEN)    00000000000059a0 0000000000000000 00000000000060a9 0000002000000000
> (XEN)    ffffffff8123c51a 000000000000e033 0000000000000293 ffff8802d2665b08
> (XEN)    000000000000e02b 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 ffff8300ca9a0000 0000000000000000
> (XEN)    0000000000000000
> (XEN) Xen call trace:
> (XEN)    [<ffff82c48015e2db>] smp_irq_move_cleanup_interrupt+0x216/0x28e
>
>
>

So vector e9 doesn't appear to be programmed in anywhere.

I am starting to get more into the realm of guessing here but, can you
use apic_verbosity=debug on the command line and copy this extra
debugging logic to send_cleanup_vector()

You should be able to conditionally trigger it on "desc->arch.vector ==
0xe9".  You will probably also want to change the BUG() to a WARN(), so
we get the interrupt and ioapic information on both sides of the cleanup
vector, as well as getting the stack trace of the codepath through Xen
as a result of vector 0xe9.

~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.