[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen-unstable: pci-passthrough "irq 16: nobody cared" on HVM guest shutdown on irq of device not passed through.



> Hi Jan,
> 
> Just tested with about the same kernel (and same hardware) on linux + kvm:
> 
> When i boot i see (on the host):
> [    1.341556] pcieport 0000:00:02.0: irq 31 for MSI/MSI-X
> [    1.341875] pcieport 0000:00:03.0: irq 32 for MSI/MSI-X
> [    1.342145] pcieport 0000:00:05.0: irq 33 for MSI/MSI-X
> [    1.342429] pcieport 0000:00:06.0: irq 34 for MSI/MSI-X
> [    1.342588] pcieport 0000:00:09.0: irq 35 for MSI/MSI-X
> [    1.342853] pcieport 0000:00:0a.0: irq 36 for MSI/MSI-X
> [    1.343125] pcieport 0000:00:0b.0: irq 37 for MSI/MSI-X
> [    1.343285] pcieport 0000:00:0c.0: irq 38 for MSI/MSI-X
> [    1.343434] pcieport 0000:00:0d.0: irq 39 for MSI/MSI-X
> [    1.343719] pcieport 0000:00:15.0: irq 40 for MSI/MSI-X

When this happend, did you see:
[  66.819396] bus: 'pci': really_probe: bound device 0000:00:0d.0 to driver 
pcieport
[   66.842286] bus: 'pci': driver_probe_device: matched device 0000:00:15.0 
with driver pcieport
[   66.868014] bus: 'pci': really_probe: probing driver pcieport with device 
0000:00:15.0
-- some other code --
[   67.757210] pcieport 0000:00:15.0: ?!?!? acpi_pci_irq_enable: PCI INT A -> 
GSI 16 (level, low) -> IRQ/rc 16


> [    1.343897] pcieport 0000:05:00.0: irq 41 for MSI/MSI-X
> [    1.344134] pcieport 0000:06:01.0: irq 42 for MSI/MSI-X
> [    1.344398] pcieport 0000:06:02.0: irq 44 for MSI/MSI-X
> 
> While the soundcard has still irq16.
> 
> Passing through 09:00.0 to a KVM guest (as primary passthrough since that 
> works 
> with KVM) it works fine and after shutting down there is no "irq 16 nobody 
> cared".
> 
> So why would linux setup those irq/gsi's for the pcieport differently on Xen 
> and on 
> baremetal (and why can a irq/gsi be bound to 2 devices) ? 

Right, that would be odd.

The IRQ can be bound to two devices if it is an 'level' type (aka, not an 
'edge').
This is something that be extrapolated from the ACPI MADT tables
and the xen_register_gsi is the one that is told about that (polarity and 
level).

Looking back at your emails:

    (XEN) [2014-09-25 13:29:04.111]    IRQ:  16 affinity:01 vec:89 
type=IO-APIC-level   status=00000030 in-flight=0 domain-list=0: 16(--

It definitly is level, so that means it is shared.

Now there is a gotcha with shared interrupts - if there are _two_
devices and one is in one guest while the other is in another guest - both
domains have to Ack it.

That is where Xen PCIBack comes in - it has its own IRQ handler that will
determine (based on a hypercall) whether the device is shared and if so
Ack it. I am wondering if it is turned off (as in, Xen thinks it should not
be shared but Linux has the device shared).

You can fiddle with the Xen pciback parameters to turn on/off this 'fake'
irq handler.

But it all seems to point that we somehow are getting the wrong impression
about the 0000:00:0d.0 and  0000:00:15.0 and using legacy interrupts
instead of MSI ones. And I think that underlaying problem is causing
these secondary ones.

Could you attach also the full dmesg under baremetal with 'debug' and all
kinds of debug enabled ? That should help a bit in figuring out why
they get MSIs under baremetal but legacy interrupts under Xen.

Thank you!
> 
> --
> Sander
> 
> 
> # cat /proc/interrupts
>             CPU0       CPU1       CPU2       CPU3       CPU4       CPU5
>    0:     100420          0          0          0          0          0  
> IR-IO-APIC-edge      timer
>    1:          1          0          0          0          0          2  
> IR-IO-APIC-edge      i8042
>    7:          1          0          0          0          0          0  
> IR-IO-APIC-edge
>    8:          0          0          0          0          0          1  
> IR-IO-APIC-edge      rtc0
>    9:          0          0          0          0          0          0  
> IR-IO-APIC-fasteoi   acpi
>   12:          0          0          0          0          0          4  
> IR-IO-APIC-edge      i8042
>   16:          0          0          0          0          0        796  
> IR-IO-APIC   16-fasteoi   snd_hda_intel
>   17:          0          0          0          0          0          4  
> IR-IO-APIC   17-fasteoi   ehci_hcd:usb1, ehci_hcd:usb2, ehci_hcd:usb3
>   18:          0          0          0          0         11       6414  
> IR-IO-APIC   18-fasteoi   ohci_hcd:usb4, ohci_hcd:usb5, ohci_hcd:usb6, 
> ohci_hcd:usb7
>   29:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      AMD-Vi
>   31:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   32:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   33:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   34:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   35:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   36:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   37:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   38:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   39:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   40:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      PCIe PME
>   45:          0          0          0          1         79      95630  
> IR-PCI-MSI-edge      ahci0
>   46:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      ahci1
>   47:          0          0          0          0         27       8643  
> IR-PCI-MSI-edge      ahci2
>   48:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      ahci3
>   49:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      ahci4
>   50:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      ahci5
>   53:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      ahci
>   55:          0          0          0          0        184      22648  
> IR-PCI-MSI-edge      eth0
>   57:          0          0          0          1        148      21232  
> IR-PCI-MSI-edge      eth1
>   59:          0          0          0          0          0          1  
> IR-PCI-MSI-edge      xhci_hcd
>   60:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   61:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   62:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   63:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   64:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   65:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   66:          0          0          0          0          0         23  
> IR-PCI-MSI-edge      xhci_hcd
>   67:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   68:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   69:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   70:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   71:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   72:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   74:          0          0          0          0         23      22099  
> IR-PCI-MSI-edge      xhci_hcd
>   75:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   76:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   77:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   78:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   79:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   80:          0          0          0          0          0          0  
> IR-PCI-MSI-edge      xhci_hcd
>   81:          0          0          0          0          0          0  
> IR-IO-APIC   23-fasteoi   cx25821[1]
>   83:          0          0          0          0          0         28  
> IR-PCI-MSI-edge      snd_hda_intel
>   85:          0          0          0          0          3        652  
> IR-PCI-MSI-edge      vfio-msi[0](0000:09:00.0)
>  NMI:          3          3          3          4          4          3   
> Non-maskable interrupts
>  LOC:      13423      25027      24547      30018      30575      46789   
> Local timer interrupts
>  SPU:          0          0          0          0          0          0   
> Spurious interrupts
>  PMI:          3          3          3          4          4          3   
> Performance monitoring interrupts
>  IWI:          0          0          0          0          1          0   IRQ 
> work interrupts
>  RTR:          0          0          0          0          0          0   
> APIC ICR read retries
>  RES:      88652      68934      52849     112934      76728      33317   
> Rescheduling interrupts
>  CAL:        295        287        327        327        311        200   
> Function call interrupts
>  TLB:       1165       1232       1122       1376       1248       1160   TLB 
> shootdowns
>  TRM:          0          0          0          0          0          0   
> Thermal event interrupts
>  THR:          0          0          0          0          0          0   
> Threshold APIC interrupts
>  MCE:          0          0          0          0          0          0   
> Machine check exceptions
>  MCP:          4          4          4          4          4          4   
> Machine check polls
>  THR:          0          0          0          0          0          0   
> Hypervisor callback interrupts
>  ERR:          1
>  MIS:          0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.