[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt



On Fri, 2015-09-04 at 01:41 -0600, Jan Beulich wrote:
> >>> On 03.09.15 at 18:52, <ackerj67@xxxxxxxxx> wrote:
> > On Thu, 2015-09-03 at 09:04 -0600, Jan Beulich wrote:
> >> >>> On 03.09.15 at 14:04, <ackerj67@xxxxxxxxx> wrote:
> >> >      I am still confused as to whether any device, or in this case 
> >> > xhci_hcd, 
> >> > can use more than one cpu at any given time. My     understanding based 
> >> > on 
> >> > David's response is that it cannot due to the event channel mapping. The 
> >> > device interrupt can be pinned     to a specific cpu by specifying the 
> >> > affinity. I was hoping there was a way to allow the driver's interrupt 
> >> > to be 
> >> > scheduled to     use more than 1 CPU at any given time. 
> >> 
> >> The problem is that you're mixing up two things: devices and
> >> interrupts. Any individual interrupt can only be serviced by a single
> >> CPU at a time, due to the way event channels get bound. Any
> >> individual device can have more than one interrupt (MSI or MSI-X),
> >> and then each of these interrupts can be serviced on different
> >> CPUs.
> > 
> > Thanks for clarifying. To the original question, with respect to my
> > limited understanding of the event channels and interrupts, each
> > interrupt can be serviced on a different CPU using irqbalance or setting
> > the affinity manually, but the same interrupt cannot be serviced by more
> > than 1 CPU at a time? If so, is there a way around the 1:1 binding when
> > loading the Dom0 kernel - a flag or option to use the native interrupt
> > scheduling for some set of or all 8 CPUs that the device can schedule
> > interrupts on when not loading the Dom0? The xhci_hcd, as one example,
> > seems to perform better when it is able to have interrupts serviced by
> > multiple CPUs. 
> 
> I don't follow - we tell you this doesn't work (multiple times and
> different people), and you ask yet another time whether this can
> be made work? Just to make this very clear once again: Under Xen
> (and leaving aside pure HVM guests), interrupt load from a single
> device can be spread across CPUs only when the device uses
> multiple interrupts.
> 
> Jan
> 



I believe the driver does support use of multiple interrupts based on
the previous explanation of the lspci output where it was established
that the device could use up to 8 interrupts which is what I see on bare
metal. What would cause the interrupt usage to be limited to a single
CPU when the hypervisor is loaded? The device mentioned, xhci_hcd can
use multiple interrupts on baremetal and the other devices including the
network interfaces are also bound to a single interrupt when the
hypervisor is loaded but can use multiple interrupts running on
baremetal. 

 70:          0          0          0          0          0          0
0        133  xen-percpu-ipi       callfuncsingle7
 71:          0          0          0          0          0          0
0          0  xen-percpu-ipi       irqwork7
 72:        100          0          0          0          0          0
0          0   xen-dyn-event     xenbus
 73:          0          0          0          0          0          0
0          0   xen-dyn-virq      xen-pcpu
 74:          0          0          0          0          0          0
0          0   xen-dyn-virq      hvc_console
 75:       6809          0          0          0          0          0
0          0  xen-pirq-msi       0000:00:1f.2
 76:     470660          0          0          0          0          0
0          0  xen-pirq-msi       xhci_hcd
 77:        608          0          0          0          0          0
0          0  xen-pirq-msi       i915
 78:         26          0          0          0          0          0
0          0  xen-pirq-msi       mei_me
 79:        243          0          0          0          0          0
0          0  xen-pirq-msi       em1
 80:        315          0          0          0          0          0
0          0  xen-pirq-msi-x     p4p1-rx-0
 81:        104          0          0          0          0          0
0          0  xen-pirq-msi-x     p4p1-tx-0
 82:          2          0          0          0          0          0
0          0  xen-pirq-msi-x     p4p1
 83:         59          0          0          0          0          0
0          0  xen-pirq-msi       snd_hda_intel
 84:         48          0          0          0          0          0
0          0   xen-dyn-event     evtchn:xenstored
 85:          0          0          0          0          0          0
0          0   xen-dyn-event     evtchn:xenstored
NMI:          0          0          0          0          0          0
0          0   Non-maskable interrupts
LOC:          0          0          0          0          0          0
0          0   Local timer interrupts
SPU:          0          0          0          0          0          0
0          0   Spurious interrupts
PMI:          0          0          0          0          0          0
0          0   Performance monitoring interrupts
IWI:          1          0          0          0          0          0
0          0   IRQ work interrupts
RTR:          0          0          0          0          0          0
0          0   APIC ICR read retries
RES:      44788      33111     267650       5641      35515       2312
10730       2479   Rescheduling interrupts
CAL:        929       1202       1164       1314       1061       1296
1156       1097   Function call interrupts
TLB:          0          0          0          0          0          0
0          0   TLB shootdowns
TRM:          0          0          0          0          0          0
0          0   Thermal event interrupts
THR:          0          0          0          0          0          0
0          0   Threshold APIC interrupts
MCE:          0          0          0          0          0          0
0          0   Machine check exceptions
MCP:          1          1          1          1          1          1
1          1   Machine check polls
HYP:     538069      43568     273140      12112      44323       7871
18949       7924   Hypervisor callback interrupts

Example of another device driver output with Hypervisor loaded:

04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
        Subsystem: Intel Corporation Device 0000
        Physical Slot: 1-3
        Flags: bus master, fast devsel, latency 0, IRQ 19
        Memory at f5600000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at b000 [size=32]
        Memory at f5620000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Endpoint, MSI 00
        Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 
        Kernel driver in use: e1000e
        Kernel modules: e1000e

Example of the same device driver output on Bare metal:

04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
        Subsystem: Intel Corporation Device 0000
        Physical Slot: 1-3
        Flags: bus master, fast devsel, latency 0, IRQ 19
        Memory at f5600000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at b000 [size=32]
        Memory at f5620000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [e0] Express Endpoint, MSI 00
        Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 
        Kernel driver in use: e1000e
        Kernel modules: e1000e

Bare metal:

cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5
CPU6       CPU7       
  0:         36          0          0          0          0          0
0          0  IR-IO-APIC-edge      timer
  8:          0          0          0          1          0          0
0          0  IR-IO-APIC-edge      rtc0
  9:          0          0          1          0          1          1
0          0  IR-IO-APIC-fasteoi   acpi
 16:          2          0         14          0         13          0
0          2  IR-IO-APIC  16-fasteoi   ehci_hcd:usb3
 18:      24420        225      39883         43     134488       1044
1085         46  IR-IO-APIC  18-fasteoi   ath9k
 23:          0          0         20          0          0          1
3         11  IR-IO-APIC  23-fasteoi   ehci_hcd:usb4
 24:          0          0          0          0          0          0
0          0  DMAR_MSI-edge      dmar0
 25:          0          0          0          0          0          0
0          0  DMAR_MSI-edge      dmar1
 26:       4119       1016       1074        519       4677        978
573        432  IR-PCI-MSI-edge      0000:00:1f.2
 27:     337125      47893     708965       4049   53940667     263303
87847       4958  IR-PCI-MSI-edge      xhci_hcd
 28:        562          0          0          0          5          1
2          0  IR-PCI-MSI-edge      i915
 29:          7          0          0          0          4          8
0          6  IR-PCI-MSI-edge      mei_me
 30:       2222         23       2721          5       5810        181
149          5  IR-PCI-MSI-edge      eth0
 31:       1504         17       2196          6       6581         95
77          6  IR-PCI-MSI-edge      eth1-rx-0
 32:          2          0          1          0          3          0
0          0  IR-PCI-MSI-edge      eth1-tx-0
 33:          0          0          0          0          0          1
0          1  IR-PCI-MSI-edge      eth1
 34:         14          6          0          0          0         38
0          2  IR-PCI-MSI-edge      snd_hda_intel
 35:          4          3          0          0          0        178
0          3  IR-PCI-MSI-edge      snd_hda_intel
NMI:         11        105         87          1         75          2
9          1   Non-maskable interrupts
LOC:     132282    1175126     905037      79133     428383      22748
118085      12513   Local timer interrupts
SPU:          0          0          0          0          0          0
0          0   Spurious interrupts
PMI:         11        105         87          1         75          2
9          1   Performance monitoring interrupts
IWI:          1          0          0          0          0          0
0          0   IRQ work interrupts
RTR:          0          0          0          0          0          0
0          0   APIC ICR read retries
RES:      36810      28341      99815        953      60355       1142
878        823   Rescheduling interrupts
CAL:       2134       1928       1793       1965       1917       1879
1877       1882   Function call interrupts
TLB:       8064        365      20958      28575      37592        113
28495        445   TLB shootdowns
TRM:          0          0          0          0          0          0
0          0   Thermal event interrupts
THR:          0          0          0          0          0          0
0          0   Threshold APIC interrupts
MCE:          0          0          0          0          0          0
0          0   Machine check exceptions
MCP:          9          9          9          9          9          9
9          9   Machine check polls
THR:          0          0          0          0          0          0
0          0   Hypervisor callback interrupts



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.