[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] xhci_hcd intterrupt affinity in Dom0/DomU limited to single interrupt



On 09/01/2015 04:56 PM, Konrad Rzeszutek Wilk wrote:
On Tue, Sep 01, 2015 at 05:39:46PM +0000, Justin Acker wrote:
Taking this to the dev list from users.

Is there a way to force or enable pirq delivery to a set of cpus as opposed to 
single device from being a assigned a single pirq so that its interrupt can be 
distributed across multiple cpus? I believe the device drivers do support 
multiple queues when run natively without the Dom0 loaded. The device in 
question is the xhci_hcd driver for which I/O transfers seem to be slowed when 
the Dom0 is loaded. The behavior seems to pass through to the DomU if pass 
through is enabled. I found some similar threads, but most relate to Ethernet 
controllers. I tried some of the x2apic and x2apic_phys dom0 kernel arguments, 
but none distributed the pirqs. Based on the reading relating to IRQs for Xen, 
I think pinning the pirqs to cpu0 is done to avoid an interrupt storm. I tried 
IRQ balance and when configured/adjusted it will balance individual pirqs, but 
not multiple interrupts.
Yes. You can do it with smp affinity:

https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt

I don't follow the "behavior seems to pass through to the DomU if pass through is 
enabled" ?



With irqbalance enabled in Dom0:
What version? There was a bug in it where it would never distribute the IRQs 
properly
across the CPUs.

Boris (CC-ed) might remember the upstream patch that made this work properly?

I think we ended up taking the latest version of irqbalance as the one that added support for Xen guests was still not quite working. Besides, that patch was for xen-dyn-events, not for pirqs.

-boris


            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       
CPU6       CPU7
  76:      11304          0     149579          0          0          0         
 0          0  xen-pirq-msi       0000:00:1f.2
  77:       1243          0          0      35447          0          0         
 0          0  xen-pirq-msi       radeon
  78:      82521          0          0          0          0          0         
 0          0  xen-pirq-msi       xhci_hcd
  79:         23          0          0          0          0          0         
 0          0  xen-pirq-msi       mei_me
  80:         11          0          0          0          0        741         
 0          0  xen-pirq-msi       em1
  81:        350          0          0          0       1671          0         
 0          0  xen-pirq-msi       iwlwifi
  82:        275          0          0          0          0          0         
 0          0  xen-pirq-msi       snd_hda_intel

With native 3.19 kernel:

Without Dom0 for the same system from the first message:

# cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       
CPU6       CPU7
   0:         33          0          0          0          0          0         
 0          0  IR-IO-APIC-edge      timer
   8:          0          0          0          0          0          0         
 1          0  IR-IO-APIC-edge      rtc0
   9:         20          0          0          0          0          1         
 1          1  IR-IO-APIC-fasteoi   acpi
  16:         15          0          8          1          4          1         
 1          1  IR-IO-APIC  16-fasteoi   ehci_hcd:usb3
  18:     703940       5678    1426226       1303    3938243     111477     
757871        510  IR-IO-APIC  18-fasteoi   ath9k
  23:         11          2          3          0          0         17         
 2          0  IR-IO-APIC  23-fasteoi   ehci_hcd:usb4
  24:          0          0          0          0          0          0         
 0          0  DMAR_MSI-edge      dmar0
  25:          0          0          0          0          0          0         
 0          0  DMAR_MSI-edge      dmar1
  26:      20419       1609      26822        567      62281       5426      
14928        395  IR-PCI-MSI-edge      0000:00:1f.2
  27:   17977230     628258   44247270     120391 1597809883   14440991  
152189328      73322  IR-PCI-MSI-edge      xhci_hcd
  28:        563          0          0          0          1          0         
 6          0  IR-PCI-MSI-edge      i915
  29:         14          0          0          4          2          4         
 0          0  IR-PCI-MSI-edge      mei_me
  30:      39514       1744      60339        157     129956      19702      
72140         83  IR-PCI-MSI-edge      eth0
  31:          3          0          0          1         54          0         
 0          2  IR-PCI-MSI-edge      snd_hda_intel
  32:      28145        284      53316         63     139165       4410      
25760         27  IR-PCI-MSI-edge      eth1-rx-0
  33:       1032         43       2392          5       1797        265       
1507         20  IR-PCI-MSI-edge      eth1-tx-0
  34:          0          1          0          0          0          1         
 2          0  IR-PCI-MSI-edge      eth1
  35:          5          0          0         12        148          6         
 2          1  IR-PCI-MSI-edge      snd_hda_intel


The USB controller is an Intel C210:

00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family 
USB xHCI Host Controller (rev 04) (prog-if 30 [XHCI])
     Subsystem: Dell Device 053e
     Flags: bus master, medium devsel, latency 0, IRQ 78
     Memory at f7f20000 (64-bit, non-prefetchable) [size=64K]
     Capabilities: [70] Power Management version 2
     Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
     Kernel driver in use: xhci_hcd
     Kernel modules: xhci_pci
      On Tuesday, September 1, 2015 11:50 AM, Ian Campbell 
<ian.campbell@xxxxxxxxxx> wrote:
  On Tue, 2015-09-01 at 13:56 +0000, Justin Acker wrote:
Thanks Ian,

I appreciate the explanation. I believe the device drivers do support
multiple queues when run natively without the Dom0 loaded. The device in
question is the xhci_hcd driver for which I/O transfers seem to be slowed
when the Dom0 is loaded. The behavior seems to pass through to the DomU
if pass through is enabled. I found some similar threads, but most relate
to Ethernet controllers. I tried some of the x2apic and x2apic_phys dom0
kernel arguments, but none distributed the pirqs. Based on the reading
relating to IRQs for Xen, I think pinning the pirqs to cpu0 is done to
avoid an I/O storm. I tried IRQ balance and when configured/adjusted it
will balance individual pirqs, but not multiple interrupts.

Is there a way to force or enable pirq delivery to a set of cpus as you
mentioned above or omit a single device from being a assigned a PIRQ so
that its interrupt can be distributed across all cpus?
A PIRQ is the way an interrupt is exposed to a PV guest, without it there
would be no interrupt at all.

I'm afraid I'm out of my depth WRT how x86/MSIs and Xen x86/PV pirqs
interact, in particular WRT configuring which set of CPUs can have the IRQ
delivered.

If no one else chimes in soon I'd suggest taking this to the dev list, at
the very least someone who knows what they are talking about (i.e. other
than me) might be able to help.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.