Xen project Mailing List

I'm trying to run Infiniband in domain zero. I have an Infiniband HCA, which uses PCI-X.

The HCA is a multifunction PCI device having also a PCI bridge.

Running 2.6.10 vanilla detects two PCI buses on the HCA where one of them is a PCI bridge.

lspci -vv output looks like this:

---------------------- lspci -vv output with vanilla kernel ---------------------------

0000:12:02.0 Ethernet controller: Intel Corp. 82544GC Gigabit Ethernet Controller (LOM) (rev 02)
Subsystem: Intel Corp. 82544GC Based Network Connection
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (63750ns min), cache line size 08
Interrupt: pin A routed to IRQ 19
Region 0: Memory at ec000000 (64-bit, non-prefetchable) [size=fe8e0000]
Region 2: Memory at eb800000 (64-bit, non-prefetchable) [size=128K]
Region 4: I/O ports at a000 [size=32]
Expansion ROM at 00020000 [disabled]
Capabilities: [dc] Power Management version 2
  Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device.
  Command: DPERE- ERO+ RBC=0 OST=0
  Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM- Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
  Address: 0000000000000000 Data: 0000

0000:13:02.0 PCI bridge: Mellanox Technology MT23108 InfiniHost HCA bridge (rev a1) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32, cache line size 08
Bus: primary=13, secondary=14, subordinate=14, sec-latency=32
Memory behind bridge: eb000000-eb7fffff
Prefetchable memory behind bridge: 00000000f0000000-00000000fe700000
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [70] PCI-X bridge device.
  Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
  Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
  : Upstream: Capacity=0, Commitment Limit=0
  : Downstream: Capacity=0, Commitment Limit=0

0000:14:00.0 InfiniBand: Mellanox Technology MT23108 InfiniHost HCA (rev a1)
Subsystem: Mellanox Technology MT23108 InfiniHost HCA
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32, cache line size 08
Interrupt: pin A routed to IRQ 17
Region 0: Memory at eb000000 (64-bit, non-prefetchable)
Region 2: Memory at fe000000 (64-bit, prefetchable) [size=8M]
Region 4: Memory at f0000000 (64-bit, prefetchable) [size=128M]
Capabilities: [40] #11 [001f]
Capabilities: [50] Vital Product Data
Capabilities: [60] Message Signalled Interrupts: 64bit+ Queue=0/5 Enable-
  Address: 0000000000000000 Data: 0000
Capabilities: [70] PCI-X non-bridge device.
  Command: DPERE- ERO- RBC=0 OST=1
  Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-

-------------------------------------------------------------------------------------------------------

My problem is that the Xen kernel (linux-2.6.10-xen0) does not detect the PCI bridge bus (bus 13) but only the second bus.

(One can also spot that during PCI scan in the kernel boot).

---------------------- lspci -vv output with 2.6.10-xen0 kernel ---------------------------

0000:12:02.0 Ethernet controller: Intel Corp. 82544GC Gigabit Ethernet Controller (LOM) (rev 02)
Subsystem: Intel Corp. 82544GC Based Network Connection
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32, cache line size 08
Interrupt: pin ? routed to IRQ 19
Region 0: Memory at ec000000 (64-bit, non-prefetchable) [size=fe8e0000]
Region 2: Memory at eb800000 (64-bit, non-prefetchable) [size=128K]
Region 4: I/O ports at a000 [size=32]
Expansion ROM at 00020000 [disabled]
Capabilities: [dc] Power Management version 2
  Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
  Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device.
  Command: DPERE- ERO+ RBC=0 OST=0
  Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM- Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
  Address: 0000000000000000 Data: 0000

0000:14:00.0 InfiniBand: Mellanox Technology MT23108 InfiniHost HCA (rev a1)
Subsystem: Mellanox Technology MT23108 InfiniHost HCA
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32, cache line size 08
Interrupt: pin ? routed to IRQ 17
Region 0: Memory at eb000000 (64-bit, non-prefetchable)
Region 2: Memory at fe000000 (64-bit, prefetchable) [size=8M]
Region 4: Memory at f0000000 (64-bit, prefetchable) [size=128M]
Capabilities: [40] #11 [001f]
Capabilities: [50] Vital Product Data
Capabilities: [60] Message Signalled Interrupts: 64bit+ Queue=0/5 Enable-
  Address: 0000000000000000 Data: 0000
Capabilities: [70] PCI-X non-bridge device.
  Command: DPERE- ERO- RBC=0 OST=1
  Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-

-----------------------------------------------------------------------------------------------------------

Other differences that I see besides the missing bus is that interrupt pin is marked with '?' in the xen kernel:

Interrupt: pin ? routed to IRQ 17
^^^^^

In the vanilla kernel it says something like:

Interrupt: pin A routed to IRQ 17

^^^^^

When I load the HCA driver, it fails (it suceeds in the vanilla kernel).

Is anybody else experiencing such problems?

Is there a fix to this problem?

Are there any known problems with the PCI support in Xen in general?

I read in the mailing list that Xen is hiding PCI bridges. Why is that?

Thanks,

Ro'ee

[Xen-devel] problem with PCI in Xen?