Re: [Xen-devel] Xen and Infiniband as dom0 (only)

[Xavier Beaudouin <xb@xxxxxxxxxxx>]

Hi there,

Le 16 mars 2010 à 16:56, Pasi Kärkkäinen a écrit :

> On Mon, Mar 15, 2010 at 01:29:54PM +0100, Xavier Beaudouin wrote:
>> Hello there,
>> 
>> I am trying to make infiniband working on a dom0.  To allow me fast access 
>> to storage, and fast migration between hypervisor.
>> 
>> I still don't want to do Ib as Hypervisor Accelerator and specialy what is 
>> shown on slide 12 of this paper : 
>> http://xen.org/files/Xen_Virtualization_InfiniBand_Mellanox_MKagan.pdf, but 
>> I need rdma to get fast access to storage :
>> - iSer (eg iscsi over infiniband)
>> - RDMA for NFS over RDMA or stuff like this.
>> 
>> On my host, without Xen (F12, with myoung kernel backported, xen 4.0.0-rc6 
>> and OFED 1.5.1-RC4), I can use correctly ipoib (IP over IB), but rdma is 
>> still broken.
>> 
>> Is there any people that has made dom0 IB working that can me some hints 
>> about what I have to fix ?
>> 
> 
> Well what actually is the problem? 
> What kind of errors do you get? 


Simple :

- 192.168.5.202 is the server (filer) with IB and OFED 1.5.1-rc5

  $ ibv_rc_pingpong 
  local address:  LID 0x0007, QPN 0x590406, PSN 0x1adab0
  remote address: LID 0x000c, QPN 0x0a0406, PSN 0x877182

- client is xen-6 (192.168.5.226) :
  $ ibv_rc_pingpong 192.168.5.202
  local address:  LID 0x000c, QPN 0x0a0406, PSN 0x877182
  remote address: LID 0x0007, QPN 0x590406, PSN 0x1adab0

Message from syslogd@xen-6 at Mar 26 12:41:56 ...
 kernel:Bad pagetable: 000f [#1] SMP 

Message from syslogd@xen-6 at Mar 26 12:41:56 ...
 kernel:last sysfs file: 
/sys/devices/pci0000:00/0000:00:0f.0/0000:04:00.0/infiniband/mthca0/node_type
Killed

dmesg on the host : 
ibv_rc_pingpong: Corrupted page table at address 7fa3eccc1010
PGD 58f1d067 PUD 58927067 PMD 53e50067 PTE fffffffffffff237
Bad pagetable: 000f [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:0f.0/0000:04:00.0/infiniband/mt
hca0/node_type
CPU 0 
Modules linked in: tun bridge stp llc nfs lockd fscache nfs_acl auth_rpcgss sunr
pc rdma_ucm ib_sdp rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_sa ipv6 ib_uverbs ib_
umad iw_nes libcrc32c iw_cxgb3 cxgb3 mlx4_ib mlx4_en mlx4_core microcode xen_evt
chn xenfs ib_mthca ib_mad forcedeth amd64_edac_mod i2c_nforce2 ib_core edac_core
 i2c_core edac_mce_amd xfs exportfs dm_multipath pata_acpi ata_generic pata_amd 
[last unloaded: scsi_wait_scan]
Pid: 4402, comm: ibv_rc_pingpong Tainted: G        W  2.6.32.10-1.2.92.xendom0.f
c12.x86_64 #1 empty
RIP: e033:[<00007fa3ec4a3f42>]  [<00007fa3ec4a3f42>] 0x7fa3ec4a3f42
RSP: e02b:00007fff80f16ca0  EFLAGS: 00010202
RAX: 000000000a000001 RBX: 0000000000000000 RCX: 00007fa3eccc1000
RDX: 0206040a0a000001 RSI: 00007fa3eccbb008 RDI: 0000000000000000
RBP: 0000000001bdc980 R08: 0000000000000000 R09: 0000000000000007
R10: 0000000000590406 R11: 0000000000000246 R12: 00007fa3ec49f010
R13: 0000000001be3170 R14: 0000000000000001 R15: 00007fff80f17000
FS:  00007fa3eccb8700(0000) GS:ffff880003e7d000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fa3eccc1010 CR3: 0000000002a12000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ibv_rc_pingpong (pid: 4402, threadinfo ffff88004f9ea000, task ffff88005b
a19760)

RIP  [<00007fa3ec4a3f42>] 0x7fa3ec4a3f42
 RSP <00007fff80f16ca0>
---[ end trace 4eaa2a86a8e2da24 ]---

Note that I have same problems with intel platform as well...

ipoib works, but I really like to get rid of these errors....

Xavier


--
Xavier Beaudouin - xb@xxxxxxxxxxx
GPG Fingerprints : A6B2 D563 F93B A3AF C08A  CBAC 6BC6 79EB DCC9 9867


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel