Re: [Xen-smartio] Problem with infiniband on Xen domU

[Jiuxing Liu <jl@xxxxxxxxxx>]

Hi Lamia,

Currently there is no direct InfiniBand
access support in the Official Xen trees. You might want
to have a look at our Xen-IB implementation.
A preliminary version can be downloaded here:
http://xenbits.xensource.com/ext/xen-smartio.hg

Some information about its design and
applications can be found here:
High Performance VMM-Bypass I/O in Virtual
Machines (Usenix 06), and
A Case for High Performance Computing
with Virtual Machines (ICS 06).
You can get tech report versions from
the OSU web site here:
http://nowlab.cse.ohio-state.edu/publications/conf-papers/2006/usenix06.pdf
and
http://nowlab.cse.ohio-state.edu/publications/conf-papers/2006/huangwei-ics06.pdf

Thanks,

-Jiuxing

"Lamia M.Youseff" <lyouseff@xxxxxxxxxxx>
Sent by: xen-smartio-bounces@xxxxxxxxxxxxxxxxxxx

05/23/2006 05:42 AM

To	xen-smartio@xxxxxxxxxxxxxxxxxxx
cc
Subject	[Xen-smartio] Problem with infiniband on Xen domU

Dear list,
I am trying to get Xen working over infiniband connectivity, and this
mailing list seems to me to be the best place to post my question. I
will greatly appreciate your help and advise.

I was able to get up Dom0 to work with infiniband, but not DomU.
I am using Xen 3.0.2 with vmlinuz-2.6.16-xen and initrd-2.6.16-xen.img
for DomU kernel and ram image respectively.

I am getting the error of  "Unknown symbol in module" when
i am trying
to insert infiniband modules ib_mthca.ko and ib_umad.ko to domU kernel.
I made sure that the kernel was configured with infiniband support.
However, the dmesg output does not show that domU recognized infiniband
hardware at bootup.

Should I be specifying anything about infiniband in domain config file?
Any help very much appreciated,
Thank you,
Lamia Youseff

-bash-3.00-domU# cat config-2.6.16-xen
..........
#
# InfiniBand support
#
CONFIG_INFINIBAND=m
CONFIG_INFINIBAND_USER_MAD=m
CONFIG_INFINIBAND_USER_ACCESS=m
CONFIG_INFINIBAND_MTHCA=m
# CONFIG_INFINIBAND_MTHCA_DEBUG is not set
CONFIG_INFINIBAND_IPOIB=m
# CONFIG_INFINIBAND_IPOIB_DEBUG is not set
CONFIG_INFINIBAND_SRP=m
-bash-3.00-domU#  dmesg | grep umad
ib_umad: Unknown symbol ib_coalesce_recv_mad
ib_umad: Unknown symbol ib_unregister_mad_agent
ib_umad: Unknown symbol ib_post_send_mad
ib_umad: Unknown symbol ib_create_send_mad
ib_umad: Unknown symbol ib_free_send_mad
ib_umad: Unknown symbol ib_free_recv_mad
ib_umad: Unknown symbol ib_register_mad_agent
ib_umad: Unknown symbol ib_coalesce_recv_mad
ib_umad: Unknown symbol ib_unregister_mad_agent
ib_umad: Unknown symbol ib_post_send_mad
ib_umad: Unknown symbol ib_create_send_mad
ib_umad: Unknown symbol ib_free_send_mad
ib_umad: Unknown symbol ib_free_recv_mad
ib_umad: Unknown symbol ib_register_mad_agent

-bash-3.00-domU#  dmesg | grep mthca
ib_mthca: Unknown symbol ib_ud_header_pack
ib_mthca: Unknown symbol ib_unregister_device
ib_mthca: Unknown symbol ib_register_device
ib_mthca: Unknown symbol ib_create_ah
ib_mthca: Unknown symbol ib_unregister_mad_agent
ib_mthca: Unknown symbol ib_post_send_mad
ib_mthca: Unknown symbol ib_create_send_mad
ib_mthca: Unknown symbol ib_dispatch_event
ib_mthca: Unknown symbol ib_ud_header_init
ib_mthca: Unknown symbol ib_get_cached_pkey
ib_mthca: Unknown symbol ib_dealloc_device
ib_mthca: Unknown symbol ib_free_send_mad
ib_mthca: Unknown symbol ib_alloc_device
ib_mthca: Unknown symbol ib_destroy_ah
ib_mthca: Unknown symbol ib_get_cached_gid
ib_mthca: Unknown symbol ib_register_mad_agent
ib_mthca: Unknown symbol ib_ud_header_pack
ib_mthca: Unknown symbol ib_unregister_device
ib_mthca: Unknown symbol ib_register_device
ib_mthca: Unknown symbol ib_create_ah
ib_mthca: Unknown symbol ib_unregister_mad_agent
ib_mthca: Unknown symbol ib_post_send_mad
ib_mthca: Unknown symbol ib_create_send_mad
ib_mthca: Unknown symbol ib_dispatch_event
ib_mthca: Unknown symbol ib_ud_header_init
ib_mthca: Unknown symbol ib_get_cached_pkey
ib_mthca: Unknown symbol ib_dealloc_device
ib_mthca: Unknown symbol ib_free_send_mad
ib_mthca: Unknown symbol ib_alloc_device
ib_mthca: Unknown symbol ib_destroy_ah
ib_mthca: Unknown symbol ib_get_cached_gid
ib_mthca: Unknown symbol ib_register_mad_agent
ib_mthca: Unknown symbol ib_unregister_mad_agent
ib_mthca: Unknown symbol ib_post_send_mad
ib_mthca: Unknown symbol ib_create_send_mad
ib_mthca: Unknown symbol ib_free_send_mad
ib_mthca: Unknown symbol ib_register_mad_agent

-bash-3.00-dom0# cat correct-xen-lvdomain.3000.config
kernel= "/boot/vmlinuz-2.6.16-xen"
ramdisk = "/boot/initrd-2.6.16-xen.img"
memory = 3000
name = "xen2-lvlinux"
ip = "10.1.1.12"
netmask=get_current_ipmask()
gateway=get_current_ipgw()
hostname="xen2-lvlinux"
disk = ['phy:new-vg/vmdisk,sda1,w']
root = "/dev/sda1 ro"
vcpus = 4



_______________________________________________
Xen-smartio mailing list
Xen-smartio@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-smartio

_______________________________________________
Xen-smartio mailing list
Xen-smartio@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-smartio