WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] Intel S3420GPLX VT-d VF question

To: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Subject: Re: [Xen-devel] Intel S3420GPLX VT-d VF question
From: "Michael A. Collins" <mike@xxxxxxxxxxx>
Date: Thu, 09 Dec 2010 22:18:45 -0500
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Delivery-date: Fri, 10 Dec 2010 02:50:39 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20101208175035.GB19417@xxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <30410a64d335f82097af217b0ed7d060@xxxxxxxxxxxxxxxx> <20101208163217.GA18739@xxxxxxxxxxxx> <ad976338520fe08e9bdb4db66598f390@xxxxxxxxxxxxxxxx> <20101208175035.GB19417@xxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Roundcube Webmail/0.4.2
On Wed, 8 Dec 2010 12:50:35 -0500, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx> wrote:
The kernel I am using is myoung's pre-built rpm from http://myoung.fedorapeople.org/dom0/x86_64/kernel-2.6.32.26-174.2.xendom0.fc12.x86_64.rpm.

Sweet. That is quite up-to-date.

I am also using the 4.0.1 rpms from
http://pasik.reaktio.net/fedora/xen-4.0.1-6.fc14.src.rpm.

Uhh, try a more modern version. The xen-unstable.hg - you will have
to compile it yourself. Pasi's awesome PVOPS Wiki mentions the appropiate
steps and I think Boris's blog has details as well.

I am following the instructions from
http://wiki.xen.org/xenwiki/Fedora13Xen4Tutorial.

The pci_pt_e820_access=on setting was taken from the following blog: http://blogs.oracle.com/wim/2010/10/playing_with_sr_iov_in_oracle.html.

Oh gosh. That is with 2.6.18 and Xen 3.4.

I was under the impression that that option had to be set for the
VFs to be seen by the kernel.

Well, they were workarounds for that verison of kernel. It might be
that they are required in 2.6.32.. but lets assume that those
are actually fixed.

I have not tried it without, but will do that tonight.  The
blacklist settings was referenced in the above article, so I stuck
it in.  I will try to load the igbvf tonight as well.

Yeah, try it. If it works you should see suddenly seven extra NICs
(ifconfig -a) in your box. If that works then we know the kernel
driver is working.

I do not see the 7 Virtual Functions in lspci.

Ah! Those "01:00.0 Ethernet controller: Intel Corporation 82576
Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network
Connection (rev 01)
01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function
(rev 01)
01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function
(rev 01)
01:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function
(rev 01)
01:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function
(rev 01)
01:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function
(rev 01)
01:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function
(rev 01)
01:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function
(rev 01)"

must show up. That is the proof that your max_ig.. parameter worked
and the card
is now able to do SR-IOV. You might have need to upgrade your lspci -
I think I had
it as "Unknown device" at some point.

Anyhow, if you see those extra devices, _then_ you can blacklist the
igvf module.


Thanks for the reply, I will try without the e820 option and loading
the igbvf module tonight if that works, I'll let you know.  If not,
I'll respond with the full xen dmesg output.

Take your time.

Seems as the igb module was loaded in the initramfs and was not reloaded using the options specified in my igb.conf file under /etc/modprobe.d. I rmmod igb and did a modprobe igb max_vfs=7 and suddenly all the Virtual Functions showed up. So that was the problem all along. Until I figure out how to edit the initramfs to load igb with max_vfs option, I'm putting the rmmod and modprobe in the rc.local file, that seems to work well and it gets executed before xend. I am build xen-4.0-testing to replace the rpms I used. Once all that is done and I test passing through one of the Virtual Functions to a vm, I'll report back.
Mike

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel