WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] PV PCI pass-through with ixgbe ?

To: "Fischer, Anna" <anna.fischer@xxxxxx>
Subject: Re: [Xen-devel] PV PCI pass-through with ixgbe ?
From: Rolf Neugebauer <rolf.neugebauer@xxxxxxxxxxxxx>
Date: Tue, 07 Apr 2009 10:48:43 +0100
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Tue, 07 Apr 2009 02:49:18 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <0199E0D51A61344794750DC57738F58E66B941F2F8@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <0199E0D51A61344794750DC57738F58E66B941F2F8@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Thunderbird 2.0.0.12 (X11/20080226)
This worked for me for a fairly recent version of xen-unstable (cs: 19429:6dcccd2dded6). I used the corresponding -xen kernel for domU. I used a freshly compiled version of the ixgbe driver (1.3.56.5-NAPI). MSI-X worked fine, though I've only tried ping, not bulk data transfers. packets looked fine when I looked at it with tcpdump.

there were quite a few changes to the MSI code recently.

Rolf


Fischer, Anna wrote:
Is it possible to use MSI-X in a PV Xen DomU (2.6.18.8) for a PCI pass-through 
device? I am running an Intel ixgbe NIC in DomU, but I do not seem to be able 
to run it with MSI-X enabled. In fact, I do not get it to work in any interrupt 
mode. The DomU/Dom0 kernel is compiled with PCI_MSI=y and I have msi=1 set in 
Xen's boot options.

I do seem to get an interrupt clash when running up the ixgbe device driver in 
DomU. Also, packets seem to be corrupted when I capture with tcpdump on the 
interfaces.

tcpdump -i eth1 -xx gives me this:

01:46:56.770226 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui 
Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
01:46:56.973564 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui 
Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
01:46:57.176902 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui 
Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
01:46:57.380238 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui 
Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 46
        0x0000:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000


I have tried to run DomU/DomU with the irqpoll boot parameter but when I do 
that then the DomU fails to boot and hangs with a kernel panic:

Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5
Copyright (c) 1999-2008 Intel Corporation.
PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
ixgbe: Interrupt Type set to 1
ixgbe: 0000:02:00.0: ixgbe_check_options: Multiple queues are not supported 
while MSI-X is disabled.  Disabling Multiple Queues.
ixgbe: 0000:02:00.0: ixgbe_check_options: RSS is not supported while multiple 
queues are disabled.  Disabling RSS.
ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue 
count = 1, Tx Queue count = 1
ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 
00:21:f7:58:01:6d
ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
BUG: unable to handle kernel NULL pointer dereference at virtual address 
0000002d
 printing eip:
c01fccaa
2c8fa000 -> *pde = 00000000:60b62001
2cfc1000 -> *pme = 00000000:00000000
Oops: 0002 [#1]
SMP
Modules linked in: ixgbe aacraid usb_storage libusual ata_piix ahci libata 
sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore
CPU:    0
EIP:    0061:[<c01fccaa>]    Not tainted VLI
EFLAGS: 00210246   (2.6.18.8-xen #5)
EIP is at pci_enable_msix+0x1da/0x510
eax: ed717800   ebx: 00000011   ecx: 00000000   edx: 00000004
esi: 00000000   edi: ed717800   ebp: ec951f80   esp: ecab9d54
ds: 007b   es: 007b   ss: 0069
Process modprobe (pid: 1566, ti=ecab8000 task=eccd2e10 task.ti=ecab8000)
Stack: 00000098 ecfd4400 c036d220 c015f905 ecfd4400 ecfd4400 00000004 ee0e639a
       00000000 00000000 00000008 ec951f60 000000b2 00000002 00030003 0000007b
       0000007b 00000000 c017682e ec951f80 00000004 00000010 ecfd4400 00000000
Call Trace:
 [<c015f905>] __kzalloc+0x15/0x50
 [<ee0e639a>] ixgbe_alloc_queues+0x8a/0x580 [ixgbe]
 [<c017682e>] __kmalloc+0xbe/0xf0
 [<ee0e6a13>] ixgbe_init_interrupt_scheme+0x183/0x340 [ixgbe]
 [<ee0e3b81>] ixgbe_read_pci_cfg_word+0x21/0x30 [ixgbe]
 [<ee0e8c17>] ixgbe_probe+0x6e7/0x9e0 [ixgbe]
 [<c017682e>] __kmalloc+0xbe/0xf0
 [<c01f5ef6>] pci_device_probe+0x56/0x80
 [<c023ecd4>] driver_probe_device+0x44/0xc0
 [<c023ee52>] __driver_attach+0x82/0x90
 [<c023e64a>] bus_for_each_dev+0x3a/0x60
 [<c023ec16>] driver_attach+0x16/0x20
 [<c023edd0>] __driver_attach+0x0/0x90
 [<c023e2bc>] bus_add_driver+0x8c/0x140
 [<c01f6097>] __pci_register_driver+0x47/0x70
 [<c0143b08>] sys_init_module+0x148/0x1b40
 [<c01ea55f>] prio_tree_insert+0x1f/0x260
 [<c01187fc>] do_page_fault+0x10c/0xc6f
 [<c010acc0>] sys_mmap2+0xd0/0xe0
 [<c01059bf>] syscall_call+0x7/0xb
Code: 00 89 54 24 38 ba ed ff ff ff 89 4c 24 3c 0b 4c 24 38 0f 85 2d 01 00 00 8b 4c 
24 28 89 d6 89 8f b0 01 00 00 e9 62 fe ff ff 89 f8 <89> 5b 1c e8 3e f2 ff ff 8b 
7c 24 18 89 44 24 54 85 ff 0f 8e 48
EIP: [<c01fccaa>] pci_enable_msix+0x1da/0x510 SS:ESP 0069:ecab9d54
 udevd-event[1564]: run_program: '/sbin/modprobe' abnormal exit


I have also tried to use legacy interrupts by setting InterruptType to 0:

Intel(R) 10 Gigabit PCI Express Network Driver - version 1.3.31.5-lro
Copyright (c) 1999-2008 Intel Corporation.
PCI: Enabling device 0000:02:00.0 (0000 -> 0003)
PCI: Setting latency timer of device 0000:02:00.0 to 64
ixgbe: Interrupt Type set to 0
ixgbe: Multiple Queue Support Disabled
ixgbe: Receive-Side Scaling (RSS) set to 0
ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
ixgbe: 0000:02:00.0: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue 
count = 1, Tx Queue count = 1
ixgbe: 0000:02:00.0: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 
00:21:f7:58:01:6d
ixgbe: eth0: ixgbe_probe: Intel(R) 10 Gigabit Network Connection
PCI: Enabling device 0000:02:00.1 (0000 -> 0003)
PCI: Setting latency timer of device 0000:02:00.1 to 64
ixgbe: Interrupt Type set to 0
ixgbe: Multiple Queue Support Disabled
ixgbe: Receive-Side Scaling (RSS) set to 0
ixgbe: Virtual Machine Device Queues (VMDQ) set to 0
ixgbe: 0000:02:00.1: ixgbe_init_interrupt_scheme: Multiqueue Disabled: Rx Queue 
count = 1, Tx Queue count = 1
ixgbe: 0000:02:00.1: ixgbe_probe: (PCI Express:2.5Gb/s:Width x8) 
00:21:f7:58:01:6c
ixgbe: eth1: ixgbe_probe: Intel(R) 10 Gigabit Network Connection

Without MSI and MSI-X the driver loads properly. However, I still end up with 
corrupted packets on tcpdump, and additionally I am getting a kernel crash when 
the guest is under high network load:

BUG: unable to handle kernel NULL pointer dereference at virtual address 
00000034
 printing eip:
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#1]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe 
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd 
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU:    0
EIP:    0061:[<c0158df7>]    Not tainted VLI
EFLAGS: 00010082   (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
 printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#2]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe 
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd 
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU:    0
EIP:    0061:[<c0145df6>]    Not tainted VLI
EFLAGS: 00010002   (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
 printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Recursive die() failure, output suppressed
 <0>Kernel panic - not syncing: Fatal exception in interrupt
 BUG: warning at 
/home/user/Download/linux-2.6.18-xen.hg/arch/i386/kernel/smp-xen.c:511/smp_call_function()
 [<c0112541>] <1>BUG: unable to handle kernel paging request at virtual address 
245c8bf1
 printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#3]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe 
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd 
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU:    0
EIP:    0061:[<c0145df6>]    Not tainted VLI
EFLAGS: 00010002   (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
 printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Oops: 0002 [#4]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe 
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd 
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU:    0
EIP:    0061:[<c0145df6>]    Not tainted VLI
EFLAGS: 00010002   (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
 printing eip:
c0145df6
2c934000 -> *pde = 00000000:60f1a001
2c8e7000 -> *pme = 00000000:00000000
Recursive die() failure, output suppressed
 <0>Kernel panic - not syncing: Fatal exception in interrupt
 <1>BUG: unable to handle kernel NULL pointer dereference at virtual address 
00000064
 printing eip:
c0158df7
2d2e1000 -> *pde = 00000000:6051f001
2d2e2000 -> *pme = 00000000:00000000
Oops: 0002 [#5]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe 
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd 
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU:    1
EIP:    0061:[<c0158df7>]    Not tainted VLI
EFLAGS: 00010282   (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
 printing eip:
c0145df6
2d2e1000 -> *pde = 00000000:6051f001
2d2e2000 -> *pme = 00000000:00000000
Oops: 0002 [#6]
SMP
Modules linked in: 8250 serial_core genrtc parport_pc lp parport pvmod ixgbe 
aacraid usb_storage libusual ata_piix ahci libata sd_mod scsi_mod ext3 jbd 
uhci_hcd ohci_hcd ehci_hcd usbcore
CPU:    1
EIP:    0061:[<c0145df6>]    Not tainted VLI
EFLAGS: 00010002   (2.6.18.8-xen #5)
BUG: unable to handle kernel paging request at virtual address 245c8bf1
 printing eip:
c0145df6
2d2e1000 -> *pde = 00000000:6051f001
2d2e2000 -> *pme = 00000000:00000000
Recursive die() failure, output suppressed

What are the correct interrupt settings for using ixgbe in PV pass-through 
mode? I have tried all three possible modes but I do not seem to get the device 
working properly under high network load.

I am using Xen 3.3.1.

release                : 2.6.18.8-xen
version                : #5 SMP Thu Mar 5 04:14:30 PST 2009
machine                : i686
nr_cpus                : 2
nr_nodes               : 1
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2200
virt_caps              : hvm
total_memory           : 2039
free_memory            : 6
node_to_cpu            : node0:0-1
node_to_memory         : node0:6
xen_major              : 3
xen_minor              : 3
xen_extra              : .1
xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p xen_scheduler : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20070925 (Red Hat 4.1.2-33)
cc_compile_by          : root
cc_compile_domain      : localdomain
cc_compile_date        : Sun Mar  8 05:40:51 PDT 2009
xend_config_format     : 4

I am using PCI pass-through to assign the NIC to my PV guest, so this is the part of my config file:
pci = [ '02:00.0','02:00.1' ]

I am also hiding the PCI devices from Dom0, so my Dom0 GRUB config looks like 
this:

title Xen 3.3
      root (hd0,1)
        kernel /boot/xen-3.3.1.gz console=vga msi=1
      module /boot/vmlinuz-2.6.18.8-xen root=LABEL=/ ro console=tty0 rhgb quiet 
pciback.permissive      pciback.hide=(02:00.0)(02:00.1) 
reassigndev=0000:02:00.0,0000:02:00.1
      module /boot/initrd-2.6.18.8-xen.img

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>