WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] RE: PCI Pass-through Difficulty

To: "Manyam, Ramesh" <Ramesh.Manyam@xxxxxxx>, "Kay, Allen M" <allen.m.kay@xxxxxxxxx>
Subject: Re: [Xen-devel] RE: PCI Pass-through Difficulty
From: Keir Fraser <keir.fraser@xxxxxxxxxxxxx>
Date: Fri, 12 Dec 2008 09:21:36 +0000
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Fri, 12 Dec 2008 01:22:01 -0800
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <BFE8DAC894D2B245AA14185EB10A2BD60B3A1BEE5A@xxxxxxxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
Thread-index: AclPGWVKxONNwPj4QmyQdrm7IY7L1gAFf7VQATZVbjAAAC9/oAAmDyKwAAEys4AAGsmqIAAYNLOAAbBHwFAAAdwKkA==
Thread-topic: [Xen-devel] RE: PCI Pass-through Difficulty
User-agent: Microsoft-Entourage/12.14.0.081024
The site is definitely up. You can go to http://xenbits.xensource.com and browse the trees.

 -- Keir

On 12/12/2008 08:33, "Manyam, Ramesh" <Ramesh.Manyam@xxxxxxx> wrote:

Allen
 
 I tried many times but I am not able to download the source code.
 
 Is there any other mirror site for this?
 
 
Ramesh
 


From: Kay, Allen M [mailto:allen.m.kay@xxxxxxxxx]
Sent: Wednesday, December 03, 2008 11:45 PM
To: Manyam, Ramesh
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: PCI Pass-through Difficulty

Ramesh,

Both staging tree and mailine xen-unstable tree are the same for the code we are talking about.

for staging tree  : hg clone http://xenbits.xensource.com/staging/xen-unstable.hg    <== I'm using this tree
for xen-unstable: hg clone http://xenbits.xensource.com/xen-unstable.hg



Allen



From: Manyam, Ramesh [mailto:Ramesh.Manyam@xxxxxxx]
Sent: Tuesday, December 02, 2008 10:47 PM
To: Kay, Allen M
Cc: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: PCI Pass-through Difficulty

Hi Allen,
           I couldn’t find map_domain_pirq() is at line 841 of xen-unstable.hg/xen/arch/x86/irq.c but I found this function in xen-unstable.hg/xen/arch/x86/physdev.c at line 55.
           

          I think you are using different source tree. Can you tell me where can I download/found your source tree?
 
Ramesh



From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Kay, Allen M
Sent: Tuesday, December 02, 2008 11:24 PM
To: Sandwell, James; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-devel] RE: PCI Pass-through Difficulty

In the staging tree, map_domain_pirq() is at line 841 of xen-unstable.hg/xen/arch/x86/irq.c.

Sounds like you are working off from a different tree.  Where did you get your source tree?

Allen



From: Sandwell, James [mailto:James.Sandwell@xxxxxxx]
Sent: Tuesday, December 02, 2008 9:18 AM
To: Kay, Allen M; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: PCI Pass-through Difficulty
In this file there is no function map_domain_pirq and I can't find anything that may be similar.  Could this be part of the problem?

Thanks,
James Sandwell



From: Kay, Allen M [mailto:allen.m.kay@xxxxxxxxx]
Sent: Monday, December 01, 2008 5:10 PM
To: Sandwell, James; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: PCI Pass-through Difficulty

> 2) Where should this code be located, we cannot find it in our xen source tree (working off of xen-unstable.hg)?
> I am assuming Linux kernel; do you still recommend doing this on the host CentOS (i.e. we're operating VxWorks as the guest OS)?

The full patch is xen-unstable.hg/xen/arch/x86/irq.c/map_domain_pirq(). Yes, you should do this if VxWorks uses MSI.

Allen



From: Sandwell, James [mailto:James.Sandwell@xxxxxxx]
Sent: Monday, December 01, 2008 3:05 PM
To: Kay, Allen M; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: PCI Pass-through Difficulty

1) Since we're running VxWorks instead of Linux we just get an 'invalid boot string' from trying to set this parameter on the guest OS. It has no effect setting it for the host CentOS.

2) Where should this code be located, we cannot find it in our xen source tree (working off of xen-unstable.hg)? I am assuming Linux kernel; do you still recommend doing this on the host CentOS (i.e. we're operating VxWorks as the guest OS)?

Thanks,
James Sandwell



From: Kay, Allen M [mailto:allen.m.kay@xxxxxxxxx]
Sent: Tuesday, November 25, 2008 12:59 PM
To: Sandwell, James; xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: RE: PCI Pass-through Difficulty
I noticed your device has MSI and MSIX capability.  You might want to try one of the following:

1) set "pci=nomsi" in the guest's grub.conf.

or

2) comment out " if ( type == MAP_PIRQ_TYPE_MSI ) return -EPERM;" in arch/x86/irq.c.

Allen



From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Sandwell, James
Sent: Tuesday, November 25, 2008 8:18 AM
To: xen-devel@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-devel] PCI Pass-through Difficulty

  We’re running into a problem with receiving interrupts from a PCI pass-through device (namely a 1068 SAS Host Board Adapter). The device has been added to be PCI pass-through in the grub.conf and a cfg script for starting the domain. We can read/write to the config registers on the PCI device but we never receive the doorbell interrupt during initialization. We’re having trouble determining root cause of this issue.



         We're using a Dell T7400 with VT-D and VT-X enabled and running Xen 3.4-unstable with patches (c/s 18430 and c/s 4761) with CentOS. We’re currently loading a VxWorks kernel as a guest operating system and can successfully execute our initialization procedures.


 
 
Various outputs (hopefully helpful – if more detailed is needed like entire logs let me know)
 
uname –a
Linux 2.6.18.8-xen #3 SMP x86_64 x86_64 x86_64 GNU/Linux
 
.cfg file
pci=['06:00.0']
 
grub.conf
title XEN (whargharbl)
        root (hd0,0)
        kernel /xen-3.4-unstable.gz acpi=force apic=on vtd=1 iommu=1
        module /vmlinuz-2.6.18.8-xen ro root=/dev/VolGroup00/LogVol00 pciback.hide=(06:00.0)
        module /initrd-2.6.18.8-xen.img
 
lspci –b -v
06:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
       Subsystem: LSI Logic / Symbios Logic Unknown device 30a0
        Flags: fast devsel, IRQ 5
        I/O ports at bc00 [disabled]
        Memory at 00000000f7cec000 (64-bit, non-prefetchable) [disabled]
        Memory at 00000000f7cf0000 (64-bit, non-prefetchable) [disabled]
        Expansion ROM at f7a00000 [disabled]
       Capabilities: [50] Power Management version 2
       Capabilities: [68] Express Endpoint IRQ 0
       Capabilities: [98] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
       Capabilities: [b0] MSI-X: Enable- Mask- TabSize=1
 
 
dmesg
GSI 26 sharing vector 0x51 and IRQ 26
ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 70 (level, low) -> IRQ 26
 
pciback 0000:06:00.0: seizing device

ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 70 (level, low) -> IRQ 26
ACPI: PCI interrupt for device 0000:06:00.0 disabled

 
xm dmesg (after we didn’t receive the doorbell interrupt we were expecting)
 
(XEN) domain_context_mapping: DEV_TYPE_PCIe_ENDPOINT
(XEN) Xen WARN at irq.c:514
(XEN) ----[ Xen-3.4-unstable  x86_64 debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:   e008:[<ffff828c8013cf0e>] pirq_guest_bind+0x2e/0x240
(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor
(XEN) rax: 0000000000000001   rbx: ffff83007ee18080   rcx: 0000000000000000
(XEN) rdx: 0000000000000001   rsi: 000000000000001a   rdi: ffff83007eff6080
(XEN) rbp: 0000000000000002   rsp: ffff828c80247ca8   r8:  000000000000001a
(XEN) r9:  ffff83007ee18490   r10: 0000000000000008   r11: 0000000000000008
(XEN) r12: 0000000000000000   r13: ffff83007eff6080   r14: 000000000000001a
(XEN) r15: 0000000000000001   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 000000006c646000   cr2: 000000000046c1f0
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff828c80247ca8:
(XEN)    0000000000000000 ffff828c80247e28 ffff828c8011b422 ffff83007ee18080
(XEN)    0000000000000002 0000000000000000 0000000000000006 000000000000001a
(XEN)    ffff83007effc080 ffff828c80125b9c 0000000000000cfc fffffffffffffff3
(XEN)    fffffffffffffffd 00007fffc86bb1d0 ffff828c80247e28 ffff828c80247e28
(XEN)    00000000004c5398 ffff828c801317a4 0000000000000cfc 0000000000000000
(XEN)    00000000800600b0 ffff83006ca8ed28 0000000000800227 0000000000800227
(XEN)    ffff83006ca8ed28 0000000000000000 0000000000000000 ffff828c80145261
(XEN)    ffff828c80247eb8 0000000000000000 0000000000000000 0000000080247f28
(XEN)    000000000006ca8e ffff828c80111eb0 ffff8284010fa630 fffffffffffffff3
(XEN)    00007fffc86bb1d0 0000000000305000 ffff828c80247e28 00000000ffffffda
(XEN)    00000000004c5398 ffff828c80104cdb 0000000020000000 0000000000000051
(XEN)    0000000000000202 ffff828c80271424 0000000000000000 0000000000000202
(XEN)    0000000500000026 0000000000d40001 000000000000001a 0000060000000001
(XEN)    0000000000000020 00000000f7cee000 0000000000000000 00000032bf209e25
(XEN)    0000000000000000 00007fffc86bb270 0000000000000006 0000000000d5b650
(XEN)    00000000004c5398 00000032bf20975c 0000000000000021 000000000000000d
(XEN)    00007fffc86bb270 0000003815b4e9a0 0000000000000000 0000000000000206
(XEN)    ffffffffffffffff 0000000000000000 00000038158cee57 0000000000000033
(XEN)    0000000000000206 ffff83007fdea080 00007fffc86bb260 0000000000305000
(XEN)    0000000000000006 00000000ffffffda 00000000004c5398 ffff828c801a9169
(XEN) Xen call trace:
(XEN)    [<ffff828c8013cf0e>] pirq_guest_bind+0x2e/0x240
(XEN)    [<ffff828c8011b422>] maybe_split+0x32/0x60
(XEN)    [<ffff828c80125b9c>] pt_irq_create_bind_vtd+0x22c/0x290
(XEN)    [<ffff828c801317a4>] arch_do_domctl+0xec4/0x1540
(XEN)    [<ffff828c80145261>] mod_l1_entry+0x971/0x990
(XEN)    [<ffff828c80111eb0>] rangeset_add_range+0x120/0x180
(XEN)    [<ffff828c80104cdb>] do_domctl+0xcdb/0xd80
(XEN)    [<ffff828c801a9169>] syscall_enter+0xa9/0xae

dmesg (after we didn’t receive the doorbell interrupt we were expecting)
 
PM: Adding info for xen-backend:vkbd-1-0
PM: Adding info for xen-backend:vfb-1-0
PM: Adding info for xen-backend:vbd-1-768
PM: Adding info for xen-backend:vif-1-0
PM: Adding info for xen-backend:pci-1-0
device vif1.0 entered promiscuous mode
eth0: port 2(vif1.0) entering learning state
eth0: topology change detected, propagating
eth0: port 2(vif1.0) entering forwarding state
ip_tables: (C) 2000-2006 Netfilter Core Team
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <maxk@xxxxxxxxxxxx>
device tap1.0 entered promiscuous mode
eth0: port 3(tap1.0) entering learning state
eth0: topology change detected, propagating
eth0: port 3(tap1.0) entering forwarding state
PM: Adding info for xen-backend:console-1-0
vif1.0: no IPv6 routers present
tap1.0: no IPv6 routers present

   Another peculiarity is that we’re not seeing the vmx flag in /proc/cpuinfo while running Xen (but booting w/o Xen it does show up).
 
Thanks,
James Sandwell
 
 
 
 
 



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel
<Prev in Thread] Current Thread [Next in Thread>