WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] pt_msix_init: Error: Can't map physical MSI-X table: In

To: Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>
Subject: Re: [Xen-devel] pt_msix_init: Error: Can't map physical MSI-X table: Invalid argument
From: Bruce Edge <bruce.edge@xxxxxxxxx>
Date: Wed, 14 Oct 2009 15:41:47 -0700
Cc: "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>
Delivery-date: Wed, 14 Oct 2009 15:42:12 -0700
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=5XTtyC38JqbYggGkZJ2hnXosH67mHFoVAhkvCYFpAqM=; b=TJy5M3xBIsLI3ED7KabUvdtzPK0COaxH02Qv7mExmj9KqFRLPZHP9zZbyFN1a/LXP/ V4+sYk3wIdod11WzY78A2EaiRH5AS/zakFThQcrzAUdjjM6NG5orsM92VWkq/2nY8AcP 4kg6NQhY8tHY8EjTCSUk7sTZjTGCq8l6p8EYo=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=wkU9UPt0ef4wsz7ZCdSQKBzOPOJxMLg9fxxOsvmRHFPFp7OApukZVqgItFX4577V23 bDw1ZIfVLrzGd1My31DUGwyClAA5bS1G0I8yYRln0y2+8lwuLxJr8t+4ahyD7ywwC61g jyFVrqcEfN9hRqVyhpvJtQbErCNl1ZUyer66g=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <alpine.DEB.2.00.0910141343050.11134@kaball-desktop>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <2B044E14371DA244B71F8BF2514563F503ECDB72@xxxxxxxxxxxxxxxxx> <alpine.DEB.2.00.0910141343050.11134@kaball-desktop>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
On Wed, Oct 14, 2009 at 5:47 AM, Stefano Stabellini
<stefano.stabellini@xxxxxxxxxxxxx> wrote:
> On Tue, 13 Oct 2009, Cinco, Dante wrote:
>> I get the above error message when I boot the domU (Linux 2.6.30.1). I'm 
>> using Xen 3.5-unstable (changeset 20303) with
>> pv_ops kernel 2.6.31.1. I use pci-stub to hide the device so that I can 
>> assign/pass it through to domU. The error is
>> coming from ioemu-xxx/hw/pt-msi.c. I think it is complaining about the 
>> table_off that is part of the mmap() call:
>>
>>     dev->msix->phys_iomem_base = mmap(0, total_entries * 16,
>>                           PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED,
>>                           fd, dev->msix->table_base + table_off);
>>
>> When I printed table_off, its value is 0x4100 (not page-aligned) which is 
>> consistent with the BAR4 offset of the PCI
>> device (see below). I think mmap() requires the last argument to be a 
>> multiple of the page size and results is a map
>> error if it is not. Here's the partial output of "lspci -vv -s 0:07:0.1" 
>> from dom0:
>>
>> 07:00.0 Fibre Channel: PMC-Sierra Inc. Device 8032 (rev 05)
>>         Subsystem: Atto Technology Device 003c
>>         Latency: 0, Cache Line Size: 64 bytes
>>         Interrupt: pin A routed to IRQ 11
>>         Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+ 
>> Queue=0/1 Enable+
>>                 Address: 00000000fee11000  Data: 405e
>>         Capabilities: [70] Express (v2) Endpoint, MSI 01
>>                 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s 
>> <4us, L1 unlimited
>>                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>>         Capabilities: [b0] MSI-X: Enable- Mask- TabSize=9
>>                 Vector table: BAR=4 offset=00004100
>>                 PBA: BAR=4 offset=00004000
>>
>> I'm able to workaround this problem by modifying pt-msi.c: force offset to 
>> zero in mmap() and on the next line, adjust
>> the pointer by table_off:
>>
>>     table_off_adjust = table_off & 0x0fff;
>>     dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + 
>> table_off_adjust,
>>                           PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED,
>>                           fd, dev->msix->table_base + table_off - 
>> table_off_adjust);
>>     dev->msix->phys_iomem_base = (void *)((char *)dev->msix->phys_iomem_base 
>> + table_off_adjust);
>>
>> Ideally, I should only be applying this to my device but since I'm not 
>> passing any other device that use MSI, I don't see
>> any negative effect. In addition, my device only uses MSI and not MSI-X.
>>
>> Obviously, this is a custom hack but I was wondering if there is a cleaner 
>> solution for this. Is a non-page aligned BAR
>> offset unusual? When I boot Linux 2.6.30.1 on bare metal (no Xen), I don't 
>> see this error.
>
> I think this workaround is not so bad, you just need to keep
> table_off_adjust around because it has to be used in munmap too.
>


Here's a patch that propagates the offset to the munmap as well. Any
chance this could be applied to the hg repository?

-Bruce

 diff -Naur ./tools/ioemu-remote/hw/pt-msi.c.orig
./tools/ioemu-remote/hw/pt-msi.c

--- ./tools/ioemu-remote/hw/pt-msi.c.orig       2009-10-13
11:54:11.000000000 -0700
+++ ./tools/ioemu-remote/hw/pt-msi.c    2009-10-14 15:16:31.000000000 -0700
@@ -535,6 +535,10 @@
                 DPCI_REMOVE_MAPPING);
 }

+/* Calculated offset to page-align MSI-X mmap
+ * Needed for munmap as well */
+static int table_off_adjust = 0;
+
 int pt_msix_init(struct pt_dev *dev, int pos)
 {
     uint8_t id;
@@ -542,6 +546,7 @@
     int i, total_entries, table_off, bar_index;
     struct pci_dev *pd = dev->pci_dev;
     int fd;
+    int err;

     id = pci_read_byte(pd, pos + PCI_CAP_LIST_ID);

@@ -584,9 +589,14 @@
         PT_LOG("Error: Can't open /dev/mem: %s\n", strerror(errno));
         goto error_out;
     }
-    dev->msix->phys_iomem_base = mmap(0, total_entries * 16,
+    PT_LOG("table_off = %llx, total_entries = %d\n",table_off,total_entries);
+    table_off_adjust = table_off & 0x0fff;
+    dev->msix->phys_iomem_base = mmap(0, total_entries * 16 + table_off_adjust,
                           PROT_WRITE | PROT_READ, MAP_SHARED | MAP_LOCKED,
-                          fd, dev->msix->table_base + table_off);
+                          fd, dev->msix->table_base + table_off -
table_off_adjust);
+    dev->msix->phys_iomem_base = (void *)((char
*)dev->msix->phys_iomem_base + table_off_adjust);
+    err = errno;
+    PT_LOG("errno = %d\n",err);
     if ( dev->msix->phys_iomem_base == MAP_FAILED )
     {
         PT_LOG("Error: Can't map physical MSI-X table: %s\n", strerror(errno));
@@ -612,7 +622,7 @@
     {
         PT_LOG("unmapping physical MSI-X table from %lx\n",
            (unsigned long)dev->msix->phys_iomem_base);
-        munmap(dev->msix->phys_iomem_base, dev->msix->total_entries * 16);
+        munmap(dev->msix->phys_iomem_base, dev->msix->total_entries *
16 + table_off_adjust);
     }

     free(dev->msix);

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel