Re: [Xen-devel] PCI passthrough issue

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Wed, 2011-02-02 at 10:24 +0000, Jean Baptiste Favre wrote:
> Hello Ian,
> 
> My domU config file:
> 
> # cat /cluster/xen/xps-106.cfg
> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
> builder      = 'linux'
> memory       = '398'
> vcpus        = '1'
> cpus         = '2'
> localtime    = 0
> serial       = 'pty'
> boot         = 'cdn'
> disk         = [ 'drbd:xps-106,xvda,w' ]
> on_poweroff  = 'destroy'
> on_reboot    = 'restart'
> on_crash     = 'restart'
> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
> console=hvc0 xencons=tty"
> pci = [ '04:00.0' ]
> name         = 'xps-106'
> hostname     = 'xps-106.clichy.jbfavre.org'
> 
> 
> Le 02/02/2011 10:27, Ian Campbell a Ãcrit :
> > On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote:
> > > Le 01/02/2011 17:23, Ian Campbell a Ãcrit :
> >
> > >> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
> > >> you post a full guest console log at some point? Comparing the logs for
> > >> the 256MB, 398MB and 512MB guest RAM case might be useful.
> > > No sure I've ever posted that logs. But I can redo my tests :)
> >
> > yes, please do that.
> Please find attached both console startup logs with 256M and 512M:
> 256M_domU_console_logs.txt
> 512M_domU_console_logs.txt

Thanks, I stripped the timestamps and cleaned up some differences due to
missing carriage returns, those versions are attached.

Interesting bits of diff (- == 256, + == 512) are:
        -Allocating PCI resources starting at 10800000 (gap: 10800000:ef800000)
        +Allocating PCI resources starting at 20800000 (gap: 20800000:df800000)

So the PCI resources are higher and smaller in the 512MB case.

        -Placing 64MB software IO TLB between cbd7ee40 - cfd7ee40
        -software IO TLB at phys 0xbd7ee40 - 0xfd7ee40
        +Placing 64MB software IO TLB between dbb1a000 - dfb1a000
        +software IO TLB at phys 0x1bb1a000 - 0x1fb1a000

Fair enough?

         sky2: driver version 1.28
        -sky2 0000:00:00.0: BAR 0: set to [mem 0xfeb00000-0xfeb03fff 64bit] 
(PCI address [0xfeb00000-0xfeb03fff])
        -sky2 0000:00:00.0: BAR 2: set to [io  0xe800-0xe8ff] (PCI address 
[0xe800-0xe8ff])
        -sky2 0000:00:00.0: enabling device (0000 -> 0003)
         sky2 0000:00:00.0: Xen PCI enabling IRQ: 18
         sky2 0000:00:00.0: Yukon-2 EC Ultra chip revision 3
         sky2 0000:00:00.0: eth0: addr 00:1f:c6:eb:71:43

So there is some sort of remapping going on in the 256MB case? (or the
512MB logs are missing a bit, which can happen). Can you post the
content of domU's /proc/{interrupts,iomem,ioports} for both cases?

> >
> > Please can you also collect and post the information from ifconfig and
> > ethtool -S which I asked for earlier.
> Attached as well:
> 256_domU_ifconfig_ping_ethtool.txt
> 512_domU_ifconfig_ping_ethtool.txt

Thanks. The interesting bit here is that the 256MB case is registering
RX dropped frames. 512_domU_ifconfig_ping_ethtool.txt did not include an
ifconfig after the experiment but I assume the dropped frames are not
present there.

The rx_dropped statistic is only cranked in a small number of places,
once in the driver and a handful in the networking core, only one of the
core cases looks likely to be relevant. The following patch should help
us figure out where the frames are dropped...

Ian.

diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7d85a38..645d9e9 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2339,6 +2339,7 @@ static struct sk_buff *receive_copy(struct sky2_port 
*sky2,
                re->skb->ip_summed = CHECKSUM_NONE;
                skb_put(skb, length);
        }
+       WARN(skb == NULL, "sky2: receive_copy failed netdev_alloc_skb_ip_align 
length %u\n", length);
        return skb;
 }
 
@@ -2386,10 +2387,15 @@ static struct sk_buff *receive_new(struct sky2_port 
*sky2,
 
        nre.skb = sky2_rx_alloc(sky2);
        if (unlikely(!nre.skb))
+       {
+               WARN(1, "sky2: receive_new failed sky2_rx_alloc\n");
                goto nobuf;
-
+       }
        if (sky2_rx_map_skb(sky2->hw->pdev, &nre, hdr_space))
+       {
+               WARN(1, "sky2: receive_new failed sky2_rx_map_skb\n");
                goto nomap;
+       }
 
        skb = re->skb;
        sky2_rx_unmap_skb(sky2->hw->pdev, re);
diff --git a/net/core/dev.c b/net/core/dev.c
index 54277df..9c99612 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1516,6 +1516,7 @@ int dev_forward_skb(struct net_device *dev, struct 
sk_buff *skb)
 
        if (unlikely(!(dev->flags & IFF_UP) ||
                     (skb->len > (dev->mtu + dev->hard_header_len + 
VLAN_HLEN)))) {
+               printk(KERN_CRIT "dev_forward_skb dropping skb\n");
                atomic_long_inc(&dev->rx_dropped);
                kfree_skb(skb);
                return NET_RX_DROP;
@@ -2700,6 +2701,7 @@ enqueue:
 
        local_irq_restore(flags);
 
+       printk(KERN_CRIT "enqueue_to_backlog dropping skb\n");
        atomic_long_inc(&skb->dev->rx_dropped);
        kfree_skb(skb);
        return NET_RX_DROP;
@@ -3125,6 +3127,7 @@ ncls:
        if (pt_prev) {
                ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
        } else {
+               printk(KERN_CRIT "__netif_receive_skb dropping skb proto 
%#x\n", type);
                atomic_long_inc(&skb->dev->rx_dropped);
                kfree_skb(skb);
                /* Jamal, now you will not able to escape explaining

Attachment: 512M_domU_console_logs.txtX
Description: Text document

Attachment: 256M_domU_console_logs.txtX
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel