This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
Home Products Support Community News


Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU

To: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Subject: Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough
From: Dante Cinco <dantecinco@xxxxxxxxx>
Date: Fri, 12 Nov 2010 10:29:57 -0800
Cc: Xen-devel <xen-devel@xxxxxxxxxxxxxxxxxxx>, "Lin, Ray" <Ray.Lin@xxxxxxx>
Delivery-date: Fri, 12 Nov 2010 10:30:48 -0800
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=QtMIMSY40V0IYx2DhwKvmuyYMq6/OpZL+pN+H1zVKV4=; b=YuJHDTIjxQjNvoJKtBiAHrTlGauJhg75a+/ugczhF+kgWLklmipDgLa30RyXLW3p0g UQcXepckqSdno8Rh4/sWAiSg+vuGasJ7t5a6PbLTWgr11sSvUqfPRiloYNSbyal7/N+G vIdx5ujfUSuqe5H+qdOPpAAAYSZnLAq7kMaV8=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=v+JuamJhLKSn2z83jMXNDh7VOWJ0zENII9wBhTb2BlbP2vQy07br0jPkL4UT7X/8Tf bG6qA8pw9HBQ1JomqyvPvcE+7utcvYk1+eAusRXZna97czt7e4Acqk1EJjKZIJmz+3Ig QCVQf6k5Pi2kGmi9rnoNtfIs9fgzQi2vHFuUE=
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20101112155659.GA5529@xxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <20101111190351.GB15530@xxxxxxxxxxxx> <EB4C61A1A2501842A04B573FE42B14D601374FBD2C@xxxxxxxxxxxxxxxxx> <20101112155659.GA5529@xxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
On Fri, Nov 12, 2010 at 7:56 AM, Konrad Rzeszutek Wilk
<konrad.wilk@xxxxxxxxxx> wrote:
> On Thu, Nov 11, 2010 at 12:42:03PM -0700, Lin, Ray wrote:
>> Konrad,
>>    See my response in red.
> Please don't top post.
>> -Ray
>> -----Original Message-----
>> From: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx 
>> [mailto:xen-devel-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Konrad Rzeszutek 
>> Wilk
>> Sent: Thursday, November 11, 2010 11:04 AM
>> To: Dante Cinco
>> Cc: Xen-devel
>> Subject: Re: [Xen-devel] swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops 
>> domU kernel with PCI passthrough
>> On Thu, Nov 11, 2010 at 10:31:48AM -0800, Dante Cinco wrote:
>> > Konrad,
>> >
>> > Without swiotlb=force, I don't see "PCI-DMA: Using software bounce
>> > buffering for IO" in /var/log/kern.log.
>> >
>> > With iommu=soft and without swiotlb=force, I see the "software bounce
>> > buffering" in /var/log/kern.log and an NMI (see below) when I load the
>> > kernel module drivers. I made sure the NMI is reproducible and not a
>> What is the kernel module doing to cause this? DMA?
> ??? What did it do?
>> > one-time event.
>> So doing 64-bit DMA causes an NMI. Do you have the Hypervisor's IOMMU VT-d 
>> enabled or disabled? (iommu=off,verbose) If you turn it off does this work?
>> We have IOMMU VT-d enabled. If we turn it off (iommu=off,verbose), the DMA 
>> doesn't work properly and the driver code is unable to detect the source of 
>> interrupt. The interrupts of our device would be disabled by kernel 
>> eventually due to nobody services the interrupts for more than 100000 times.
> That does not sound right. You should be able to use the PCI passthrough 
> without the IOMMU. Since it
> is an interrupt issue it sounds like that you are using x2APIC and that is 
> enabled without the IOMMU.
> Had you tried disabling IOMMU and x2apic? (this is all on the hypervisor 
> line?)

I set the hypervisor boot options to iommu=0 x2apic=0. I booted the
pvops domU with swiotlb=force initially since that's the option that
worked in the past. Not long after loading the kernel module drivers,
domU hung/froze but dom0 stayed up. I checked the Xen interrupt
bindings and I see the PCI-passthrough devices have either (PS--) or

(XEN)    IRQ:  66 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:72
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  67 affinity:00000000,00000000,00000000,00000200 vec:3b
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  68 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:8a
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  69 affinity:00000000,00000000,00000000,00000800 vec:43
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  70 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:9a
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  71 affinity:fffffff,ffffffff,ffffffff,ffffffff vec:a2
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  72 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:aa
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  73 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:b2
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  74 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ba
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  75 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:c2
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  76 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:ca
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  77 affinity:00000000,00000000,00000000,00080000 vec:4b
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  78 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:da
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  79 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:23
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  80 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:2b
type=PCI-MSI         status=00000010 in-flight=0
(XEN)    IRQ:  81 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:33
type=PCI-MSI         status=00000010 in-flight=0

I will reboot the pvops domU with iommu=soft and without swiotlb=force next.

- Dante

Xen-devel mailing list

<Prev in Thread] Current Thread [Next in Thread>