WARNING - OLD ARCHIVES

This is an archived copy of the Xen.org mailing list, which we have preserved to ensure that existing links to archives are not broken. The live archive, which contains the latest emails, can be found at http://lists.xen.org/
   
 
 
Xen 
 
Home Products Support Community News
 
   
 

xen-devel

Re: [Xen-devel] megasas stops I/O when running kernel as dom0 under xen4

To: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
Subject: Re: [Xen-devel] megasas stops I/O when running kernel as dom0 under xen4.1/4.2
From: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
Date: Wed, 24 Aug 2011 18:20:03 +0100
Cc: Andreas Olsowski <andreas.olsowski@xxxxxxxxxxx>, "Keir@xxxxxxxxxxxxxxxxxxxx" <Keir@xxxxxxxxxxxxxxxxxxxx>, "xen-devel@xxxxxxxxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxxxxxxxx>, Fraser <keir.xen@xxxxxxxxx>
Delivery-date: Wed, 24 Aug 2011 10:20:53 -0700
Envelope-to: www-data@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20110824170919.GA14696@xxxxxxxxxxxx>
List-help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-id: Xen developer discussion <xen-devel.lists.xensource.com>
List-post: <mailto:xen-devel@lists.xensource.com>
List-subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
List-unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>, <mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
References: <4E4916A3.9070106@xxxxxxxxxxx> <4E4E56EE.2070801@xxxxxxxxxx> <4E4E705E.3040505@xxxxxxxxxxx> <4E4E79E8.3020808@xxxxxxxxxx> <4E4E913C.40809@xxxxxxxxxxx> <4E4E9423.8010904@xxxxxxxxxx> <4E4EA725.30405@xxxxxxxxxxx> <4E521BD5.2040609@xxxxxxxxxx> <4E54E93D.7060301@xxxxxxxxxx> <4E552D62.2000002@xxxxxxxxxx> <20110824170919.GA14696@xxxxxxxxxxxx>
Sender: xen-devel-bounces@xxxxxxxxxxxxxxxxxxx
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.18) Gecko/20110617 Lightning/1.0b2 Thunderbird/3.1.11

On 24/08/11 18:09, Konrad Rzeszutek Wilk wrote:
> On Wed, Aug 24, 2011 at 05:57:06PM +0100, Andrew Cooper wrote:
>> On 24/08/11 13:06, Andrew Cooper wrote:
>>> On 22/08/11 10:05, Andrew Cooper wrote:
>>>> On 19/08/11 19:10, Andreas Olsowski wrote:
>>>>> Am 19.08.2011 18:49, schrieb Andrew Cooper:
>>>>>
>>>>>> The only change you need to make is in megasas_probe_one() in
>>>>>> megaraid_sas_base.c
>>>>>>
>>>>>> Add a call to pci_enable_msi(pdev) immediately after the current
>>>>> call to
>>>>>> pci_set_master(pdev);
>>>>>>
>>>>>> ~Andrew
>>>>>>
>>>>> Yep, that works fine. Removed the module option as well.
>>>>>
>>>>> root@tarballerina:~# cat /proc/interrupts  |grep mega
>>>>> 2236:      69010          0          0          0          0         
>>>>> 0          0          0  xen-pirq-msi       megasas
>>>>>
>>>>> The same procedure that would have lead to almost instant errors has
>>>>> not brought them to appear again.
>>>>>
>>>> Good.  This is what we are seeing as well.  I am still awaiting a reply
>>>> from LSI on this topic.
>>>>
>>>> Unfortunately, this does point to a regression in the way Xen deals with
>>>> legacy interrupts.
>>> Out of interest, on all 3 of your boxes with the megaraid_sas cards,
>>> could you gather the io_apic information?
>>>
>>> It is the z xen debug key on the serial console (or alternatively put
>>> apic_verbosity=debug on the xen commandline and the information gets
>>> dumped into the dmesg)
>> You can ignore this - it is not relevant.
>>
>> I have narrowed the problem to a bug in the interrupt migration code.
> Goodies!
>> The bug occurs when the move pending flag is set, and somehow another
>> interrupt comes in on the old pcpu without triggering the move
>> completion code.  This leaves the IO_APIC with ack'd but not EOI'd
>> interrupt from the megaraid_sas device.
> Ah, so the interrupt is delievered to Dom0 on the old per_cpu
> event which is ignored. Ignored b/c we have rebinded the event channel
> to the other CPU, right?

The interrupt is not ignored - it seems to be being serviced by the
device driver in dom0.   I will admit that my debugging code may be a
bit flaky - I started by trying to match IRQ35 (which is always claimed
by PCI INTA on this server - very useful for debugging) between do_IRQ
and its related PHYSDEVOP_eoi.

I am currently trying to track the exact order of events around this
interrupt which misses the move completion code.

> Is there any code in the Hypervisor to turn off interrupt migration code?

Not that I have found, although playing around with vcpu and task
pinning should work.  My debugging shows that Xen-4.1.1 is migrating
this interrupt between PCPUs on average once every 4 real interrupts
when dom0 is under any load whatsoever.

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

<Prev in Thread] Current Thread [Next in Thread>