Re: [Xen-devel] [PATCH 0/5] Add MSI support to XEN

[Keir Fraser <keir.fraser@xxxxxxxxxxxxx>]

On 28/3/08 08:40, "Jiang, Yunhong" <yunhong.jiang@xxxxxxxxx> wrote:

> The investigation result is,
> 1) if mask and ack the interrupt, the interrupt will happen 3 times, the
> last 2 is masked because they happened when the first one is still
> pending for ISR's handler, the system is ok.

How can you tell it happened three times? If the interrupt is pending in the
ISR then only one further pending interrupt can become visible to software
as there is only one pending bit per vector in the IRR.

> So I suppose the problem happens only if trigger the interrupt by
> software. I consulted the HW engineer also but didn't get confirmation,
> the only answer I got is, the PCI-E need a rising edge before send the
> 2nd interrupt :(

That answer means very little to me. One interesting question to have
answered would be: is this a closed-loop or open-loop interrupt storm? I.e.,
does the device somehow detect API EOI and then trigger re-send of the MSI
(closed loop) or is this an initialisation-time-only open-loop storm where
the device is spitting out the MSI regularly until some device register gets
written by the interrupt service routine?

Given the circumstances, I'm inclined to think it is the latter. Especially
since I think the former is impossible as EPIC EOI is not visible outside
the processor unless the interrupt came from a level-triggered IO-APIC pin,
and even then the EOI would not be visible across the PCI bus!

Also it seems *very* likely that this is just an initialisation-time thing,
and the device probably behaves very nicely after it is bootstrapped. In
light of this I think we should treat MSI sources as ACKTYPE_NONE in Xen
(i.e, require no callback from guest to hypervisor on completion of the
interrupt handler). We can then handle the interrupt storm entirely within
the hypervisor by detecting the storm and masking the interrupt and only
unmasking on some timeout.

In your tests, how aggressive was the IRQ storm? If you looked at the
interrupted EIP on each interrupt, was it immediately after the APIC was
EOIed and EFLAGS.IF set back to 1, or was it some time after? This tells us
how aggressively the device is sending out EOIs, and may determine how
cunning we need to be regarding interrupt storm detection.

> I'm not sure if there are any other BRAIN-DEAD device like this, I only
> have this device to test MSI-X function, but we may need make sure it
> will not break the whole system.

Yes, we have to handle this case, unfortunately.

> The call-back to guest because we are using the ACK-new method to work
> around this issue. Yes, it is expensive, Also, this ACK-new method may
> cause deadlock as Haitao suggested in the mail.

Yes, that sucks. See my previous email -- if possible it would be great to
teach Xen enough about the PCI config space to be able to mask MSIs.

> But if we move the config space to HV, then we don't need this ACK-new
> method, that should be ok, but admittedly, that should be the last
> method we we turn to, since config-space should be owned by domain0.

A partial movement into the hypervisor may be the best of a choice of evils.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel