Re: [Xen-devel] IOMMU initialization failure when using linux-as-bootloader

[Sylvain Munaut <s.munaut@xxxxxxxxxxxxxxxxxxxx>]

Hi,

>> (XEN) DMAR_IQA_REG = 47ad78002
>> (XEN) DMAR_IQH_REG = 440
>> (XEN) DMAR_IQT_REG = 20
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 0:
>> (XEN) queue invalidate wait descriptor was not executed
>> (XEN) ****************************************
>>
>
> We gave a queued invalidation request to the IOMMU, and it didn't respond.
>
> Given the position in the log, this is likely the very first time we have
> tried interacting with the IOMMU, and given the fact you used kexec to get
> here, it is likely that it is already set up from Linux's use.
>
> It is probably something like we aren't resetting it, and either interacting
> with it in a mismatch between polled or interrupt mode, or that it has
> interrupts set up and they are going to an unexpected location.

Ah, thanks for the hint.

I tried to boot the first stage using intremap=off to prevent it from
using interrupt remapping and that seemed to work around the issue !

I had a quick look into the xen code and I could see two path that
enable interrupt remapping :
 - Through iommu_enable_x2apic_IR
 - Through init_vtd_hw

In the first path, Xen will force disable interrupt remapping and
queued invalidation in case the BIOS had them on and then will
re-initialize them properly.
In the second path, Xen doesn't seem to do that.

And presumably in my system the second path is taken because the BIOS
opts-out of x2APIC according to the linux boot messages.

So I tried to just do the same pre-cleanup in init_vtd_hw than in
iommu_enable_x2apic_IR (see diff below) and this seemed to work just
fine.

I'll test this in more detail next week to make sure things really
work as expected (atm I'm just testing if dom0 boots properly). And if
things turn OK and you think this is a proper fix for the issue and
safe to apply, I can send the patch properly to the list.


Cheers,

    Sylvain Munaut

-----

diff --git a/xen/drivers/passthrough/vtd/iommu.c
b/xen/drivers/passthrough/vtd/iommu.c
index 48f120b..5f79609 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -2102,6 +2102,9 @@ static int __must_check init_vtd_hw(void)

         clear_fault_bits(iommu);

+        disable_intremap(iommu);
+        disable_qinval(iommu);
+
         spin_lock_irqsave(&iommu->register_lock, flags);
         sts = dmar_readl(iommu->reg, DMAR_FECTL_REG);
         sts &= ~DMA_FECTL_IM;

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel