Re: [PATCH v2 1/5] x86/shutdown: offline APs with interrupts disabled on all CPUs

[Jan Beulich <jbeulich@xxxxxxxx>]

On 10.02.2025 11:46, Roger Pau Monné wrote:
> On Mon, Feb 10, 2025 at 11:20:20AM +0100, Jan Beulich wrote:
>> On 06.02.2025 16:06, Roger Pau Monne wrote:
>>> The current shutdown logic in smp_send_stop() will disable the APs while
>>> having interrupts enabled on the BSP or possibly other APs. On AMD systems
>>> this can lead to local APIC errors:
>>>
>>> APIC error on CPU0: 00(08), Receive accept error
>>>
>>> Such error message can be printed in a loop, thus blocking the system from
>>> rebooting.  I assume this loop is created by the error being triggered by
>>> the console interrupt, which is further stirred by the ESR handler
>>> printing to the console.
>>>
>>> Intel SDM states:
>>>
>>> "Receive Accept Error.
>>>
>>> Set when the local APIC detects that the message it received was not
>>> accepted by any APIC on the APIC bus, including itself. Used only on P6
>>> family and Pentium processors."
>>>
>>> So the error shouldn't trigger on any Intel CPU supported by Xen.
>>>
>>> However AMD doesn't make such claims, and indeed the error is broadcasted
>>> to all local APICs when an interrupt targets a CPU that's already offline.
>>>
>>> To prevent the error from stalling the shutdown process perform the
>>> disabling of APs and the BSP local APIC with interrupts disabled on all
>>> CPUs in the system, so that by the time interrupts are unmasked on the BSP
>>> the local APIC is already disabled.  This can still lead to a spurious:
>>>
>>> APIC error on CPU0: 00(00)
>>>
>>> As a result of an LVT Error getting injected while interrupts are masked on
>>> the CPU, and the vector only handled after the local APIC is already
>>> disabled.
>>
>> Isn't this bogus, too? As in: Error interrupt without any ESR bits set? Since
>> I can already see our QA folks report this as another issue, can we perhaps
>> somehow amend the log message in that case, indicating we think it's bogus?
> 
> Note that the disable_local_APIC() call in __stop_this_cpu() does also
> call clear_local_APIC(), which will attempt to clear ESR also.

Hmm, that's odd. I wonder whether we shouldn't suppress this when called
from disable_local_APIC().

> Further patches in the series prevent the error from triggering in the
> first place, since an attempt is made to disable or mask all possible
> external interrupt sources Xen knows about before doing AP shutdown.

Right.

Jan