Re: [Xen-devel] [PATCH v2] x86/boot: enable NMIs after traps init

[Ross Philipson <ross.philipson@xxxxxxxxx>]

On 10/23/2018 11:31 AM, Jason Andryuk wrote:
> On Tue, Oct 23, 2018 at 10:46 AM Andrew Cooper
> <andrew.cooper3@xxxxxxxxxx> wrote:
>> On 23/10/18 15:01, Jason Andryuk wrote:
>>> On Tue, Oct 23, 2018 at 7:15 AM Andrew Cooper <andrew.cooper3@xxxxxxxxxx> 
>>> wrote:
>>>> On 23/10/18 11:59, Sergey Dyasli wrote:
>>>>> In certain scenarios, NMIs might be disabled during Xen boot process.
>>>>> Such situation will cause alternative_instructions() to:
>>>>>
>>>>>     panic("Timed out waiting for alternatives self-NMI to hit");
>>>>>
>>>>> This bug was originally seen when using Tboot to boot Xen.
>>>>>
>>>>> To prevent this from happening, enable NMIs during cpu_init() and
>>>>> during __start_xen() for BSP.
>>>>>
>>>>> Signed-off-by: Sergey Dyasli <sergey.dyasli@xxxxxxxxxx>
>>>> Reviewed-by: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
>>> FYI, Ross and Eric came up with a tboot patch recently added to OpenXT:
>>> https://github.com/OpenXT/xenclient-oe/blob/master/recipes-openxt/tboot/tboot-1.9.6/0023-tboot-Unmask-NMI-potentially-masked-during-SENTER.patch
>>>
>>> Using this Xen patch with the tboot one reverted works too.
>>>
>>> Tested-by: Jason Andryuk <jandryuk@xxxxxxxxx>
>> :(
>>
>> Can bugs like this please be reported upstream?  Given the observation
>> of "Tboot hands off with NMIs disabled", the fix is very easy.
> I'm not opposed to reporting upstream.  In this case, I at least
> assumed it was something we did in our EFI & tboot combo.  An Ivy
> Bridge legacy boot system with tboot & Xen worked fine.  For me, it
> was only a newer Skylake (or Kaby Lake) machine that had issue when
> booting our EFI & tboot combo.  So it wasn't clear that tboot always
> left NMIs disabled.  Yes, we should have reported something upstream
> as a heads up for other tboot/Xen users.

According to the specs, NMIs and SMIs are disabled post launch on the BSP and
after AP wakeup is done. The TBOOT code explicitly re-enables SMIs but currently
not NMIs. Any IRET later on would have re-enabled them so it might explain how
they incidentally get re-enabled in certain configurations.

Personally I think the fix to re-enable them in TBOOT should go upstream.

Thanks
Ross

>
> Regards,
> Jason
>
>> On the subject of having NMIs disabled, it is definitely a more robust
>> way of handing off between components.  Until Xen has transitioned the
>> BSP and APs into 64bit mode and fully set the IDT up, NMIs are fatal to
>> the system.
>>
>> Furthermore, I've got some plans to suggest we do this on the kexec path
>> as well.  XenServer has one example where we crash from a GPU related
>> error, and a slightly delayed NMI arrives, usually when we're in purgatory.
>>
>> ~Andrew


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel