Re: [Xen-devel] Xen Introspection, KPTI, and CR3 bit 63 leads to guest VMENTRY failures during introspection

[Razvan Cojocaru <rcojocaru@xxxxxxxxxxxxxxx>]

On 01/25/2018 02:25 PM, Razvan Cojocaru wrote:
> On 01/25/2018 02:15 AM, Andrew Cooper wrote:
>> On 24/01/2018 22:31, Bitweasil . wrote:
>>> I've recently discovered that if you attempt to use introspection to
>>> capture CR3 changes with the new KPTI enabled kernels, the guest dies
>>> shortly after the start of introspection with failed VM entry due to
>>> invalid guest state.
>>>
>>> I believe the invalid state here is the high bit being set in CR3 -
>>> while this is how one indicates that PCID should not invalidate the
>>> various page table caches, introspection leads to this being set in
>>> the VMCS, which appears to be wrong.
>>>
>>> With the XenServer 4.7.1 code base (which is my working code base at
>>> the moment), I have not found a way around this, as the
>>> vm_event_set_registers function (xen/arch/x86/vm_event.c) does not set
>>> the CR3 value, and vm_event_register_write_resume only allows
>>> inhibiting the write, not writing a modified value.
>>>
>>> I've attempted several ways to work around this with a livepatch, and
>>> have not (yet) been successful.
>>>
>>> Masking at the top of hvm_set_cr3 allows the guest to continue, but
>>> appears to do the wrong thing with regards to the guest (tasks begin
>>> dying quickly from invalid opcode errors).
>>>
>>> In any case, Andrew mentions that this appears to still be an issue in
>>> staging, so this likely needs addressing.  At this point in time, I
>>> believe guests with KPTI enabled cannot be introspected if that
>>> introspection involves capturing CR3 changes.
>>>
>>> Please let me know if you need any more details on this issue!
>>
>> Just as an FYI to people reading this, that is actually XenServer 7.1's
>> hypervisor which is Xen 4.7.1-based but the fact that the HVM CR3 code
>> has little-to-no clue about PCID appears to be unchanged into staging. 
>> Sadly, it doesn't appear to be trivial to fix.
> 
> Well, FWIW we do support masks for CR events since Xen 4.10 - we can
> simply mask whatever bits we _don't_ want events sent out for. I don't
> know if this solves Bitweasil's problem, but it's certainly something to
> take into consideration.
> 
> For example, looking at xen-access.c:
> 
> 629     if ( write_ctrlreg_cr4 )
> 630     {
> 631         /* Mask the CR4.PGE bit so no events will be generated for
> global TLB flushes. */
> 632         rc = xc_monitor_write_ctrlreg(xch, domain_id,
> VM_EVENT_X86_CR3, 1, 1,
> 633                                       X86_CR3_PGE, 1);

This line says X86_CR4_PGE (as it should) in the original xen-access.c -
I was just messing with it and forgot. Sorry.


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel