Re: [Xen-devel] Xen Introspection, KPTI, and CR3 bit 63 leads to guest VMENTRY failures during introspection

[Razvan Cojocaru <rcojocaru@xxxxxxxxxxxxxxx>]

On 01/25/2018 02:15 AM, Andrew Cooper wrote:
> On 24/01/2018 22:31, Bitweasil . wrote:
>> I've recently discovered that if you attempt to use introspection to
>> capture CR3 changes with the new KPTI enabled kernels, the guest dies
>> shortly after the start of introspection with failed VM entry due to
>> invalid guest state.
>>
>> I believe the invalid state here is the high bit being set in CR3 -
>> while this is how one indicates that PCID should not invalidate the
>> various page table caches, introspection leads to this being set in
>> the VMCS, which appears to be wrong.
>>
>> With the XenServer 4.7.1 code base (which is my working code base at
>> the moment), I have not found a way around this, as the
>> vm_event_set_registers function (xen/arch/x86/vm_event.c) does not set
>> the CR3 value, and vm_event_register_write_resume only allows
>> inhibiting the write, not writing a modified value.
>>
>> I've attempted several ways to work around this with a livepatch, and
>> have not (yet) been successful.
>>
>> Masking at the top of hvm_set_cr3 allows the guest to continue, but
>> appears to do the wrong thing with regards to the guest (tasks begin
>> dying quickly from invalid opcode errors).
>>
>> In any case, Andrew mentions that this appears to still be an issue in
>> staging, so this likely needs addressing.  At this point in time, I
>> believe guests with KPTI enabled cannot be introspected if that
>> introspection involves capturing CR3 changes.
>>
>> Please let me know if you need any more details on this issue!
> 
> Just as an FYI to people reading this, that is actually XenServer 7.1's
> hypervisor which is Xen 4.7.1-based but the fact that the HVM CR3 code
> has little-to-no clue about PCID appears to be unchanged into staging. 
> Sadly, it doesn't appear to be trivial to fix.

Well, FWIW we do support masks for CR events since Xen 4.10 - we can
simply mask whatever bits we _don't_ want events sent out for. I don't
know if this solves Bitweasil's problem, but it's certainly something to
take into consideration.

For example, looking at xen-access.c:

629     if ( write_ctrlreg_cr4 )
630     {
631         /* Mask the CR4.PGE bit so no events will be generated for
global TLB flushes. */
632         rc = xc_monitor_write_ctrlreg(xch, domain_id,
VM_EVENT_X86_CR3, 1, 1,
633                                       X86_CR3_PGE, 1);
634         if ( rc < 0 )
635         {
636             ERROR("Error %d setting write control register trapping
with vm_event\n", rc);
637             goto exit;
638         }
639     }
640


Thanks,
Razvan

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel