Xen project Mailing List

Razvan -

Yes, that patch resolves the issues. Performance is abysmal (as expected with a CR3 switch on every syscall), but things behave properly.

Jan -

It turns out that unrelated code in my livepatch was causing the crashing when I filtered the high bit - I was not properly gating some other CR3 processing code in the livepatch, and that was causing the invalid opcode errors, not the filtered CR3 value. You are correct that masking the bit should not cause any correctness issues, only performance issues.

Will one of you take up submitting this patch into the proper places? I'm afraid I'm not very familiar with the Xen patch submission process.

Thank you!

-Bit

On Thu, Jan 25, 2018 at 8:07 AM, Razvan Cojocaru <rcojocaru@xxxxxxxxxxxxxxx> wrote:

On 01/25/2018 12:31 AM, Bitweasil . wrote:
> I've recently discovered that if you attempt to use introspection to
> capture CR3 changes with the new KPTI enabled kernels, the guest dies
> shortly after the start of introspection with failed VM entry due to
> invalid guest state.
>
> I believe the invalid state here is the high bit being set in CR3 -
> while this is how one indicates that PCID should not invalidate the
> various page table caches, introspection leads to this being set in the
> VMCS, which appears to be wrong.
>
> With the XenServer 4.7.1 code base (which is my working code base at the
> moment), I have not found a way around this, as the
> vm_event_set_registers function (xen/arch/x86/vm_event.c) does not set
> the CR3 value, and vm_event_register_write_resume only allows inhibiting
> the write, not writing a modified value.
>
> I've attempted several ways to work around this with a livepatch, and
> have not (yet) been successful.
>
> Masking at the top of hvm_set_cr3 allows the guest to continue, but
> appears to do the wrong thing with regards to the guest (tasks begin
> dying quickly from invalid opcode errors).
>
> In any case, Andrew mentions that this appears to still be an issue in
> staging, so this likely needs addressing. At this point in time, I
> believe guests with KPTI enabled cannot be introspected if that
> introspection involves capturing CR3 changes.
>
> Please let me know if you need any more details on this issue!

I've managed to reproduce the problem and this patch has fixed it for me
with Xen staging:

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index db282b5..7be962e 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -2320,6 +2320,9 @@ int hvm_set_cr3(unsigned long value, bool_t may_defer)
}
}

+ if ( hvm_pcid_enabled(v) )
+ value &= ((1ull << 63) - 1);
+
if ( hvm_paging_enabled(v) && !paging_mode_hap(v->domain) &&
(value != v->arch.hvm_vcpu.guest_cr[3]) )
{

Would you mind giving this a spin and confirm it also solves your problems?

Thanks,
Razvan

Re: [Xen-devel] Xen Introspection, KPTI, and CR3 bit 63 leads to guest VMENTRY failures during introspection