Re: [Xen-devel] Xen Introspection, KPTI, and CR3 bit 63 leads to guest VMENTRY failures during introspection

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 24/01/2018 22:31, Bitweasil . wrote:
> I've recently discovered that if you attempt to use introspection to
> capture CR3 changes with the new KPTI enabled kernels, the guest dies
> shortly after the start of introspection with failed VM entry due to
> invalid guest state.
>
> I believe the invalid state here is the high bit being set in CR3 -
> while this is how one indicates that PCID should not invalidate the
> various page table caches, introspection leads to this being set in
> the VMCS, which appears to be wrong.
>
> With the XenServer 4.7.1 code base (which is my working code base at
> the moment), I have not found a way around this, as the
> vm_event_set_registers function (xen/arch/x86/vm_event.c) does not set
> the CR3 value, and vm_event_register_write_resume only allows
> inhibiting the write, not writing a modified value.
>
> I've attempted several ways to work around this with a livepatch, and
> have not (yet) been successful.
>
> Masking at the top of hvm_set_cr3 allows the guest to continue, but
> appears to do the wrong thing with regards to the guest (tasks begin
> dying quickly from invalid opcode errors).
>
> In any case, Andrew mentions that this appears to still be an issue in
> staging, so this likely needs addressing.  At this point in time, I
> believe guests with KPTI enabled cannot be introspected if that
> introspection involves capturing CR3 changes.
>
> Please let me know if you need any more details on this issue!

Just as an FYI to people reading this, that is actually XenServer 7.1's
hypervisor which is Xen 4.7.1-based but the fact that the HVM CR3 code
has little-to-no clue about PCID appears to be unchanged into staging. 
Sadly, it doesn't appear to be trivial to fix.

~Andrew

>
> -Bit
>
>
> (XEN) [19458.318035] Failed vm entry (exit reason 0x80000021) caused
> by invalid guest state (0).
> (XEN) [19458.318042] ************* VMCS Area **************
> (XEN) [19458.318050] *** Guest State ***
> (XEN) [19458.318056] CR0: actual=0x000000008005003b,
> shadow=0x0000000080050033, gh_mask=ffffffffffffffff
> (XEN) [19458.318062] CR4: actual=0x0000000000362670,
> shadow=0x0000000000360670, gh_mask=ffffffffffffffff
> (XEN) [19458.318069] CR3 = 0x800000001ded7080
> (XEN) [19458.318076] PDPTE0 = 0x0000000000020000  PDPTE1 =
> 0x000006f800150018
> (XEN) [19458.318082] PDPTE2 = 0x0000000000000000  PDPTE3 =
> 0x000006f800150018
> (XEN) [19458.318089] RSP = 0xffff880015b87f50 (0xffff880015b87f50) 
> RIP = 0xffffffff81845857 (0xffffffff81845857)
> (XEN) [19458.318095] RFLAGS=0x00000082 (0x00000082)  DR7 =
> 0x0000000000000400
> (XEN) [19458.318101] Sysenter RSP=0000000000000000
> CS:RIP=0010:ffffffff8184a220
> (XEN) [19458.318105]        sel  attr  limit   base
> (XEN) [19458.318112]   CS: 0010 0a09b ffffffff 0000000000000000
> (XEN) [19458.318119]   DS: 0000 1c000 ffffffff 0000000000000000
> (XEN) [19458.318126]   SS: 0018 0c093 ffffffff 0000000000000000
> (XEN) [19458.318133]   ES: 0000 1c000 ffffffff 0000000000000000
> (XEN) [19458.318140]   FS: 0000 1c000 ffffffff 00007fde038ba700
> (XEN) [19458.318147]   GS: 0000 1c000 ffffffff ffff88001ba00000
> (XEN) [19458.318152] GDTR:            0000007f ffff88001ba0c000
> (XEN) [19458.318158] LDTR: 0000 1c000 ffffffff 0000000000000000
> (XEN) [19458.318164] IDTR:            00000fff ffffffffff574000
> (XEN) [19458.318169]   TR: 0040 0008b 00002087 ffff88001ba048c0
> (XEN) [19458.318175] EFER = 0x0000000000000000  PAT = 0x0407010600070106
> (XEN) [19458.318179] PreemptionTimer = 0x00000000  SM Base = 0x00000000
> (XEN) [19458.318185] DebugCtl = 0x0000000000000000  DebugExceptions =
> 0x0000000000000000
> (XEN) [19458.318233] PerfGlobCtl = 0x0000000000000000  BndCfgS =
> 0x0000000000000000
> (XEN) [19458.318297] Interruptibility = 00000000  ActivityState = 00000000
> (XEN) [19458.318324] *** Host State ***
> (XEN) [19458.318329] RIP = 0xffff82d0801ee100
> (vmx_asm_vmexit_handler)  RSP = 0xffff8300bfcfff90
> (XEN) [19458.318333] CS=e008 SS=0000 DS=0000 ES=0000 FS=0000 GS=0000
> TR=e040
> (XEN) [19458.318335] FSBase=0000000000000000 GSBase=0000000000000000
> TRBase=ffff82d08035e780
> (XEN) [19458.318337] GDTBase=ffff82d0802d9000 IDTBase=ffff82d080357ce0
> (XEN) [19458.318339] CR0=000000008005003b CR3=000000010f001000
> CR4=00000000003526e0
> (XEN) [19458.318341] Sysenter RSP=ffff8300bfcfffc0
> CS:RIP=e008:ffff82d08022bb30
> (XEN) [19458.318343] EFER = 0x0000000000000000  PAT = 0x0000050100070406
> (XEN) [19458.318344] *** Control State ***
> (XEN) [19458.318347] PinBased=0000003f CPUBased=b6a0e5fa
> SecondaryExec=001014ea
> (XEN) [19458.318348] EntryControls=000153ff ExitControls=008fefff
> (XEN) [19458.318350] ExceptionBitmap=00060082 PFECmask=00000000
> PFECmatch=00000000
> (XEN) [19458.318352] VMEntry: intr_info=00000000 errcode=00000000
> ilen=00000000
> (XEN) [19458.318353] VMExit: intr_info=00000000 errcode=00000000
> ilen=00000003
> (XEN) [19458.318355]         reason=80000021
> qualification=0000000000000000
> (XEN) [19458.318357] IDTVectoring: info=00000000 errcode=00000000
> (XEN) [19458.318359] TSC Offset = 0xffffd23bbd8772ac  TSC Multiplier =
> 0x0000000000000000
> (XEN) [19458.318361] TPR Threshold = 0x00  PostedIntrVec = 0x00
> (XEN) [19458.318365] EPT pointer = 0x000000010ee9501e  EPTP index = 0x0000
> (XEN) [19458.318396] PLE Gap=00000080 Window=00001000
> (XEN) [19458.318402] Virtual processor ID = 0xccd3 VMfunc controls =
> 0000000000000000
> (XEN) [19458.318406] **************************************
> (XEN) [19458.318412] domain_crash called from
> vmx_vmexit_handler+0x4ab/0x19f5
> (XEN) [19458.318417] Domain 15 (vcpu#0) crashed on cpu#0:
> (XEN) [19458.318443] ----[ Xen-4.7.1-1.0  x86_64  debug=n  Not tainted
> ]----
> (XEN) [19458.318448] CPU:    0
> (XEN) [19458.318453] RIP:    0010:[<ffffffff81845857>]
> (XEN) [19458.318458] RFLAGS: 0000000000000082   CONTEXT: hvm guest (d15v0)
> (XEN) [19458.318466] rax: 800000001ded7080   rbx: 0000000000000000  
> rcx: 00007fde033ce730
> (XEN) [19458.318470] rdx: 00000000000000fa   rsi: 0000000000000002  
> rdi: 00007ffd8ee85250
> (XEN) [19458.318484] rbp: 00007ffd8ee85410   rsp: ffff880015b87f50  
> r8:  0000000000000000
> (XEN) [19458.318498] r9:  0000000000000017   r10: 0000000000000000  
> r11: 0000000000000246
> (XEN) [19458.318502] r12: 00007ffd8ee85250   r13: 0000000000000000  
> r14: 0000000000000004
> (XEN) [19458.318525] r15: 000055a8503b3828   cr0: 0000000080050033  
> cr4: 0000000000360670
> (XEN) [19458.318538] cr3: 800000001ded7080   cr2: 00007ffef290a090
> (XEN) [19458.318552] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss:
> 0018   cs: 0010
>


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel