[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen Introspection, KPTI, and CR3 bit 63 leads to guest VMENTRY failures during introspection

  • To: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, "Bitweasil ." <bitweasil@xxxxxxxxxxxxxx>, xen-devel@xxxxxxxxxxxxx
  • From: Razvan Cojocaru <rcojocaru@xxxxxxxxxxxxxxx>
  • Date: Thu, 25 Jan 2018 14:27:34 +0200
  • Cc: tamas@xxxxxxxxxxxxx, jbeulich@xxxxxxxx
  • Comment: DomainKeys? See http://domainkeys.sourceforge.net/
  • Delivery-date: Thu, 25 Jan 2018 12:27:45 +0000
  • Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=bitdefender.com; b=UnU1BwK8R6R0Wz1OstJKRPeUyxWMOcc/HKq9i286Zxx+mPvN7b0evTnKW8p+6xvzhqIhAezTYmEQYxQv9Tc12Y6FvuX5ACc9pZDhIZmpvbwGMU5Q/f2vJyX98ShpjDIvBmQ7jurxXPd6+J5N36fbTsHT6MH+nuTyRu3xky629T7ArcJ1kBUpvq0qOpVm70A9sjDurcZx06AT53cX0FBxxpLHLnBaqRaWDaZnl0sjzeCETAd1g7Yb+MDlWyiv03VzWXqmRlQw8V+sahFIMAKpazSZwmbU2PPFFCEI+6XR4eD/Piix9McxYvkvwvc38Oc53zRBY8czr8Ww9EJu0JFG5g==; h=Received:Received:Received:Received:Subject:From:To:Cc:References:Message-ID:Date:User-Agent:MIME-Version:In-Reply-To:Content-Type:Content-Language:Content-Transfer-Encoding;
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 01/25/2018 02:25 PM, Razvan Cojocaru wrote:
> On 01/25/2018 02:15 AM, Andrew Cooper wrote:
>> On 24/01/2018 22:31, Bitweasil . wrote:
>>> I've recently discovered that if you attempt to use introspection to
>>> capture CR3 changes with the new KPTI enabled kernels, the guest dies
>>> shortly after the start of introspection with failed VM entry due to
>>> invalid guest state.
>>> I believe the invalid state here is the high bit being set in CR3 -
>>> while this is how one indicates that PCID should not invalidate the
>>> various page table caches, introspection leads to this being set in
>>> the VMCS, which appears to be wrong.
>>> With the XenServer 4.7.1 code base (which is my working code base at
>>> the moment), I have not found a way around this, as the
>>> vm_event_set_registers function (xen/arch/x86/vm_event.c) does not set
>>> the CR3 value, and vm_event_register_write_resume only allows
>>> inhibiting the write, not writing a modified value.
>>> I've attempted several ways to work around this with a livepatch, and
>>> have not (yet) been successful.
>>> Masking at the top of hvm_set_cr3 allows the guest to continue, but
>>> appears to do the wrong thing with regards to the guest (tasks begin
>>> dying quickly from invalid opcode errors).
>>> In any case, Andrew mentions that this appears to still be an issue in
>>> staging, so this likely needs addressing.  At this point in time, I
>>> believe guests with KPTI enabled cannot be introspected if that
>>> introspection involves capturing CR3 changes.
>>> Please let me know if you need any more details on this issue!
>> Just as an FYI to people reading this, that is actually XenServer 7.1's
>> hypervisor which is Xen 4.7.1-based but the fact that the HVM CR3 code
>> has little-to-no clue about PCID appears to be unchanged into staging. 
>> Sadly, it doesn't appear to be trivial to fix.
> Well, FWIW we do support masks for CR events since Xen 4.10 - we can
> simply mask whatever bits we _don't_ want events sent out for. I don't
> know if this solves Bitweasil's problem, but it's certainly something to
> take into consideration.
> For example, looking at xen-access.c:
> 629     if ( write_ctrlreg_cr4 )
> 630     {
> 631         /* Mask the CR4.PGE bit so no events will be generated for
> global TLB flushes. */
> 632         rc = xc_monitor_write_ctrlreg(xch, domain_id,
> VM_EVENT_X86_CR3, 1, 1,
> 633                                       X86_CR3_PGE, 1);

This line says X86_CR4_PGE (as it should) in the original xen-access.c -
I was just messing with it and forgot. Sorry.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.