[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen Introspection, KPTI, and CR3 bit 63 leads to guest VMENTRY failures during introspection

  • To: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, "Bitweasil ." <bitweasil@xxxxxxxxxxxxxx>, xen-devel@xxxxxxxxxxxxx
  • From: Razvan Cojocaru <rcojocaru@xxxxxxxxxxxxxxx>
  • Date: Thu, 25 Jan 2018 14:25:03 +0200
  • Cc: tamas@xxxxxxxxxxxxx, jbeulich@xxxxxxxx
  • Comment: DomainKeys? See http://domainkeys.sourceforge.net/
  • Delivery-date: Thu, 25 Jan 2018 12:25:19 +0000
  • Domainkey-signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=bitdefender.com; b=r9jZmZ/JbhWgHZ3Kdozz35OJelLZ3oyNk4CfzmsYtcqXbaxLtFS0EYkqRMM53dDcNYybYSqd8GqzlaSs8thGgu/adoba/U8/zZibuiqnfYrEbhW6nngNXwDul6OZTVjlaLK8k+u8TUVck4nmdTQtC0r4o0UnX/duCGgd8/IRsN3W71iT6QWZ/L4ITZKT4a82Dz44MDmDrves9VpyssRboG+cQrHog/B3f5lKaFtkaGlel8uiCUAbJ/fPrLLol/IiKQoTDAMwAc6/OITJ9hS/TnVk/EZkffHGIx1ACvzWQH4dW5r80HnV4Mc9dm/i0CjRh0Tr78xJjMta96u+mos77w==; h=Received:Received:Received:Received:Subject:To:Cc:References:From:Message-ID:Date:User-Agent:MIME-Version:In-Reply-To:Content-Type:Content-Language:Content-Transfer-Encoding;
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

On 01/25/2018 02:15 AM, Andrew Cooper wrote:
> On 24/01/2018 22:31, Bitweasil . wrote:
>> I've recently discovered that if you attempt to use introspection to
>> capture CR3 changes with the new KPTI enabled kernels, the guest dies
>> shortly after the start of introspection with failed VM entry due to
>> invalid guest state.
>> I believe the invalid state here is the high bit being set in CR3 -
>> while this is how one indicates that PCID should not invalidate the
>> various page table caches, introspection leads to this being set in
>> the VMCS, which appears to be wrong.
>> With the XenServer 4.7.1 code base (which is my working code base at
>> the moment), I have not found a way around this, as the
>> vm_event_set_registers function (xen/arch/x86/vm_event.c) does not set
>> the CR3 value, and vm_event_register_write_resume only allows
>> inhibiting the write, not writing a modified value.
>> I've attempted several ways to work around this with a livepatch, and
>> have not (yet) been successful.
>> Masking at the top of hvm_set_cr3 allows the guest to continue, but
>> appears to do the wrong thing with regards to the guest (tasks begin
>> dying quickly from invalid opcode errors).
>> In any case, Andrew mentions that this appears to still be an issue in
>> staging, so this likely needs addressing.  At this point in time, I
>> believe guests with KPTI enabled cannot be introspected if that
>> introspection involves capturing CR3 changes.
>> Please let me know if you need any more details on this issue!
> Just as an FYI to people reading this, that is actually XenServer 7.1's
> hypervisor which is Xen 4.7.1-based but the fact that the HVM CR3 code
> has little-to-no clue about PCID appears to be unchanged into staging. 
> Sadly, it doesn't appear to be trivial to fix.

Well, FWIW we do support masks for CR events since Xen 4.10 - we can
simply mask whatever bits we _don't_ want events sent out for. I don't
know if this solves Bitweasil's problem, but it's certainly something to
take into consideration.

For example, looking at xen-access.c:

629     if ( write_ctrlreg_cr4 )
630     {
631         /* Mask the CR4.PGE bit so no events will be generated for
global TLB flushes. */
632         rc = xc_monitor_write_ctrlreg(xch, domain_id,
VM_EVENT_X86_CR3, 1, 1,
633                                       X86_CR3_PGE, 1);
634         if ( rc < 0 )
635         {
636             ERROR("Error %d setting write control register trapping
with vm_event\n", rc);
637             goto exit;
638         }
639     }


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.