Re: [Xen-devel] [PATCH RFC] x86/emulate: implement hvmemul_cmpxchg() with an actual CMPXCHG

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 09.04.17 at 00:15, <rcojocaru@xxxxxxxxxxxxxxx> wrote:
> On 04/03/2017 09:36 PM, Razvan Cojocaru wrote:
>> On 04/03/2017 09:20 PM, Razvan Cojocaru wrote:
>>> On 04/01/2017 07:56 PM, Razvan Cojocaru wrote:
>>>> On 03/31/2017 06:04 PM, Jan Beulich wrote:
>>>>>>>> On 31.03.17 at 17:01, <rcojocaru@xxxxxxxxxxxxxxx> wrote:
>>>>>> On 03/31/2017 05:46 PM, Jan Beulich wrote:
>>>>>>>>>> On 31.03.17 at 11:56, <rcojocaru@xxxxxxxxxxxxxxx> wrote:
>>>>>>>> On 03/31/2017 10:34 AM, Jan Beulich wrote:
>>>>>>>>>>>> On 31.03.17 at 08:17, <rcojocaru@xxxxxxxxxxxxxxx> wrote:
>>>>>>>>>> On 03/30/2017 06:47 PM, Jan Beulich wrote:
>>>>>>>>>>>> Speaking of emulated MMIO, I've got this when the guest was 
>>>>>>>>>>>> crashing
>>>>>>>>>>>> immediately (pre RETRY loop):
>>>>>>>>>>>>
>>>>>>>>>>>>  MMIO emulation failed: d3v8 32bit @ 0008:82679f3c -> f0 0f ba 30 
>>>>>>>>>>>> 00 72
>>>>>>>>>>>> 07 8b cb e8 da 4b ff ff 8b 45
>>>>>>>>>>>
>>>>>>>>>>> That's a BTR, which we should be emulating fine. More information
>>>>>>>>>>> would need to be collected to have a chance to understand what
>>>>>>>>>>> might be going one (first of all the virtual and physical memory
>>>>>>>>>>> address this was trying to act on).
>>>>>>>>>>
>>>>>>>>>> Right, the BTR part should be fine, but I think the LOCK part is 
>>>>>>>>>> what's
>>>>>>>>>> causing the issue. I've done a few more test runs to see what return
>>>>>>>>>> RETRY (dumping the instruction with an "(r)" prefix to distinguish 
>>>>>>>>>> from
>>>>>>>>>> the UNHANDLEABLE dump), and a couple of instructions return RETRY 
>>>>>>>>>> (BTR
>>>>>>>>>> and XADD, both LOCK-prefixed, which means they now involve CMPXCHG
>>>>>>>>>> handler, which presumably now fails - possibly simply because it's
>>>>>>>>>> always LOCKed in my patch):
>>>>>>>>>
>>>>>>>>> Well, all of that looks to be expected behavior. I'm afraid I don't 
>>>>>>>>> see
>>>>>>>>> how this information helps understanding the MMIO emulation failure
>>>>>>>>> above.
>>>>>>>>
>>>>>>>> I've managed to obtain this log of emulation errors:
>>>>>>>> https://pastebin.com/Esy1SkHx 
>>>>>>>>
>>>>>>>> The "virtual address" lines that are not followed by any "Mem event"
>>>>>>>> line correspond to CMXCHG_FAILED return codes.
>>>>>>>>
>>>>>>>> The very last line is a MMIO emulation failed.
>>>>>>>>
>>>>>>>> It's probably important that this happens with the model where
>>>>>>>> hvm_emulate_one_vm_event() does _not_ re-try the emulation until it
>>>>>>>> succeeds. The other model allows me to go further with the guest, but
>>>>>>>> eventually I get timeout-related BSODs or the guest becomes 
>>>>>>>> unresponsive.
>>>>>>>
>>>>>>> Interesting. You didn't clarify what the printed "offset" values are,
>>>>>>> and it doesn't look like these have any correlation with the underlying
>>>>>>> (guest) physical address, which we would also want to see. And then
>>>>>>> it strikes me as odd that in these last lines
>>>>>>>
>>>>>>> (XEN) Mem event (RETRY) emulation failed: d5v8 32bit @ 0008:826bb861 -> 
>>>>>>> f0 0f 
> 
>>>>>> ba 30 00 72 07 8b cb e8 da 4b ff ff 8b 45
>>>>>>> (XEN) virtual address: 0xffd080f0, offset: 4291854576
>>>>>>> (XEN) MMIO emulation failed: d5v8 32bit @ 0008:82655f3c -> f0 0f ba 30 
>>>>>>> 00 72 
>>>>>> 07 8b cb e8 da 4b ff ff 8b 45
>>>>>>>
>>>>>>> the instruction pointers and virtual addresses are different, but the
>>>>>>> code bytes are exactly the same. This doesn't seem very likely, so I
>>>>>>> wonder whether there's an issue with us wrongly re-using previously
>>>>>>> fetched insn bytes. (Of course I'd be happy to be proven wrong with
>>>>>>> this guessing, by you checking the involved binary/ies.)
>>>>>>
>>>>>> Offset is the actual value of the "offset" parameter of
>>>>>> hvmemul_cmpxchg().
>>>>>
>>>>> That's not very useful then, as for flat segments "offset" ==
>>>>> "virtual address" (i.e. you merely re-print in decimal what you've
>>>>> already printed in hex).
>>>>
>>>> The attached patch (a combination of your patch and mine) produces the
>>>> following output when booting a Windows 7 32-bit guest with monitoring:
>>>> https://pastebin.com/ayiFmj1N 
>>>>
>>>> The failed MMIO emulation is caused by a mapping failure due to the
>>>> "!nestedhvm_vcpu_in_guestmode(curr) && hvm_mmio_internal(gpa)" condition
>>>> being true in hvmemul_vaddr_to_mfn(). I've ripped that off from
>>>> __hvm_copy() but it looks like that might not be the right way to use it.
>>>
>>> Sorry to reply to this email instead of your original reply but I've
>>> "left it" in my computer at work. Here's the last part of the log, with
>>> the VCPU number logged for the GFN as well:
>>>
>>> (XEN) [8] gfn: 0x2781
>>> (XEN) [8] virtual address: 0x827810a8, rc: 0
>>> (XEN) [8] gfn: 0x2781
>>> (XEN) [8] virtual address: 0x827810a8, rc: 0
>>> (XEN) [8] gfn: 0x2781
>>> (XEN) [8] virtual address: 0x827810cc, rc: 0
>>> (XEN) [8] gfn: 0x2781
>>> (XEN) [8] virtual address: 0x8278109c, rc: 0
>>> (XEN) [8] gfn: 0x2781
>>> (XEN) [8] virtual address: 0x827810d0, rc: 0
>>> (XEN) [11] gfn: 0x2781
>>> (XEN) [8] gfn: 0x2781
>>> (XEN) [11] virtual address: 0x8278109c, rc: 0
>>> (XEN) [8] virtual address: 0x8278109c, rc: 4
>>> (XEN) Dump follows for VCPU 8
>>> (XEN) Mem event (RETRY) emulation failed: d3v8 32bit @ 0008:826b5c7c ->
>>> f0 0f c1 08 85 c9 74 1f f6 c1 02 75 1a 41 8d 41
>>> (XEN) [11] gfn: 0x2781
>>> (XEN) [11] virtual address: 0x827810a8, rc: 0
>>> (XEN) [11] gfn: 0x2781
>>> (XEN) [11] virtual address: 0x827810a8, rc: 0
>>> (XEN) [8] gfn: 0xfed00
>>> (XEN) !page
>>> (XEN) hvmemul_vaddr_to_mfn() fail
>>> (XEN) [8] virtual address: 0xffd080f0, rc: 1
>>> (XEN) MMIO emulation failed: d3v8 32bit @ 0008:8264ff3c -> f0 0f c1 08
>>> 85 c9 74 1f f6 c1 02 75 1a 41 8d 41
>>>
>>> The code does look the same for the last two failures on VCPU 8, for
>>> clearly different GFNs and virtual addresses. The first time it hits a
>>> protected page, we try to emulate the instruction and it fails with
>>> X86EMUL_CMPXCHG_FAILED (rc: 4). Then it somehow pops up again and this
>>> time it's MMIO-emulated, and that fails as well (with UNHANDLEABLE,
>>> since we can't seem to be able to map the memory).
>> 
>> Another log, making sure that the MMIO emulation fail really happens on
>> the same processor as the preceding (identical bytes) fail:
>> 
>> (XEN) [8] gfn: 0x276f
>> (XEN) [11] virtual address: 0x8276f09c, rc: 0
>> (XEN) [8] virtual address: 0x8276f09c, rc: 4
>> (XEN) Dump follows for VCPU 8
>> (XEN) Mem event (RETRY) emulation failed: d3v8 32bit @ 0008:826a3861 ->
>> f0 0f ba 30 00 72 07 8b cb e8 da 4b ff ff 8b 45
>> (XEN) [11] gfn: 0x276f
>> (XEN) [11] virtual address: 0x8276f0d0, rc: 0
>> (XEN) [11] gfn: 0x276f
>> (XEN) [11] virtual address: 0x8276f09c, rc: 0
>> (XEN) [11] gfn: 0x276f
>> (XEN) [11] virtual address: 0x8276f0cc, rc: 0
>> (XEN) [11] gfn: 0x276f
>> (XEN) [11] virtual address: 0x8276f09c, rc: 0
>> (XEN) [8] gfn: 0xfed00
>> (XEN) !page
>> (XEN) hvmemul_vaddr_to_mfn() fail
>> (XEN) [8] virtual address: 0xffd080f0, rc: 1
>> (XEN) Dump follows for VCPU 8
>> (XEN) MMIO emulation failed: d3v8 32bit @ 0008:8263df3c -> f0 0f ba 30
>> 00 72 07 8b cb e8 da 4b ff ff 8b 45
>> (XEN) [11] gfn: 0x276f
>> (XEN) [11] virtual address: 0x8276f0d0, rc: 0
> 
> As you've suggested, this does indeed seem to happen because of the
> current model of emulating because of vm_event replies: an instruction
> needs to be emulated in hvm_do_resume(), but inside
> hvm_emulate_one_vm_event(), hvm_emulate_one() returns RETRY, presumably
> because of the new cmpxchg handler. This causes
> hvm_emulate_one_vm_event() to simply return, doing nothing else. Then,
> the guest resumes execution at the same place (since RIP has not been
> modified), then hvm_do_resume() gets called again, except this time this
> code:
> 
>  481     if ( !handle_hvm_io_completion(v) )
>  482         return;
> 
> ends up trying to MMIO emulate the current instruction by calling
> handle_mmio().
> 
> I've had hvm_emulate_one_vm_event() return what hvm_emulate_one()
> returns, and loop while ( rc == RETRY ) in hvm_do_resume() - this is
> functionally equivalent to what I've been doing before by wrapping the
> hvm_emulate_one_vm_event() code in a do {} while ( rc == RETRY ); loop
> as previously discussed. However, as stated before, at a much later
> point I still get rare BSODs, or the guest becomes unresponsive. Digging
> a bit more, I've found that what seems to be happening is that emulating
> some instructions returns EXCEPTION at some point (for instructions
> using the new cmpxchg handler, this can happen if paging_gva_to_gfn()
> returns INVALID_GFN and neither PFEC_page_paged nor PFEC_page_shared are
> set).

Apparently in line with your later reply - if you got INVALID_GFN
from paging_gva_to_gfn(), the earlier read of the same memory
should have failed already (unless something plays with the P2M
behind your back).

Jan

> Since this causes hvm_emulate_one_vm_event() to
> hvm_inject_event(), I'm not sure that it does not mess with the
> interrupt part of hvm_do_resume(). This will need more testing to figure
> out exactly what's going wrong.
> 
> 
> Thanks,
> Razvan



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel