Re: [Xen-devel] xsave=0 workaround needed on 3.2 kernels with Xen 4.1 or Xen-unstable.

[AP <apxeng@xxxxxxxxx>]

On Mon, May 7, 2012 at 7:15 AM, Jan Beulich <JBeulich@xxxxxxxx> wrote:
>
> >>> On 04.05.12 at 21:30, AP <apxeng@xxxxxxxxx> wrote:
> > From the above I realized that X86_CR4_OSXSAVE was never getting set
> > in v->arch.pv_vcpu.ctrlreg[4].
>
> Yes, that was the observation in the previous thread too, but the
> reporter didn't seem interested in continuing on from there.
>
>
> > So I tried the following patch:
> >
> > diff -r 5a0d60bb536b xen/arch/x86/domain.c
> > --- a/xen/arch/x86/domain.c   Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/arch/x86/domain.c   Fri May 04 12:23:57 2012 -0700
> > @@ -691,8 +691,6 @@ unsigned long pv_guest_cr4_fixup(const s
> >          hv_cr4_mask &= ~X86_CR4_DE;
> >      if ( cpu_has_fsgsbase && !is_pv_32bit_domain(v->domain) )
> >          hv_cr4_mask &= ~X86_CR4_FSGSBASE;
> > -    if ( xsave_enabled(v) )
> > -        hv_cr4_mask &= ~X86_CR4_OSXSAVE;
> >
> >      if ( (guest_cr4 & hv_cr4_mask) != (hv_cr4 & hv_cr4_mask) )
> >          gdprintk(XENLOG_WARNING,
> > diff -r 5a0d60bb536b xen/include/asm-x86/domain.h
> > --- a/xen/include/asm-x86/domain.h    Fri Apr 27 21:10:59 2012 -0700
> > +++ b/xen/include/asm-x86/domain.h    Fri May 04 12:23:57 2012 -0700
> > @@ -530,7 +530,7 @@ unsigned long pv_guest_cr4_fixup(const s
> >       & ~X86_CR4_DE)
> >  #define real_cr4_to_pv_guest_cr4(c)                         \
> >      ((c) & ~(X86_CR4_PGE | X86_CR4_PSE | X86_CR4_TSD        \
> > -             | X86_CR4_OSXSAVE | X86_CR4_SMEP))
> > +             | X86_CR4_SMEP))
> >
> >  void domain_cpuid(struct domain *d,
> >                    unsigned int  input,
>
> No, this is specifically the wrong thing. From what we know so far
> (i.e. the outcome of the above printing you added) the problem in
> in the Dom0 kernel (in it never setting CR4.OSXSAVE prior to
> attempting XSETBV). What your patch efectively does is take away
> control from the guest kernels to control the (virtual) CR4 flag...
>
> > That allowed the system to boot successfully though I did see the
> > following message:
> > (XEN) domain.c:698:d0 Attempt to change CR4 flags 00042660 -> 00002660
>
> ... which is what this message is telling you.
>
> > Not sure if the above patch is right fix but I hope it was at least
> > helpful in pointing at where the problem might be.
> >
> > BTW, I see the same invalid op issue with Xen 4.1.2 if I boot with
> > xsave=1.
>
> Sure, as it's a kernel problem. It's the kernel that needs logging added,
> to find out why the CR4 write supposedly happening immediately
> prior to the XSETBV (set_in_cr4(X86_CR4_OSXSAVE)) doesn't actually
> happen, or doesn't set the flag. Perhaps something fishy going on

xen_write_cr4 explicitly turns off X86_CR4_OSXSAVE.

static void xen_write_cr4(unsigned long cr4)
{
    cr4 &= ~X86_CR4_PGE;
    cr4 &= ~X86_CR4_PSE;
    cr4 &= ~X86_CR4_OSXSAVE;

    native_write_cr4(cr4);
}

I added some prints to the Ubuntu kernel (3.0.0-17) to confirm. Here
is what I see:

<snip>
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    6.834419] xstate_enable: Setting X86_CR4_OSXSAVE
[    6.838041] set_in_cr4: cr4: 0x42660
[    6.841743] xen_write_cr4: cr4: 0x2660
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x2660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x2660 write CR4: 0x426f0
[    6.860546] xstate_enable: Exec xsetbv
(XEN) traps.c:2254:d0 XSETBV: lock: 0 rep_prefix: 0 opsize_prefix: 0 cr4: 0x2660
[    6.870509] invalid opcode: 0000 [#1] SMP
<snip>

Removing the explicit turning off of X86_CR4_OSXSAVE allowed the system to boot.

(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    7.554262] Setting X86_CR4_OSXSAVE
[    7.557869] set_in_cr4: cr4: 0x42660
[    7.561551] xen_write_cr4: cr4: 0x42660
(XEN) domain.c:704:d0 vcpu[0] hv_cr4: 0x2660 hv_cr4_mask:
0xfffffffffffbfff3 returning cr4: 0x42660
(XEN) traps.c:2409:d0 vcpu[0] pv cr4: 0x42660 write CR4: 0x426f0
[    7.580522] Exec xsetbv
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
(XEN) traps.c:874:d0 vcpu[0] cpuid XSAVE supported
[    7.597071] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340

Thanks,
AP

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel