Re: [Xen-devel] Sporadic PV guest malloc.c assertion failures and segfaults unless pv-l1tf=false is set

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 25/11/2018 09:14, Andy Smith wrote:
> Hello,
>
> On Sun, Nov 25, 2018 at 06:18:49AM +0000, Andy Smith wrote:
>> In the text for XSA-273 it says:
>>
>>     "Shadowing comes with a workload-dependent performance hit to
>>     the guest.  Once the guest kernel software updates have been
>>     applied, a well behaved guest will not write vulnerable PTEs,
>>     and will therefore avoid the performance penalty (or crash)
>>     entirely."
>>
>> Does anyone have a reference to what is needed in the Linux kernel
>> for that?
> Perhaps stupidly, I have only just now thought to check whether the
> one guest I have an easy reproducer on (predictable failure of
> php-fpm) was actually running an up to date kernel. It was not.
>
> It is Debian stretch and was running kernel package
> linux-image-4.9.0-7-amd64 version 4.9.110-3+deb9u2. The guest's
> administrator obviously had not done any upgrades since install time
> because updated kernel linux-image-4.9.0-8-amd64 version 4.9.130-2
> was available.
>
> After installing and booting with that, the guest no longer causes
> "L1TF-vulnerable L1e 000000006a6ff960 - Shadowing" to be emitted in
> the hypervisor dmesg, and the problems I described disappear.
>
> I assume this is because of:
>
> https://metadata.ftp-master.debian.org/changelogs//main/l/linux/linux_4.9.130-2_changelog
>
>     linux (4.9.110-3+deb9u3) stretch-security; urgency=high
>
>       [ Salvatore Bonaccorso ]
>         * Add L1 Terminal Fault fixes (CVE-2018-3620, CVE-2018-3646)
>
> So, I can tell affected guest administrators to upgrade their
> kernels to include Linux's L1TF protections and their problems
> should go away.
>
> Do you care that without those fixes there appear to be memory
> corruption issues? If so, I can keep this reproducer guest around
> and debug it further.

Yes please - we'd like to get to the bottom of this.  Shadow pagetables
should function the same as not using them in the first place, and
clearly there is a bug here.

In terms of your previous question concerning the default for dom0, I
did eventually get to the bottom of that.

https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg00943.html

We had some shadow bugs (different across different releases) when
trying to shadow PV guests with superpages, and this definitely is
specific to dom0.


In terms of debugging this issue, I'm afraid that will be a little more
complicated.

Fundamentally, it will either be insufficient/incorrect TLB flushing, or
something is causing the shadow pagetables to become wrong WRT the
guests tables.  My gut feeling is the former.

Which are your two types of Intel server?  You say that you only see
this with 64bit Debian kernels?  Another dimension here is the use of
PCID as a meltdown mitigation for 64bit PV guests.

Could you experiment with disabling PCID (`pcid=0` on the xen command
line) and seeing if that affects the reproducibility.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel