Re: alternatives+livepatch testing

["Doebel, Bjoern" <doebel@xxxxxxxxx>]

Hi,

> Hello,
>
> The recent hiccup with CET-IBT, and discovery that livepatch-build-tools
> have been broken for several releases, demonstrates that we do not have
> remotely adequate testing in place.  We need to address, and ensure we
> don't end up in the same position again.
>
> Alternatives and Livepatching have a number of overlapping test
> requirements, so how about the following plan:
>
> 1) Introduce a new $arch/scm-tests.c, with something akin to the
> existing stub_selftest().  I'm tempted to move stub_selftest() out of
> initcall and call it from init_done() (before we clobber .init.text)
> because that gets shstk testing included.
>
> Even without livepatching, we've got various requirements such as
> endbr's only existing where expected, and getting clobbered when
> suitably annotated, and altcalls turning into UD for a still-NULL pointer.
>
> Items not yet upstream but on the radar include inlining of retpoline
> thunks and SLS workarounds, which would also fit into this.
>
> 2) Provide (in xen.git) a patch to scm-tests.c which OSSTest/other can
> use livepatch-build-tools on to generate a real livepatch, and a new
> livepatching subop which can be invoked from xen-livepatch in userspace
> that will run the same kind of consistency checks as 1) on the patched
> content.
>
> This lets us create specific constructs and confirm that they get
> patched correctly, without having to specifically execute the result.  I
> (think) we can do everything needed without reference to the livepatch
> metadata, which simplifies things.
>
> Providing a patch isn't totally ideal from a "maintaining xen" point of
> view, but I think we can have a build-time test which confirms the patch
> is still good, and it is definitely the right primitive to use for the
> end-to-end testing.
I can share a bit what we are doing with our internal livepatch testing:

* We have a simple test livepatch, similar to 
xen/test/livepatch/xen_prepost_hooks.c. We build and load this livepatch 
both in our nightly test setup as well as when testing production 
builds. This test only validates livepatch functionality in the 
hypervisor works and I think it would not have detected the build problem.
* To detect the build issue, we need an actual diff-based livepatch. We 
are actually using a non-trivial patch here, which enables FEP in Xen at 
load time, because we have tests that make use of FEP later on.
* If you are concerned about maintainability, I'd go for a very simple 
diff to a location that won't change a lot. For my CET-IBT adaption, I 
went for
--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -405,6 +405,8 @@ static int cf_check vmx_domain_initialise(struct 
domain *d)
         .tail = vmx_do_resume,
     };

+    printk("vmx_domain_initialise called\n");
+
     d->arch.ctxt_switch = &csw;

     /*

==> This has the disadvantage of being arch-specific. There are likely 
more generic locations in Xen that haven't been changed for years, 
indicating likelihood of not requiring too many updates in the future.
* From a testing perspective it is a good idea to have a patch like the 
one above with a visible side effect, because that allows you to later 
also have testing that loads that actual patch and triggers the 
functionality to observe the side effect.
So in summary, your plan sounds reasonable.

Bjoern



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879