Re: [Xen-devel] [PATCH] mm/page_alloc: always scrub pages given to the allocator

[Sergey Dyasli <sergey.dyasli@xxxxxxxxxx>]

On Mon, 2018-10-01 at 14:54 +0100, George Dunlap wrote:
> On 10/01/2018 02:44 PM, Sergey Dyasli wrote:
> > On Mon, 2018-10-01 at 07:38 -0600, Jan Beulich wrote:
> > > > > > On 01.10.18 at 15:12, <andrew.cooper3@xxxxxxxxxx> wrote:
> > > > 
> > > > On 01/10/18 12:13, Jan Beulich wrote:
> > > > > > > > On 01.10.18 at 11:58, <sergey.dyasli@xxxxxxxxxx> wrote:
> > > > > > 
> > > > > > Having the allocator return unscrubbed pages is a potential security
> > > > > > concern: some domain can be given pages with memory contents of 
> > > > > > another
> > > > > > domain. This may happen, for example, if a domain voluntarily 
> > > > > > releases
> > > > > > its own memory (ballooning being the easiest way for doing this).
> > > > > 
> > > > > And we've always said that in this case it's the domain's 
> > > > > responsibility
> > > > > to scrub the memory of secrets it cares about. Therefore I'm at the
> > > > > very least missing some background on this change of expectations.
> > > > 
> > > > You were on the call when this was discussed, along with the synchronous
> > > > scrubbing in destroydomain.
> > > 
> > > Quite possible, but it has been a while.
> > > 
> > > > Put simply, the current behaviour is not good enough for a number of
> > > > security sensitive usecases.
> > > 
> > > Well, I'm looking forward for Sergey to expand on this in the commit
> > > message.
> > > 
> > > > The main reason however for doing this is the optimisations it enables,
> > > > and in particular, not double scrubbing most of our pages.
> > > 
> > > Well, wait - scrubbing != zeroing (taking into account also what you
> > > say further down).
> > > 
> > > > > > Change the allocator to always scrub the pages given to it by:
> > > > > > 
> > > > > > 1. free_xenheap_pages()
> > > > > > 2. free_domheap_pages()
> > > > > > 3. online_page()
> > > > > > 4. init_heap_pages()
> > > > > > 
> > > > > > Performance testing has shown that on multi-node machines bootscrub
> > > > > > vastly outperforms idle-loop scrubbing. So instead of marking all 
> > > > > > pages
> > > > > > dirty initially, introduce bootscrub_done to track the completion of
> > > > > > the process and eagerly scrub all allocated pages during boot.
> > > > > 
> > > > > I'm afraid I'm somewhat lost: There still is active boot time 
> > > > > scrubbing,
> > > > > or at least I can't see how that might be skipped (other than due to
> > > > > "bootscrub=0"). I was actually expecting this to change at some
> > > > > point. Am I perhaps simply mis-reading this part of the description?
> > > > 
> > > > No.  Sergey tried that, and found a massive perf difference between
> > > > scrubbing in the idle loop and scrubbing at boot.  (1.2s vs 40s iirc)
> > > 
> > > That's not something you can reasonably compare, imo: For one,
> > > it is certainly expected for the background scrubbing to be slower,
> > > simply because of other activity on the system. And then 1.2s
> > > looks awfully small for a multi-Tb system. Yet it is mainly large
> > > systems where the synchronous boot time scrubbing is a problem.
> > 
> > Let me throw in some numbers.
> > 
> > Performance of current idle loop scrubbing is just not good enough:
> > on 8 nodes, 32 CPUs and 512GB RAM machine it takes ~40 seconds to scrub
> > all the memory instead of ~8 seconds for current bootscrub implementation.
> > 
> > This was measured while synchronously waiting for CPUs to scrub all the
> > memory in idle-loop. But scrubbing can happen in background, of course.
> 
> Right, the whole point of idle loop scrubbing is that you *don't*
> syncronously wait for *all* the memory to finish scrubbing before you
> can use part of it.  So why is this an issue for you guys -- what
> concrete problem did it cause, that the full amount of memory took 40s
> to finish scrubbing rather than only 8s?

There is no issue at the moment. Using idle loop to scrub all the memory
is just not viable: it doesn't scale. How long does it currently take
to bootscrub a multi-Tb system? If it takes a few minutes then I fear
that it might take an hour to idle-scrub it.

-- 
Thanks,
Sergey
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel