Re: [Xen-devel] [PATCH] x86/nmi: lower initial watchdog frequency to avoid boot hangs

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 08/02/18 09:12, Jan Beulich wrote:
>>>> On 07.02.18 at 18:08, <andrew.cooper3@xxxxxxxxxx> wrote:
>> On 07/02/18 15:06, Jan Beulich wrote:
>>> Also you completely ignore my argument against the seemingly
>>> random division by 10, including the resulting question of what you
>>> mean to do once 10Hz also turns out too high a frequency.
>> We've got to pick a frequency.  The current 100Hz is just as arbitrary
>> as the proposed new 10Hz.
> Not exactly - the 100Hz is simply the frequency we run the main tick
> at, so while random it is not as random as any further derived value
> which has no proper reason behind it.
>
> There's one more point wrt your argument of overhead: If servicing
> an NMI takes that long on those boxes, you're basically saying you
> are happy to waste at least 1% of a core's bandwidth on a
> debugging feature. Is that reasonable for a production setup? And
> considering that I'd expect the patch to have chosen e.g. HZ / 5,
> HZ / 4, or even HZ / 2 if that worked reliably, I could even conclude
> you're happy to spend somewhere between 5 and 10% of one
> core's bandwidth. (FAOD all this is based on the 1Hz frequency we
> - iirc - run the NMI at later on.) To me this is another clear argument
> to turn off the watchdog on those systems, rather than trying to
> "fix" its probing.

It is not a debugging feature; it's a reliability feature.  With
clustered storage in particular, it is absolutely paramount to guarantee
that a struggling host fences itself cleanly, or you lose the entire
cluster.

This particular issue is a failure to boot, but by far the most common
issue we see in the field is a fence when all-but-one CPU is waiting in
the calibration rendezvous, by which point the host has effectively been
dead for 5 seconds already.  Turning the watchdog off isn't a viable or
reasonable solution to the problem.

We switch the NMI frequency to ~2Hz after the calibration, but that is
after having run the BSP at 100Hz for a long period of time, and the APs
at this rate for a short while.  Irrespective of the exact fix here, it
is simply not a good idea to be running with this NMI frequency, other
than possibly during the immediate calibration logic.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel