[PATCH RFC v2 14/15] rcu-tasks: Retire switched-out tasks with no trampoline nesting at scan time

[Josef Bacik <josef@xxxxxxxxxxxxxx>]

The previous patch lets a task report its own quiescent state when it is
preempted with rcu_tramp_nesting == 0, but a task that was preempted
before the grace period started and simply has not run since is still
listed as a holdout until it next passes through __schedule().  As Paul
pointed out, the grace-period kthread can settle that case itself.

For a task that is switched out, task_call_func() pins it and
!task_curr() tells us it left the CPU through __schedule(), whose
locking orders its last rcu_tramp_nesting and rcu_tasks_irq_ip updates
before ours; a task preempted from irq exit inside trampoline text has
the count held non-zero across the switch by irqentry_preempt().  So a
pinned, not-running task with a zero count is neither in nor called
from a trampoline, and with rcu_tasks_irq_ip_holds() also clear (the
kprobe jump window, which can open after the task was switched out) it
is quiescent now, whether or not it ever runs again.  Check that in
rcu_tasks_pertask() so such tasks never become holdouts, and in
check_holdout_task() so they are retired on the next scan.

This is the only remote reader of the count, and it only reads it for a
pinned, switched-out task, so the trampoline-side increment and
decrement stay plain on every preemption model, PREEMPT_RT included.

Only under CONFIG_RCU_TASKS_PREEMPT_QS; other architectures are
unchanged.

Suggested-by: Paul E. McKenney <paulmck@xxxxxxxxxx>
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@xxxxxxxxxxxxxx>
---
 kernel/rcu/tasks.h | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)

diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index bab08a666dc0..ba432bd922e2 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -1014,10 +1014,39 @@ static bool rcu_tasks_is_holdout(struct task_struct *t)
        return true;
 }
 
+#ifdef CONFIG_RCU_TASKS_PREEMPT_QS
+/* task_call_func() callback: is @t switched out with no trampoline in play? */
+static int rcu_tasks_switched_out_clean(struct task_struct *t, void *arg)
+{
+       /*
+        * With @t pinned, !task_curr() means it last left the CPU through
+        * __schedule(), so its rcu_tramp_nesting and rcu_tasks_irq_ip are
+        * stable and ordered before the rq lock we hold.  A task preempted
+        * from irq exit inside trampoline text has the count held non-zero
+        * across the switch by irqentry_preempt(), so zero here means neither
+        * in nor called from a trampoline; rcu_tasks_irq_ip_holds() covers the
+        * one case that can become true after the task was switched out (a
+        * kprobe jump-optimization window).  Both clear: already quiescent,
+        * whether or not it ever runs again.
+        */
+       return !task_curr(t) && !READ_ONCE(t->rcu_tramp_nesting) &&
+              !rcu_tasks_irq_ip_holds(t);
+}
+
+/* Is @t, right now, switched out somewhere that is a quiescent state? */
+static bool rcu_tasks_preempted_qs(struct task_struct *t)
+{
+       return task_call_func(t, rcu_tasks_switched_out_clean, NULL);
+}
+#else
+static bool rcu_tasks_preempted_qs(struct task_struct *t) { return false; }
+#endif
+
 /* Per-task initial processing. */
 static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop)
 {
-       if (t != current && rcu_tasks_is_holdout(t)) {
+       if (t != current && rcu_tasks_is_holdout(t) &&
+           !rcu_tasks_preempted_qs(t)) {
                get_task_struct(t);
                t->rcu_tasks_nvcsw = READ_ONCE(t->nvcsw);
                WRITE_ONCE(t->rcu_tasks_holdout, true);
@@ -1181,6 +1210,7 @@ static void check_holdout_task(struct task_struct *t,
        if (!READ_ONCE(t->rcu_tasks_holdout) ||
            t->rcu_tasks_nvcsw != READ_ONCE(t->nvcsw) ||
            !rcu_tasks_is_holdout(t) ||
+           rcu_tasks_preempted_qs(t) ||
            (IS_ENABLED(CONFIG_NO_HZ_FULL) &&
             !is_idle_task(t) && READ_ONCE(t->rcu_tasks_idle_cpu) >= 0)) {
                WRITE_ONCE(t->rcu_tasks_holdout, false);

-- 
2.55.0