Bug 1217849

Created attachment 1021037 [details]
xfs: Disable preemption when grabbing all icsb counter locks

I submitted this patch for upstream realtime too.

Created attachment 1025110 [details]
xfs: Disable percpu SB on PREEMPT_RT_FULL

Running a test on a large CPU count box with xfs, I hit a live lock
with the following backtraces on several CPUs:

 Call Trace:
  [<ffffffff812c34f8>] __const_udelay+0x28/0x30
  [<ffffffffa033ab9a>] xfs_icsb_lock_cntr+0x2a/0x40 [xfs]
  [<ffffffffa033c871>] xfs_icsb_modify_counters+0x71/0x280 [xfs]
  [<ffffffffa03413e1>] xfs_trans_reserve+0x171/0x210 [xfs]
  [<ffffffffa0378cfd>] xfs_create+0x24d/0x6f0 [xfs]
  [<ffffffff8124c8eb>] ? avc_has_perm_flags+0xfb/0x1e0
  [<ffffffffa0336eeb>] xfs_vn_mknod+0xbb/0x1e0 [xfs]
  [<ffffffffa0337043>] xfs_vn_create+0x13/0x20 [xfs]
  [<ffffffff811b0edd>] vfs_create+0xcd/0x130
  [<ffffffff811b21ef>] do_last+0xb8f/0x1240
  [<ffffffff811b39b2>] path_openat+0xc2/0x490

Looking at the code I see it was stuck at:

STATIC void
xfs_icsb_lock_cntr(
        xfs_icsb_cnts_t *icsbp)
{
        while (test_and_set_bit(XFS_ICSB_FLAG_LOCK, &icsbp->icsb_flags)) {
                ndelay(1000);
        }
}

In xfs_icsb_modify_counters() the code is fine. There's a
preempt_disable() called when taking this bit spinlock and a
preempt_enable() after it is released. The issue is that not all
locations are protected by preempt_disable() when PREEMPT_RT is set.
Namely the places that grab all CPU cntr locks.

STATIC void
xfs_icsb_lock_all_counters(
        xfs_mount_t     *mp)
{
        xfs_icsb_cnts_t *cntp;
        int             i;

        for_each_online_cpu(i) {
                cntp = (xfs_icsb_cnts_t *)per_cpu_ptr(mp->m_sb_cnts, i);
                xfs_icsb_lock_cntr(cntp);
        }
}

STATIC void
xfs_icsb_disable_counter()
{
        [...]
        xfs_icsb_lock_all_counters(mp);
        [...]
        xfs_icsb_unlock_all_counters(mp);
}

STATIC void
xfs_icsb_balance_counter_locked()
{
        [...]
        xfs_icsb_disable_counter();
        [...]
}

STATIC void
xfs_icsb_balance_counter(
        xfs_mount_t     *mp,
        xfs_sb_field_t  fields,
        int             min_per_cpu)
{
        spin_lock(&mp->m_sb_lock);
        xfs_icsb_balance_counter_locked(mp, fields, min_per_cpu);
        spin_unlock(&mp->m_sb_lock);
}

Now, when PREEMPT_RT is not enabled, that spin_lock() disables
preemption. But for PREEMPT_RT, it does not. Although with my test box I
was not able to produce a task state of all tasks, but I'm assuming that
some task called the xfs_icsb_lock_all_counters() and was preempted by
an RT task and could not finish, causing all callers of that lock to
block indefinitely.

Dave Chinner has stated that the scalability of that code will probably
be negated by PREEMPT_RT, and that it is probably best to just disable
the code in question. Also, this code has been rewritten in newer kernels.

Link: http://lkml.kernel.org/r/20150504004844.GA21261@dastard

Please try the new patch. Dave Chinner suggested disabling the code that causes the deadlock, as PREEMPT_RT will have more issues with it which would negate the benefits for it.

*** Bug 1224430 has been marked as a duplicate of this bug. ***

addressed in kernel-rt-3.10.0-229.rt56.153.el6rt

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1138.html