Description of problem: While testing for latencies, the hp-dl580g8 system I was running on locked up and spit out a bunch of NMI back traces of all the CPUs. Looking into it, it seems that xfs has its own home grown bit spinlock that is mostly held with preemption disabled. But there is a few cases that preemption is not disabled when PREEMPT_RT is enabled (spin_locks() do not disable preemption in that case), and if a holder of the bit spinlock is preempted and is prevented by a higher priority task from running, then it is possible that the bit spinlock will never be released, locking up all new users that try to grab it.
Created attachment 1021037 [details] xfs: Disable preemption when grabbing all icsb counter locks I submitted this patch for upstream realtime too.
Created attachment 1025110 [details] xfs: Disable percpu SB on PREEMPT_RT_FULL Running a test on a large CPU count box with xfs, I hit a live lock with the following backtraces on several CPUs: Call Trace: [<ffffffff812c34f8>] __const_udelay+0x28/0x30 [<ffffffffa033ab9a>] xfs_icsb_lock_cntr+0x2a/0x40 [xfs] [<ffffffffa033c871>] xfs_icsb_modify_counters+0x71/0x280 [xfs] [<ffffffffa03413e1>] xfs_trans_reserve+0x171/0x210 [xfs] [<ffffffffa0378cfd>] xfs_create+0x24d/0x6f0 [xfs] [<ffffffff8124c8eb>] ? avc_has_perm_flags+0xfb/0x1e0 [<ffffffffa0336eeb>] xfs_vn_mknod+0xbb/0x1e0 [xfs] [<ffffffffa0337043>] xfs_vn_create+0x13/0x20 [xfs] [<ffffffff811b0edd>] vfs_create+0xcd/0x130 [<ffffffff811b21ef>] do_last+0xb8f/0x1240 [<ffffffff811b39b2>] path_openat+0xc2/0x490 Looking at the code I see it was stuck at: STATIC void xfs_icsb_lock_cntr( xfs_icsb_cnts_t *icsbp) { while (test_and_set_bit(XFS_ICSB_FLAG_LOCK, &icsbp->icsb_flags)) { ndelay(1000); } } In xfs_icsb_modify_counters() the code is fine. There's a preempt_disable() called when taking this bit spinlock and a preempt_enable() after it is released. The issue is that not all locations are protected by preempt_disable() when PREEMPT_RT is set. Namely the places that grab all CPU cntr locks. STATIC void xfs_icsb_lock_all_counters( xfs_mount_t *mp) { xfs_icsb_cnts_t *cntp; int i; for_each_online_cpu(i) { cntp = (xfs_icsb_cnts_t *)per_cpu_ptr(mp->m_sb_cnts, i); xfs_icsb_lock_cntr(cntp); } } STATIC void xfs_icsb_disable_counter() { [...] xfs_icsb_lock_all_counters(mp); [...] xfs_icsb_unlock_all_counters(mp); } STATIC void xfs_icsb_balance_counter_locked() { [...] xfs_icsb_disable_counter(); [...] } STATIC void xfs_icsb_balance_counter( xfs_mount_t *mp, xfs_sb_field_t fields, int min_per_cpu) { spin_lock(&mp->m_sb_lock); xfs_icsb_balance_counter_locked(mp, fields, min_per_cpu); spin_unlock(&mp->m_sb_lock); } Now, when PREEMPT_RT is not enabled, that spin_lock() disables preemption. But for PREEMPT_RT, it does not. Although with my test box I was not able to produce a task state of all tasks, but I'm assuming that some task called the xfs_icsb_lock_all_counters() and was preempted by an RT task and could not finish, causing all callers of that lock to block indefinitely. Dave Chinner has stated that the scalability of that code will probably be negated by PREEMPT_RT, and that it is probably best to just disable the code in question. Also, this code has been rewritten in newer kernels. Link: http://lkml.kernel.org/r/20150504004844.GA21261@dastard
Please try the new patch. Dave Chinner suggested disabling the code that causes the deadlock, as PREEMPT_RT will have more issues with it which would negate the benefits for it.
*** Bug 1224430 has been marked as a duplicate of this bug. ***
addressed in kernel-rt-3.10.0-229.rt56.153.el6rt
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1138.html