Bug 1292927 - rcuc starvation leads to rcu stall
rcuc starvation leads to rcu stall
Status: ON_QA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel-rt (Show other bugs)
7.3
x86_64 Linux
high Severity high
: rc
: 7.3
Assigned To: Scott Wood
Jianlin Shi
: ZStream
Depends On:
Blocks: 1420851 1442258 1449577 1293229 1295885
  Show dependency treegraph
 
Reported: 2015-12-18 13:56 EST by Daniel Bristot de Oliveira
Modified: 2017-09-18 12:14 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1293229 (view as bug list)
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
rt: rcu: Boost rcuc if it has 2 jiffies before splatting (3.18 KB, patch)
2015-12-18 14:14 EST, Clark Williams
no flags Details | Diff
rcu: Boost rcuc if it has 4 jiffies before splatting (3.25 KB, patch)
2016-01-06 21:43 EST, Luis Claudio R. Goncalves
no flags Details | Diff
softirq: Perform softirqs in local_bh_enable() for a limited amount of time (3.64 KB, patch)
2016-01-06 21:44 EST, Luis Claudio R. Goncalves
no flags Details | Diff
Server side script for testing the NAPI-POLL sfc issue (1.29 KB, text/plain)
2017-04-19 11:46 EDT, Luis Claudio R. Goncalves
no flags Details
Client side script for testing the NAPI-poll sfc issue (1.64 KB, text/plain)
2017-04-19 11:49 EDT, Luis Claudio R. Goncalves
no flags Details

  None (edit)
Description Daniel Bristot de Oliveira 2015-12-18 13:56:40 EST
Description of problem:

Kernel emits RCU stall messages if a RT task with priority higher than or equals
to FIFO:2 runs for more than 60 seconds.

This occurs due to the starvation of the rcuc/ threads, that runs with priority
FIFO:2.

How reproducible:
Always

Steps to Reproduce:
1. set rcu_nocbs=$CPU kernel parameter for a given CPU. For instance, CPU 3:

   Edit the file /etc/sysconfig/grub, and add the "rcu_nocbs=3" parameter in the
   GRUB_CMDLINE_LINUX= option.

   Apply grub's config using the following command:
   # grub2-mkconfig > /etc/grub2.cfg

2. Reboot the system

3. Move all rcu threads to a house keeping CPU. For example, to the CPU 0:
   # for i in `pgrep rcu[^c]` ; do taskset -pc 0 $i > /dev/null ; done

4. Run hackbench (from rt-tests package) on the CPU configured as rcu_nocbs,
   with a FIFO prio >= rcuc/CPU thread's prio. On or example, the CPU 3:

   # taskset -c 3 chrt -f 10 hackbench -g 2 -f 2 -s 1000 -l 10000000000 > /dev/null 2&>1
   
5. Wait for 61 seconds, kill hackbench, and run dmesg to see RCU stall messages.

Actual results:
   RCU stall messages in the CPU that is running hackbench.

Expected results:
   No RCU stall messages.

Additional info:
  Workaround: Increase rcuc threads priority.
Comment 2 Clark Williams 2015-12-18 14:14 EST
Created attachment 1107358 [details]
rt: rcu: Boost rcuc if it has 2 jiffies before splatting

Patch to prevent RCU starvation on RT kernel
Comment 5 Luis Claudio R. Goncalves 2016-01-06 21:43 EST
Created attachment 1112322 [details]
rcu: Boost rcuc if it has 4 jiffies before splatting
Comment 6 Luis Claudio R. Goncalves 2016-01-06 21:44 EST
Created attachment 1112323 [details]
softirq: Perform softirqs in local_bh_enable() for a limited amount of time
Comment 8 Jianlin Shi 2016-09-29 04:11:54 EDT
RCU stall messages still appear when run reproducer on 3.10.0-510.


[root@ibm-x3650m5-03 ~]# uname -a
Linux ibm-x3650m5-03.rhts.eng.pek2.redhat.com 3.10.0-510.rt56.415.el7.x86_64 #1 SMP PREEMPT RT Wed Sep 21 16:48:53 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@ibm-x3650m5-03 ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-510.rt56.415.el7.x86_64 root=/dev/mapper/rhel_ibm--x3650m5--03-root ro crashkernel=auto rd.lvm.lv=rhel_ibm-x3650m5-03/root rd.lvm.lv=rhel_ibm-x3650m5-03/swap console=ttyS0,115200n81 LANG=en_US.UTF-8 rcu_nocbs=3
[root@ibm-x3650m5-03 ~]# for i in `pgrep rcu[^c]` ; do taskset -pc 0 $i > /dev/null ; done
[root@ibm-x3650m5-03 ~]# taskset -c 3 chrt -f 10 hackbench -g 2 -f 2 -s 1000 -l 10000000000 > /dev/null 2&>1


ibm-x3650m5-03 login: [  201.042244] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=60000 jiffies g=3269 c=3268 q=5479)
[  201.042245] Task dump for CPU 3:
[  201.042246] hackbench       R  running task        0  3208   3207 0x00000080
[  201.042260]  ffff88086cc7c080 00000000d580a3e4 ffff88046fac3d90 ffffffff810bc1a6
[  201.042261]  0000000000000003 ffffffff81a01f00 ffff88046fac3da8 ffffffff810bffd9
[  201.042262]  0000000000000004 ffff88046fac3dd8 ffffffff8112fdc0 ffff88046fad20e0
[  201.042262] Call Trace:
[  201.042269]  <IRQ>  [<ffffffff810bc1a6>] sched_show_task+0xb6/0x120
[  201.042271]  [<ffffffff810bffd9>] dump_cpu_task+0x39/0x70
[  201.042274]  [<ffffffff8112fdc0>] rcu_dump_cpu_stacks+0x90/0xd0
[  201.042276]  [<ffffffff8113467a>] rcu_check_callbacks+0x49a/0x840
[  201.042279]  [<ffffffff8108f0e2>] update_process_times+0x42/0x70
[  201.042281]  [<ffffffff810e98d5>] tick_sched_handle.isra.18+0x25/0x60
[  201.042282]  [<ffffffff810e9aa4>] tick_sched_timer+0x44/0x70
[  201.042285]  [<ffffffff810ab605>] __run_hrtimer+0x85/0x270
[  201.042286]  [<ffffffff810e9a60>] ? tick_sched_do_timer+0x50/0x50
[  201.042288]  [<ffffffff810ac460>] hrtimer_interrupt+0x120/0x2a0
[  201.042291]  [<ffffffff810437f7>] local_apic_timer_interrupt+0x37/0x60
[  201.042295]  [<ffffffff8169292f>] smp_apic_timer_interrupt+0x3f/0x60
[  201.042297]  [<ffffffff8169109d>] apic_timer_interrupt+0x6d/0x80
[  201.042300]  <EOI>  [<ffffffff810a3320>] ? task_work_run+0xe0/0xe0
[  201.042303]  [<ffffffff812a5d59>] ? sock_has_perm+0x49/0xc0
[  201.042305]  [<ffffffff812a5eb3>] selinux_socket_recvmsg+0x23/0x30
[  201.042307]  [<ffffffff812a2fb6>] security_socket_recvmsg+0x16/0x20
[  201.042310]  [<ffffffff8154accc>] sock_aio_read.part.7+0xdc/0x160
[  201.042312]  [<ffffffff8154ad71>] sock_aio_read+0x21/0x30
[  201.042314]  [<ffffffff811f2b9d>] do_sync_read+0x8d/0xd0
[  201.042316]  [<ffffffff811f340d>] vfs_read+0x14d/0x170
[  201.042317]  [<ffffffff811f3f2f>] SyS_read+0x7f/0xe0
[  201.042318]  [<ffffffff81690409>] system_call_fastpath+0x16/0x1b


BTW, also tried on 3.10.0-327.10.1.rt56.211.el7_2.x86_64, no RCU stall messages appeared.
Comment 9 Clark Williams 2016-09-29 11:05:25 EDT
(In reply to Jianlin Shi from comment #8)
> RCU stall messages still appear when run reproducer on 3.10.0-510.
> 

Looks like I was missing a section of the patch which modified kernel/rcutree.c (the else clause). I've updated the patch and when my current -511 build finishes, I'll kick off a new build (also -511 based) which should fix this issue. 

I'll move to Modified when the build is done.
Comment 21 Luis Claudio R. Goncalves 2017-04-19 11:46 EDT
Created attachment 1272681 [details]
Server side script for testing the NAPI-POLL  sfc issue

This script should run at x3650m2-01.farm.hsv.redhat.com (RT test box in the Huntsville lab).

From a console session run:

./SERVER

This test requires netperf2, already in the box. There is a note in the script showing how to get netperf2 code if necessary.
Comment 22 Luis Claudio R. Goncalves 2017-04-19 11:49 EDT
Created attachment 1272682 [details]
Client side script for testing the NAPI-poll sfc issue

This script mut run at rhelrt-17.farm.hsv.redhat.com (RT test box at the Huntsville lab).

From and ssh session run:

./CLIENT_17

Go to the console session and wait for the error.
Comment 26 Scott Wood 2017-06-20 23:58:35 EDT
Setting back to MODIFIED as there are patches in the tree to fix the originally reported issue, which I cannot reproduce in the current tree.  The netdev stall is not RT-specific and not related to the previous issues in this BZ.  If there are any other scenarios that lead to RCU stalls in the current tree, please file a new BZ with specific reproduction instructions.

Note You need to log in before you can comment on or make changes to this bug.