Bug 751975

Summary: F16 Alpha:rcu_sched_state detected stall on CPU 2 (t=12662100 jiffies)
Product: [Fedora] Fedora Reporter: IBM Bug Proxy <bugproxy>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 16CC: gansalmon, itamar, jonathan, kernel-maint, madhu.chinakonda, pknirsch, wgomerin
Target Milestone: ---   
Target Release: ---   
Hardware: ppc64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-13 19:00:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description IBM Bug Proxy 2011-11-08 08:20:48 UTC 
Problem Description
------------------------------------------------
While running ltpstress test on Fedora16 on P7 Juno-L systems,noticed following message appear on console .While the console still echoes characters system was not accessible at all , went to hung state .To reclaim the system , I did a system reboot from HMC.

[253857.643634] INFO: rcu_sched_state detected stall on CPU 2 (t=12662100 jiffies)
[253857.643634] INFO: rcu_sched_state detected stall on CPU 2 (t=12662100 jiffies)
[253857.643634] INFO: rcu_sched_state detected stall on CPU 2 (t=12662100 jiffies)

Also oom killer got produced noticed in /var/log/messages
Nov  7 02:17:55 c57f1ju0204 kernel: [241004.795551] Out of memory: Kill process 2346 (genload) score 61 or sacrifice child
Nov  7 02:17:55 c57f1ju0204 kernel: [241004.795557] Killed process 2346 (genload) total-vm:1052672kB, anon-rss:544896kB, file-rss:0kB
Nov  7 02:18:03 c57f1ju0204 kernel: [241013.722158] genload invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722186] genload cpuset=/ mems_allowed=0-1
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722195] Call Trace:
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722208] [c0000000fa017370] [c000000000014c64] .show_stack+0x94/0x144 (unreliable)
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722225] [c0000000fa017430] [c00000000066df0c] .dump_stack+0x24/0x2c
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722239] [c0000000fa0174b0] [c00000000015aaac] .dump_header+0xac/0x1dc
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722252] [c0000000fa0175c0] [c00000000015aedc] .oom_kill_process+0x68/0x2b0
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722265] [c0000000fa0176a0] [c00000000015b7a0] .out_of_memory+0x368/0x3e4
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722277] [c0000000fa017780] [c0000000001601a4] .__alloc_pages_nodemask+0x650/0x7d4
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722291] [c0000000fa017910] [c00000000019ab2c] .alloc_pages_vma+0x168/0x188
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722306] [c0000000fa0179c0] [c00000000017d808] .handle_pte_fault+0x1d0/0xb84
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722319] [c0000000fa017ac0] [c00000000017f180] .handle_mm_fault+0x1a4/0x1b4
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722332] [c0000000fa017b80] [c0000000006677ac] .do_page_fault+0x474/0x704
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722345] [c0000000fa017e30] [c000000000006438] handle_page_fault+0x20/0x74
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722356] Mem-Info:
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722363] Node 0 DMA per-cpu:
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722372] CPU    0: hi:    6, btch:   1 usd:   0
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722379] CPU    1: hi:    6, btch:   1 usd:   2
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722388] CPU    2: hi:    6, btch:   1 usd:   0
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722397] CPU    3: hi:    6, btch:   1 usd:   1
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722405] CPU    4: hi:    6, btch:   1 usd:   0
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722413] CPU    5: hi:    6, btch:   1 usd:   0
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722421] CPU    6: hi:    6, btch:   1 usd:   0
Nov  7 02:18:04 c57f1ju0204 kernel: [241013.722428] CPU    7: hi:    6, btch:   1 usd:   0


 Additional information 
-------------------------------------------------------
[root@c57f1ju0204 ~]# uname -a
Linux c57f1ju0204.ppd.pok.ibm.com 3.1.0-0.rc9.git0.2.fc16.kh.ppc64 #1 SMP Wed Oct 12 22:41:01 UTC 2011 ppc64 ppc64 ppc64 GNU/Linux

[root@c57f1ju0204 ~]# cat /proc/meminfo 
MemTotal:        8318144 kB
MemFree:         7523136 kB
Buffers:           42368 kB
Cached:           416640 kB
SwapCached:            0 kB
Active:           342976 kB
Inactive:         237824 kB
Active(anon):     137792 kB
Inactive(anon):   117696 kB
Active(file):     205184 kB
Inactive(file):   120128 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2047936 kB
SwapFree:        2047936 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        123840 kB
Mapped:            44224 kB
Shmem:            133248 kB
Slab:              81536 kB
SReclaimable:      17728 kB
SUnreclaim:        63808 kB
KernelStack:        2400 kB
PageTables:        13952 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6206976 kB
Committed_AS:     515328 kB
VmallocTotal:   8589934592 kB
VmallocUsed:       30784 kB
VmallocChunk:   8589884160 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:      16384 kB

[root@c57f1ju0204 ~]# lscpu 
Architecture:          ppc64
Byte Order:            Big Endian
CPU(s):                16
On-line CPU(s) list:   0-7
Off-line CPU(s) list:  8-15
Thread(s) per core:    4
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          2
Model:                 IBM,8246-L2B
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              4096K
NUMA node0 CPU(s):     0-3
NUMA node1 CPU(s):     4-7
Comment 1 Phil Knirsch 2012-01-13 19:00:40 UTC 
Fixed.

Thanks & regards, Phil