Bug 703578

Summary:	Kernel panic in cache_alloc_refill
Product:	Red Hat Enterprise Linux 6	Reporter:	Neal Kim <nkim>
Component:	kernel	Assignee:	Thomas Graf <tgraf>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Network QE <network-qe>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	6.0	CC:	james.brown, jwest, kzhang, mepley, rkhan, sbest, sforsber
Target Milestone:	rc
Target Release:	6.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2011-10-25 10:45:26 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Neal Kim 2011-05-10 17:53:16 UTC

Description of problem:

Very commonly run into this in a test with transparent proxy (TPROXY) under a load of around 600-700 Mbps at 2000 connections/sec. from around 25000 unique IP addresses.

[root@s01b01 ~]# ------------[ cut here ]------------
kernel BUG at mm/slab.c:3065!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/cpu23/cache/index2/shared_cpu_map
CPU 19
Modules linked in: xt_TPROXY xt_socket nf_conntrack nf_defrag_ipv4 nf_tproxy_core ip6table_mangle ip6_tables iptable_mangle xt_MARK ip_tables bmnet(P)(U) bmnetpub(U) 8021q garp stp llc bonding ipv6 tcp_westwood dm_mod tun kvm_intel kvm uinput cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core mptscsih mptbase bnx2 ixgbe(U) dca mdio ext4 mbcache jbd2

Modules linked in: xt_TPROXY xt_socket nf_conntrack nf_defrag_ipv4 nf_tproxy_core ip6table_mangle ip6_tables iptable_mangle xt_MARK ip_tables bmnet(P)(U) bmnetpub(U) 8021q garp stp llc bonding ipv6 tcp_westwood dm_mod tun kvm_intel kvm uinput cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core mptscsih mptbase bnx2 ixgbe(U) dca mdio ext4 mbcache jbd2
Pid: 6290, comm: cftclient Tainted: P ---------------- 2.6.32-71.15.1.el6.x86_64 #1 IBM System x -[7871AC1]-
RIP: 0010:[<ffffffff81156984>] [<ffffffff81156984>] cache_alloc_refill+0x1e4/0x240
RSP: 0018:ffff880561ebfcf8 EFLAGS: 00010046
RAX: 0000000000000005 RBX: ffff880c64250a40 RCX: 000000000000003b
RDX: ffff880c22177000 RSI: ffff880c64f5ee40 RDI: ffff880c22fa5000
RBP: ffff880561ebfd58 R08: ffff880c22177000 R09: 0000000000000000
R10: ffffffff81c9a600 R11: 0000000000000000 R12: ffff880c64fe9c00
R13: ffff880c64f5ee40 R14: 0000000000000005 R15: ffff880c22fa5000
FS: 00007f39d3634720(0000) GS:ffff8806954e0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000369b2da060 CR3: 0000000bba30f000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process cftclient (pid: 6290, threadinfo ffff880561ebe000, task ffff880513a414a0)
Stack:
ffff880561ebfd78 00000001812071b1 ffff880c64f5ee80 0004122004445860
<0> ffff880c64f5ee60 ffff880c64f5ee50 00000000ffffffff ffffffff81c9a600
<0> 0000000000000020 ffff880c64250a40 0000000000000020 0000000000000246
Call Trace:
[<ffffffff8115741f>] kmem_cache_alloc+0x15f/0x190
[<ffffffff8144a4cc>] inet_bind_bucket_create+0x2c/0x80
[<ffffffff8144cc1b>] inet_csk_get_port+0x22b/0x4a0
[<ffffffff814729ca>] inet_bind+0x10a/0x1f0
[<ffffffff813ff610>] sys_bind+0xd0/0xf0
[<ffffffff813ff6f6>] ? sys_setsockopt+0xc6/0xe0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b
Code: 89 ff e8 60 29 11 00 eb 99 66 0f 1f 44 00 00 41 c7 45 60 01 00 00 00 4d 8b 7d 20 4c 39 7d c0 0f 85 f2 fe ff ff eb 84 0f 0b eb fe <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 eb f4 8b 55 ac 8b 75 bc 31
RIP [<ffffffff81156984>] cache_alloc_refill+0x1e4/0x240
RSP <ffff880561ebfcf8>
---[ end trace 784289e9b623fdfb ]---
Kernel panic - not syncing: Fatal exception in interrupt
Pid: 6290, comm: cftclient Tainted: P D ---------------- 2.6.32-71.15.1.el6.x86_64 #1
Call Trace:
[<ffffffff814c8633>] panic+0x78/0x137
[<ffffffff814cc712>] oops_end+0xf2/0x100
[<ffffffff8101733b>] die+0x5b/0x90
[<ffffffff814cbfb4>] do_trap+0xc4/0x160
[<ffffffff81014ee5>] do_invalid_op+0x95/0xb0
[<ffffffff81156984>] ? cache_alloc_refill+0x1e4/0x240
[<ffffffff81013f5b>] invalid_op+0x1b/0x20
[<ffffffff81156984>] ? cache_alloc_refill+0x1e4/0x240
[<ffffffff811568fb>] ? cache_alloc_refill+0x15b/0x240
[<ffffffff8115741f>] kmem_cache_alloc+0x15f/0x190
[<ffffffff8144a4cc>] inet_bind_bucket_create+0x2c/0x80
[<ffffffff8144cc1b>] inet_csk_get_port+0x22b/0x4a0
[<ffffffff814729ca>] inet_bind+0x10a/0x1f0
[<ffffffff813ff610>] sys_bind+0xd0/0xf0
[<ffffffff813ff6f6>] ? sys_setsockopt+0xc6/0xe0
[<ffffffff81013172>] system_call_fastpath+0x16/0x1b

Version-Release number of selected component (if applicable):

kernel-2.6.32-71.15.1.el6.x86_64


How reproducible:

Very commonly run into this in a test with transparent proxy (TPROXY) under a load of around 600-700 Mbps at 2000 connections/sec. from around 25000 unique IP addresses.

  
Actual results:

Kernel panic.


Expected results:

No kernel panic.

Comment 1 Neil Horman 2011-05-10 18:06:08 UTC

Neal, is this using your reproducer, or does this only happen at the customer site?  If its here, can you provide me access to the system in question?

Comment 3 Neal Kim 2011-05-10 18:15:28 UTC

Hi Neil,

This only happens in the customer's environment. I have been unable to reproduce this panic in our local environment.

Comment 4 RHEL Program Management 2011-05-11 06:01:36 UTC

Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 16 Thomas Graf 2011-10-25 10:45:26 UTC

This issue is fixed in 6.1 and 6.0.z