Hide Forgot
Description of problem: Very commonly run into this in a test with transparent proxy (TPROXY) under a load of around 600-700 Mbps at 2000 connections/sec. from around 25000 unique IP addresses. [root@s01b01 ~]# ------------[ cut here ]------------ kernel BUG at mm/slab.c:3065! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/system/cpu/cpu23/cache/index2/shared_cpu_map CPU 19 Modules linked in: xt_TPROXY xt_socket nf_conntrack nf_defrag_ipv4 nf_tproxy_core ip6table_mangle ip6_tables iptable_mangle xt_MARK ip_tables bmnet(P)(U) bmnetpub(U) 8021q garp stp llc bonding ipv6 tcp_westwood dm_mod tun kvm_intel kvm uinput cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core mptscsih mptbase bnx2 ixgbe(U) dca mdio ext4 mbcache jbd2 Modules linked in: xt_TPROXY xt_socket nf_conntrack nf_defrag_ipv4 nf_tproxy_core ip6table_mangle ip6_tables iptable_mangle xt_MARK ip_tables bmnet(P)(U) bmnetpub(U) 8021q garp stp llc bonding ipv6 tcp_westwood dm_mod tun kvm_intel kvm uinput cdc_ether usbnet mii serio_raw i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support shpchp ioatdma i7core_edac edac_core mptscsih mptbase bnx2 ixgbe(U) dca mdio ext4 mbcache jbd2 Pid: 6290, comm: cftclient Tainted: P ---------------- 2.6.32-71.15.1.el6.x86_64 #1 IBM System x -[7871AC1]- RIP: 0010:[<ffffffff81156984>] [<ffffffff81156984>] cache_alloc_refill+0x1e4/0x240 RSP: 0018:ffff880561ebfcf8 EFLAGS: 00010046 RAX: 0000000000000005 RBX: ffff880c64250a40 RCX: 000000000000003b RDX: ffff880c22177000 RSI: ffff880c64f5ee40 RDI: ffff880c22fa5000 RBP: ffff880561ebfd58 R08: ffff880c22177000 R09: 0000000000000000 R10: ffffffff81c9a600 R11: 0000000000000000 R12: ffff880c64fe9c00 R13: ffff880c64f5ee40 R14: 0000000000000005 R15: ffff880c22fa5000 FS: 00007f39d3634720(0000) GS:ffff8806954e0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000369b2da060 CR3: 0000000bba30f000 CR4: 00000000000026e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process cftclient (pid: 6290, threadinfo ffff880561ebe000, task ffff880513a414a0) Stack: ffff880561ebfd78 00000001812071b1 ffff880c64f5ee80 0004122004445860 <0> ffff880c64f5ee60 ffff880c64f5ee50 00000000ffffffff ffffffff81c9a600 <0> 0000000000000020 ffff880c64250a40 0000000000000020 0000000000000246 Call Trace: [<ffffffff8115741f>] kmem_cache_alloc+0x15f/0x190 [<ffffffff8144a4cc>] inet_bind_bucket_create+0x2c/0x80 [<ffffffff8144cc1b>] inet_csk_get_port+0x22b/0x4a0 [<ffffffff814729ca>] inet_bind+0x10a/0x1f0 [<ffffffff813ff610>] sys_bind+0xd0/0xf0 [<ffffffff813ff6f6>] ? sys_setsockopt+0xc6/0xe0 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b Code: 89 ff e8 60 29 11 00 eb 99 66 0f 1f 44 00 00 41 c7 45 60 01 00 00 00 4d 8b 7d 20 4c 39 7d c0 0f 85 f2 fe ff ff eb 84 0f 0b eb fe <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 eb f4 8b 55 ac 8b 75 bc 31 RIP [<ffffffff81156984>] cache_alloc_refill+0x1e4/0x240 RSP <ffff880561ebfcf8> ---[ end trace 784289e9b623fdfb ]--- Kernel panic - not syncing: Fatal exception in interrupt Pid: 6290, comm: cftclient Tainted: P D ---------------- 2.6.32-71.15.1.el6.x86_64 #1 Call Trace: [<ffffffff814c8633>] panic+0x78/0x137 [<ffffffff814cc712>] oops_end+0xf2/0x100 [<ffffffff8101733b>] die+0x5b/0x90 [<ffffffff814cbfb4>] do_trap+0xc4/0x160 [<ffffffff81014ee5>] do_invalid_op+0x95/0xb0 [<ffffffff81156984>] ? cache_alloc_refill+0x1e4/0x240 [<ffffffff81013f5b>] invalid_op+0x1b/0x20 [<ffffffff81156984>] ? cache_alloc_refill+0x1e4/0x240 [<ffffffff811568fb>] ? cache_alloc_refill+0x15b/0x240 [<ffffffff8115741f>] kmem_cache_alloc+0x15f/0x190 [<ffffffff8144a4cc>] inet_bind_bucket_create+0x2c/0x80 [<ffffffff8144cc1b>] inet_csk_get_port+0x22b/0x4a0 [<ffffffff814729ca>] inet_bind+0x10a/0x1f0 [<ffffffff813ff610>] sys_bind+0xd0/0xf0 [<ffffffff813ff6f6>] ? sys_setsockopt+0xc6/0xe0 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b Version-Release number of selected component (if applicable): kernel-2.6.32-71.15.1.el6.x86_64 How reproducible: Very commonly run into this in a test with transparent proxy (TPROXY) under a load of around 600-700 Mbps at 2000 connections/sec. from around 25000 unique IP addresses. Actual results: Kernel panic. Expected results: No kernel panic.
Neal, is this using your reproducer, or does this only happen at the customer site? If its here, can you provide me access to the system in question?
Hi Neil, This only happens in the customer's environment. I have been unable to reproduce this panic in our local environment.
Since RHEL 6.1 External Beta has begun, and this bug remains unresolved, it has been rejected as it is not proposed as exception or blocker. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
This issue is fixed in 6.1 and 6.0.z