Created attachment 474367 [details]
Picture of another panic

Wasn't able to capture the full text since this occurred during shutdown.

In kfree():

        if (unlikely(!PageSlab(page))) {
                BUG_ON(!PageCompound(page));

Has this been happening all along or did it start with a recent kernel update?

(In reply to comment #3)
> Has this been happening all along or did it start with a recent kernel update?

I haven't tested thoroughly with the other 2.6.35 kernels, but it does look like the issue might exist in 2.6.35.9-64. I am about to start testing 2.6.35.6-48 to see if it has similar problems.

Not sure if this is helpful but I have another stack from 2.6.35.10-74.

Jan 20 15:46:01 o12-30 [28597.628126] kernel BUG at mm/slub.c:2834!
Jan 20 15:46:02 o12-30 [28597.628126] invalid opcode: 0000 [#1] 
Jan 20 15:46:02 o12-30 SMP
Jan 20 15:46:02 o12-30 
Jan 20 15:46:02 o12-30 [28597.628126] last sysfs file: /sys/module/ipmi_si/refcnt
Jan 20 15:46:02 o12-30 [28597.628126] CPU 1 
Jan 20 15:46:02 o12-30 
Jan 20 15:46:02 o12-30 [28597.628126] Modules linked in:
Jan 20 15:46:02  nfsd
Jan 20 15:46:02  lockd
Jan 20 15:46:02  nfs_acl
Jan 20 15:46:02  auth_rpcgss
Jan 20 15:46:02  sunrpc
Jan 20 15:46:02  exportfs
Jan 20 15:46:02  ipt_REDIRECT
Jan 20 15:46:02  xt_NOTRACK
Jan 20 15:46:02  iptable_raw
Jan 20 15:46:02  fuse
Jan 20 15:46:02  ipmi_si(-)
Jan 20 15:46:02  ipmi_msghandler
Jan 20 15:46:02  ipv6
Jan 20 15:46:02  iptable_nat
Jan 20 15:46:02  nf_nat
Jan 20 15:46:02  dm_multipath
Jan 20 15:46:02  netconsole
Jan 20 15:46:02  configfs
Jan 20 15:46:02  radeon
Jan 20 15:46:02  ttm
Jan 20 15:46:02  drm_kms_helper
Jan 20 15:46:02  drm
Jan 20 15:46:02  mptsas
Jan 20 15:46:02  i2c_algo_bit
Jan 20 15:46:02  i2c_core
Jan 20 15:46:02  iTCO_wdt
Jan 20 15:46:02  tg3
Jan 20 15:46:02  mptscsih
Jan 20 15:46:02  dcdbas
Jan 20 15:46:02  iTCO_vendor_support
Jan 20 15:46:02  hed
Jan 20 15:46:02  mptbase
Jan 20 15:46:02  i5100_edac
Jan 20 15:46:02  serio_raw
Jan 20 15:46:02  edac_core
Jan 20 15:46:02  scsi_transport_sas
Jan 20 15:46:02  [last unloaded: ipmi_devintf]
Jan 20 15:46:02 o12-30 
Jan 20 15:46:02 o12-30 [28597.628126]
Jan 20 15:46:02 o12-30 [28597.628126] Pid: 21798, comm: modprobe Not tainted 2.6.35.10-74.fc14.x86_64 #1 0TY179/PowerEdge R300
Jan 20 15:46:02 o12-30 [28597.628126] RIP: 0010:[<ffffffff81108a6e>] 
Jan 20 15:46:02  [<ffffffff81108a6e>] kfree+0x69/0xd7
Jan 20 15:46:02 o12-30 [28597.628126] RSP: 0018:ffff88021e8f9d88  EFLAGS: 00010246
Jan 20 15:46:02 o12-30 [28597.628126] RAX: 0040000000100078 RBX: ffff880200000000 RCX: 000000000015000e
Jan 20 15:46:02 o12-30 [28597.628126] RDX: 0040000000100078 RSI: ffffea00076d56d0 RDI: ffff880200000000
Jan 20 15:46:03 o12-30 [28597.628126] RBP: ffff88021e8f9db8 R08: ffff88021e8f9cc8 R09: ffff880002015500
Jan 20 15:46:03 o12-30 [28597.628126] R10: ffff880200000001 R11: ffff88021e8f9d18 R12: ffff88021f3d66f8
Jan 20 15:46:03 o12-30 [28597.628126] R13: ffffffff81219e79 R14: ffffea0007000000 R15: 000000000040f190
Jan 20 15:46:03 o12-30 [28597.628126] FS:  00007fe42bfa1720(0000) GS:ffff880002080000(0000) knlGS:0000000000000000
Jan 20 15:46:03 o12-30 [28597.628126] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 20 15:46:03 o12-30 [28597.628126] CR2: 00007fe42bb411a0 CR3: 0000000225cdd000 CR4: 00000000000406e0
Jan 20 15:46:03 o12-30 [28597.628126] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 20 15:46:03 o12-30 [28597.628126] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 20 15:46:03 o12-30 [28597.628126] Process modprobe (pid: 21798, threadinfo ffff88021e8f8000, task ffff880225b2c5c0)
Jan 20 15:46:03 o12-30 [28597.628126] Stack:
Jan 20 15:46:03 o12-30 [28597.628126] ffff880200000000
Jan 20 15:46:03  ffff88021f3d66c0
Jan 20 15:46:03  ffff88021f3d66f8
Jan 20 15:46:03  ffffffff81a90210
Jan 20 15:46:03 o12-30 
Jan 20 15:46:03 o12-30 [28597.628126] <0>
Jan 20 15:46:03  ffff880200000000
Jan 20 15:46:04  000000000040f190
Jan 20 15:46:04  ffff88021e8f9de8
Jan 20 15:46:04  ffffffff81219e79
Jan 20 15:46:04 o12-30 
Jan 20 15:46:04 o12-30 [28597.628126] <0>
Jan 20 15:46:04  ffff88021f3d66f8
Jan 20 15:46:04  ffffffff81219d50
Jan 20 15:46:04  ffffffff81a7bf70
Jan 20 15:46:04  ffffffffa010f870
Jan 20 15:46:04 o12-30 
Jan 20 15:46:04 o12-30 [28597.628126] Call Trace:
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff81219e79>] kobject_release+0x129/0x1d9
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff81219d50>] ? kobject_release+0x0/0x1d9
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff8121b382>] kref_put+0x43/0x4d
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff81219cc7>] kobject_put+0x47/0x4b
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff812d7b08>] bus_remove_driver+0xbd/0xed
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff812d8463>] driver_unregister+0x6c/0x74
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff81233795>] pci_unregister_driver+0x44/0x89
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffffa010d4dc>] cleanup_ipmi_si+0x30/0xa8 [ipmi_si]
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff8107af85>] sys_delete_module+0x1ca/0x23b
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff8146cfdd>] ? do_page_fault+0x238/0x265
Jan 20 15:46:04 o12-30 [28597.628126] [<ffffffff81009cf2>] system_call_fastpath+0x16/0x1b
Jan 20 15:46:05 o12-30 [28597.628126] Code: 
Jan 20 15:46:05 o12-30 ff
Jan 20 15:46:05 o12-30 14
Jan 20 15:46:05 o12-30 24
Jan 20 15:46:05 o12-30 49
Jan 20 15:46:05 o12-30 83
Jan 20 15:46:05 o12-30 c4
Jan 20 15:46:05 o12-30 10
Jan 20 15:46:05 o12-30 49
Jan 20 15:46:05 o12-30 83
Jan 20 15:46:05 o12-30 3c
Jan 20 15:46:05 o12-30 24
Jan 20 15:46:05 o12-30 00
Jan 20 15:46:05 o12-30 eb
Jan 20 15:46:05 o12-30 e4
Jan 20 15:46:05 o12-30 48
Jan 20 15:46:05 o12-30 83
Jan 20 15:46:05 o12-30 fb
Jan 20 15:46:05 o12-30 10
Jan 20 15:46:05 o12-30 76
Jan 20 15:46:05 o12-30 79
Jan 20 15:46:05 o12-30 48
Jan 20 15:46:05 o12-30 89
Jan 20 15:46:05 o12-30 df
Jan 20 15:46:05 o12-30 e8
Jan 20 15:46:05 o12-30 1b
Jan 20 15:46:05 o12-30 e9
Jan 20 15:46:05 o12-30 ff
Jan 20 15:46:05 o12-30 ff
Jan 20 15:46:05 o12-30 49
Jan 20 15:46:05 o12-30 89
Jan 20 15:46:05 o12-30 c6
Jan 20 15:46:05 o12-30 48
Jan 20 15:46:05 o12-30 8b
Jan 20 15:46:05 o12-30 00
Jan 20 15:46:05 o12-30 a8
Jan 20 15:46:05 o12-30 80
Jan 20 15:46:05 o12-30 75
Jan 20 15:46:05 o12-30 11
Jan 20 15:46:05 o12-30 f6
Jan 20 15:46:05 o12-30 c4
Jan 20 15:46:05 o12-30 c0
Jan 20 15:46:05 o12-30 75
Jan 20 15:46:05 o12-30 02
Jan 20 15:46:05 o12-30 f>
Jan 20 15:46:05 o12-30 0b
Jan 20 15:46:05 o12-30 4c
Jan 20 15:46:05 o12-30 89
Jan 20 15:46:05 o12-30 f7
Jan 20 15:46:05 o12-30 e8
Jan 20 15:46:05 o12-30 a7
Jan 20 15:46:05 o12-30 37
Jan 20 15:46:05 o12-30 fd
Jan 20 15:46:05 o12-30 ff
Jan 20 15:46:05 o12-30 eb
Jan 20 15:46:05 o12-30 56
Jan 20 15:46:05 o12-30 4d
Jan 20 15:46:05 o12-30 8b
Jan 20 15:46:05 o12-30 6e
Jan 20 15:46:05 o12-30 10
Jan 20 15:46:05 o12-30 4c
Jan 20 15:46:05 o12-30 8b
Jan 20 15:46:05 o12-30 7d
Jan 20 15:46:05 o12-30 08
Jan 20 15:46:05 o12-30 e8
Jan 20 15:46:05 o12-30 
Jan 20 15:46:05 o12-30 [28597.628126] RIP 
Jan 20 15:46:05  [<ffffffff81108a6e>] kfree+0x69/0xd7
Jan 20 15:46:05 o12-30 [28597.628126] RSP <ffff88021e8f9d88>
Jan 20 15:46:05 o12-30 [28601.071743] ---[ end trace 9dc380e62a5d64dc ]---

The issue exists in 2.6.35.6-48

Jan 21 02:35:26 o12-28 [ 554.258001] kernel BUG at mm/slub.c:2834!
Jan 21 02:35:26 o12-28 [ 554.258001] invalid opcode: 0000 [#1] 
Jan 21 02:35:26 o12-28 SMP
Jan 21 02:35:26 o12-28 
Jan 21 02:35:26 o12-28 [ 554.258001] last sysfs file: /sys/devices/pci0000:00/0000:00:04.0/0000:05:00.0/host4/port-4:1/end_device-4:1/target4:0:1/4:0:1:0/block/sdb/dev
Jan 21 02:35:27 o12-28 [ 554.258001] CPU 0 
Jan 21 02:35:27 o12-28 
Jan 21 02:35:27 o12-28 [ 554.258001] Modules linked in:
Jan 21 02:35:27  nfsd
Jan 21 02:35:27  lockd
Jan 21 02:35:27  nfs_acl
Jan 21 02:35:27  auth_rpcgss
Jan 21 02:35:27  sunrpc
Jan 21 02:35:27  exportfs
Jan 21 02:35:27  ipt_REDIRECT
Jan 21 02:35:27  xt_NOTRACK
Jan 21 02:35:27  iptable_raw
Jan 21 02:35:27  fuse
Jan 21 02:35:27  ipmi_devintf
Jan 21 02:35:27  ipmi_si
Jan 21 02:35:27  ipmi_msghandler
Jan 21 02:35:27  ipv6
Jan 21 02:35:27  iptable_nat
Jan 21 02:35:27  nf_nat
Jan 21 02:35:27  dm_multipath
Jan 21 02:35:27  netconsole
Jan 21 02:35:27  configfs
Jan 21 02:35:27  radeon
Jan 21 02:35:27  ttm
Jan 21 02:35:27  drm_kms_helper
Jan 21 02:35:27  mptsas
Jan 21 02:35:27  drm
Jan 21 02:35:27  mptscsih
Jan 21 02:35:27  mptbase
Jan 21 02:35:27  i2c_algo_bit
Jan 21 02:35:27  scsi_transport_sas
Jan 21 02:35:27  i2c_core
Jan 21 02:35:27  hed
Jan 21 02:35:27  dcdbas
Jan 21 02:35:27  serio_raw
Jan 21 02:35:27  iTCO_wdt
Jan 21 02:35:27  iTCO_vendor_support
Jan 21 02:35:27  i5100_edac
Jan 21 02:35:27  tg3
Jan 21 02:35:27  edac_core
Jan 21 02:35:27  [last unloaded: scsi_wait_scan]
Jan 21 02:35:27 o12-28 
Jan 21 02:35:27 o12-28 [ 554.258001] 
Jan 21 02:35:27 o12-28 [ 554.258001] Pid: 15, comm: events/0 Not tainted 2.6.35.6-48.fc14.x86_64 #1 0TY179/PowerEdge R300
Jan 21 02:35:27 o12-28 [ 554.258001] RIP: 0010:[<ffffffff81108796>] 
Jan 21 02:35:27  [<ffffffff81108796>] kfree+0x69/0xd7
Jan 21 02:35:27 o12-28 [ 554.258001] RSP: 0018:ffff880226d4fde0  EFLAGS: 00010246
Jan 21 02:35:27 o12-28 [ 554.258001] RAX: 0040000000100068 RBX: ffff880200000000 RCX: dead000000100100
Jan 21 02:35:27 o12-28 [ 554.258001] RDX: 0040000000100068 RSI: ffffffff81dc6574 RDI: ffff880200000000
Jan 21 02:35:27 o12-28 [ 554.258001] RBP: ffff880226d4fe10 R08: ffff8802255bfec0 R09: ffff8802224d7800
Jan 21 02:35:28 o12-28 [ 554.258001] R10: 0000000000000000 R11: 0000000000000246 R12: ffff880226d60000
Jan 21 02:35:28 o12-28 [ 554.258001] R13: ffffffff811d5b0d R14: ffffea0007000000 R15: ffff880002018588
Jan 21 02:35:28 o12-28 [ 554.258001] FS:  0000000000000000(0000) GS:ffff880002000000(0000) knlGS:0000000000000000
Jan 21 02:35:28 o12-28 [ 554.258001] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 21 02:35:28 o12-28 [ 554.258001] CR2: 00007f7375a617e8 CR3: 000000022413a000 CR4: 00000000000406f0
Jan 21 02:35:28 o12-28 [ 554.258001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 21 02:35:28 o12-28 [ 554.258001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 21 02:35:28 o12-28 [ 554.258001] Process events/0 (pid: 15, threadinfo ffff880226d4e000, task ffff880226d60000)
Jan 21 02:35:28 o12-28 [ 554.258001] Stack:
Jan 21 02:35:28 o12-28 [ 554.258001]  ffff880226d4fdf0
Jan 21 02:35:28  ffff8802255bfe40
Jan 21 02:35:28  ffff880226d60000
Jan 21 02:35:28  ffff880226d60000
Jan 21 02:35:28 o12-28 
Jan 21 02:35:28 o12-28 [ 554.258001] <0>
Jan 21 02:35:28  ffff880226d60000
Jan 21 02:35:28  ffff880002018588
Jan 21 02:35:28  ffff880226d4fe30
Jan 21 02:35:28  ffffffff811d5b0d
Jan 21 02:35:28 o12-28 
Jan 21 02:35:28 o12-28 [ 554.258001] <0>
Jan 21 02:35:28  ffff880226d60000
Jan 21 02:35:28  ffff880002018580
Jan 21 02:35:28  ffff880226d4fee0
Jan 21 02:35:28  ffffffff81062571
Jan 21 02:35:28 o12-28 
Jan 21 02:35:28 o12-28 [ 554.258001] Call Trace:
Jan 21 02:35:29 o12-28 [ 554.258001]  [<ffffffff811d5b0d>] key_cleanup+0xdb/0xef
Jan 21 02:35:29 o12-28 [ 554.258001]  [<ffffffff81062571>] worker_thread+0x1c5/0x251
Jan 21 02:35:29 o12-28 [ 554.258001]  [<ffffffff811d5a32>] ? key_cleanup+0x0/0xef
Jan 21 02:35:29 o12-28 [ 554.258001]  [<ffffffff810663c3>] ? autoremove_wake_function+0x0/0x39
Jan 21 02:35:29 o12-28 [ 554.258001]  [<ffffffff810623ac>] ? worker_thread+0x0/0x251
Jan 21 02:35:29 o12-28 [ 554.258001]  [<ffffffff81065f29>] kthread+0x7f/0x87
Jan 21 02:35:29 o12-28 [ 554.258001]  [<ffffffff8100aae4>] kernel_thread_helper+0x4/0x10
Jan 21 02:35:29 o12-28 [ 554.258001]  [<ffffffff81065eaa>] ? kthread+0x0/0x87
Jan 21 02:35:29 o12-28 [ 554.258001]  [<ffffffff8100aae0>] ? kernel_thread_helper+0x0/0x10
Jan 21 02:35:29 o12-28 [ 554.258001] Code: 
Jan 21 02:35:29 o12-28 ff
Jan 21 02:35:29 o12-28 14
Jan 21 02:35:29 o12-28 24
Jan 21 02:35:29 o12-28 49
Jan 21 02:35:29 o12-28 83
Jan 21 02:35:29 o12-28 c4
Jan 21 02:35:29 o12-28 10
Jan 21 02:35:29 o12-28 49
Jan 21 02:35:29 o12-28 83
Jan 21 02:35:29 o12-28 3c
Jan 21 02:35:29 o12-28 24
Jan 21 02:35:29 o12-28 00
Jan 21 02:35:29 o12-28 eb
Jan 21 02:35:29 o12-28 e4
Jan 21 02:35:29 o12-28 48
Jan 21 02:35:29 o12-28 83
Jan 21 02:35:29 o12-28 fb
Jan 21 02:35:29 o12-28 10
Jan 21 02:35:29 o12-28 76
Jan 21 02:35:29 o12-28 79
Jan 21 02:35:29 o12-28 48
Jan 21 02:35:29 o12-28 89
Jan 21 02:35:29 o12-28 df
Jan 21 02:35:29 o12-28 e8
Jan 21 02:35:29 o12-28 1b
Jan 21 02:35:29 o12-28 e9
Jan 21 02:35:29 o12-28 ff
Jan 21 02:35:29 o12-28 ff
Jan 21 02:35:29 o12-28 49
Jan 21 02:35:29 o12-28 89
Jan 21 02:35:29 o12-28 c6
Jan 21 02:35:29 o12-28 48
Jan 21 02:35:29 o12-28 8b
Jan 21 02:35:29 o12-28 00
Jan 21 02:35:29 o12-28 a8
Jan 21 02:35:29 o12-28 80
Jan 21 02:35:29 o12-28 75
Jan 21 02:35:29 o12-28 11
Jan 21 02:35:29 o12-28 f6
Jan 21 02:35:29 o12-28 c4
Jan 21 02:35:29 o12-28 c0
Jan 21 02:35:29 o12-28 75
Jan 21 02:35:29 o12-28 02
Jan 21 02:35:29 o12-28 f>
Jan 21 02:35:29 o12-28 0b
Jan 21 02:35:29 o12-28 4c
Jan 21 02:35:29 o12-28 89
Jan 21 02:35:29 o12-28 f7
Jan 21 02:35:29 o12-28 e8
Jan 21 02:35:29 o12-28 43
Jan 21 02:35:29 o12-28 38
Jan 21 02:35:29 o12-28 fd
Jan 21 02:35:29 o12-28 ff
Jan 21 02:35:29 o12-28 eb
Jan 21 02:35:29 o12-28 56
Jan 21 02:35:29 o12-28 4d
Jan 21 02:35:29 o12-28 8b
Jan 21 02:35:29 o12-28 6e
Jan 21 02:35:29 o12-28 10
Jan 21 02:35:29 o12-28 4c
Jan 21 02:35:29 o12-28 8b
Jan 21 02:35:29 o12-28 7d
Jan 21 02:35:29 o12-28 08
Jan 21 02:35:29 o12-28 e8
Jan 21 02:35:29 o12-28 
Jan 21 02:35:29 o12-28 [ 554.258001] RIP 
Jan 21 02:35:29  [<ffffffff81108796>] kfree+0x69/0xd7
Jan 21 02:35:29 o12-28 [ 554.258001]  RSP <ffff880226d4fde0>

Is there anything you need from me to help resolve this issue?

I grabbed the 2.6.36.2-12.rc1.fc15.x86_64 kernel from http://kojipkgs.fedoraproject.org/packages/kernel/ and it seems to have resolved the issue. The same nodes that were hitting the BUG on 2.6.35 have lasted over 3x longer on 2.6.36. Hope this helps to narrow down what could be the cause.

Nate, have you moved your systems to f15 at this point?

(In reply to comment #9)
> Nate, have you moved your systems to f15 at this point?

The systems were moved to a vanilla 2.6.36.4 and have been running that for the past few months, so I can not comment on f15's kernel.

Closing as fixed in F15 per comment #8

Hi,

Do we know what exact fix was made for this in the kernel ?
Any patch or commit id will be helpful. 

Thanks,
Manish