Ok, I looked at this bz.

Following is the function in question.

struct blkio_group *blkiocg_lookup_group(struct blkio_cgroup *blkcg, void *key)
{
        struct blkio_group *blkg;
        struct hlist_node *n;
        void *__key;

        hlist_for_each_entry_rcu(blkg, n, &blkcg->blkg_list, blkcg_node) {
                __key = blkg->key;
                if (__key == key)
                        return blkg;
        }

        return NULL;
}

I think we have been passed blkcg=NULL that's why when it tries to access &blkcg->blkg_list it crashes. Note blkg_list is at offset 0x28 in blkio_cgroup
structure.

Above function is called here.

static struct throtl_grp * throtl_find_alloc_tg(struct throtl_data *td,
                        struct cgroup *cgroup)
{
        struct blkio_cgroup *blkcg = cgroup_to_blkio_cgroup(cgroup);

..
..
..

        if (blkcg == &blkio_root_cgroup)
                tg = &td->root_tg;
        else
                tg = tg_of_blkg(blkiocg_lookup_group(blkcg, key));

..
..
..
}

So it looks like somehow  cgroup_to_blkio_cgroup() returned NULL. And at this point of time I have no idea why that can happen. To me this is in the context of process and if process still part of the cgroup, then process is alive
and cgroup can not go away. So above function should not return null.

It smells of a generic cgroup layer bug which can be triggered in some corner case. We shall have to keep trying it and see if we can come up with a way to reproduce it.

Here is another one:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000028 
IP: [<ffffffff81254fb9>] blkiocg_lookup_group+0x9/0x40 
PGD 795e7067 PUD 2c4ee067 PMD 0  
Oops: 0000 [#1] SMP  
last sysfs file: /sys/devices/system/node/has_normal_memory 
CPU 1  
Modules linked in: sunrpc cpufreq_ondemand acpi_cpufreq freq_table ipv6 dm_mirror dm_region_hash dm_log tg3 microcode dcdbas serio_raw i2c_i801 sg iTCO_wdt iTCO_vendor_support i3000_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core dm_mod [last unloaded: scsi_wait_scan] 
 
Modules linked in: sunrpc cpufreq_ondemand acpi_cpufreq freq_table ipv6 dm_mirror dm_region_hash dm_log tg3 microcode dcdbas serio_raw i2c_i801 sg iTCO_wdt iTCO_vendor_support i3000_edac edac_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core dm_mod [last unloaded: scsi_wait_scan] 
Pid: 18182, comm: rmdir Not tainted 2.6.32-131.0.1.el6.x86_64 #1 PowerEdge SC440               
RIP: 0010:[<ffffffff81254fb9>]  [<ffffffff81254fb9>] blkiocg_lookup_group+0x9/0x40 
RSP: 0018:ffff880079d8b8b8  EFLAGS: 00010007 
RAX: ffff88007886f540 RBX: ffff88007afb8980 RCX: 000000000000ffff 
RDX: 000000000000649b RSI: ffff880037665800 RDI: 0000000000000000 
RBP: ffff880079d8b8b8 R08: 0000000000000000 R09: 0000000000000025 
R10: 0000000000000000 R11: 0000000000000000 R12: ffff880037665828 
R13: ffff88007886f540 R14: ffff880037665800 R15: 0000000000000010 
FS:  00007f752fd62700(0000) GS:ffff880002040000(0000) knlGS:0000000000000000 
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
CR2: 0000000000000028 CR3: 0000000037a5f000 CR4: 00000000000006e0 
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
Process rmdir (pid: 18182, threadinfo ffff880079d8a000, task ffff880045594a80) 
Stack: 
 ffff880079d8b978 ffffffff812578da ffffc90000307020 ffffea00017cc990 
<0> ffff880045594a80 000000500002bb00 ffff880045594a80 0000000000020058 
<0> ffff880079d8ba18 ffffffff8111fc01 ffff8800000126c0 0000000000000000 
Call Trace: 
 [<ffffffff812578da>] blk_throtl_bio+0xba/0x550 
 [<ffffffff8111fc01>] ? __alloc_pages_nodemask+0x111/0x8b0 
 [<ffffffff81249002>] generic_make_request+0x1f2/0x5b0 
 [<ffffffff8110d1fe>] ? find_get_page+0x1e/0xa0 
 [<ffffffff811a2c6f>] ? __find_get_block_slow+0xaf/0x130 
 [<ffffffff8124944f>] submit_bio+0x8f/0x120 
 [<ffffffff811a2996>] submit_bh+0xf6/0x150 
 [<ffffffff811a4313>] ll_rw_block+0x143/0x150 
 [<ffffffff811a434e>] __breadahead+0x2e/0x40 
 [<ffffffffa0236f7e>] __ext4_get_inode_loc+0x33e/0x3b0 [ext4] 
 [<ffffffffa0237076>] ext4_iget+0x86/0x7e0 [ext4] 
 [<ffffffff81205ffb>] ? security_d_instantiate+0x1b/0x30 
 [<ffffffffa023f3d5>] ext4_lookup+0xa5/0x140 [ext4] 
 [<ffffffff8118068b>] do_lookup+0x18b/0x220 
 [<ffffffff81180c89>] __link_path_walk+0x569/0x820 
 [<ffffffff8118160a>] path_walk+0x6a/0xe0 
 [<ffffffff811817db>] do_path_lookup+0x5b/0xa0 
 [<ffffffff81173d31>] ? get_empty_filp+0xa1/0x170 
 [<ffffffff8118244b>] do_filp_open+0xfb/0xd90 
 [<ffffffff810415d4>] ? __do_page_fault+0x1e4/0x480 
 [<ffffffff81262791>] ? cpumask_any_but+0x31/0x50 
 [<ffffffff8113cca0>] ? unmap_region+0x110/0x130 
 [<ffffffff8118f0b2>] ? alloc_fd+0x92/0x160 
 [<ffffffff8116f859>] do_sys_open+0x69/0x140 
 [<ffffffff8116f970>] sys_open+0x20/0x30 
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b 
Code: 14 45 85 c0 75 e4 83 78 18 02 75 de 48 85 c0 74 09 8b 40 20 c9 c3 0f 1f 40 00 8b 47 20 c9 c3 0f 1f 00 55 48 89 e5 0f 1f 44 00 00 <48> 8b 57 28 48 85 d2 75 0e eb 24 0f 1f 40 00 48 8b 12 48 85 d2  
RIP  [<ffffffff81254fb9>] blkiocg_lookup_group+0x9/0x40 
 RSP <ffff880079d8b8b8> 
CR2: 0000000000000028 
---[ end trace fa132e811bbf87a1 ]--- 
Kernel panic - not syncing: Fatal exception 
Pid: 18182, comm: rmdir Tainted: G      D    ----------------   2.6.32-131.0.1.el6.x86_64 #1 
Call Trace: 
 [<ffffffff814daaa1>] ? panic+0x78/0x143 
 [<ffffffff814deae4>] ? oops_end+0xe4/0x100 
 [<ffffffff81040cdb>] ? no_context+0xfb/0x260 
 [<ffffffff8137316f>] ? ata_bmdma_start+0x2f/0x40 
 [<ffffffff81040f65>] ? __bad_area_nosemaphore+0x125/0x1e0 
 [<ffffffff813633c1>] ? ata_qc_issue+0x1d1/0x340 
 [<ffffffff8104108e>] ? bad_area+0x4e/0x60 
 [<ffffffff8136b440>] ? ata_scsi_rw_xlat+0x0/0x1f0 
 [<ffffffff810417b3>] ? __do_page_fault+0x3c3/0x480 
 [<ffffffff8110fa45>] ? mempool_alloc_slab+0x15/0x20 
 [<ffffffff8110fb53>] ? mempool_alloc+0x63/0x140 
 [<ffffffff8125fe5f>] ? cfq_set_request+0x18f/0x520 
 [<ffffffff814e0aae>] ? do_page_fault+0x3e/0xa0 
 [<ffffffff814dde55>] ? page_fault+0x25/0x30 
 [<ffffffff81254fb9>] ? blkiocg_lookup_group+0x9/0x40 
 [<ffffffff812578da>] ? blk_throtl_bio+0xba/0x550 
 [<ffffffff8111fc01>] ? __alloc_pages_nodemask+0x111/0x8b0 
 [<ffffffff81249002>] ? generic_make_request+0x1f2/0x5b0 
 [<ffffffff8110d1fe>] ? find_get_page+0x1e/0xa0 
 [<ffffffff811a2c6f>] ? __find_get_block_slow+0xaf/0x130 
 [<ffffffff8124944f>] ? submit_bio+0x8f/0x120 
 [<ffffffff811a2996>] ? submit_bh+0xf6/0x150 
 [<ffffffff811a4313>] ? ll_rw_block+0x143/0x150 
 [<ffffffff811a434e>] ? __breadahead+0x2e/0x40 
 [<ffffffffa0236f7e>] ? __ext4_get_inode_loc+0x33e/0x3b0 [ext4] 
 [<ffffffffa0237076>] ? ext4_iget+0x86/0x7e0 [ext4] 
 [<ffffffff81205ffb>] ? security_d_instantiate+0x1b/0x30 
 [<ffffffffa023f3d5>] ? ext4_lookup+0xa5/0x140 [ext4] 
 [<ffffffff8118068b>] ? do_lookup+0x18b/0x220 
 [<ffffffff81180c89>] ? __link_path_walk+0x569/0x820 
 [<ffffffff8118160a>] ? path_walk+0x6a/0xe0 
 [<ffffffff811817db>] ? do_path_lookup+0x5b/0xa0 
 [<ffffffff81173d31>] ? get_empty_filp+0xa1/0x170 
 [<ffffffff8118244b>] ? do_filp_open+0xfb/0xd90 
 [<ffffffff810415d4>] ? __do_page_fault+0x1e4/0x480 
 [<ffffffff81262791>] ? cpumask_any_but+0x31/0x50 
 [<ffffffff8113cca0>] ? unmap_region+0x110/0x130 
 [<ffffffff8118f0b2>] ? alloc_fd+0x92/0x160 
 [<ffffffff8116f859>] ? do_sys_open+0x69/0x140 
 [<ffffffff8116f970>] ? sys_open+0x20/0x30 
 [<ffffffff8100b172>] ? system_call_fastpath+0x16/0x1b 
panic occurred, switching back to text console

Jan,

Is it possible to enable kdump in your recipe and capture vmcore when this bug is reproduced?

Vivek

Jan,

Can you please try running your tests with following kernel. I have put a fix there and hoping this issue should be resolved.

http://download.lab.bos.redhat.com/brewroot/scratch/vgoyal/task_3291406/

(In reply to comment #15)
> Jan,
> 
> Can you please try running your tests with following kernel. I have put a fix
> there and hoping this issue should be resolved.
> 
> http://download.lab.bos.redhat.com/brewroot/scratch/vgoyal/task_3291406/

Vivek,

The test_10 from LTP's cgroup_regressions is now running for >6 days on your testing kernel from #15 without any crash. Previously this test hit the problem within couple of hours, which makes me believe that fix and suspicion about rebind_subsytems() and rcu protection is correct.

static int rebind_subsystems(struct cgroupfs_root *root,
...
1022                        cgrp->subsys[i] = NULL;

(In reply to comment #16)
> (In reply to comment #15)
> > Jan,
> > 
> > Can you please try running your tests with following kernel. I have put a fix
> > there and hoping this issue should be resolved.
> > 
> > http://download.lab.bos.redhat.com/brewroot/scratch/vgoyal/task_3291406/
> 
> Vivek,
> 
> The test_10 from LTP's cgroup_regressions is now running for >6 days on your
> testing kernel from #15 without any crash. Previously this test hit the problem
> within couple of hours, which makes me believe that fix and suspicion about
> rebind_subsytems() and rcu protection is correct.
> 
> static int rebind_subsystems(struct cgroupfs_root *root,
> ...
> 1022                        cgrp->subsys[i] = NULL;

Thanks Jan. I will send the fix to upstream first and then pull it in rhel6.

This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.

Patch(es) available on kernel-2.6.32-160.el6

After having run the cgroup test suite from LTP and others which reproduced the bug in the past since July on numerous systems I think we can consider this issue verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2011-1530.html