Bug 612805

Hi Jan, Cai,

Is there any chance this is a regression?

Has this ever worked in RHEL? Does it work if you remount? This works in upstream kernels.

remount failed too,
# mkdir -p /cgroup/lab1
# mount -t cgroup -o memory lab1 /cgroup/lab1
# mount -t cgroup -o remount,cpuset,memory,net_cls lab1 /cgroup/lab1
mount: /cgroup/lab1 is busy

Works for me:

[root@dell-pe2950-02 ~]# mkdir /dev/lab1
[root@dell-pe2950-02 ~]# mount -t cgroup -o cpuset,memory,net_cls lab1 /dev/lab1
[root@dell-pe2950-02 ~]# uname -a
Linux dell-pe2950-02.lab.bos.redhat.com 2.6.32-44.1.el6.x86_64 #1 SMP Wed Jul 14 18:51:29 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-pe2950-02 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.0 Beta (Santiago)
[root@dell-pe2950-02 ~]#

Same for remount:

[root@dell-pe2950-02 ~]# mount -t cgroup -o remount,cpuset,net_cls lab1 /dev/lab1
[root@dell-pe2950-02 ~]# mount
[...]
lab1 on /dev/lab1 type cgroup (rw,cpuset,net_cls)
[root@dell-pe2950-02 ~]#

I also tried another i386 system. It is the same failure.

Reserving a i386 system, this may be arch specific.

The original problem was reproduced in x86_64. Interesting, when trying in on KVM guests - x86_64 failed; i386 working.

So are you running this in a kvm guest exclusively or does it also fail on the host? This isn't mentioned in the bugzilla, so far I've only been trying to reproduce this on a default RHEL kernel.

Both. Reproduced on T60/X200 i386/x86_64 hosts also.

I could also reproduce it using 2.6.35-rc5.

Ah I think libcgroup is preventing anyone to create new cgroups.

try "rpm -e libvirt", "rpm -e libcgroup"

then follow the command given above.. and the mounting will succeed.

That makes sense, thanks Linda.

So can we close this?

OK, then we will fix the doc to point the right way to create cgroups with libcgroup installed? Also, libcgroup is such a core component that lots of users will be affected by this.

libcgroup does not prevent anyone to create anything. Only if properly configured, libcgroup can mount controllers automatically on startup. This happens e.g. when libvirt is installed. Simple 'mount' command does not show these mounts, check /proc/mounts instead.

'service cgconfig stop' unmounts everything that libcgroup created and provides starting ground for your custom mounting experiments.

Maybe this behavior should be emphasized in the Resource Guide.

I don't understand is also this part,
# mount -t cgroup -o cpuset,memory,net_cls lab1 /dev/lab1 - EBUSY
# mount -t cgroup -o cpuset lab1 /dev/lab1 - PASS (any one subsystem)

(In reply to comment #16)
> I don't understand is also this part,
> # mount -t cgroup -o cpuset,memory,net_cls lab1 /dev/lab1 - EBUSY
> # mount -t cgroup -o cpuset lab1 /dev/lab1 - PASS (any one subsystem)    

That *probably* indicates that one of these controllers was already mounted and was unmounted, while it contained some groups. If you unmount an hiearachy which is not empty (i.e. has some groups), this hierarchy is not destroyed, it still lives in kernel, it is just not available from userspace. So, mount cpuset, memory and net_cls one by one, look if they contain groups and delete them. Then unmount them and try to mount them together - it should work now.

Or it might be caused by totally unrelated problem - kernel does not provide sane error reporting here. It's nice that the interface is via virtual filesystem, but it really sucks when it cannot tell me *what* is wrong :(.

I did not do anything apart from the default setting for libcgroup.

# cat /proc/mounts
...
cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /cgroup/net_cls cgroup rw,relatime,net_cls 0 0
cgroup /cgroup/blkio cgroup rw,relatime,blkio 0 0
...

It was perfectly fine to mount them one by one,

#  mount -t cgroup -o cpuset lab1 /dev/lab1
#  mount -t cgroup -o memory lab1 /dev/lab2
#  mount -t cgroup -o net_cls lab1 /dev/lab3

Even if I umounted all cgroups in /proc/mounts one by one, it still failed for this command,

# mount -t cgroup -o net_cls,cpuacct lab1 /dev/lab3

(In reply to comment #18)
> Even if I umounted all cgroups in /proc/mounts one by one, it still failed for
> this command,
> 
> # mount -t cgroup -o net_cls,cpuacct lab1 /dev/lab3    

As I wrote above, simple umnout does not destroy the hierarchies. They still live inside kernel!

Libcgroup has nice command to destroy them - cgclear (see its man page). It deletes only mounted hierarchies, so do not forget to mount them before.

See also content of /proc/cgroups, it tells you how the controllers are grouped together in hierarchies. If any of the controller has nozero value in column 'num_cgroups', it means it is used, i.e. it is either mounted or it was unmounted when it contained some groups (=i.e. it still lives in kernel).

IMO, it is error prone to mix up with direct cgroup modification and via libcgroup. As you said I need to mount all cgroups before we can cgclear. Also, the daemons failed,
# /etc/init.d/cgconfig start
Starting cgconfig service: Loading configuration file /etc/cgconfig.conf failed
Permission denied
Failed to parse /etc/cgconfig.conf                         [FAILED]

Alternative, we can write down only libcgroup ways to manage cgroups in the doc to avoid pains points mixed up those.

After running cgclear, mount and /proc/mounts are inconsistent with various of cgroup entries. Is that normal?

# mount
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/VolGroup-lv_home on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
none on /cgroup/ns type cgroup (rw,ns)
none on /cgroup/cpu type cgroup (rw,cpu)
none on /cgroup/cpuacct type cgroup (rw,cpuacct)
none on /cgroup/memory type cgroup (rw,memory)
none on /cgroup/devices type cgroup (rw,devices)
none on /cgroup/freezer type cgroup (rw,freezer)
none on /cgroup/net_cls type cgroup (rw,net_cls)
none on /cgroup/blkio type cgroup (rw,blkio)


# cat /proc/mounts 
rootfs / rootfs rw 0 0
/proc /proc proc rw,relatime 0 0
/sys /sys sysfs rw,seclabel,relatime 0 0
udev /dev devtmpfs rw,seclabel,relatime,size=1902064k,nr_inodes=475516,mode=755 0 0
devpts /dev/pts devpts rw,seclabel,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,seclabel,relatime 0 0
/dev/mapper/VolGroup-lv_root / ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
none /selinux selinuxfs rw,relatime 0 0
udev /dev devtmpfs rw,seclabel,relatime,size=1902064k,nr_inodes=475516,mode=755 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/sda1 /boot ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
/dev/mapper/VolGroup-lv_home /home ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
/etc/auto.misc /misc autofs rw,relatime,fd=7,pgrp=1700,timeout=300,minproto=5,maxproto=5,indirect 0 0
-hosts /net autofs rw,relatime,fd=13,pgrp=1700,timeout=300,minproto=5,maxproto=5,indirect 0 0

# cat /proc/cgroups 
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	1	1
ns	0	1	1
cpu	0	1	1
cpuacct	0	1	1
memory	0	1	1
devices	0	1	1
freezer	0	1	1
net_cls	0	1	1
blkio	0	1	1

Hello,
add comment 21 there is rule - each controller can be only in one hierarchy, so if libcgroup put it to one (mount it to separate directory), then you can't do any other with it = mount it to directory with some other controller. So you can mix up direct modification and libcgroup configuration but you have to keep this rule. But it is really complicated to use both.

There is no need to mount all cgroups before you use cgclear. The error message which you get is "Permission denied" so it not the problem with mounting of controllers.

add 21:
it is ok if mount and /proc/mount are identical, then if you do mount with "-n" option then it should not be in mount
(e.g. try 
mount -t cgroup -n -o cpuset,memory,net_cls lab1 /dev/lab1
and 
mount -t cgroup -o cpuset,memory,net_cls lab1 /dev/lab1   )

srry add comment 21 should be add comment 20

I'm closing this bug, if there will be some problem regarding this issue please reopen it.

I'd suggest this can be a doc improvement item mentioned in comment #15 and comment #20.

(In reply to comment #25)
> I'd suggest this can be a doc improvement item mentioned in comment #15 and
> comment #20.

Comment #15:

I moved the description of cgconfig to the start of the section (section 2.1 as of build 0.11-1) to emphasise that this is our recommended way of creating hierarchies and attaching subsystems to them. I noted the effects of starting and stopping the service in section 2.1:

"The default cgconfig.conf file installed with the libcgroup package creates and mounts an individual hierarchy for each subsystem, and attaches the subsystems to these hierarchies.
If you stop the cgconfig service (with service cgconfig stop), it unmounts all the hierarchies that it mounted."

and in an expanded warning at the start of "2.2. Creating a Hierarchy and Attaching Subsystems":

"On a system on which control groups are already configured (either manually, or by the cgconfig service) these commands will fail unless you first unmount existing hierarchies, which will affect the operation of the system. Do not experiment with these instructions on production systems."

Comment #20

I have reordered the presentation of material throughout chapter 2 to ensure that any description of direct modification of hierarchies and cgroups with shell commands is clearly identified as an "alternative method" and is presented only after the libcgroup method is described. I think it would be detrimental to remove this material entirely from the guide, both because it might prove useful to some users under some circumstances, but also because I think presenting the underlying shell commands helps describe the cgroup mechanism better.

Cheers
Rudi

Verified in Red_Hat_Enterprise_Linux-Resource_Management_Guide-6-en-US-1.0-3.

------- Comment From lnx1138.ibm.com 2010-11-03 15:12 EDT-------
(In reply to comment #12)
> Hello,
> From comment 22 it would appear this is working as designed. Please let me know
> what you would like done with this issue.

Hi Joe,

The RHBZ bug does seem to point in that direction. What we would like to know is what modification to the LTP testcase is needed to make it succeed on RHEL 6. What we tried didn't seem to work.

------- Comment From sradodla.com 2010-11-09 05:08 EDT-------
(In reply to comment #15)
> Hello,
> Were does this script come from?
> Please also let me know the last release were this script worked correctly?
> Thank You
> Joe Kachuck
>
> Status set to: Waiting on Client

Hello Joe,

The script is taken from LTP (ltp-full-20100430/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh).

The srripts have worked correctly on ltp-full-20090930.tgz release.

Regards,
- Sravan.

------- Comment From sradodla.com 2010-11-09 12:06 EDT-------

> Hello Joe,
>
> The script is taken from LTP
> (ltp-full-20100430/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh).
>
> The srripts have worked correctly on ltp-full-20090930.tgz release.
>
> Regards,
> - Sravan.

Hello Joe,

I have installed sep release ltp(ltp-full-20090930.tgz) and have come across the same issue.
So the scripts that have worked fine on rhel6 beta1 does not seem to work on rhel6 rc1.

Regards,
- Sravan.

------- Comment From iranna.ankad.com 2010-11-10 01:58 EDT-------

> Hello Joe,
>
> I have installed sep release ltp(ltp-full-20090930.tgz) and have come across
> the same issue.
> So the scripts that have worked fine on rhel6 beta1 does not seem to work on
> rhel6 rc1.

That means regression (in cgroup) introduced between beta & RC/GA kernel.

------- Comment From lnx1138.ibm.com 2010-11-15 12:04 EDT-------
(In reply to comment #20)
> Please let me know if it would be acceptable with IBM to connect this issue to
> BZ 612805.

Hi Joe,

Yes, please and when you do is there a way to provide a short synopsis of what we are seeing and have tried in the event we can get a suggestion on how to best change the test to avoid the problem or whether it is a slightly different issue we are seeing? Thanks.

------- Comment From lnx1138.ibm.com 2011-01-17 16:17 EDT-------
(In reply to comment #23)
> Verified in Red_Hat_Enterprise_Linux-Resource_Management_Guide-6-en-US-1.0-3.
>

Is this guide available online that contains the update? If so, can we get a URL please? If not, it should be available in RHEL 6.1? Thanks.

------- Comment From prem.karat.ibm.com 2011-02-24 09:11 EDT-------
> (In reply to comment #23)
> > Verified in Red_Hat_Enterprise_Linux-Resource_Management_Guide-6-en-US-1.0-3.
> >
>
> Is this guide available online that contains the update? If so, can we get a
> URL please? If not, it should be available in RHEL 6.1? Thanks.

Hi Luciano,

This is available under Revision.

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/appe-Resource_Management_Guide-Revision_History.html

Cheers,
Prem

Issue Registered (Severity: 3)
Status set to: Waiting on Tech
Added Customer Reference: 67984
From comment 22 it would appear this is working as designed. Please let me know what you would like done with this issue.

It also appears from the BZ that the docs will be updated to make this more clear. Would you like this attached to this BZ?

Thank You
Joe Kachuck

jkachuck assigned to issue for IBM-LTC.

Status set to: Waiting on Client

Status set to: Waiting on Tech
Hello,
Were does this script come from?
Please also let me know the last release were this script worked correctly?

Thank You
Joe Kachuck

Status set to: Waiting on Client

Status set to: Waiting on Tech

Hello,
Please let me know if it would be acceptable with IBM to connect this issue to BZ 612805.

Thank You
Joe Kachuck

Status set to: Waiting on Client

Status set to: Waiting on Tech
These changes made by mdoyle.
Bugzilla comment added:

Bugzilla status changed from 'ON_QA' to 'VERIFIED'

https://bugzilla.redhat.com/show_bug.cgi?id=612805