Bug 612805 - cgroup: mount: none already mounted or /cgroups busy
Summary: cgroup: mount: none already mounted or /cgroups busy
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: doc-Resource_Management_Guide
Version: 6.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Douglas Silas
QA Contact: ecs-bugs
URL:
Whiteboard:
Keywords: Documentation, Reopened
Depends On:
Blocks: 684940
TreeView+ depends on / blocked
 
Reported: 2010-07-09 05:50 UTC by Qian Cai
Modified: 2018-11-14 15:14 UTC (History)
7 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2011-05-06 07:14:46 UTC


Attachments (Terms of Use)

Description Qian Cai 2010-07-09 05:50:31 UTC
Description of problem:
By following the instructions to setup cgroup, it failed
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6-Beta/html/Resource_Management_Guide/ch02.html

# mount
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/vda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

# mkdir -p /cgroup/lab1
# mount -t cgroup -o cpuset,memory,net_cls lab1 /cgroup/lab1
mount: lab1 already mounted or /cgroup/lab1 busy

Version-Release number of selected component (if applicable):
kernel-2.6.32-44.el6.x86_64
libcgroup-0.36.1-5.el6.x86_64
RHEL6.0-20100707.4

How reproducible:
always

Comment 1 Mike Gahagan 2010-07-09 15:03:16 UTC
Hi Jan, Cai,

Is there any chance this is a regression?

Comment 2 Thomas Graf 2010-07-16 10:34:55 UTC
Has this ever worked in RHEL? Does it work if you remount? This works in upstream kernels.

Comment 3 Qian Cai 2010-07-16 10:47:46 UTC
remount failed too,
# mkdir -p /cgroup/lab1
# mount -t cgroup -o memory lab1 /cgroup/lab1
# mount -t cgroup -o remount,cpuset,memory,net_cls lab1 /cgroup/lab1
mount: /cgroup/lab1 is busy

Comment 4 Thomas Graf 2010-07-16 11:04:58 UTC
Works for me:

[root@dell-pe2950-02 ~]# mkdir /dev/lab1
[root@dell-pe2950-02 ~]# mount -t cgroup -o cpuset,memory,net_cls lab1 /dev/lab1
[root@dell-pe2950-02 ~]# uname -a
Linux dell-pe2950-02.lab.bos.redhat.com 2.6.32-44.1.el6.x86_64 #1 SMP Wed Jul 14 18:51:29 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-pe2950-02 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.0 Beta (Santiago)
[root@dell-pe2950-02 ~]#

Comment 5 Thomas Graf 2010-07-16 11:07:10 UTC
Same for remount:

[root@dell-pe2950-02 ~]# mount -t cgroup -o remount,cpuset,net_cls lab1 /dev/lab1
[root@dell-pe2950-02 ~]# mount
[...]
lab1 on /dev/lab1 type cgroup (rw,cpuset,net_cls)
[root@dell-pe2950-02 ~]#

Comment 6 Qian Cai 2010-07-16 11:15:43 UTC
I also tried another i386 system. It is the same failure.

Comment 7 Thomas Graf 2010-07-16 11:33:13 UTC
Reserving a i386 system, this may be arch specific.

Comment 8 Qian Cai 2010-07-16 11:40:15 UTC
The original problem was reproduced in x86_64. Interesting, when trying in on KVM guests - x86_64 failed; i386 working.

Comment 9 Thomas Graf 2010-07-16 14:00:38 UTC
So are you running this in a kvm guest exclusively or does it also fail on the host? This isn't mentioned in the bugzilla, so far I've only been trying to reproduce this on a default RHEL kernel.

Comment 10 Qian Cai 2010-07-16 14:08:12 UTC
Both. Reproduced on T60/X200 i386/x86_64 hosts also.

Comment 11 Qian Cai 2010-07-16 16:14:56 UTC
I could also reproduce it using 2.6.35-rc5.

Comment 12 Linda Wang 2010-07-16 19:43:25 UTC
Ah I think libcgroup is preventing anyone to create new cgroups.

try "rpm -e libvirt", "rpm -e libcgroup"

then follow the command given above.. and the mounting will succeed.

Comment 13 Thomas Graf 2010-07-16 21:45:18 UTC
That makes sense, thanks Linda.

So can we close this?

Comment 14 Qian Cai 2010-07-18 07:41:50 UTC
OK, then we will fix the doc to point the right way to create cgroups with libcgroup installed? Also, libcgroup is such a core component that lots of users will be affected by this.

Comment 15 Jan Safranek 2010-07-19 13:59:08 UTC
libcgroup does not prevent anyone to create anything. Only if properly configured, libcgroup can mount controllers automatically on startup. This happens e.g. when libvirt is installed. Simple 'mount' command does not show these mounts, check /proc/mounts instead.

'service cgconfig stop' unmounts everything that libcgroup created and provides starting ground for your custom mounting experiments.

Maybe this behavior should be emphasized in the Resource Guide.

Comment 16 Qian Cai 2010-07-19 14:07:38 UTC
I don't understand is also this part,
# mount -t cgroup -o cpuset,memory,net_cls lab1 /dev/lab1 - EBUSY
# mount -t cgroup -o cpuset lab1 /dev/lab1 - PASS (any one subsystem)

Comment 17 Jan Safranek 2010-07-19 14:35:38 UTC
(In reply to comment #16)
> I don't understand is also this part,
> # mount -t cgroup -o cpuset,memory,net_cls lab1 /dev/lab1 - EBUSY
> # mount -t cgroup -o cpuset lab1 /dev/lab1 - PASS (any one subsystem)    

That *probably* indicates that one of these controllers was already mounted and was unmounted, while it contained some groups. If you unmount an hiearachy which is not empty (i.e. has some groups), this hierarchy is not destroyed, it still lives in kernel, it is just not available from userspace. So, mount cpuset, memory and net_cls one by one, look if they contain groups and delete them. Then unmount them and try to mount them together - it should work now.

Or it might be caused by totally unrelated problem - kernel does not provide sane error reporting here. It's nice that the interface is via virtual filesystem, but it really sucks when it cannot tell me *what* is wrong :(.

Comment 18 Qian Cai 2010-07-19 15:11:35 UTC
I did not do anything apart from the default setting for libcgroup.

# cat /proc/mounts
...
cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /cgroup/net_cls cgroup rw,relatime,net_cls 0 0
cgroup /cgroup/blkio cgroup rw,relatime,blkio 0 0
...

It was perfectly fine to mount them one by one,

#  mount -t cgroup -o cpuset lab1 /dev/lab1
#  mount -t cgroup -o memory lab1 /dev/lab2
#  mount -t cgroup -o net_cls lab1 /dev/lab3

Even if I umounted all cgroups in /proc/mounts one by one, it still failed for this command,

# mount -t cgroup -o net_cls,cpuacct lab1 /dev/lab3

Comment 19 Jan Safranek 2010-07-20 06:46:10 UTC
(In reply to comment #18)
> Even if I umounted all cgroups in /proc/mounts one by one, it still failed for
> this command,
> 
> # mount -t cgroup -o net_cls,cpuacct lab1 /dev/lab3    

As I wrote above, simple umnout does not destroy the hierarchies. They still live inside kernel!

Libcgroup has nice command to destroy them - cgclear (see its man page). It deletes only mounted hierarchies, so do not forget to mount them before.

See also content of /proc/cgroups, it tells you how the controllers are grouped together in hierarchies. If any of the controller has nozero value in column 'num_cgroups', it means it is used, i.e. it is either mounted or it was unmounted when it contained some groups (=i.e. it still lives in kernel).

Comment 20 Qian Cai 2010-07-20 07:12:56 UTC
IMO, it is error prone to mix up with direct cgroup modification and via libcgroup. As you said I need to mount all cgroups before we can cgclear. Also, the daemons failed,
# /etc/init.d/cgconfig start
Starting cgconfig service: Loading configuration file /etc/cgconfig.conf failed
Permission denied
Failed to parse /etc/cgconfig.conf                         [FAILED]

Alternative, we can write down only libcgroup ways to manage cgroups in the doc to avoid pains points mixed up those.

Comment 21 Qian Cai 2010-07-20 13:32:30 UTC
After running cgclear, mount and /proc/mounts are inconsistent with various of cgroup entries. Is that normal?

# mount
/dev/mapper/VolGroup-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/VolGroup-lv_home on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
none on /cgroup/ns type cgroup (rw,ns)
none on /cgroup/cpu type cgroup (rw,cpu)
none on /cgroup/cpuacct type cgroup (rw,cpuacct)
none on /cgroup/memory type cgroup (rw,memory)
none on /cgroup/devices type cgroup (rw,devices)
none on /cgroup/freezer type cgroup (rw,freezer)
none on /cgroup/net_cls type cgroup (rw,net_cls)
none on /cgroup/blkio type cgroup (rw,blkio)


# cat /proc/mounts 
rootfs / rootfs rw 0 0
/proc /proc proc rw,relatime 0 0
/sys /sys sysfs rw,seclabel,relatime 0 0
udev /dev devtmpfs rw,seclabel,relatime,size=1902064k,nr_inodes=475516,mode=755 0 0
devpts /dev/pts devpts rw,seclabel,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,seclabel,relatime 0 0
/dev/mapper/VolGroup-lv_root / ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
none /selinux selinuxfs rw,relatime 0 0
udev /dev devtmpfs rw,seclabel,relatime,size=1902064k,nr_inodes=475516,mode=755 0 0
/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0
/dev/sda1 /boot ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
/dev/mapper/VolGroup-lv_home /home ext4 rw,seclabel,relatime,barrier=1,data=ordered 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
/etc/auto.misc /misc autofs rw,relatime,fd=7,pgrp=1700,timeout=300,minproto=5,maxproto=5,indirect 0 0
-hosts /net autofs rw,relatime,fd=13,pgrp=1700,timeout=300,minproto=5,maxproto=5,indirect 0 0

# cat /proc/cgroups 
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	1	1
ns	0	1	1
cpu	0	1	1
cpuacct	0	1	1
memory	0	1	1
devices	0	1	1
freezer	0	1	1
net_cls	0	1	1
blkio	0	1	1

Comment 22 Ivana Varekova 2010-07-20 15:16:43 UTC
Hello,
add comment 21 there is rule - each controller can be only in one hierarchy, so if libcgroup put it to one (mount it to separate directory), then you can't do any other with it = mount it to directory with some other controller. So you can mix up direct modification and libcgroup configuration but you have to keep this rule. But it is really complicated to use both.

There is no need to mount all cgroups before you use cgclear. The error message which you get is "Permission denied" so it not the problem with mounting of controllers.

add 21:
it is ok if mount and /proc/mount are identical, then if you do mount with "-n" option then it should not be in mount
(e.g. try 
mount -t cgroup -n -o cpuset,memory,net_cls lab1 /dev/lab1
and 
mount -t cgroup -o cpuset,memory,net_cls lab1 /dev/lab1   )

Comment 23 Ivana Varekova 2010-07-20 15:17:21 UTC
srry add comment 21 should be add comment 20

Comment 24 Ivana Varekova 2010-07-23 06:22:26 UTC
I'm closing this bug, if there will be some problem regarding this issue please reopen it.

Comment 25 Qian Cai 2010-07-23 06:50:01 UTC
I'd suggest this can be a doc improvement item mentioned in comment #15 and comment #20.

Comment 26 Ruediger Landmann 2010-11-08 15:04:59 UTC
(In reply to comment #25)
> I'd suggest this can be a doc improvement item mentioned in comment #15 and
> comment #20.

Comment #15:

I moved the description of cgconfig to the start of the section (section 2.1 as of build 0.11-1) to emphasise that this is our recommended way of creating hierarchies and attaching subsystems to them. I noted the effects of starting and stopping the service in section 2.1:

"The default cgconfig.conf file installed with the libcgroup package creates and mounts an individual hierarchy for each subsystem, and attaches the subsystems to these hierarchies.
If you stop the cgconfig service (with service cgconfig stop), it unmounts all the hierarchies that it mounted."

and in an expanded warning at the start of "2.2. Creating a Hierarchy and Attaching Subsystems":

"On a system on which control groups are already configured (either manually, or by the cgconfig service) these commands will fail unless you first unmount existing hierarchies, which will affect the operation of the system. Do not experiment with these instructions on production systems."

Comment #20

I have reordered the presentation of material throughout chapter 2 to ensure that any description of direct modification of hierarchies and cgroups with shell commands is clearly identified as an "alternative method" and is presented only after the libcgroup method is described. I think it would be detrimental to remove this material entirely from the guide, both because it might prove useful to some users under some circumstances, but also because I think presenting the underlying shell commands helps describe the cgroup mechanism better.

Cheers
Rudi

Comment 32 Michael Doyle 2010-11-18 00:09:54 UTC
Verified in Red_Hat_Enterprise_Linux-Resource_Management_Guide-6-en-US-1.0-3.

Comment 35 IBM Bug Proxy 2011-03-26 16:13:57 UTC
------- Comment From lnx1138@linux.vnet.ibm.com 2010-11-03 15:12 EDT-------
(In reply to comment #12)
> Hello,
> From comment 22 it would appear this is working as designed. Please let me know
> what you would like done with this issue.

Hi Joe,

The RHBZ bug does seem to point in that direction. What we would like to know is what modification to the LTP testcase is needed to make it succeed on RHEL 6. What we tried didn't seem to work.

------- Comment From sradodla@in.ibm.com 2010-11-09 05:08 EDT-------
(In reply to comment #15)
> Hello,
> Were does this script come from?
> Please also let me know the last release were this script worked correctly?
> Thank You
> Joe Kachuck
>
> Status set to: Waiting on Client

Hello Joe,

The script is taken from LTP (ltp-full-20100430/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh).

The srripts have worked correctly on ltp-full-20090930.tgz release.

Regards,
- Sravan.

------- Comment From sradodla@in.ibm.com 2010-11-09 12:06 EDT-------

> Hello Joe,
>
> The script is taken from LTP
> (ltp-full-20100430/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh).
>
> The srripts have worked correctly on ltp-full-20090930.tgz release.
>
> Regards,
> - Sravan.

Hello Joe,

I have installed sep release ltp(ltp-full-20090930.tgz) and have come across the same issue.
So the scripts that have worked fine on rhel6 beta1 does not seem to work on rhel6 rc1.

Regards,
- Sravan.

------- Comment From iranna.ankad@in.ibm.com 2010-11-10 01:58 EDT-------

> Hello Joe,
>
> I have installed sep release ltp(ltp-full-20090930.tgz) and have come across
> the same issue.
> So the scripts that have worked fine on rhel6 beta1 does not seem to work on
> rhel6 rc1.

That means regression (in cgroup) introduced between beta & RC/GA kernel.

------- Comment From lnx1138@linux.vnet.ibm.com 2010-11-15 12:04 EDT-------
(In reply to comment #20)
> Please let me know if it would be acceptable with IBM to connect this issue to
> BZ 612805.

Hi Joe,

Yes, please and when you do is there a way to provide a short synopsis of what we are seeing and have tried in the event we can get a suggestion on how to best change the test to avoid the problem or whether it is a slightly different issue we are seeing? Thanks.

------- Comment From lnx1138@linux.vnet.ibm.com 2011-01-17 16:17 EDT-------
(In reply to comment #23)
> Verified in Red_Hat_Enterprise_Linux-Resource_Management_Guide-6-en-US-1.0-3.
>

Is this guide available online that contains the update? If so, can we get a URL please? If not, it should be available in RHEL 6.1? Thanks.

------- Comment From prem.karat@linux.vnet.ibm.com 2011-02-24 09:11 EDT-------
> (In reply to comment #23)
> > Verified in Red_Hat_Enterprise_Linux-Resource_Management_Guide-6-en-US-1.0-3.
> >
>
> Is this guide available online that contains the update? If so, can we get a
> URL please? If not, it should be available in RHEL 6.1? Thanks.

Hi Luciano,

This is available under Revision.

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/appe-Resource_Management_Guide-Revision_History.html

Cheers,
Prem

Comment 36 IBM Bug Proxy 2016-06-20 10:28:49 UTC
Issue Registered (Severity: 3)
Status set to: Waiting on Tech
Added Customer Reference: 67984
From comment 22 it would appear this is working as designed. Please let me know what you would like done with this issue.

It also appears from the BZ that the docs will be updated to make this more clear. Would you like this attached to this BZ?

Thank You
Joe Kachuck

jkachuck assigned to issue for IBM-LTC.

Status set to: Waiting on Client

Status set to: Waiting on Tech
Hello,
Were does this script come from?
Please also let me know the last release were this script worked correctly?

Thank You
Joe Kachuck

Status set to: Waiting on Client

Status set to: Waiting on Tech

Hello,
Please let me know if it would be acceptable with IBM to connect this issue to BZ 612805.

Thank You
Joe Kachuck

Status set to: Waiting on Client

Status set to: Waiting on Tech
These changes made by mdoyle@redhat.com.
Bugzilla comment added:

Bugzilla status changed from 'ON_QA' to 'VERIFIED'

https://bugzilla.redhat.com/show_bug.cgi?id=612805


Note You need to log in before you can comment on or make changes to this bug.