Description of problem: cgconfigparser will always fail on my system when I try to mount more than one controller on a mount point. How reproducible: ##########Manually: # mount -t cgroup -o cpuset,memory,devices,blkio,freezer,net_cls,cpu,cpuacct,clone_children cgroup /mnt/lxc # mount -t cgroup -o cpuset,memory,devices,blkio,freezer,net_cls,cpu,cpuacct,clone_children cgroup /mnt/lxc2 Both work OK. #########cgconfig (Example 1 from man cgconfig.conf page): #########WORKS: mount { cpu = /mnt/cgroups/cpu; cpuacct = /mnt/cgroups/cpu; } #########ALSO WORKS: mount { cpu = /mnt/cgroups/cpu; cpu = /sys/fs/cgroup/cpu; } #########NEVER WORKS: mount { cpu = /sys/fs/cgroup/cpu; cpu = /mnt/cgroups/cpu; cpuacct = /mnt/cgroups/cpu; } will always result in # cgconfigparser -l /etc/cgconfig.conf Loading configuration file /etc/cgconfig.conf failed Cgroup mounting failed Additional info: man cgconfig.conf clearly states: > Having multiple hierarchies is perfectly valid and > Libcgroup merges all subsystems mounted to the same directory (see Example 1) and the directory is mounted only once. strace (probably/might) reveal the cause: # strace cgconfigparser -l /etc/cgconfig.conf execve("/sbin/cgconfigparser", ["cgconfigparser", "-l", "/etc/cgconfig.conf"], [/* 26 vars */]) = 0 brk(0) = 0x232c000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3713d9000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=37560, ...}) = 0 mmap(NULL, 37560, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb3713cf000 close(3) = 0 open("/lib64/libcgroup.so.1", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2409\0{;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=74560, ...}) = 0 mmap(0x3b7b000000, 4627152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3b7b000000 mprotect(0x3b7b011000, 2093056, PROT_NONE) = 0 mmap(0x3b7b210000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x10000) = 0x3b7b210000 mmap(0x3b7b212000, 2456272, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3b7b212000 close(3) = 0 open("/lib64/libpthread.so.0", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000k\300z;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=145840, ...}) = 0 mmap(0x3b7ac00000, 2208760, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3b7ac00000 mprotect(0x3b7ac17000, 2093056, PROT_NONE) = 0 mmap(0x3b7ae16000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x3b7ae16000 mmap(0x3b7ae18000, 13304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3b7ae18000 close(3) = 0 open("/lib64/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\25Bz;\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1968968, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3713ce000 mmap(0x3b7a400000, 3785880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3b7a400000 mprotect(0x3b7a592000, 2097152, PROT_NONE) = 0 mmap(0x3b7a792000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x192000) = 0x3b7a792000 mmap(0x3b7a797000, 21656, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x3b7a797000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3713cd000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3713cb000 arch_prctl(ARCH_SET_FS, 0x7fb3713cbb40) = 0 mprotect(0x3b7ae16000, 4096, PROT_READ) = 0 mprotect(0x3b7a792000, 16384, PROT_READ) = 0 mprotect(0x3b79e20000, 4096, PROT_READ) = 0 munmap(0x7fb3713cf000, 37560) = 0 set_tid_address(0x7fb3713cbe10) = 6455 set_robust_list(0x7fb3713cbe20, 0x18) = 0 rt_sigaction(SIGRTMIN, {0x3b7ac06690, [], SA_RESTORER|SA_SIGINFO, 0x3b7ac0f3c0}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0x3b7ac06720, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x3b7ac0f3c0}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0 brk(0) = 0x232c000 brk(0x234d000) = 0x234d000 brk(0) = 0x234d000 open("/etc/cgconfig.conf", O_RDONLY|O_CLOEXEC) = 3 mmap(NULL, 315392, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb37137e000 ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff21c5f308) = -1 ENOTTY (Inappropriate ioctl for device) fstat(3, {st_mode=S_IFREG|0644, st_size=702, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3713d8000 read(3, "#\n# Copyright IBM Corporation. "..., 8192) = 702 read(3, "", 4096) = 0 read(3, "", 8192) = 0 ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff21c5f308) = -1 ENOTTY (Inappropriate ioctl for device) stat("/sys/fs/cgroup/cpu", {st_mode=S_IFDIR|0755, st_size=60, ...}) = 0 mount("cgroup", "/sys/fs/cgroup/cpu", "cgroup", 0, "cpu") = 0 stat("/mnt/cgroups/cpu", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0 mount("cgroup", "/mnt/cgroups/cpu", "cgroup", 0, "cpu,cpuacct") = -1 EBUSY (Device or resource busy) umount("/sys/fs/cgroup/cpu", 0) = 0 rmdir("/sys/fs/cgroup/cpu") = -1 ENOTEMPTY (Directory not empty) umount("/mnt/cgroups/cpu", 0) = 0 rmdir("/mnt/cgroups/cpu") = 0 munmap(0x7fb37137e000, 315392) = 0 close(3) = 0 munmap(0x7fb3713d8000, 4096) = 0 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb3713d8000 write(1, "Loading configuration file /etc/"..., 53Loading configuration file /etc/cgconfig.conf failed ) = 53 write(1, "Cgroup mounting failed\n", 23Cgroup mounting failed ) = 23 exit_group(3) = ?
mount { cpu = /sys/fs/cgroup/cpu; cpu = /mnt/cgroups/cpu; cpuacct = /mnt/cgroups/cpu; } This is a broken configuration.. This is translated as mount -t cgroup -o cpu none /sys/fs/cgroup/cpu mount -t cgroup -o cpu,cpuacct none /mnt/cgroups/cpu This sequence will fail. While I agree the error message can be (and should be) clearer, I am not convinced we should really go about adding more code to handle corner cases like this. Jan, any comments? Would you prefer to add an error message, or add more code to actually handle this case?
> Jan, any comments? Would you prefer to add an error message, or add more code > to actually handle this case? I think error message should be enough here.
Wait... man cgconfig.conf clearly states: > Having multiple hierarchies is perfectly valid and > Libcgroup merges all subsystems mounted to the same directory (see Example 1) and the directory is mounted only once. So am I reading this incorrectly? Doesn't this specifically say that that is exactly how it's supposed to work?
We will core(In reply to comment #3) > Wait... > > man cgconfig.conf clearly states: > > Having multiple hierarchies is perfectly valid > and > > Libcgroup merges all subsystems mounted to the same directory (see Example 1) and the directory is mounted only once. > > So am I reading this incorrectly? Doesn't this specifically say that that is > exactly how it's supposed to work? We should correct the manpages. The thing is, this can lead us to situations like mount { cpu = /sys/fs/cgroup/cpu; cpu = /mnt/cgroups/cpu; memory = /mnt/cgroups/mem; cpuset = /mnt/cgroups/mem; cpuacct = /mnt/cgroups/cpu; cpuset = /sys/fs/cgroup/cpu; } Which now starts getting quite ambiguous. Yes, it can be fixed, but not worth the pain IMO. (unless you have a patch for it ;-) )
Well, the only usage scenario I can think of in this respect is, which is exactly why that is exactly what I'm trying to achieve, is LXC. LXC hardcoded checks for it's own mountpoint AND needs the appropriate controllers directly on it AFAICT. So my preferred setup would be for example mount { blkio = /sys/fs/cgroup/blkio; cpu = /sys/fs/cgroup/cpu; cpuacct = /sys/fs/cgroup/cpuacct; cpuset = /sys/fs/cgroup/cpuset; devices = /sys/fs/cgroup/devices; freezer = /sys/fs/cgroup/freezer; memory = /sys/fs/cgroup/memory; net_cls = /sys/fs/cgroup/net_cls; blkio = /sys/fs/cgroup/lxc; cpuacct = /sys/fs/cgroup/lxc; devices = /sys/fs/cgroup/lxc; freezer = /sys/fs/cgroup/lxc; memory = /sys/fs/cgroup/lxc; net_cls = /sys/fs/cgroup/lxc; } Tell the LXC people to start decently using CGroups instead of parsing through mtab string-matching against 'cgroup' and 'lxc' and this won't be a problem.
(In reply to comment #5) > Well, the only usage scenario I can think of in this respect is, which is > exactly why that is exactly what I'm trying to achieve, is LXC. > > LXC hardcoded checks for it's own mountpoint AND needs the appropriate > controllers directly on it AFAICT. > > So my preferred setup would be for example > > mount { > blkio = /sys/fs/cgroup/blkio; > cpu = /sys/fs/cgroup/cpu; > cpuacct = /sys/fs/cgroup/cpuacct; > cpuset = /sys/fs/cgroup/cpuset; > devices = /sys/fs/cgroup/devices; > freezer = /sys/fs/cgroup/freezer; > memory = /sys/fs/cgroup/memory; > net_cls = /sys/fs/cgroup/net_cls; > blkio = /sys/fs/cgroup/lxc; > cpuacct = /sys/fs/cgroup/lxc; > devices = /sys/fs/cgroup/lxc; > freezer = /sys/fs/cgroup/lxc; > memory = /sys/fs/cgroup/lxc; > net_cls = /sys/fs/cgroup/lxc; > } > Please tell me how you expect cgconfigparser to parse it? > Tell the LXC people to start decently using CGroups instead of parsing through > mtab string-matching against 'cgroup' and 'lxc' and this won't be a problem. I am going to close this as NOTABUG. I really cannot be fixing another project's use of cgroups in a way that does not make any sense. Please talk to lxc developers on using easily usable mechanisms. libcgroup provides APIs to find mount points. systemd also uses cgroups in a sensible fashion. There is no need for them to reinvent the wheel.
> I really cannot be fixing another project's use of cgroups in a way that does not make any sense. I understand that of course ;-) I certainly agree that LXC's way of using CGroups doesn't seem to make much sense in light of the current practices. OTOH, they might have their perfectly valid reasons for it. > Please tell me how you expect cgconfigparser to parse it? The expected result would be: mount -t cgroup -o cpu none /sys/fs/cgroup/cpu mount -t cgroup -o blkio none /sys/fs/cgroup/blkio mount -t cgroup -o cpuacct none /sys/fs/cgroup/cpuacct mount -t cgroup -o devices none /sys/fs/cgroup/devices etc. mount -t cgroup -o cpu,devices,net_cls,freezer,memory,blkio,etc... none /sys/fs/cgroup/lxc The idea, as mentioned, would be that all of the controllers necessary for LXC would be under a single mountpoint hierarchy, while retaining the default system hierarchy used by other components. If you choose not to support this setup, that's fine by me of course, but I do think it's necessary to reflect this in the documentation. The explicit mention of 'Having multiple hierarchies is perfectly valid' apparently no longer holds true, and it'd be nice if the cgconfigparser would give better errors about a violation of this kind. Just to be clear before I contact the LXC people about this; the single mount per controller limit is of the control groups themselves or just the userspace libraries such as cgconfigparser?
(In reply to comment #7) > > I really cannot be fixing another > project's use of cgroups in a way that does not make any sense. > > I understand that of course ;-) I certainly agree that LXC's way of using > CGroups doesn't seem to make much sense in light of the current practices. > OTOH, they might have their perfectly valid reasons for it. > > > Please tell me how you expect cgconfigparser to parse it? > The expected result would be: > > mount -t cgroup -o cpu none /sys/fs/cgroup/cpu > mount -t cgroup -o blkio none /sys/fs/cgroup/blkio > mount -t cgroup -o cpuacct none /sys/fs/cgroup/cpuacct > mount -t cgroup -o devices none /sys/fs/cgroup/devices > etc. > mount -t cgroup -o cpu,devices,net_cls,freezer,memory,blkio,etc... none > /sys/fs/cgroup/lxc > > The idea, as mentioned, would be that all of the controllers necessary for LXC > would be under a single mountpoint hierarchy, while retaining the default > system hierarchy used by other components. This is simply not possible. Not because of libcgroup, but because of kernel itself. Have you tried these mount commands? The last one will fail, it's not possible to mount two *different* hierarchies with the same controller twice. It's only possible to mount *one* hierarchy with the same controllers multiple times. And that's what libcgroup supports.