Well, quite a few controllers don't work particularly well if used in conjunction with others. The cgroups kernel folks usually reply with "well, then don't do that". 

I think mounting multiple cgroup controllers in the same hierarchy is just broken, and one of the key libcgroup features is it that libcgroup will do the right thing when everything is kept seperate. And we should make use of this.

So, I guess what I want to say: this is something where we should just come up with a sane default policy. And that sane default policy is that we mount all hierarchies seperately and let libcgroup figure out the details.

I will now add a configuration option to systemd which allows turning off the automatic mounting of all cgroup hierarchies. But still I believe that we should have a good default setup, and when people want to shoot themselves in the foot they can do that, but we shouldn't encourage them by shipping things by default in a way that is borked.

(In reply to comment #1)
> Well, quite a few controllers don't work particularly well if used in
> conjunction with others. The cgroups kernel folks usually reply with "well,
> then don't do that". 

I'd let users decide this. Some controllers work well when mounted together.

> I will now add a configuration option to systemd which allows turning off the
> automatic mounting of all cgroup hierarchies. But still I believe that we
> should have a good default setup, and when people want to shoot themselves in
> the foot they can do that, but we shouldn't encourage them by shipping things
> by default in a way that is borked.

There already *is* good default setup, default cgconfig.conf mounts everything on boot, in very similar manner as systemd does, except:
1) it is off by default, nothing is mounted for regular user
2) it is documented
3) it can be easily configured to mount things differently (+create groups, move processes around etc.)

Please don't try to do everything in systemd, there are already tools available for mounting control groups for few Fedora releases.

Note that sooner or later I plan to make at least some of the controlers configurable in the systemd unit files. Then I will need the controllers mounted anyway, because I need them right from the beginning.

In fact, that's the main reason I added the current mounting code. So far I haven't added the config options that then allow you to configure any of the cgroup properties, but I actually planned to add a couple before F14.

You claimed this would break libcgroup. How so? libcgroup is able to discover the cgroup mount points on initialization just fine?

Also, we don't allow the user configuration of the path for /sys or /proc either. Why should we for the cgroup controllers?

> You claimed this would break libcgroup. How so? libcgroup is able to discover
> the cgroup mount points on initialization just fine?

I have one known bug for sure, and taht has to fixed ASAP. I am a bit occupied otherwise now, but if someone else can grab it, nothing like it. I will reach it in a few days time.

Run systemd, and do a service cgconfig stop and i am pretty sure stuff is going to break.

It breaks existing configuration in /etc/cgconfig.conf

(In reply to comment #4)
> Also, we don't allow the user configuration of the path for /sys or /proc
> either. Why should we for the cgroup controllers?

Because there are enough use cases where it makes sense to have some controllers mounted together, and there are others where it does not make sense.

Having said that, once the /sys mount point is up and going in upstream, it will be by default be wrt to the /sys mount point path.

I'm trying to understand the interaction here a little better. So, is the following all correct?

1. libcgroup is both the library, and a set of tools which use that library.

2. These include both command line tools which are intended for e.g. staring processes within a control group *and* daemons which preserve state and enforce policy.

3. Some of the tools assume certain config files; others 

4. The daemons are not started automatically in fedora, but *are* in RHEL6.

4. systemd uses cgroups, but not libcgroup or its tools

5. systemd's current use of cgroups is interfering with use of the tools which use libcgroup, in certain non-default configurations.

6. Running some commands, like cgclear, will probably pretty awesomely break systemd

Correct so far?


(An aside: From a quick look, it appears that systemd's documentation is already well ahead of that for libcgroup -- the man pages and official online documentation are awful, so thank goodness for http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6-Beta/html/Resource_Management_Guide/ch02.html
And the config files are, well, kinda ugly -- human editable and at least they're not XML, but of the sort designed to store state for programs, not for easy administration.)

Then, for the questions. :)

Crucially: is systemd intended to replace some or all of the libcgroup functionality? If so, what parts?

If systemd is to replace some of this functionality (and it seems like a natural fit, honestly -- the current tools are rough and not well established), how can it be made to coexist nicely with the existing tools until it meets or exceeds the use cases?

Note that in my next systemd upload the cgroup moint point will be moved to /sys/fs/cgroup.

Also note that to ensure that the user cannot unmount the cgroup fs during runtime systemd will actually keep around an open fd for the cgroup root dir at all times (we just keep it open for this one purpose, it is not used otherwise). Attempts to unmount it will hence result in EBUSY.

(In reply to comment #8)

Hmmm. The middle of my list got mangled somehow. Let me try that again, hopefully more clearly. And I'm going to switch to letters to avoid *more* confusion.

Is this all correct?

A. libcgroup is both the library, and a set of tools which use that library.

B. These include both command line tools which are intended for e.g. starting
   processes within a control group *and* daemons which preserve state and 
   enforce  policy.

C. Some of the tools assume certain config files (e.g. cgclassify and the 
   daemons); others just act on the specified arguments without needing 
   configuration (e.g. cgexec, cgset).

D. The daemons are not started automatically in fedora, but *are* in RHEL6.

E. systemd uses cgroups, but not libcgroup or its tools

F. systemd's current use of cgroups is interfering with use of the tools which
   use libcgroup, in certain non-default configurations.

G. Running some commands, like cgclear, will probably pretty awesomely break
   systemd.

assumption G is not true. systemd will most likely not care, and it protects its own hierarchy with the mentioned open fd.

(In reply to comment #11)
> assumption G is not true. systemd will most likely not care, and it protects
> its own hierarchy with the mentioned open fd.

So, in other words, G should be reversed: if you're running systemd, cgclear will not be able to remove all cgroups. (This may not have any practical implications, other than the fact that the libcgroup utilities seem to assume that they own and completely control all cgroups, whereas that's clearly not everyone's assumption.)

Sooooo, in the name of science, I tried it on this system. The result is that all subtrees of /cgroup/systemd are removed and all tasks moved to the top-level systemd:/ group.

Cool, huh? So, I think the original "G" is true to at least some degree. Now, please excuse me while I see if my system will still reboot cleanly under these conditions. :)

Annnd, the answer is no. Not too surprised. Welllll, that's why they call it development.

Note that the init script for cgconfig calls cgclear on stop. That means that having it enabled will *probably* break clean shutdown, and will also hork your system if you happen to shut it down manually.

Again (D, above), this is not enabled by default in Fedora, but is in RHEL6. So it might not have been caught in alpha testing, but once RHEL6 ships and people are configuring Fedora boxes to do RHEL6-like stuff, well.... not so good.

I boot Fedora 14 Alpha using the kernel boot parameter " cgroup_disable=memory" because I covet the 41MB that it would require to administer my 4GB system (1% of RAM!!).  Probably because of this, I see "Failed to mount /cgroup/memory: Invalid argument" in red text when booting.  I can't tell if there are any other effects.

cgclear should stay away from the systemd hierarchy. It's systemd's turf, not cgclear's turf!

File this bug now against libcgroup:

https://bugzilla.redhat.com/show_bug.cgi?id=627378

(In reply to comment #17)
> cgclear should stay away from the systemd hierarchy. It's systemd's turf, not
> cgclear's turf!

The program as documented and implemented controls _all_ cgroups. Like I said, that seems to be the design of the suite of tools. In other words, cgroups conflicts with all other programs that want to use cgroups.

That conflict needs to be resolved.

(In reply to comment #8)
> I'm trying to understand the interaction here a little better.

Try not to FUD later on.

> So, is the
> following all correct?
> 
> 1. libcgroup is both the library, and a set of tools which use that library.
> 
> 2. These include both command line tools which are intended for e.g. staring
> processes within a control group *and* daemons which preserve state and enforce
> policy.
> 

Yes to both.

> 3. Some of the tools assume certain config files; others 
> 

By default, yes. Can be configured

> 4. The daemons are not started automatically in fedora, but *are* in RHEL6.
> 
> 4. systemd uses cgroups, but not libcgroup or its tools
> 

It uses libcgroup

> 5. systemd's current use of cgroups is interfering with use of the tools which
> use libcgroup, in certain non-default configurations.
> 

No. Just breaks one tool

> 6. Running some commands, like cgclear, will probably pretty awesomely break
> systemd
> 
> Correct so far?
> 

Partially. What's your point?

> 
> (An aside: From a quick look, it appears that systemd's documentation is
> already well ahead of that for libcgroup -- the man pages and official online
> documentation are awful, so thank goodness for
> http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6-Beta/html/Resource_Management_Guide/ch02.html
> And the config files are, well, kinda ugly -- human editable and at least
> they're not XML, but of the sort designed to store state for programs, not for
> easy administration.)
> 
> Then, for the questions. :)
> 
> Crucially: is systemd intended to replace some or all of the libcgroup
> functionality? If so, what parts?
> 

No.

> If systemd is to replace some of this functionality (and it seems like a
> natural fit, honestly -- the current tools are rough and not well established),

How does systemd make a natural fit? Your honesty does not tell me any of the design aspects.

(In reply to comment #13)
> Sooooo, in the name of science, I tried it on this system.

Of course the developer's comment earlier on in the day that cgclear would break systemd's use of cgroups was not enough. Seriously, it would help if you would tone down as opposed to riling things up.

> The result is that
> all subtrees of /cgroup/systemd are removed and all tasks moved to the
> top-level systemd:/ group.
> 
> Cool, huh? So, I think the original "G" is true to at least some degree. Now,
> please excuse me while I see if my system will still reboot cleanly under these
> conditions. :)

(In reply to comment #16)
> I boot Fedora 14 Alpha using the kernel boot parameter " cgroup_disable=memory"
> because I covet the 41MB that it would require to administer my 4GB system (1%
> of RAM!!).  Probably because of this, I see "Failed to mount /cgroup/memory:
> Invalid argument" in red text when booting.  I can't tell if there are any
> other effects.

Which is one the reasons why systemd should not be messing around with mnoutning cgroups. The administrator knows which groups can be mounted and not, and also how he wants to mount them. This is why the configuration utilities exist.

> And the config files are, well, kinda ugly -- human editable and at least
> they're not XML, but of the sort designed to store state for programs, not for
> easy administration.)

specific suggestions are appreciated. Developer time is appreciated. Please don't handwave.

(In reply to comment #21)
> (In reply to comment #13)
> > Sooooo, in the name of science, I tried it on this system.
> 
> Of course the developer's comment earlier on in the day that cgclear would
> break systemd's use of cgroups was not enough. Seriously, it would help if you
> would tone down as opposed to riling things up.

I apologize; you appear to have misread my tone. I meant to keep things light, but I see that you have interpreted that as making fun. Seriously, not my intention. I don't mean to be disparaging to either libcgroup or systemd.

(In reply to comment #20)
> > 4. systemd uses cgroups, but not libcgroup or its tools
> It uses libcgroup

It does? It doesn't seem to link against it, and there doesn't appear to be an RPM dependency.

> > 5. systemd's current use of cgroups is interfering with use of the tools which
> > use libcgroup, in certain non-default configurations.
> No. Just breaks one tool

Which one, and how?

> > 6. Running some commands, like cgclear, will probably pretty awesomely break
> > systemd
> > Correct so far?
> Partially. What's your point?

I thought that was pretty clear. There is a design conflict between libcgroup and systemd that needs to be resolved.


> > Crucially: is systemd intended to replace some or all of the libcgroup
> > functionality? If so, what parts?
> No.

I'm not sure you can answer that question -- it's for the systemd developers. 


> > If systemd is to replace some of this functionality (and it seems like a
> > natural fit, honestly -- the current tools are rough and not well established),
> How does systemd make a natural fit? Your honesty does not tell me any of the
> design aspects.

It seems like a natural fit because systemd intends to be a centralized place to control processes and their resources. And it already does things like place all processes spawned in user accounts into a per-user cgroup.

(In reply to comment #22)
> Which is one the reasons why systemd should not be messing around with
> mnoutning cgroups. The administrator knows which groups can be mounted and not,
> and also how he wants to mount them. This is why the configuration utilities
> exist.

So, *one* possible solution here is for systemd to be refactored to use libcgroups, rather than trying to sit beside it. But I don't know if that will be adequate for everything systemd is trying to do. Lennart?

(In reply to comment #23)
> > And the config files are, well, kinda ugly -- human editable and at least
> > they're not XML, but of the sort designed to store state for programs, not for
> > easy administration.)
> 
> specific suggestions are appreciated. Developer time is appreciated. Please
> don't handwave.

I guess cgrules.conf is fairly sensible, although it's not clear to me when and how the rules will be enforced. (Is it atomically when a matching process is spawned? If the daemon isn't running, there's nothing to enforce the rules, right? There don't appear to be man pages for the daemons.)

The on-topic question is: what will happen if this matches a process systemd has already put into a separate control group? The big design difference here is that the cgrules.conf philosophy appears to be based on static, admin-defined cgroup names, whereas systemd creates them on the fly based on usernames and on its unit definitions, which can be dynamic.

But the cgconfig.conf has an entirely different sort of syntax from the cgrules.conf file. That syntax is full of nested { }, and different types of sections each have their own different possible parameters, some of them confusingly named. It looks like there's really nothing here that couldn't be specified with a more flat file, which would 1) better match the syntax of the other libcgroup conf files and 2) be much easier for a human to understand on first read.
(This whole discussion is separate from the issue, though -- we can take it offline if you like -- please feel free to send me an e-mail.)


Another point is: libcgroup, although it is designed to control cgroups in general, does not appear to be ambitious in terms of policy. It has some simple defaults, and assumes that the sysadmin or distribution will set something up. And Fedora has set nothing up -- we just ship the package. In my experience, this sort of thing is relegated to niche use -- expert sysadmins at large institutions will use it, everyone else will ignore it, because it becomes Yet Another Thing To Configure After Install -- *unless* the distro takes an active interest in providing a good, useful default policy. This does not reflect badly on libcgroups, but it's worth noting in the discussion that no one in Fedora seems to have, so far, been excited about working on such a policy as a feature.

Whereas systemd, although it may be buried under all the other things systemd is trying to do, very much is interested in making every system come with a useful default categorization policy.

(In reply to comment #25)
> (In reply to comment #20)
> > > 4. systemd uses cgroups, but not libcgroup or its tools
> > It uses libcgroup
> 
> It does? It doesn't seem to link against it, and there doesn't appear to be an
> RPM dependency.
> 

It did at some point. I guess he will do it again once we fix up his requirements.

> > > 5. systemd's current use of cgroups is interfering with use of the tools which
> > > use libcgroup, in certain non-default configurations.
> > No. Just breaks one tool
> 
> Which one, and how?
> 

cgclear as it has been mentioned quite often in this bug.

> > > 6. Running some commands, like cgclear, will probably pretty awesomely break
> > > systemd
> > > Correct so far?
> > Partially. What's your point?
> 
> I thought that was pretty clear. There is a design conflict between libcgroup
> and systemd that needs to be resolved.
> 

This is not the place for it then. It should be handled upstream, and I think both set of developers are interested in handling it and can do without this needless mongering around.

> 
> > > Crucially: is systemd intended to replace some or all of the libcgroup
> > > functionality? If so, what parts?
> > No.
> 
> I'm not sure you can answer that question -- it's for the systemd developers. 
> 

Again, systemd aims to solve a completely different problem.

> 
> > > If systemd is to replace some of this functionality (and it seems like a
> > > natural fit, honestly -- the current tools are rough and not well established),
> > How does systemd make a natural fit? Your honesty does not tell me any of the
> > design aspects.
> 
> It seems like a natural fit because systemd intends to be a centralized place
> to control processes and their resources. And it already does things like place
> all processes spawned in user accounts into a per-user cgroup.

You do realize there are a number of other scenarios as opposed to a per-user cgroup. I repeat again, systemd is an init system. It does that *very* well. It is not supposed to be setting cgroups for other subsystems. Even if systemd wants to use the memory/cpu cgroups for some use, it will have to use it nicely with *other* users and systemd cannot assume anything about it.

I repeat again, systemd is not the only user of cgroups, so think beyond that.

(In reply to comment #26)
> (In reply to comment #22)
> > Which is one the reasons why systemd should not be messing around with
> > mnoutning cgroups. The administrator knows which groups can be mounted and not,
> > and also how he wants to mount them. This is why the configuration utilities
> > exist.
> 
> So, *one* possible solution here is for systemd to be refactored to use
> libcgroups, rather than trying to sit beside it. But I don't know if that will
> be adequate for everything systemd is trying to do. Lennart?

And if you had been following upstream development, Lennart has been kind enough to give us a list of deficiencies, which are being worked on. As mentioned elsewhere, I am in the middle of some serious travel, and so am not able to do any development work right now.

(In reply to comment #27)
> (In reply to comment #23)
> > > And the config files are, well, kinda ugly -- human editable and at least
> > > they're not XML, but of the sort designed to store state for programs, not for
> > > easy administration.)
> > 
> > specific suggestions are appreciated. Developer time is appreciated. Please
> > don't handwave.
> 
> I guess cgrules.conf is fairly sensible, although it's not clear to me when and
> how the rules will be enforced. (Is it atomically when a matching process is
> spawned? If the daemon isn't running, there's nothing to enforce the rules,
> right? There don't appear to be man pages for the daemons.)
> 
> The on-topic question is: what will happen if this matches a process systemd
> has already put into a separate control group? The big design difference here
> is that the cgrules.conf philosophy appears to be based on static,
> admin-defined cgroup names, whereas systemd creates them on the fly based on
> usernames and on its unit definitions, which can be dynamic.
> 

Again, please do check out cgred. It is quite dynamic, and with the configuration files, allows the user to set it up as per their requirements. Please, *read the code* and then comment.

> But the cgconfig.conf has an entirely different sort of syntax from the
> cgrules.conf file. That syntax is full of nested { }, and different types of
> sections each have their own different possible parameters, some of them
> confusingly named. It looks like there's really nothing here that couldn't be
> specified with a more flat file, which would 1) better match the syntax of the
> other libcgroup conf files and 2) be much easier for a human to understand on
> first read.

Please send your suggestions to libcg-devel.net . You will find that upstream is quite open to good ideas. Ideas nicely formed will be appreciated, not handwaving though. Do note the list is subscriber only to handle the sf spam.

> (This whole discussion is separate from the issue, though -- we can take it
> offline if you like -- please feel free to send me an e-mail.)
> 

Please email libcg-devel.

> 
> Another point is: libcgroup, although it is designed to control cgroups in
> general, does not appear to be ambitious in terms of policy. It has some simple
> defaults, and assumes that the sysadmin or distribution will set something up.
> And Fedora has set nothing up -- we just ship the package. In my experience,
> this sort of thing is relegated to niche use -- expert sysadmins at large
> institutions will use it, everyone else will ignore it, because it becomes Yet
> Another Thing To Configure After Install -- *unless* the distro takes an active
> interest in providing a good, useful default policy. This does not reflect
> badly on libcgroups, but it's worth noting in the discussion that no one in
> Fedora seems to have, so far, been excited about working on such a policy as a
> feature.
> 

Right. So again, talk about what you think a sensible default will be, and mention it. Open a new bug, and you will again find that Jan is extremely responsive to Fedora bugs and we will handle it. However I certainly do *not* appreciate having this hostile discussion where you seem to have decided to attack libcgroup. It does make more inclined to ignore you.

> Whereas systemd, although it may be buried under all the other things systemd
> is trying to do, very much is interested in making every system come with a
> useful default categorization policy.

(In reply to comment #30)
> Right. So again, talk about what you think a sensible default will be, and
> mention it. Open a new bug, and you will again find that Jan is extremely
> responsive to Fedora bugs and we will handle it. However I certainly do *not*
> appreciate having this hostile discussion where you seem to have decided to
> attack libcgroup. It does make more inclined to ignore you.

I assure you I am quite baffled by your defensive reaction. I have no idea how my questions are "hostile". If the development process is this sensitive, then, well, no thanks.

(In reply to comment #27)
> (In reply to comment #23)
> > > And the config files are, well, kinda ugly -- human editable and at least
> > > they're not XML, but of the sort designed to store state for programs, not for
> > > easy administration.)
> > 

XML is hard to read, but easy to validate. I did not want to enforce having an XML parser to get libcgroups working. IMHO XML parsing is heavy and can take some expertise to debug.


> But the cgconfig.conf has an entirely different sort of syntax from the
> cgrules.conf file. That syntax is full of nested { }, and different types of
> sections each have their own different possible parameters, some of them
> confusingly named. It looks like there's really nothing here that couldn't be
> specified with a more flat file, which would 1) better match the syntax of the
> other libcgroup conf files and 2) be much easier for a human to understand on
> first read.

nested {} was my idea, I am to blame, but I was inspired by dhcp*conf/dhclient*conf. System administrators already use that form of configuration for dhcp, unless they use a GUI. I'd be interested in knowing what is confusingly named - I think we can take this to libcg-devel

Please see my comments on the systemd+libcgroup situation on fedora-devel:

http://lists.fedoraproject.org/pipermail/devel/2010-August/141970.html

(and the follow-ups)

I actually believe there's only very little that needs fixing:

- cgclear should not touch the systemd tree (https://bugzilla.redhat.com/show_bug.cgi?id=627378)

- the defaults in the libcgroup configuration should be fixed for the new kernel mount point. (https://bugzilla.redhat.com/show_bug.cgi?id=627381)

- It's probably wise if the libcgroup configuration would comment the mount lines by default. (same bug)

I think, if these three bugs are fixed we have pretty good cooperation between the two packages.

I am tacking the liberty to turn this now into a tracker bug for systemd + libcgroup cooperation.

btw, can we still use this bug to stop havign systemd mounting other hierarchies it really has not business messing about with? :-)

libcgroup-0.36.2-4.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/libcgroup-0.36.2-4.fc14

libcgroup-0.36.2-4.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libcgroup'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/libcgroup-0.36.2-4.fc14

libcgroup-0.36.2-4.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.