Bug 919793

Description of problem:

/etc/security/limits.d/90-nproc.conf was introduced in pam package in RH6. It contains this line:

# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024


This line overrides the conventionally set /etc/security/limits.conf value of the same name. Years of expected behavior are thrown out the window and honest system administrators are exposed to outages on Redhat 6!

BZ 432903 states many contradictory things in 2008:

"The obvious argument against this is that it doesn't prevent a malicious user
from raising their soft limit and forkbombing the system."

And:

"For typical desktop use, 256 would probably be fine, but it might annoy
power-users, and cause problems for highly-threaded java servlets that aren't so
highly-threaded to warrant limits.conf configuration currently."


The original Bugzilla starts showing some rebellion in 2009, and in 2011, 2012, and 2013, various users complain about their outages when using the previously-working /etc/security/limits.conf settings.

The entire methodology of migrating widely-impacting config settings into separate subdirectory files is suspect and open to abuse by any package maintainer.

An earlier model for this (/etc/modprobe.d) does not have the same logic -- because values for a package will be specific to that particular module. Setting module parameters for a foreign module would be considered rude at best.

But different package maintainers will have different philosophical and practical considerations in adjusting ulimits for all or some users. There is much room for conflict there!

And thus, nproc * soft 1024 is just the first conflict.


Version-Release number of selected component (if applicable):

Redhat 6 all versions. 6.1-6.3 tested at the least.

How reproducible:

100%

Steps to Reproduce:
1. vi /etc/security/limits.conf and put in desired value for nproc. Verify settings:
# grep '^[^#]*soft.*nproc' /etc/security/limits.conf /etc/security/limits.d/90-nproc.conf
/etc/security/limits.conf:*             soft    nproc   16384
/etc/security/limits.d/90-nproc.conf:*          soft    nproc     1024

2. Re-login and ulimit -Su:
# ulimit -Su
1024

3. vi /etc/security/limits.d/90-nproc.conf and comment out the line:
# grep '^[^#]*soft.*nproc' /etc/security/limits.conf /etc/security/limits.d/90-nproc.conf
/etc/security/limits.conf:*             soft    nproc   16384

4. Re-login and ulimit -Su:
# ulimit -Su
16384

Actual results:

# ulimit -Su
Returns the value in /etc/security/limits.d/90-nproc.conf and the default package install overrides the custom (and customary) value in /etc/security/limits.conf.

Expected results:

/etc/security/limits.conf is the conventional location for ulimit settings.

Creating /etc/security/limits.d/90-nproc.conf and similar files creates a "Who is the master file conundrum?"

Meaning, if you have a value in each of these files, who is the real master? What about other files in /etc/security/limits.d?

Each side of the argument has merit, but the convention is that limits.conf is the master config location. With RH6, this is no longer the case, and package maintainers are free to abuse the new convention by dropping in their own philosophical values.

In this case, 90-nproc.conf has a default value placed in by the pam rpm, and that value is taken as the overriding value, obviating years of experience.

This results in outages. Perhaps only in the upper 5% of use cases, but is that where we want to concentrate avoidable failures?


Additional info:

Invisibly overriding conventional default configuration locations is always a bad engineering practice. The only way a normal sysadmin will know about this setting is after their first, second, or third outage. Presumably they lost their job if there was a fourth outage.

Additional problem: The default pam package's /etc/security/limits.conf does not even reference the new /etc/security/limits.d/ structure in its comments -- missing a critical "heads-up" mechanism to sysadmins everywhere.

Additional problem: The /usr/share/doc/pam-1.1.1/txts/README.pam_limits file references the /etc/security/limits.d/ structure, but does not make plain the consequences of a default 90-nproc.conf file: it will override your normal limits.conf setting. This paragraph is too coy to be comprehended -- only the most paranoid will understand the trap of the 90-nproc.conf file:

"By default limits are taken from the /etc/security/limits.conf config file.
Then individual *.conf files from the /etc/security/limits.d/ directory are
read. The files are parsed one after another in the order of "C" locale. The
effect of the individual files is the same as if all the files were
concatenated together in the order of parsing. If a config file is explicitly
specified with a module option then the files in the above directory are not
parsed."  [snooze]

Additional problem: In standard RPM config, there is the macro %config(noreplace), which has the effect of "don't blow away folks' configs even if we have better stuff". By duplicating a config value out of the normal location into somewhere else, you have circumvented the %config vs %config(noreplace) logic. A new macro would logically need to preserve the desirable "don't blow away folks' configs" ethos: %config(don'toverridethisotherfile,"/etc/security/limits.conf"). I don't see this happening.

Final problem: This creates a mechanism whereby random package maintainers have an easy drop-in method to override local /etc/security/limits.conf settings, aside from %{POSTIN}. All RH6 sysadmins must now always defensively manage /etc/security/limits.d, and ANY RPM CAN DROP A * VALUE THERE. As an example, look at the rhevm rpm adding /etc/security/limits.d/10-ovirt-engine.conf:

ovirt - nofile 65535

Different packages will have different priorities. As referenced in the original BZ 432903, the "make" maintainer may take to heart the comments about "make -j" as a threat to proper system administration, and add 95-make-nproc.conf to the make rpm with another value mentioned in BZ 432903:

*             soft    nproc   256

Think of the outages!

Meanwhile, the maintainer of httpd might see the comments form BZ 432903 from 2011:

"There should be some documentation about this setting for Apache admins going nuts about their MaxClients setting being ignored!"

and then the httpd maintainer says, "I must save my people, er, rpm users!" and puts in 99-httpd-nproc.conf:

*             soft    nproc   106496

Thus, two different package maintainers each can read the same BZ and come up with two mutually-exclusive values to put in. They are both valid from their own packages' standpoint, but they are at odds with each other. Shall we debate whether all the ulimits truly address resource limit exhaustion cases?

But, what is the logic of who wins the /etc/security/limits.d/ race?

Numeric/alpha ordering. A determined and phonebook-savvy package maintainer would then always create his/her limits.d file as: 99-zzzzzzzzzz.conf. And maybe put %config(noreplace) on it.

A better solution would be to isolate the values by their true linear breakdowns:

/etc/limits.d/[user]/[soft|hard]/[ulimit].conf

So, the ovirt value would be set up in these two files:

/etc/limits.d/ovirt/soft/nofile.conf
/etc/limits.d/ovirt/hard/nofile.conf

Each file contains only the value:

65535

Should comments be a good idea? Hmmm...

But what to do with the really protected stuff (*)? It would be in this directory:

/etc/limits.d/*/soft/
/etc/limits.d/*/hard/

By "*" I mean really, a directory named "*".

This could get ugly, when one considers netgroup entries (@admin) -- but then again, it might be an easier way to manage netgroups as well, just leave that to the end users...