Bug 754285 - Hint that /etc/security/limits.conf does not apply to systemd services
Hint that /etc/security/limits.conf does not apply to systemd services
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: pam (Show other bugs)
16
All Linux
unspecified Severity high
: ---
: ---
Assigned To: Tomas Mraz
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-15 18:25 EST by Andrew Okhmat
Modified: 2015-09-21 15:36 EDT (History)
14 users (show)

See Also:
Fixed In Version: pam-1.1.5-5.fc15
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-12-21 04:14:37 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Andrew Okhmat 2011-11-15 18:25:13 EST
Description of problem:
  By default Fedora use small ulimit for open files (1024). On servers with high load, this value should be increased. In earlier versions of Fedora, we set the values ​​in limits.conf. But now, this method doesn't work. When systemd starts processes, it completely ignores these settings.


Steps to Reproduce:

1. add new limits to limits.conf
   # echo "mysql           hard     nofile          131070" >> /etc/security/limits.conf
   # echo "mysql           soft     nofile          131070" >> /etc/security/limits.conf
2. Set big values for mysql
   # echo -e "table_open_cache=16384\nmax_connections=1700" >> /etc/my.cnf
3. start mysqld server
   # systemctl start mysqld.service
  
Actual results:
    Now we need to look in /var/log/mysqld.log.
    It is seen that the number of available descriptors left by default - 1024

===cut on===   
111115 13:18:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
111115 13:18:36 [Warning] Changed limits: max_open_files: 1024  max_connections: 214  table_cache: 400
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
111115 13:18:36 [Note] Plugin 'FEDERATED' is disabled.
111115 13:18:36 InnoDB: The InnoDB memory heap is disabled
111115 13:18:36 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111115 13:18:36 InnoDB: Compressed tables use zlib 1.2.5
111115 13:18:36 InnoDB: Using Linux native AIO
111115 13:18:36 InnoDB: Initializing buffer pool, size = 24.0G
111115 13:18:37 InnoDB: Completed initialization of buffer pool
111115 13:18:37 InnoDB: highest supported file format is Barracuda.
111115 13:18:38  InnoDB: Waiting for the background threads to start
111115 13:18:39 InnoDB: 1.1.8 started; log sequence number 196542194396
111115 13:18:39 [Note] Event Scheduler: Loaded 0 events
111115 13:18:39 [Note] /usr/libexec/mysqld: ready for connections.
===cut off===   

Expected results:
   Start mysqld with 131070 descriptors, without warnings.


Additional info:
   If mysqld start manually, everything works without problems. Ex:
   # ulimit -n 131070
   # su - mysql
   $ ulimit -n
      131070
   $ /usr/bin/mysqld_safe --nowatch --basedir=/usr
Comment 1 Kay Sievers 2011-11-15 19:05:06 EST
Systemd does not support global limits, the file is intentionally ignored.

LimitNOFILE= in the service file can be set to specify the number of open
file descriptors for a specific service.
Comment 2 Kay Sievers 2011-11-15 19:06:45 EST
Details are in:
  man systemd.exec
Comment 3 Andrew Okhmat 2011-11-15 19:38:15 EST
Maybe it makes sense to add a small notice to the man page systemd.exec, f.e. "file limits.conf is ignored". I think we are not alone who face to this problem.

Thank you for quick reply. Ticket may be closed.
Comment 4 Michal Schmidt 2011-11-16 04:46:37 EST
It would make sense to put a warning in the packaged /etc/security/limits.conf itself:

# NOTE:
# These limits affect only user logins, not services spawned by systemd.
# To set the limits for a service, use the Limit*= options in the service's
# systemd unit file as documented in systemd.exec(5).
Comment 5 Andrig Miller 2011-12-07 13:33:14 EST
I ran into the same problem, but following the man page instructions, where I modified the mysqld.service file in /lib/systemd/system to change the limits did nothing to actually fix the problem.

I even ran systemctl --system daemon-reload after modifying it.  I even tried rebooting just to see if that would help, which to no surprise it didn't.

I tried using infinity, "infinity", unlimited, and specific values, like 8192, but to no avail.
Comment 6 Andrig Miller 2011-12-07 15:03:02 EST
I ran into the same problem, but following the man page instructions, where I modified the mysqld.service file in /lib/systemd/system to change the limits did nothing to actually fix the problem.

I even ran systemctl --system daemon-reload after modifying it.  I even tried rebooting just to see if that would help, which to no surprise it didn't.

I tried using infinity, "infinity", unlimited, and specific values, like 8192, but to no avail.
Comment 7 Andrew Haveland-Robinson 2011-12-20 21:50:57 EST
I spent hours trying to solve a mysql resources problem, and came here after becoming convinced it was a bug - and a pretty serious one.

I upgraded a server from fc15 to fc16 yesterday (12GB i7). It's a hot spare and node of a multiple master mysql cluster here at home over a VPN for handling overflows from my live webserver and doing slave backups/reporting.

Last night, after spending many hours upgrading fc15 to fc16, getting it booting properly with all necessary services running again, the automatic mysql backup failed with hundreds of these lines:

"Out of resources when opening file '/dev/shm/#sql_23a7_2.MYI' (Errcode: 24) when trying to dump tablespaces"

Looked like file descriptors running out, but 8192 was enough when running under fc15 the day before, and my.cnf hadn't changed.
I tried adding ulimit -n 8192 to rc.local, reboot. Still no joy. Backup failed after a few databases.

Here is some info:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
uname -a
Linux 3.1.5-6.fc16.x86_64 #1 SMP Thu Dec 15 16:14:44 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

rpm -qa | grep -i mysql-server
mysql-server-5.5.18-1.fc16.x86_64

ps aux | grep mysqld
* /usr/libexec/mysqld * --open-files-limit=8192 *

ulimit -n;
8192

(never needed to do this before)
/etc/security/limits.conf
*                soft    nofile          8192
*                hard    nofile          8192

mysql -p -Be "show variables like 'open_files_limit';"
Variable_name         Value
open_files_limit      1024
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Another two fc16 slaves show the same things.

mysqld starts with --open-files-limit=8192 as desired, and presumably ulimit -n 8192 is being called in the /usr/bin/mysqld_safe as root, but something is still preventing mysqld exceeding 1024, and this makes the database unusable with the desired table cache. Reducing the table cache to 500 allowed the backup process to complete, although it needs a table cache of a few thousand, and 2-3x more file descriptors. 1024 is totally inadequate as a default.

The same commands on the live fc15 server (16GB i7 2600K):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
uname -r;
2.6.40.6-0.fc15.x86_64

rpm -qa | grep -i mysql-server;
mysql-server-5.5.18-1.fc15.x86_64

ps aux | grep mysqld;
mysql * /usr/libexec/mysqld * --open-files-limit=8192 *

ulimit -n;
1024

mysql -p -Be "show variables like 'open_files_limit';"
Variable_name           Value
open_files_limit        16494
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Interesting, that open files limit is more than double than asked for, and this page explains why nicely:
http://blogs.oracle.com/rainy/entry/changing_process_max_file_descriptor

mysql-server is the same on both fc15 and 16.
The fc15 system doesn't have any modifications to /etc/security/limits.conf
ulimit -n is not set in rc.local

So, instead of a systemd problem, which seems to have done its job correctly in launching mysql with the right parameters, perhaps it is a kernel regression, or something else interfering?

I'm not yet ready to risk upgrading fc15->16 on my remote live server after this and other experiences.

Hope this info helps.
Comment 8 Tomas Mraz 2011-12-21 04:14:37 EST
I've added a comment to the limits.conf about that it applies only to the user login sessions.

If setting LimitNOFILE=8192 in mysqld.service file does not help, please open a new bug against systemd.
Comment 9 Andrew Haveland-Robinson 2011-12-21 08:34:06 EST
Thanks Tomas, it worked! but I would like to understand why.
It fixed it so I have enough files now, but puristically the behaviour has changed: LimitNOFILE is a hard setting, open-files-limit in my.cnf is ignored and mysqld doesn't set it based on table_cache, so my.cnf is less centralized.

I added to mysqld.service
[Service]
LimitNOFILE=32768

my.cnf:
table_cache      = 8192
open-files-limit = 16384

systemctl --system daemon-reload
systemctl restart mysqld.service

mysql -u root -p -Be "show variables like 'open_files_limit'"
Variable_name      Value
open_files_limit   32768

Whatever value of LimitNOFILE I use is what mysqld gets.

The same settings running under fc15 (sysv) show expected results:
(2 x table_cache) + max_connections + 10
mysql -u root -p -Be "show variables like 'open_files_limit'"
Variable_name      Value
open_files_limit   16494

This is likely to bite a few moderately heavy users.
mysqld should be allowed to set the limits in my.cnf
Should I add this as a new systemd bug?
Comment 10 Tomas Mraz 2011-12-21 09:11:50 EST
Perhaps the open_files_limit was somehow handled by the old mysqld sysv init script? In that case I'd report it against mysql because the systemd mysql unit is deficient.
Comment 11 Tom Lane 2012-01-27 17:29:34 EST
(In reply to comment #10)
> Perhaps the open_files_limit was somehow handled by the old mysqld sysv init
> script? In that case I'd report it against mysql because the systemd mysql unit
> is deficient.

Pre-systemd, the mysqld_safe script was launched as root, so it could and did increase the open files ulimit value.  In the systemd unit file we are not launching the script as root, so it can't increase ulimit; if you want something above the default hard limit then you *must* set LimitNOFILE to a value at least as large as what you need.  This is not a bug but an intentional change to improve security.  The systemd environment already forces other changes in the way to configure mysql, so the fact that there's one more doesn't bother me particularly.
Comment 12 Andrew Haveland-Robinson 2012-01-28 21:58:13 EST
Fair enough Tom, I can live with the change of policy.
However, it cost me several hours of digging around, so to avoid others going down the same route, I suggest adding this to the supplied mysqld.service file, thus:

[Service]
#Uncomment if you need to increase open-files-limit
#LimitNOFILE=1024
Comment 13 Michal Schmidt 2012-01-29 08:37:02 EST
(In reply to comment #12)
> I suggest adding this to the supplied mysqld.service file, thus:
> 
> [Service]
> #Uncomment if you need to increase open-files-limit
> #LimitNOFILE=1024

You should not edit /lib/systemd/system/* files directly. They will be overwritten on updates. You need to put your modifications under /etc.

https://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F
Comment 14 Fedora Update System 2012-01-31 15:57:53 EST
pam-1.1.5-5.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/pam-1.1.5-5.fc15
Comment 15 Fedora Update System 2012-01-31 15:58:11 EST
pam-1.1.5-5.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/pam-1.1.5-5.fc16
Comment 16 Fedora Update System 2012-02-08 18:01:31 EST
pam-1.1.5-5.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 17 Fedora Update System 2012-02-14 04:06:02 EST
pam-1.1.5-5.fc15 has been pushed to the Fedora 15 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 18 Saso Tavcar 2012-11-04 13:27:38 EST
Also check SELinux status. Variable LimitNOFILE does not apply with SELinux enabled. 

It works for me after:

# setenforce 0
# systemctl disable mysql.service
# vi /etc/systemd/system/mysqld.service
....

# cat /etc/systemd/system/mysqld.service
.include /lib/systemd/system/mysqld.service

[Service]
LimitNOFILE=65535

# systemctl enable mysql.service
# systemctl restart mysqld.service


mysql> SHOW VARIABLES LIKE 'open_files_limit';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| open_files_limit | 65535 |
+------------------+-------+
1 row in set (0.00 sec)
Comment 19 Tomas Mraz 2012-11-05 03:24:56 EST
That would be a SELinux policy bug, can you please open a new bug report against it?
Comment 20 Michael Bayer 2015-09-13 13:32:57 EDT
Just noting that users are still confused about this issue: http://serverfault.com/questions/628610/increasing-nproc-for-processes-launched-by-systemd-on-centos-7

Looking at a Fedora 20 machine, I see the file has this:

#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.

This is an accurate message, however I think it really doesn't provide enough clues for the typical user that doesn't deal with these limits very often to know what's going on.  It does not even mention the word "systemd" as something to help users recognize what we're talking about.  A message that includes more of the following might help to alleviate ongoing confusion:

# This file *only* refers to resource limits for the users logged in via PAM.
# It does *not* refer to resource limits of the system services, in other 
# words all services that run via systemd, as well as all processes
# forked off by systemd processes. In order to set resource limits 
# for systemd-initiated services and their child processes, refer 
# to process-specific limits in the appropriate 
# /etc/systemd/system/<servicename>.service file; see 
# <link to official docuemntation> for details.
Comment 21 Tomas Mraz 2015-09-14 04:11:29 EDT
I do not think that referencing systemd-specific details in PAM configuration is appropriate. So I could modify it to contain:

# This file *only* refers to resource limits for the users logged in via PAM.
# It does *not* refer to resource limits of the system services, in other 
# words all services that run via systemd, as well as all processes
# forked off by systemd processes.

If you really think this makes the message more clear.

But I am not going to add the rest as that belongs to systemd or Fedora administrator guide documentation.
Comment 22 Michael Bayer 2015-09-14 10:09:52 EDT
I think adding the note about "systemd" would be great, probably would have been enough for me to avoid needing that stackexchange issue.  I understand why you don't want to get into the systemd docs, I wouldn't either.  

how about:

# This file *only* refers to resource limits for the users logged in via PAM.
# It does *not* refer to resource limits of the system services, in other 
# words all services that run via systemd, as well as all processes
# forked off by systemd processes; please refer to systemd's configurational
# systems in order to modify these limits.

so there's some notion of an action to take, kind of helps with "no really, there's another way to do it for that", as opposed to, "it's not possible?"
Comment 23 Tomas Mraz 2015-09-16 10:37:12 EDT
> # forked off by systemd processes; please refer to systemd's configurational
> # systems in order to modify these limits.
'systemd services' configuration files' ?
Comment 24 Michal Sekletar 2015-09-21 15:19:15 EDT
(In reply to Michael Bayer from comment #22)

> I think adding the note about "systemd" would be great, probably would have
> been enough for me to avoid needing that stackexchange issue.

I don't understand why current note in the config file needs any changes at all. I think it was perfectly accurate and applied not only to Fedora but to other distros without systemd as well. It simply pointed out that one should consult documentation of init system to see how to specify limits for system services. Now we make it sound like that there was change in behavior caused by move to systemd. AFAICT, that is not case, as one would have the same problem with upstart and sysvinit.
Comment 25 Michael Bayer 2015-09-21 15:36:39 EDT
> Now we make it sound like that there was change in behavior caused by move to systemd.

there is truth to this, in that limits.conf would often be ignored anyway by many services.   However, limits.conf would still in many cases, including that of MySQL, serve as the top-level limit that "ulimit" could be set toward, and by starting MySQL with systemd now this is a change in behavior; see comment 11 here.

Note You need to log in before you can comment on or make changes to this bug.