Description of problem: By default Fedora use small ulimit for open files (1024). On servers with high load, this value should be increased. In earlier versions of Fedora, we set the values in limits.conf. But now, this method doesn't work. When systemd starts processes, it completely ignores these settings. Steps to Reproduce: 1. add new limits to limits.conf # echo "mysql hard nofile 131070" >> /etc/security/limits.conf # echo "mysql soft nofile 131070" >> /etc/security/limits.conf 2. Set big values for mysql # echo -e "table_open_cache=16384\nmax_connections=1700" >> /etc/my.cnf 3. start mysqld server # systemctl start mysqld.service Actual results: Now we need to look in /var/log/mysqld.log. It is seen that the number of available descriptors left by default - 1024 ===cut on=== 111115 13:18:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 111115 13:18:36 [Warning] Changed limits: max_open_files: 1024 max_connections: 214 table_cache: 400 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 111115 13:18:36 [Note] Plugin 'FEDERATED' is disabled. 111115 13:18:36 InnoDB: The InnoDB memory heap is disabled 111115 13:18:36 InnoDB: Mutexes and rw_locks use GCC atomic builtins 111115 13:18:36 InnoDB: Compressed tables use zlib 1.2.5 111115 13:18:36 InnoDB: Using Linux native AIO 111115 13:18:36 InnoDB: Initializing buffer pool, size = 24.0G 111115 13:18:37 InnoDB: Completed initialization of buffer pool 111115 13:18:37 InnoDB: highest supported file format is Barracuda. 111115 13:18:38 InnoDB: Waiting for the background threads to start 111115 13:18:39 InnoDB: 1.1.8 started; log sequence number 196542194396 111115 13:18:39 [Note] Event Scheduler: Loaded 0 events 111115 13:18:39 [Note] /usr/libexec/mysqld: ready for connections. ===cut off=== Expected results: Start mysqld with 131070 descriptors, without warnings. Additional info: If mysqld start manually, everything works without problems. Ex: # ulimit -n 131070 # su - mysql $ ulimit -n 131070 $ /usr/bin/mysqld_safe --nowatch --basedir=/usr
Systemd does not support global limits, the file is intentionally ignored. LimitNOFILE= in the service file can be set to specify the number of open file descriptors for a specific service.
Details are in: man systemd.exec
Maybe it makes sense to add a small notice to the man page systemd.exec, f.e. "file limits.conf is ignored". I think we are not alone who face to this problem. Thank you for quick reply. Ticket may be closed.
It would make sense to put a warning in the packaged /etc/security/limits.conf itself: # NOTE: # These limits affect only user logins, not services spawned by systemd. # To set the limits for a service, use the Limit*= options in the service's # systemd unit file as documented in systemd.exec(5).
I ran into the same problem, but following the man page instructions, where I modified the mysqld.service file in /lib/systemd/system to change the limits did nothing to actually fix the problem. I even ran systemctl --system daemon-reload after modifying it. I even tried rebooting just to see if that would help, which to no surprise it didn't. I tried using infinity, "infinity", unlimited, and specific values, like 8192, but to no avail.
I spent hours trying to solve a mysql resources problem, and came here after becoming convinced it was a bug - and a pretty serious one. I upgraded a server from fc15 to fc16 yesterday (12GB i7). It's a hot spare and node of a multiple master mysql cluster here at home over a VPN for handling overflows from my live webserver and doing slave backups/reporting. Last night, after spending many hours upgrading fc15 to fc16, getting it booting properly with all necessary services running again, the automatic mysql backup failed with hundreds of these lines: "Out of resources when opening file '/dev/shm/#sql_23a7_2.MYI' (Errcode: 24) when trying to dump tablespaces" Looked like file descriptors running out, but 8192 was enough when running under fc15 the day before, and my.cnf hadn't changed. I tried adding ulimit -n 8192 to rc.local, reboot. Still no joy. Backup failed after a few databases. Here is some info: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ uname -a Linux 3.1.5-6.fc16.x86_64 #1 SMP Thu Dec 15 16:14:44 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux rpm -qa | grep -i mysql-server mysql-server-5.5.18-1.fc16.x86_64 ps aux | grep mysqld * /usr/libexec/mysqld * --open-files-limit=8192 * ulimit -n; 8192 (never needed to do this before) /etc/security/limits.conf * soft nofile 8192 * hard nofile 8192 mysql -p -Be "show variables like 'open_files_limit';" Variable_name Value open_files_limit 1024 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Another two fc16 slaves show the same things. mysqld starts with --open-files-limit=8192 as desired, and presumably ulimit -n 8192 is being called in the /usr/bin/mysqld_safe as root, but something is still preventing mysqld exceeding 1024, and this makes the database unusable with the desired table cache. Reducing the table cache to 500 allowed the backup process to complete, although it needs a table cache of a few thousand, and 2-3x more file descriptors. 1024 is totally inadequate as a default. The same commands on the live fc15 server (16GB i7 2600K): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ uname -r; 2.6.40.6-0.fc15.x86_64 rpm -qa | grep -i mysql-server; mysql-server-5.5.18-1.fc15.x86_64 ps aux | grep mysqld; mysql * /usr/libexec/mysqld * --open-files-limit=8192 * ulimit -n; 1024 mysql -p -Be "show variables like 'open_files_limit';" Variable_name Value open_files_limit 16494 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interesting, that open files limit is more than double than asked for, and this page explains why nicely: http://blogs.oracle.com/rainy/entry/changing_process_max_file_descriptor mysql-server is the same on both fc15 and 16. The fc15 system doesn't have any modifications to /etc/security/limits.conf ulimit -n is not set in rc.local So, instead of a systemd problem, which seems to have done its job correctly in launching mysql with the right parameters, perhaps it is a kernel regression, or something else interfering? I'm not yet ready to risk upgrading fc15->16 on my remote live server after this and other experiences. Hope this info helps.
I've added a comment to the limits.conf about that it applies only to the user login sessions. If setting LimitNOFILE=8192 in mysqld.service file does not help, please open a new bug against systemd.
Thanks Tomas, it worked! but I would like to understand why. It fixed it so I have enough files now, but puristically the behaviour has changed: LimitNOFILE is a hard setting, open-files-limit in my.cnf is ignored and mysqld doesn't set it based on table_cache, so my.cnf is less centralized. I added to mysqld.service [Service] LimitNOFILE=32768 my.cnf: table_cache = 8192 open-files-limit = 16384 systemctl --system daemon-reload systemctl restart mysqld.service mysql -u root -p -Be "show variables like 'open_files_limit'" Variable_name Value open_files_limit 32768 Whatever value of LimitNOFILE I use is what mysqld gets. The same settings running under fc15 (sysv) show expected results: (2 x table_cache) + max_connections + 10 mysql -u root -p -Be "show variables like 'open_files_limit'" Variable_name Value open_files_limit 16494 This is likely to bite a few moderately heavy users. mysqld should be allowed to set the limits in my.cnf Should I add this as a new systemd bug?
Perhaps the open_files_limit was somehow handled by the old mysqld sysv init script? In that case I'd report it against mysql because the systemd mysql unit is deficient.
(In reply to comment #10) > Perhaps the open_files_limit was somehow handled by the old mysqld sysv init > script? In that case I'd report it against mysql because the systemd mysql unit > is deficient. Pre-systemd, the mysqld_safe script was launched as root, so it could and did increase the open files ulimit value. In the systemd unit file we are not launching the script as root, so it can't increase ulimit; if you want something above the default hard limit then you *must* set LimitNOFILE to a value at least as large as what you need. This is not a bug but an intentional change to improve security. The systemd environment already forces other changes in the way to configure mysql, so the fact that there's one more doesn't bother me particularly.
Fair enough Tom, I can live with the change of policy. However, it cost me several hours of digging around, so to avoid others going down the same route, I suggest adding this to the supplied mysqld.service file, thus: [Service] #Uncomment if you need to increase open-files-limit #LimitNOFILE=1024
(In reply to comment #12) > I suggest adding this to the supplied mysqld.service file, thus: > > [Service] > #Uncomment if you need to increase open-files-limit > #LimitNOFILE=1024 You should not edit /lib/systemd/system/* files directly. They will be overwritten on updates. You need to put your modifications under /etc. https://fedoraproject.org/wiki/Systemd#How_do_I_customize_a_unit_file.2F_add_a_custom_unit_file.3F
pam-1.1.5-5.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/pam-1.1.5-5.fc15
pam-1.1.5-5.fc16 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/pam-1.1.5-5.fc16
pam-1.1.5-5.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.
pam-1.1.5-5.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.
Also check SELinux status. Variable LimitNOFILE does not apply with SELinux enabled. It works for me after: # setenforce 0 # systemctl disable mysql.service # vi /etc/systemd/system/mysqld.service .... # cat /etc/systemd/system/mysqld.service .include /lib/systemd/system/mysqld.service [Service] LimitNOFILE=65535 # systemctl enable mysql.service # systemctl restart mysqld.service mysql> SHOW VARIABLES LIKE 'open_files_limit'; +------------------+-------+ | Variable_name | Value | +------------------+-------+ | open_files_limit | 65535 | +------------------+-------+ 1 row in set (0.00 sec)
That would be a SELinux policy bug, can you please open a new bug report against it?
Just noting that users are still confused about this issue: http://serverfault.com/questions/628610/increasing-nproc-for-processes-launched-by-systemd-on-centos-7 Looking at a Fedora 20 machine, I see the file has this: #This file sets the resource limits for the users logged in via PAM. #It does not affect resource limits of the system services. This is an accurate message, however I think it really doesn't provide enough clues for the typical user that doesn't deal with these limits very often to know what's going on. It does not even mention the word "systemd" as something to help users recognize what we're talking about. A message that includes more of the following might help to alleviate ongoing confusion: # This file *only* refers to resource limits for the users logged in via PAM. # It does *not* refer to resource limits of the system services, in other # words all services that run via systemd, as well as all processes # forked off by systemd processes. In order to set resource limits # for systemd-initiated services and their child processes, refer # to process-specific limits in the appropriate # /etc/systemd/system/<servicename>.service file; see # <link to official docuemntation> for details.
I do not think that referencing systemd-specific details in PAM configuration is appropriate. So I could modify it to contain: # This file *only* refers to resource limits for the users logged in via PAM. # It does *not* refer to resource limits of the system services, in other # words all services that run via systemd, as well as all processes # forked off by systemd processes. If you really think this makes the message more clear. But I am not going to add the rest as that belongs to systemd or Fedora administrator guide documentation.
I think adding the note about "systemd" would be great, probably would have been enough for me to avoid needing that stackexchange issue. I understand why you don't want to get into the systemd docs, I wouldn't either. how about: # This file *only* refers to resource limits for the users logged in via PAM. # It does *not* refer to resource limits of the system services, in other # words all services that run via systemd, as well as all processes # forked off by systemd processes; please refer to systemd's configurational # systems in order to modify these limits. so there's some notion of an action to take, kind of helps with "no really, there's another way to do it for that", as opposed to, "it's not possible?"
> # forked off by systemd processes; please refer to systemd's configurational > # systems in order to modify these limits. 'systemd services' configuration files' ?
(In reply to Michael Bayer from comment #22) > I think adding the note about "systemd" would be great, probably would have > been enough for me to avoid needing that stackexchange issue. I don't understand why current note in the config file needs any changes at all. I think it was perfectly accurate and applied not only to Fedora but to other distros without systemd as well. It simply pointed out that one should consult documentation of init system to see how to specify limits for system services. Now we make it sound like that there was change in behavior caused by move to systemd. AFAICT, that is not case, as one would have the same problem with upstart and sysvinit.
> Now we make it sound like that there was change in behavior caused by move to systemd. there is truth to this, in that limits.conf would often be ignored anyway by many services. However, limits.conf would still in many cases, including that of MySQL, serve as the top-level limit that "ulimit" could be set toward, and by starting MySQL with systemd now this is a change in behavior; see comment 11 here.
(In reply to Kay Sievers from comment #1) > Systemd does not support global limits, the file is intentionally ignored. > > LimitNOFILE= in the service file can be set to specify the number of open > file descriptors for a specific service. I came here from the Internet and wanted to comment for people looking for this: You can set a global limit for systemd; simply put your limit (e.g. "DefaultLimitCORE=1000000") into /etc/systemd/system.conf instead of your specific .service file, then it becomes global.