Description of problem: On a RHEL2.2 host, I have 256 luns mapped each with 4 FCP paths giving a total of 1024 paths. The FCP HBA driver on the host properly detects all these paths i.e. all 1024 entries are present in the sysfs. But when I start multipathd with the option max_fds set to "unlimited" in multipath.conf file, the daemon fails to start up: # /etc/init.d/multipathd start Starting multipathd daemon: [ OK ] # /etc/init.d/multipathd status multipathd dead but pid file exists I see the following message in /var/log/messages -- multipathd: can't set open fds limit to -1 : Operation not permitted Version-Release number of selected component (if applicable): # rpm -qa | grep device device-mapper-event-1.02.24-1.el5 device-mapper-1.02.24-1.el5 device-mapper-1.02.24-1.el5 device-mapper-multipath-0.4.7-17.el5 How reproducible: Always Steps to Reproduce: 1. Map 256 luns to a RHEL5.2 host with 4 FCP paths each thereby giving a total of 1024 paths. 2. Add the option "max_fds unlimited" in defaults section in multipath.conf file. 3. Configure maps for all the devices, by running "multipath -v3" command. 4. Start the multipathd daemon. Actual results: The multipathd daemon gets killed during startup for the above setup. # /etc/init.d/multipathd status multipathd dead but pid file exists Expected results: The multipathd daemon should start properly, with fd limit set to a possible maximum value. Additional info: 1. If max_fds option is configured with a sufficiently high numerical value, multipathd starts up fine. 2. /etc/multipath.conf settings are attached.
Created attachment 312991 [details] multipath.conf settings
Sorry for the typo in the description section, its "On a RHEL5.2 host...".
Ben's proposed action from that bug 251346 - remove the "unlimited", provide a hard upper limit (based on the kernel's limit) and call it "max". Comment #20 From Ben Marzinski (bmarzins) on 2008-04-22 13:42 EST [reply] Private Unfortunately, unlimited does seem like it will not work do to an in-kernel limit of 1048576 open file descriptors. However, for all practical purposes, this limit shouldn't be a big problem. I'll remove the "unlimited" option in future releases. It might be best to replace it with a "max" option, that simply sets the limit to the in-kernel max.
tagging for inclusion in the RHEL5.3 release notes, including as quoted in "Release Notes" field of this bug.
Added New Release Notes Contents.
Fixed. I've changed "unlimited" to "max" which sets the number of fds to the NR_OPEN kernel value.
Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,3 +1 @@ -In /etc/multipath.conf, setting max_fds to unlimited will prevent the multipathd +In /etc/multipath.conf, setting max_fds to unlimited will prevent the multipathd daemon from starting up properly. As such, you should use a sufficiently high value instead for this setting.-daemon from starting up properly. As such, you should use a sufficiently high -value instead for this setting.
Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -In /etc/multipath.conf, setting max_fds to unlimited will prevent the multipathd daemon from starting up properly. As such, you should use a sufficiently high value instead for this setting.+In /etc/multipath.conf, "umlimited" is no longer an appropriate value for the "max_fds" option. Instead, you should use the value "max" which sets the number of open file descriptors to the system maximum.
Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1,3 @@ -In /etc/multipath.conf, "umlimited" is no longer an appropriate value for the "max_fds" option. Instead, you should use the value "max" which sets the number of open file descriptors to the system maximum.+In /etc/multipath.conf, "umlimited" is no longer an appropriate value for the "max_fds" option. Instead, you should use the value "max" which sets the number of open file descriptors to the system maximum. + +Previously, setting max_fds to unlimited in /etc/multipath.conf would prevent the multipathd daemon from starting.
In response to comment #12, multipathd does now start up with over 1K LUNs, so 251346 is no longer an issue. This bug (457226) is now resolved. All the customers need to know is that they can use "max" now to do what "unlimited" was supposed to do. The updated release note in comment #13 looks great.
Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,3 +1 @@ -In /etc/multipath.conf, "umlimited" is no longer an appropriate value for the "max_fds" option. Instead, you should use the value "max" which sets the number of open file descriptors to the system maximum. +Previously, setting max_fds to "unlimited" in /etc/multipath.conf would prevent the multipathd daemon from starting. If number of open file descriptors needd to be set to the system maximum, max_fds should be set to "max"- -Previously, setting max_fds to unlimited in /etc/multipath.conf would prevent the multipathd daemon from starting.
I have verified this by setting the max_fds option to "max" in RHEL5U3 Beta, but still the multipathd goes dead when I start the multipathd with 1024 paths. # /etc/init.d/multipathd status multipathd dead but pid file exists If a change the max_fds to some high value (say 4096), multipathd starts correctly.
This is fixed. I'm rebuilding a package now.
I have verified it in the RHEL5.3 GASnapshot2 (device-mapper-multipath-0.4.7-22.el5). This issue is fixed.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2009-0232.html