Bug 457226

Summary: [NetApp 5.3 bug] multipathd fails to start if max_fds option is configured as "unlimited" in multipath.conf with 1024 paths.
Product: Red Hat Enterprise Linux 5 Reporter: Rajashekhar M A <rajashekhar.a>
Component: device-mapper-multipathAssignee: Ben Marzinski <bmarzins>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 5.2CC: agk, andriusb, bmarzins, bmr, christophe.varoqui, cmarthal, coughlan, cward, ddomingo, dwysocha, edamato, egoggin, heinzm, junichi.nomura, kueda, lmb, mbroz, naveenr, prockai, tranlan, xdl-redhat-bugzilla
Target Milestone: rcKeywords: OtherQA
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, setting max_fds to "unlimited" in /etc/multipath.conf would prevent the multipathd daemon from starting. If number of open file descriptors needd to be set to the system maximum, max_fds should be set to "max"
 Story Points: ---
Clone Of: Environment:
Last Closed: 2009-01-20 22:09:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 251346    
Bug Blocks: 373081, 454962, 457228    
Attachments:
Description Flags
multipath.conf settings none

Description Rajashekhar M A 2008-07-30 12:24:30 UTC 
Description of problem:

On a RHEL2.2 host, I have 256 luns mapped each with 4 FCP paths giving a total
of 1024 paths. The FCP HBA driver on the host properly detects all these paths
i.e. all 1024 entries are present in the sysfs. But when I start multipathd with
the option max_fds set to "unlimited" in multipath.conf file, the daemon fails
to start up:

# /etc/init.d/multipathd start
Starting multipathd daemon:                                [  OK  ]
# /etc/init.d/multipathd status
multipathd dead but pid file exists

I see the following message in /var/log/messages --
multipathd: can't set open fds limit to -1 : Operation not permitted


Version-Release number of selected component (if applicable):

# rpm -qa | grep device
device-mapper-event-1.02.24-1.el5
device-mapper-1.02.24-1.el5
device-mapper-1.02.24-1.el5
device-mapper-multipath-0.4.7-17.el5


How reproducible:

Always


Steps to Reproduce:

1. Map 256 luns to a RHEL5.2 host with 4 FCP paths each thereby giving a total
of 1024 paths.
2. Add the option "max_fds unlimited" in defaults section in multipath.conf file.
3. Configure maps for all the devices, by running "multipath -v3" command.
4. Start the multipathd daemon.

Actual results:

The multipathd daemon gets killed during startup for the above setup.
# /etc/init.d/multipathd status
multipathd dead but pid file exists


Expected results:

The multipathd daemon should start properly, with fd limit set to a possible
maximum value.

Additional info:

1. If max_fds option is configured with a sufficiently high numerical value,
multipathd starts up fine.

2. /etc/multipath.conf settings are attached.
Comment 1 Rajashekhar M A 2008-07-30 12:24:30 UTC 
Created attachment 312991 [details]
multipath.conf settings
Comment 2 Rajashekhar M A 2008-07-30 12:26:22 UTC 
Sorry for the typo in the description section, its "On a RHEL5.2 host...".
Comment 5 Dave Wysochanski 2008-07-30 16:39:33 UTC 
Ben's proposed action from that bug 251346 - remove the "unlimited", provide a
hard upper limit (based on the kernel's limit) and call it "max".

Comment #20 From Ben Marzinski (bmarzins)  	 on 2008-04-22 13:42
EST  	[reply]   	 Private

Unfortunately, unlimited does seem like it will not work do to an in-kernel
limit of 1048576 open file descriptors. However, for all practical purposes,
this limit shouldn't be a big problem.  I'll remove the "unlimited" option in
future releases. It might be best to replace it with a "max" option, that simply
sets the limit to the in-kernel max.
Comment 6 Don Domingo 2008-08-03 22:29:20 UTC 
tagging for inclusion in the RHEL5.3 release notes, including as quoted in "Release Notes" field of this bug.
Comment 7 Don Domingo 2008-08-03 22:29:20 UTC 
Added New Release Notes Contents.
Comment 8 Ben Marzinski 2008-09-02 22:34:05 UTC 
Fixed.  I've changed "unlimited" to "max" which sets the number of fds to the NR_OPEN kernel value.
Comment 10 Andrius Benokraitis 2008-09-03 13:52:50 UTC 
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,3 +1 @@
-In /etc/multipath.conf, setting max_fds to unlimited will prevent the multipathd
+In /etc/multipath.conf, setting max_fds to unlimited will prevent the multipathd daemon from starting up properly. As such, you should use a sufficiently high value instead for this setting.-daemon from starting up properly. As such, you should use a sufficiently high
-value instead for this setting.
Comment 11 Ben Marzinski 2008-09-03 17:46:50 UTC 
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1 @@
-In /etc/multipath.conf, setting max_fds to unlimited will prevent the multipathd daemon from starting up properly. As such, you should use a sufficiently high value instead for this setting.+In /etc/multipath.conf, "umlimited" is no longer an appropriate value for the "max_fds" option. Instead, you should use the value "max" which sets the number of open file descriptors to the system maximum.
Comment 13 Ryan Lerch 2008-09-04 00:37:28 UTC 
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1,3 @@
-In /etc/multipath.conf, "umlimited" is no longer an appropriate value for the "max_fds" option. Instead, you should use the value "max" which sets the number of open file descriptors to the system maximum.+In /etc/multipath.conf, "umlimited" is no longer an appropriate value for the "max_fds" option. Instead, you should use the value "max" which sets the number of open file descriptors to the system maximum.
+
+Previously, setting max_fds to unlimited in /etc/multipath.conf would prevent the multipathd daemon from starting.
Comment 14 Ben Marzinski 2008-09-04 20:46:53 UTC 
In response to comment #12, multipathd does now start up with over 1K LUNs, so 251346 is no longer an issue.  This bug (457226) is now resolved.  All the customers need to know is that they can use "max" now to do what "unlimited" was supposed to do.

The updated release note in comment #13 looks great.
Comment 16 Ryan Lerch 2008-10-24 00:07:46 UTC 
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,3 +1 @@
-In /etc/multipath.conf, "umlimited" is no longer an appropriate value for the "max_fds" option. Instead, you should use the value "max" which sets the number of open file descriptors to the system maximum.
+Previously, setting max_fds to "unlimited" in /etc/multipath.conf would prevent the multipathd daemon from starting. If number of open file descriptors needd to be set to the system maximum,  max_fds should be set to "max"-
-Previously, setting max_fds to unlimited in /etc/multipath.conf would prevent the multipathd daemon from starting.
Comment 17 Naveen Reddy 2008-10-30 16:23:33 UTC 
I have verified this by setting the max_fds option to "max" in RHEL5U3 Beta, but still the multipathd goes dead when I start the multipathd with 1024 paths.

# /etc/init.d/multipathd status
multipathd dead but pid file exists

If a change the max_fds to some high value (say 4096), multipathd starts correctly.
Comment 18 Ben Marzinski 2008-11-04 15:50:14 UTC 
This is fixed. I'm rebuilding a package now.
Comment 20 Naveen Reddy 2008-11-14 06:35:43 UTC 
I have verified it in the RHEL5.3 GASnapshot2 (device-mapper-multipath-0.4.7-22.el5). This issue is fixed.
Comment 23 errata-xmlrpc 2009-01-20 22:09:07 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-0232.html