Bug 1040418

Summary:	NFS: Scalability issue with subdir authentication
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	santosh pradhan <spradhan>
Component:	doc-Console_Administration_Guide	Assignee:	rjoseph
Status:	CLOSED CANTFIX	QA Contact:	Anil Shah <ashah>
Severity:	high	Docs Contact:
Priority:	medium
Version:	2.1	CC:	ashah, divya, mhideo, nlevinki, poelstra, rhs-bugs, rjoseph, rwheeler, sdharane, sharne, shtripat, storage-doc, vagarwal, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	The length of the argument to nfs.export-dir (or any other gluster set option) is limited to internal buffer size of Linux shell. In a typical setup the default size of this buffer is 131072.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-02-04 10:23:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1035040

Description santosh pradhan 2013-12-11 11:50:15 UTC

Description of problem:


Version-Release number of selected component (if applicable):


Gluster NFS has subdir authentication feature. It needs to be validated against possible scalability issues. e.g.

1. NFS Volume file size

2. rpc size during volfile change

3. Time for re-configure

4. Watch for mount timeouts

5. ~64K configurations (try with 64k entries) 

6. Vary the number of hostnames/IP-addresses (range from 16 - 256)

7. RHS-C 

This is in fact not a defect but a tracker bug for scalability of subdir authentication RFE.

Comment 1 Vivek Agarwal 2013-12-12 11:28:23 UTC

Can you get some performance numbers with the above parameters?

Comment 2 Anil Shah 2013-12-17 06:58:25 UTC

gluster vol set nfs.export-dir is working fine with 4k no of directories.

,/dir3878(10.70.36.240),/dir3879(10.70.36.240),/dir3880(10.70.36.240),/dir3881(10.70.36.240),/dir3882(10.70.36.240),/dir3883(10.70.36.240),/dir3884(10.70.36.240),/dir3885(10.70.36.240),/dir3886(10.70.36.240),/dir3887(10.70.36.240),/dir3888(10.70.36.240),/dir3889(10.70.36.240),/dir3890(10.70.36.240),/dir3891(10.70.36.240),/dir3892(10.70.36.240),/dir3893(10.70.36.240),/dir3894(10.70.36.240),/dir3895(10.70.36.240),/dir3896(10.70.36.240),/dir3897(10.70.36.240),/dir3898(10.70.36.240),/dir3899(10.70.36.240),/dir3900(10.70.36.240),/dir3901(10.70.36.240),/dir3902(10.70.36.240),/dir3903(10.70.36.240),/dir3904(10.70.36.240),/dir3905(10.70.36.240),/dir3906(10.70.36.240),/dir3907(10.70.36.240),/dir3908(10.70.36.240),/dir3909(10.70.36.240),/dir3910(10.70.36.240),/dir3911(10.70.36.240),/dir3912(10.70.36.240),/dir3913(10.70.36.240),/dir3914(10.70.36.240),/dir3915(10.70.36.240),/dir3916(10.70.36.240),/dir3917(10.70.36.240),/dir3918(10.70.36.240),/dir3919(10.70.36.240),/dir3920(10.70.36.240),/dir3921(10.70.36.240),/dir3922(10.70.36.240),/dir3923(10.70.36.240),/dir3924(10.70.36.240),/dir3925(10.70.36.240),/dir3926(10.70.36.240),/dir3927(10.70.36.240),/dir3928(10.70.36.240),/dir3929(10.70.36.240),/dir3930(10.70.36.240),/dir3931(10.70.36.240),/dir3932(10.70.36.240),/dir3933(10.70.36.240),/dir3934(10.70.36.240),/dir3935(10.70.36.240),/dir3936(10.70.36.240),/dir3937(10.70.36.240),/dir3938(10.70.36.240),/dir3939(10.70.36.240),/dir3940(10.70.36.240),/dir3941(10.70.36.240),/dir3942(10.70.36.240),/dir3943(10.70.36.240),/dir3944(10.70.36.240),/dir3945(10.70.36.240),/dir3946(10.70.36.240),/dir3947(10.70.36.240),/dir3948(10.70.36.240),/dir3949(10.70.36.240),/dir3950(10.70.36.240),/dir3951(10.70.36.240),/dir3952(10.70.36.240),/dir3953(10.70.36.240),/dir3954(10.70.36.240),/dir3955(10.70.36.240),/dir3956(10.70.36.240),/dir3957(10.70.36.240),/dir3958(10.70.36.240),/dir3959(10.70.36.240),/dir3960(10.70.36.240),/dir3961(10.70.36.240),/dir3962(10.70.36.240),/dir3963(10.70.36.240),/dir3964(10.70.36.240),/dir3965(10.70.36.240),/dir3966(10.70.36.240),/dir3967(10.70.36.240),/dir3968(10.70.36.240),/dir3969(10.70.36.240),/dir3970(10.70.36.240),/dir3971(10.70.36.240),/dir3972(10.70.36.240),/dir3973(10.70.36.240),/dir3974(10.70.36.240),/dir3975(10.70.36.240),/dir3976(10.70.36.240),/dir3977(10.70.36.240),/dir3978(10.70.36.240),/dir3979(10.70.36.240),/dir3980(10.70.36.240),/dir3981(10.70.36.240),/dir3982(10.70.36.240),/dir3983(10.70.36.240),/dir3984(10.70.36.240),/dir3985(10.70.36.240),/dir3986(10.70.36.240),/dir3987(10.70.36.240),/dir3988(10.70.36.240),/dir3989(10.70.36.240),/dir3990(10.70.36.240),/dir3991(10.70.36.240),/dir3992(10.70.36.240),/dir3993(10.70.36.240),/dir3994(10.70.36.240),/dir3995(10.70.36.240),/dir3996(10.70.36.240),/dir3997(10.70.36.240),/dir3998(10.70.36.240),/dir3999(10.70.36.240),/dir4000(10.70.36.240)"
volume set: success

real	1m1.550s
user	0m54.144s
sys	0m0.169s

Comment 3 Anil Shah 2013-12-17 06:59:49 UTC

however,
gluster vol set nfs.export-dir command for 8k and 16K no. of directories is falling with below error.

2.197|10.70.42.101|10.70.42.200),/dir15989(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15990(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15991(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15992(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15993(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15994(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15995(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15996(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15997(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15998(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15999(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir16000(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200)"
./export-dir.sh: line 68: /usr/sbin/gluster: Argument list too long

Comment 4 santosh pradhan 2013-12-17 17:12:39 UTC


./export-dir.sh: line 68: /usr/sbin/gluster: Argument list too long

The error is thrown by the SHELL itself, not by the gluster. Rajesh will share the data, he collected from the SHELL. (bash)

Comment 5 rjoseph 2013-12-18 05:15:17 UTC

I tested nfs.export-dir command in my Fedora 19 machine and the command fails with "Argument list too long" as observed by Anil and Santosh. Then I checked the current limit for the maximum allowed commandline argument length (using xargs --show-limits) and I got the following result

POSIX upper limit on argument length (this system): 2091957
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2088810
Size of command buffer we are actually using: 131072


On RHEL 6.5 I got the following result.
Your environment variables take up 2009 bytes
POSIX upper limit on argument length (this system): 2617383
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2615374
Size of command buffer we are actually using: 131072

So it seems like as of now the limiting factor is command buffer size which is 131072.

Comment 6 santosh pradhan 2013-12-19 11:51:29 UTC

Reviewed on:
https://code.engineering.redhat.com/gerrit/13847

Comment 7 Vivek Agarwal 2013-12-24 08:29:16 UTC

per discussion 12/23, this will go as a known issue as it is related to shell limitation. Removing it from the must fix list for corbett

Comment 8 Shalaka 2014-01-03 06:48:59 UTC

As this bug is identified as Known Issue, please add the doctext.

Comment 9 santosh pradhan 2014-02-04 06:16:54 UTC

May be Rajesh can rectify the text if he wants.

Comment 10 santosh pradhan 2014-02-05 05:41:40 UTC

The modified text looks good to me.

Comment 12 Vivek Agarwal 2015-02-04 10:23:48 UTC

Based on discussion with ndevos (dev) and asriram (doc), closing this and raising a doc bug for this to be known issue.