Bug 1040418 - NFS: Scalability issue with subdir authentication
Summary: NFS: Scalability issue with subdir authentication
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: doc-Console_Administration_Guide
Version: 2.1
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: ---
Assignee: rjoseph
QA Contact: Anil Shah
URL:
Whiteboard:
Depends On:
Blocks: 1035040
TreeView+ depends on / blocked
 
Reported: 2013-12-11 11:50 UTC by santosh pradhan
Modified: 2015-02-04 10:23 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The length of the argument to nfs.export-dir (or any other gluster set option) is limited to internal buffer size of Linux shell. In a typical setup the default size of this buffer is 131072.
Clone Of:
Environment:
Last Closed: 2015-02-04 10:23:48 UTC
Embargoed:


Attachments (Terms of Use)

Description santosh pradhan 2013-12-11 11:50:15 UTC
Description of problem:


Version-Release number of selected component (if applicable):


Gluster NFS has subdir authentication feature. It needs to be validated against possible scalability issues. e.g.

1. NFS Volume file size

2. rpc size during volfile change

3. Time for re-configure

4. Watch for mount timeouts

5. ~64K configurations (try with 64k entries) 

6. Vary the number of hostnames/IP-addresses (range from 16 - 256)

7. RHS-C 

This is in fact not a defect but a tracker bug for scalability of subdir authentication RFE.

Comment 1 Vivek Agarwal 2013-12-12 11:28:23 UTC
Can you get some performance numbers with the above parameters?

Comment 2 Anil Shah 2013-12-17 06:58:25 UTC
gluster vol set nfs.export-dir is working fine with 4k no of directories.

,/dir3878(10.70.36.240),/dir3879(10.70.36.240),/dir3880(10.70.36.240),/dir3881(10.70.36.240),/dir3882(10.70.36.240),/dir3883(10.70.36.240),/dir3884(10.70.36.240),/dir3885(10.70.36.240),/dir3886(10.70.36.240),/dir3887(10.70.36.240),/dir3888(10.70.36.240),/dir3889(10.70.36.240),/dir3890(10.70.36.240),/dir3891(10.70.36.240),/dir3892(10.70.36.240),/dir3893(10.70.36.240),/dir3894(10.70.36.240),/dir3895(10.70.36.240),/dir3896(10.70.36.240),/dir3897(10.70.36.240),/dir3898(10.70.36.240),/dir3899(10.70.36.240),/dir3900(10.70.36.240),/dir3901(10.70.36.240),/dir3902(10.70.36.240),/dir3903(10.70.36.240),/dir3904(10.70.36.240),/dir3905(10.70.36.240),/dir3906(10.70.36.240),/dir3907(10.70.36.240),/dir3908(10.70.36.240),/dir3909(10.70.36.240),/dir3910(10.70.36.240),/dir3911(10.70.36.240),/dir3912(10.70.36.240),/dir3913(10.70.36.240),/dir3914(10.70.36.240),/dir3915(10.70.36.240),/dir3916(10.70.36.240),/dir3917(10.70.36.240),/dir3918(10.70.36.240),/dir3919(10.70.36.240),/dir3920(10.70.36.240),/dir3921(10.70.36.240),/dir3922(10.70.36.240),/dir3923(10.70.36.240),/dir3924(10.70.36.240),/dir3925(10.70.36.240),/dir3926(10.70.36.240),/dir3927(10.70.36.240),/dir3928(10.70.36.240),/dir3929(10.70.36.240),/dir3930(10.70.36.240),/dir3931(10.70.36.240),/dir3932(10.70.36.240),/dir3933(10.70.36.240),/dir3934(10.70.36.240),/dir3935(10.70.36.240),/dir3936(10.70.36.240),/dir3937(10.70.36.240),/dir3938(10.70.36.240),/dir3939(10.70.36.240),/dir3940(10.70.36.240),/dir3941(10.70.36.240),/dir3942(10.70.36.240),/dir3943(10.70.36.240),/dir3944(10.70.36.240),/dir3945(10.70.36.240),/dir3946(10.70.36.240),/dir3947(10.70.36.240),/dir3948(10.70.36.240),/dir3949(10.70.36.240),/dir3950(10.70.36.240),/dir3951(10.70.36.240),/dir3952(10.70.36.240),/dir3953(10.70.36.240),/dir3954(10.70.36.240),/dir3955(10.70.36.240),/dir3956(10.70.36.240),/dir3957(10.70.36.240),/dir3958(10.70.36.240),/dir3959(10.70.36.240),/dir3960(10.70.36.240),/dir3961(10.70.36.240),/dir3962(10.70.36.240),/dir3963(10.70.36.240),/dir3964(10.70.36.240),/dir3965(10.70.36.240),/dir3966(10.70.36.240),/dir3967(10.70.36.240),/dir3968(10.70.36.240),/dir3969(10.70.36.240),/dir3970(10.70.36.240),/dir3971(10.70.36.240),/dir3972(10.70.36.240),/dir3973(10.70.36.240),/dir3974(10.70.36.240),/dir3975(10.70.36.240),/dir3976(10.70.36.240),/dir3977(10.70.36.240),/dir3978(10.70.36.240),/dir3979(10.70.36.240),/dir3980(10.70.36.240),/dir3981(10.70.36.240),/dir3982(10.70.36.240),/dir3983(10.70.36.240),/dir3984(10.70.36.240),/dir3985(10.70.36.240),/dir3986(10.70.36.240),/dir3987(10.70.36.240),/dir3988(10.70.36.240),/dir3989(10.70.36.240),/dir3990(10.70.36.240),/dir3991(10.70.36.240),/dir3992(10.70.36.240),/dir3993(10.70.36.240),/dir3994(10.70.36.240),/dir3995(10.70.36.240),/dir3996(10.70.36.240),/dir3997(10.70.36.240),/dir3998(10.70.36.240),/dir3999(10.70.36.240),/dir4000(10.70.36.240)"
volume set: success

real	1m1.550s
user	0m54.144s
sys	0m0.169s

Comment 3 Anil Shah 2013-12-17 06:59:49 UTC
however,
gluster vol set nfs.export-dir command for 8k and 16K no. of directories is falling with below error.

2.197|10.70.42.101|10.70.42.200),/dir15989(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15990(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15991(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15992(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15993(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15994(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15995(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15996(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15997(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15998(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir15999(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200),/dir16000(10.70.42.200|10.70.42.197|10.70.42.101|10.70.42.200)"
./export-dir.sh: line 68: /usr/sbin/gluster: Argument list too long

Comment 4 santosh pradhan 2013-12-17 17:12:39 UTC

./export-dir.sh: line 68: /usr/sbin/gluster: Argument list too long

The error is thrown by the SHELL itself, not by the gluster. Rajesh will share the data, he collected from the SHELL. (bash)

Comment 5 rjoseph 2013-12-18 05:15:17 UTC
I tested nfs.export-dir command in my Fedora 19 machine and the command fails with "Argument list too long" as observed by Anil and Santosh. Then I checked the current limit for the maximum allowed commandline argument length (using xargs --show-limits) and I got the following result

POSIX upper limit on argument length (this system): 2091957
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2088810
Size of command buffer we are actually using: 131072


On RHEL 6.5 I got the following result.
Your environment variables take up 2009 bytes
POSIX upper limit on argument length (this system): 2617383
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2615374
Size of command buffer we are actually using: 131072

So it seems like as of now the limiting factor is command buffer size which is 131072.

Comment 6 santosh pradhan 2013-12-19 11:51:29 UTC
Reviewed on:
https://code.engineering.redhat.com/gerrit/13847

Comment 7 Vivek Agarwal 2013-12-24 08:29:16 UTC
per discussion 12/23, this will go as a known issue as it is related to shell limitation. Removing it from the must fix list for corbett

Comment 8 Shalaka 2014-01-03 06:48:59 UTC
As this bug is identified as Known Issue, please add the doctext.

Comment 9 santosh pradhan 2014-02-04 06:16:54 UTC
May be Rajesh can rectify the text if he wants.

Comment 10 santosh pradhan 2014-02-05 05:41:40 UTC
The modified text looks good to me.

Comment 12 Vivek Agarwal 2015-02-04 10:23:48 UTC
Based on discussion with ndevos (dev) and asriram (doc), closing this and raising a doc bug for this to be known issue.


Note You need to log in before you can comment on or make changes to this bug.