Bug 1266357

Summary: Distributed Disperse volume random file list error in client after adding new bricks
Product: [Community] GlusterFS Reporter: amudhan83
Component: disperseAssignee: Shyamsundar <srangana>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.7.3CC: bugs, gluster-bugs, mdfakkeer, pkarampu
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-29 08:40:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
log from client point none

Description amudhan83 2015-09-25 06:14:21 UTC
Created attachment 1076872 [details]
log from client point

Description of problem:
when new bricks added to a existing running cluster, client1 (fuse) not able to list file and log throws error but same file able to list after remount  and now again some other file is not listing. 

but able to list files in another client2 (fuse) when client1 is not bale to list file.

Version-Release number of selected component (if applicable):


How reproducible:
random

Steps to Reproduce:
1.20 node with each 36 bricks
2.create disperse vol with 8 + 2 adding node{1..10}:/brick1 node{1..10}:/brick2 likewise
3.create folder hierarchy like this.

/root1/subroot1/aaaa/{a..z}/par{1..500}/chil{1..3}
/root1/subroot1/bbbb/{a..z}/par{1..500}/chil{1..3}
/root1/subroot2/aaaa/{a..z}/par{1..500}/chil{1..3}
/root1/subroot2/bbbb/{a..z}/par{1..500}/chil{1..3}
/root2/aaaa/{a..z}/par{1..500}/chil{1..3}
/root3/aaaa/{a..z}/par{1..500}/chil{1..3}
/root3/bbbb/{a..z}/par{1..500}/chil{1..3}
/root3/bbbb/{a..z}/par{1..500}/chil{1..3}

in child folder create 10 files

in existing cluster add new bricks.

add brick node{21..30}:/brick1 node{21..30}:/brick2 likewise

but don't start rebalance or fix-layout now.

also ensure in existing brick usage in cluster are above 80% so data will start to move to new bricks.

start a transfering data to volume, now files will be written to newly added bricks.
now read files from any of the chil folders it would be missing.

Actual results:

randomly files are not listing

Expected results:

show list files without any issue

Additional info:

Comment 1 Backer 2015-09-26 06:33:23 UTC
Version-Release number of selected component (if applicable):
3.7.3

Comment 2 Backer 2015-09-29 12:46:56 UTC
This issue is resolved after changing the ulimit -n value from 1024 to 4096. If the brick count is more than 1024, we need to modify the ulimit -n value on clients as well as cluster nodes. So we can close this ticket.

Comment 3 Pranith Kumar K 2015-12-29 08:40:01 UTC
Closing as per comment-2