Bug 1760713

Summary: Remove-brick : linkto file is missing for certain files from hash subvol and hence cannot access those files from client node.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sayalee <saraut>
Component: distributeAssignee: Susant Kumar Palai <spalai>
Status: CLOSED DEFERRED QA Contact: Sayalee <saraut>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.5CC: amukherj, nchilaka, rhinduja, rhs-bugs, sheggodu, spalai, storage-qa-internal, vdas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
When the cluster.lookup-optimize option is enabled, running a rename operation after a rebalance can lead to some files being inaccessible from the client. Workaround: Disable cluster.lookup-optimize on the volume. # gluster volume set VOLNAME cluster.lookup-optimize off
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-16 07:46:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1696815    

Description Sayalee 2019-10-11 07:29:59 UTC
Description of problem:
Performed remove brick on a volume, with rename of files and remove of files running in parallel as IO, there are remove-brick failures due to ENOENTs. 
Also, the linkto-file of some files go missing from the backend bricks and so those files become inaccessible from the client node.

Version-Release number of selected component (if applicable):
6.0.15

How reproducible:
1/1

Steps to Reproduce:
1. Create a 5X3 volume. 
2. The volume is mounted at :
   mnt1 --> RHEL 7.7 mahcine, FUSE mount
   mnt2 --> RHEL 8.1 machine, GNFS mount
3. On the client node, created 5 dirs:
   dir1 : contains 500 block files
   dir2 : contains 500 device file 
   dir3 : contained an untar of linux kernel, started removing all the files and directories under dir3 using rm-rf
   dir4 - contains 500 pipe files
   dir5 - created 100 files, started renaming 100 files such that each file will be renamed 2000 times.
4. While the rename of files and rm -rf was still in progress, started remove-brick operation on the volume to   convert it into 4X3.
5. Once the remove-brick migration was completed, committed the remove-brick process.

Actual results:
* Remove-brick migration had failures due to ENOENTs.
* The number of block, device and pipe files was same, before and after remove-brick.
* No issues w.r.t. rm -rf of files under dir3.
* All files under dir5 were renamed, but some files became inaccessible. For these files, the linkto-file was missing from the hash subvol.

Expected results:
* All files should be accessible from the client node.

Additional info:
sos-reports will be shared.