Bug 1248998
| Summary: | [AFR]: Files not available in the mount point after converting Distributed volume type to Replicated one. | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Byreddy <bsrirama> | |
| Component: | replicate | Assignee: | Mohit Agrawal <moagrawa> | |
| Status: | CLOSED ERRATA | QA Contact: | Nag Pavan Chilakam <nchilaka> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | rhgs-3.1 | CC: | amukherj, asrivast, moagrawa, nchilaka, pkarampu, ravishankar, rcyriac, rhinduja, rhs-bugs, sabose | |
| Target Milestone: | --- | Keywords: | ZStream | |
| Target Release: | RHGS 3.2.0 | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.8.4-1 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1365455 1366440 1366444 (view as bug list) | Environment: | ||
| Last Closed: | 2017-03-23 05:22:39 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1276203, 1320020 | |||
| Bug Blocks: | 1240658, 1351522, 1365455, 1366440, 1366444 | |||
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions Patch links for downstream : https://code.engineering.redhat.com/gerrit/#/c/71426/ https://code.engineering.redhat.com/gerrit/#/c/71424/ Following is the planned QA Test Plan(QATP) to verify this bug(TC#3 is the main case to test the fix):
TC#1:automatic heal should be triggered and all files must be availble on both bricks( data,metadata and entry heals must pass )
1:create a single brick volume
2:now start volume, and add some files and directories and note them
3:now add-brick such that this brick makes the volume a replica vol 1x2 by using below command gluster v add-brick <vname> rep 2 <newbirck>
4:Now from the mount point check if you are able to see all the files and dirs created in step 2 and check that they are accessible too
5. Now check the heal info command to see that all heals are complete
6. Now check the backend brick to make sure all files are replicated
7. Now create new files and dirs and make sure they are replicated to both bricks
8. Make sure data,metadata and entry heals pass
TC#2:bringing down brick of the original brick must not cause any IO issue(As it is now a AFR volume)-->test on both fuse and nfs --->Testcase failed on fuse client
step1,2,3 same as above
4:Now check if the heal is complete using heal info
5. After heal completes, now bring down the first brick(which was used to create the actual old volume)
6.without doing any lookup on the fuse mount, try to create new file ; make sure the file gets created
Failure details: Case failed at the first file create at step 6 with transport end point error
TC#3:the old brick or legacy brick must be the source and not the new brick(tests the actual fix)
step 1 to 4 same as TC#1
5. Now make sure that all the files and directories are created on the new brick
6. The mount point should be able to display all the files and their contents(not just file names, but even the contents)
TC#4:afr client bits must be visible on the old brick or legacy brick (with client bits set for the client1, ie the newly added brick)
step 1 to 4 same as TC#1
5. Now even before self heal is triggered, check the xattrs on any of the file and dir on the old brick. it must have following information of its other pair brick(the newly added brick) trusted.afr.jan-client-1=0x000000010000000100000000
TC#5:lookup(named self heal) should trigger heal of the file
step 1 to 6 same as TC#1
5. Now do a cat of a file created previously and see if this heals the file by checking the size of file in backend brick on both bricks. Also check md5sum which should be same
TC#6:manual heal command should trigger heal of the file
step 1 to 6 same as TC#17.
7. trigger a heal on the volume, this should heal --->fails because this cmnd "heal operation to perform index self heal on volume" but the index itself is not getting created
TC#7:manual heal full command should trigger heal of the file
step 1 to 6 same as TC#17.
7. trigger a heal full on the volume, this should heal
Run all the above cases on below configurations:
Fuse mount
NFS mount
x3
dist-rep volume
Validation on 3.7.9-2 build [root@dhcp35-191 feb]# rpm -qa|grep gluster glusterfs-client-xlators-3.7.9-2.el7rhgs.x86_64 glusterfs-server-3.7.9-2.el7rhgs.x86_64 python-gluster-3.7.5-19.el7rhgs.noarch gluster-nagios-addons-0.2.5-1.el7rhgs.x86_64 vdsm-gluster-4.16.30-1.3.el7rhgs.noarch glusterfs-3.7.9-2.el7rhgs.x86_64 glusterfs-api-3.7.9-2.el7rhgs.x86_64 glusterfs-cli-3.7.9-2.el7rhgs.x86_64 glusterfs-geo-replication-3.7.9-2.el7rhgs.x86_64 gluster-nagios-common-0.2.3-1.el7rhgs.noarch glusterfs-libs-3.7.9-2.el7rhgs.x86_64 glusterfs-fuse-3.7.9-2.el7rhgs.x86_64 glusterfs-rdma-3.7.9-2.el7rhgs.x86_64 from the QATP I found the following issues or pass/fail status On FUSE Mount: TC#1:FAIL ->raised bug 1330526 - adding brick to a single brick volume to convert to replica is not triggering self heal Tc#2:FAIL Testcase failed on fuse client --->bug raised:1330399 - "Transport endpoint is not connected" error on fuse mount when we bring down the legacy brick of a volume after converting it to replicate tc#3:PASS TC#4:PASS TC#5:PASS TC#6:FAIL due to bug raised in TC#1 TC#7:PASS On NFS Mount: Hit the following issue: saw an issue where the first ls shows the file size as zero for all files, however second ls does read correctly raised bug 1330555 - file sizes showing as zero on issuing ls on an nfs mount after adding brick to convert a single brick to replica volume(1x2) Since, the TC#1 failed with self heal not triggering and after discussing with dev moving to failed_qa The patches sent for this BZ were supposed to address 2 things : 1) Allow increasing the replica count from 2 to 3 such that heal to the newly added brick is done automatically. 2) Allow converting a distribute volume to replicate one. The patches address point 1 but not point 2. Brief explanation of the fix and why it fails for point 2 : a) When a new brick is added to a volume such that replica count is increased, the existing bricks accuse the new bricks indicating some heal needs to be performed. b) This accusing is 2 step process : Marking pending xattrs on '/' of the old bricks and subsequently adding an index link file in <brickpath>/.glusterfs/indices/xattrop directory. This index file is used by self-heal daemon to pick up the files to be healed. c) The mentioned index file is added by "index" xlator. This xlator keeps track of a watchlist which contains xattr key patterns, when ever a non-zero value is provided against any of the xattr key patterns in watchlist, this xlator add the index file. d) This watchlist is populated during init of this xlator based on the xlator option provided to it in a *replicate* volume. So the fix works fine for a replicate volume whose replica count is increased. But in case of a distribute volume being converted to replicate, index xlator doesn't have this watchlist. e) So upon adding a new brick to the volume, second part mentioned in point (b) fails. Resulting in 0 files that need healing (as seen from indices/xattrop) even though pending markers are set. f) Hence, the healing fails. Proposed solution : When new bricks are added to a volume to increase replica count, reconfigure the existing bricks such that index xlator residing on these bricks is made aware of the watchlist and index files can be added. Planning to make this fix for upcoming releases. Dropping it from 3.1.3. removed fixed-in version due to failed_Qa Patch is merged in upstream http://review.gluster.org/#/c/15118/ Regards Mohit Agrawal Have retest above QATP and found that the main case#1 due to which this bug was failed, is working, ie converting a single brick volume to a 1x2 was not triggering self heal Now it work hence mvoing to verified [root@dhcp35-192 ~]# rpm -qa|grep gluster glusterfs-client-xlators-3.8.4-3.el7rhgs.x86_64 glusterfs-server-3.8.4-3.el7rhgs.x86_64 glusterfs-3.8.4-3.el7rhgs.x86_64 glusterfs-fuse-3.8.4-3.el7rhgs.x86_64 glusterfs-cli-3.8.4-3.el7rhgs.x86_64 glusterfs-rdma-3.8.4-3.el7rhgs.x86_64 glusterfs-libs-3.8.4-3.el7rhgs.x86_64 glusterfs-api-3.8.4-3.el7rhgs.x86_64 glusterfs-debuginfo-3.8.4-3.el7rhgs.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html |
Issue: ====== Files created in the mount point when the volume type is Distributed are not available after converting the volume type to Replicated. RHGS Version: ============= Red Hat Gluster Storage Server 3.1 ( glusterfs-3.7.1-11.el6rhs ) Steps to Reproduce: =================== 1. Create a distributed volume type with one brick. 2. Mount the volume and create some files (Eg: touch file{1.30}) 3. Convert distributed volume type to Replicated one by adding one more brick ( 1x2 Config ) 4. Check the files present in the mount point created in step-2. //Files created in step-2 are not available. Actual Result: ============== Files which are created in the mount point when the volume type is Distributed are not available in the mount point after converting volume type to Replicated one. Expected Result: =============== Files created in one volume type should be present after converting volume to different possible type (Eg Distributed to Replicated , Distributed to Distributed-Replicated )