Bug 1248998 - [AFR]: Files not available in the mount point after converting Distributed volume type to Replicated one.
Summary: [AFR]: Files not available in the mount point after converting Distributed vo...
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: replicate
Version: rhgs-3.1
Hardware: x86_64
OS: Linux
Target Milestone: ---
: RHGS 3.2.0
Assignee: Mohit Agrawal
QA Contact: nchilaka
Depends On: 1276203 1320020
Blocks: 1240658 1351522 1365455 1366440 1366444
TreeView+ depends on / blocked
Reported: 2015-07-31 10:00 UTC by Byreddy
Modified: 2017-03-23 05:22 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.8.4-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1365455 1366440 1366444 (view as bug list)
Last Closed: 2017-03-23 05:22:39 UTC

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description Byreddy 2015-07-31 10:00:28 UTC
Files created in the mount point  when the volume type is Distributed are not available after converting the volume type to Replicated.

RHGS Version:
Red Hat Gluster Storage Server 3.1 ( glusterfs-3.7.1-11.el6rhs )

Steps to Reproduce:
1. Create a distributed volume type with one brick.
2. Mount the volume and create some files (Eg: touch file{1.30})
3. Convert distributed volume type to Replicated one by adding one more brick ( 1x2 Config )
4. Check the files present in the mount point created in step-2. //Files created in step-2 are not available.

Actual Result:
Files which are created in the mount point when the volume type is Distributed are not available in the mount point after converting volume type to Replicated one.

Expected Result:
Files created in one volume type should be present after converting volume to different possible type (Eg Distributed to Replicated , Distributed to  Distributed-Replicated )

Comment 6 Mike McCune 2016-03-28 22:17:27 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions

Comment 9 nchilaka 2016-04-26 12:03:32 UTC
Following is the planned QA Test Plan(QATP) to verify this bug(TC#3 is the main case to test the fix):
TC#1:automatic heal should be triggered and all files must be availble on both bricks( data,metadata  and entry heals must pass )
    1:create a single brick volume
    2:now start volume, and add some files and directories and note them
    3:now add-brick such that this brick makes the volume a replica vol 1x2 by using below command  gluster v add-brick <vname> rep  2   <newbirck>
    4:Now from the mount point check if you are able to see all the files and dirs created in  step 2 and check that they are accessible too
    5. Now  check the heal info command to see that all heals are complete
    6. Now check the backend brick to make sure all files are replicated
    7. Now create new files and dirs and make sure they are replicated to both bricks
    8. Make sure data,metadata and entry heals pass

TC#2:bringing down brick of the original brick must not cause any IO issue(As it is now a AFR volume)-->test on both fuse and nfs --->Testcase failed on fuse client 
    step1,2,3 same as above
    4:Now check if the heal is complete using heal info
    5. After heal completes, now bring down the first brick(which was used to create the actual old volume)
    6.without doing any lookup on the fuse mount, try to create new file ; make sure the file gets created
    Failure details: Case failed at the first file create at step 6 with transport end point error

TC#3:the old brick or legacy brick must be the source and not the new brick(tests the actual fix)
    step 1 to 4 same as TC#1
    5. Now make sure that all the files and directories are created on the new brick
    6. The mount point should be able to display all the files and their contents(not just file names, but even the contents)

TC#4:afr client bits must be visible on the old brick or legacy brick (with client bits set for the client1, ie the newly added brick)
    step 1 to 4 same as TC#1
    5. Now even before self heal is triggered, check the xattrs on any of the file and dir on the old brick. it must have following information of its other pair brick(the newly added brick) trusted.afr.jan-client-1=0x000000010000000100000000

TC#5:lookup(named self heal) should trigger heal of the file
    step 1 to 6 same as TC#1
    5. Now do a cat of a file created previously and see if this heals the file by checking the size of file in backend brick on both bricks. Also check md5sum which should be same
    TC#6:manual heal command should trigger heal of the file
    step 1 to 6 same as TC#17. 
    7. trigger a heal on the volume, this should heal --->fails because this cmnd "heal operation to perform index self heal on volume" but the index itself is not getting created

TC#7:manual heal full command  should trigger heal of the file
    step 1 to 6 same as TC#17. 
    7.  trigger a heal full on the volume, this should heal 

Run all the above cases on below configurations:
    Fuse mount
    NFS mount
    dist-rep volume

Comment 10 nchilaka 2016-04-26 12:55:26 UTC
Validation on 3.7.9-2 build
[root@dhcp35-191 feb]# rpm -qa|grep gluster

from the QATP I found the following issues or pass/fail status
On FUSE Mount:
TC#1:FAIL ->raised bug 1330526 - adding brick to a single brick volume to convert to replica is not triggering self heal
Tc#2:FAIL Testcase failed on fuse client --->bug raised:1330399 -  "Transport endpoint is not  connected" error on fuse mount when we bring down the legacy brick of a  volume after converting it to replicate      
TC#6:FAIL due to bug raised in TC#1

On NFS Mount: Hit the following issue:
saw an issue where the first ls shows the file size as zero for all files, however second ls does read correctly raised bug 1330555 -        file sizes showing as zero on issuing ls on an nfs mount after adding brick to convert a single brick to replica volume(1x2)

Since, the TC#1 failed with self heal not triggering and after discussing with dev moving to failed_qa

Comment 11 Anuradha 2016-04-28 06:26:50 UTC
The patches sent for this BZ were supposed to address 2 things :

1) Allow increasing the replica count from 2 to 3 such that heal to the newly added brick is done automatically.

2) Allow converting a distribute volume to replicate one.

The patches address point 1 but not point 2.

Brief explanation of the fix and why it fails for point 2 :
a) When a new brick is added to a volume such that replica count is increased, the existing bricks accuse the new bricks indicating some heal needs to be performed.

b) This accusing is 2 step process : Marking pending xattrs on '/' of the old bricks and subsequently adding an index link file in <brickpath>/.glusterfs/indices/xattrop directory. This index file is used by self-heal daemon to pick up the files to be healed.

c) The mentioned index file is added by "index" xlator. This xlator keeps track of a watchlist which contains xattr key patterns, when ever a non-zero value is provided against any of the xattr key patterns in watchlist, this xlator add the index file.

d) This watchlist is populated during init of this xlator based on the xlator option provided to it in a *replicate* volume. So the fix works fine for a replicate volume whose replica count is increased. But in case of a distribute volume being converted to replicate, index xlator doesn't have this watchlist.

e) So upon adding a new brick to the volume, second part mentioned in point (b) fails. Resulting in 0 files that need healing (as seen from indices/xattrop) even though pending markers are set.

f) Hence, the healing fails.

Proposed solution :
When new bricks are added to a volume to increase replica count, reconfigure the existing bricks such that index xlator residing on these bricks is made aware of the watchlist and index files can be added.

Planning to make this fix for upcoming releases. Dropping it from 3.1.3.

Comment 13 nchilaka 2016-05-02 10:42:24 UTC
removed fixed-in version due to failed_Qa

Comment 15 Mohit Agrawal 2016-08-12 02:01:25 UTC
Patch is merged in upstream http://review.gluster.org/#/c/15118/

Mohit Agrawal

Comment 21 nchilaka 2016-10-26 10:39:44 UTC
Have retest above QATP and found that the main case#1 due to which this bug was failed, is working, ie converting a single brick volume to a 1x2 was not triggering self heal
Now it work hence mvoing to verified

[root@dhcp35-192 ~]# rpm -qa|grep gluster

Comment 23 errata-xmlrpc 2017-03-23 05:22:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.