Bug 1125824

Summary: rebalance is not resulting in the hash layout changes being available to nfs client
Product: [Community] GlusterFS Reporter: Vivek Agarwal <vagarwal>
Component: distributeAssignee: bugs <bugs>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: urgent    
Version: mainlineCC: aavati, bugs, gluster-bugs, nbalacha, nsathyan, pcuzner, poelstra, rcyriac, rgowdapp, sankarshan, shmohan, spalai, ssaha, ssamanta, vagarwal, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.7.0 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1120456
: 1138393 1139997 1140338 (view as bug list) Environment:
Last Closed: 2015-05-14 17:26:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1120456    
Bug Blocks: 1117822, 1125958, 1138393, 1139997, 1140338    

Comment 1 Anand Avati 2014-08-12 15:51:19 UTC
REVIEW: http://review.gluster.org/8462 (cluster/dht: Fix dht_access treating directory like files) posted (#2) for review on master by Shyamsundar Ranganathan (srangana@redhat.com)

Comment 2 Anand Avati 2014-08-12 17:51:14 UTC
REVIEW: http://review.gluster.org/8462 (cluster/dht: Fix dht_access treating directory like files) posted (#3) for review on master by Shyamsundar Ranganathan (srangana@redhat.com)

Comment 3 Susant Kumar Palai 2014-08-13 07:44:07 UTC
Analysis on the regression failure of the patch:

After adding another 2 subvolumes found the following messages:
[2014-08-13 03:13:40.757715] W [MSGID: 108008] [afr-read-txn.c:218:afr_read_txn] 0-patchy-replicate-2: Unreadable subvolume -1 found with event generation 2. (Possible split-brain)
[2014-08-13 03:13:40.758077] W [MSGID: 108008] [afr-read-txn.c:218:afr_read_txn] 0-patchy-replicate-3: Unreadable subvolume -1 found with event generation 2. (Possible split-brain)

++++ 
@messageid 108008
 * @diagnosis There is an inconsistency in the file's data/metadata/gfid
 * amongst the bricks of a replica set.
 * @recommendedaction Resolve the split brain by clearing the AFR changelog
 * attributes from the appropriate brick and trigger self-heal.
 */
#define AFR_MSG_SPLIT_BRAIN             (GLFS_COMP_BASE_AFR + 8)
+++++

And I could see repetitive logs related to ESTALE for opendir on client 4,5,6,7 which were added later to the volume. Possible that lookup was not received by the inodes and hence dht_selfheal has not created the parents and hence ESTALE was seen.

[2014-08-13 03:13:41.235328] W [client-rpc-fops.c:2677:client3_3_opendir_cbk] 0-patchy-client-4: remote operation failed: Stale file handle. Path: <gfid:d4497edf-eed6-4b1c-ac6f-039d1c92ccd6> (d4497edf-eed6-4b1c-ac6f-039d1c92ccd6)
[2014-08-13 03:13:41.235387] W [client-rpc-fops.c:2677:client3_3_opendir_cbk] 0-patchy-client-5: remote operation failed: Stale file handle. Path: <gfid:d4497edf-eed6-4b1c-ac6f-039d1c92ccd6> (d4497edf-eed6-4b1c-ac6f-039d1c92ccd6)
[2014-08-13 03:13:41.235435] W [client-rpc-fops.c:2677:client3_3_opendir_cbk] 0-patchy-client-6: remote operation failed: Stale file handle. Path: <gfid:d4497edf-eed6-4b1c-ac6f-039d1c92ccd6> (d4497edf-eed6-4b1c-ac6f-039d1c92ccd6)
[2014-08-13 03:13:41.235671] W [client-rpc-fops.c:2677:client3_3_opendir_cbk] 0-patchy-client-7: remote operation failed: Stale file handle. Path: <gfid:d4497edf-eed6-4b1c-ac6f-039d1c92ccd6> (d4497edf-eed6-4b1c-ac6f-039d1c92ccd6)


LOGS FOR ENOEMPTY
========================
[2014-08-13 03:13:54.131642] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: fd80ffa4: /dir0001 => -1 (Directory not empty)
[2014-08-13 03:13:54.377206] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 5a81ffa4: /dir0002 => -1 (Directory not empty)
[2014-08-13 03:13:54.630304] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: b781ffa4: /dir0003 => -1 (Directory not empty)
[2014-08-13 03:13:54.875241] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 1382ffa4: /dir0004 => -1 (Directory not empty)
[2014-08-13 03:13:55.130447] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 7882ffa4: /dir0005 => -1 (Directory not empty)
[2014-08-13 03:13:55.387722] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: d182ffa4: /dir0006 => -1 (Directory not empty)
[2014-08-13 03:13:55.641347] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 3183ffa4: /dir0007 => -1 (Directory not empty)
[2014-08-13 03:13:55.900819] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 9783ffa4: /dir0008 => -1 (Directory not empty)
[2014-08-13 03:13:56.157796] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: fa83ffa4: /dir0009 => -1 (Directory not empty)
[2014-08-13 03:13:56.397986] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 5984ffa4: /dir0010 => -1 (Directory not empty)
[2014-08-13 03:13:56.608139] W [socket.c:530:__socket_rwv] 0-patchy-client-7: readv on 23.253.200.127:49173 failed (No data available)
[2014-08-13 03:13:56.608218] I [client.c:2215:client_rpc_notify] 0-patchy-client-7: disconnected from patchy-client-7. C

I could not hit the issue on my vm. Hence, will rerun the regression test to get stat of the files left on the bricks(if finds) and add another rm -rf to the end to check whether the subsequent rm -rf works or not.

Comment 4 Susant Kumar Palai 2014-08-13 08:50:18 UTC
Got a machine on which the issue is reproduced consistently. Will debug and update. Hence, not sending any new test case.

Comment 5 Anand Avati 2014-08-13 14:32:19 UTC
REVIEW: http://review.gluster.org/8462 (cluster/dht: Fix dht_access treating directory like files) posted (#4) for review on master by Shyamsundar Ranganathan (srangana@redhat.com)

Comment 6 Anand Avati 2014-08-18 13:31:27 UTC
REVIEW: http://review.gluster.org/8462 (cluster/dht: Fix dht_access treating directory like files) posted (#5) for review on master by Shyamsundar Ranganathan (srangana@redhat.com)

Comment 7 Anand Avati 2014-08-18 17:28:22 UTC
COMMIT: http://review.gluster.org/8462 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 6630fff4812f4e8617336b98d8e3ac35976e5990
Author: Shyam <srangana@redhat.com>
Date:   Tue Aug 12 10:48:27 2014 -0400

    cluster/dht: Fix dht_access treating directory like files
    
    When the cluster topology changes due to add-brick, all sub
    volumes of DHT will not contain the directories till a rebalance
    is completed. Till the rebalance is run, if a caller bypasses
    lookup and calls access due to saved/cached inode information
    (like NFS server does) then, dht_access misreads the error
    (ESTALE/ENOENT) from the new subvolumes and incorrectly tries
    to handle the inode as a file. This results in the directories
    in memory state in DHT to be corrupted and not heal even post
    a rebalance.
    
    This commit fixes the problem in dht_access thereby preventing
    DHT from misrepresenting a directory as a file in the case
    presented above.
    
    Change-Id: Idcdaa3837db71c8fe0a40ec0084a6c3dbe27e772
    BUG: 1125824
    Signed-off-by: Shyam <srangana@redhat.com>
    Reviewed-on: http://review.gluster.org/8462
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Comment 8 Anand Avati 2014-09-23 09:31:30 UTC
REVIEW: http://review.gluster.org/8721 (cluster/dht: Fix dht_access treating directory like files) posted (#2) for review on release-3.5 by N Balachandran (nbalacha@redhat.com)

Comment 9 Niels de Vos 2015-05-14 17:26:57 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 10 Niels de Vos 2015-05-14 17:35:31 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 11 Niels de Vos 2015-05-14 17:37:53 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 12 Niels de Vos 2015-05-14 17:43:04 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user