Bug 1125824 - rebalance is not resulting in the hash layout changes being available to nfs client
Summary: rebalance is not resulting in the hash layout changes being available to nfs ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: mainline
Hardware: aarch64
OS: Linux
urgent
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On: 1120456
Blocks: glusterfs-3.6.0 1125958 1138393 1139997 1140338
TreeView+ depends on / blocked
 
Reported: 2014-08-01 07:32 UTC by Vivek Agarwal
Modified: 2016-02-18 00:03 UTC (History)
16 users (show)

Fixed In Version: glusterfs-3.7.0
Doc Type: Bug Fix
Doc Text:
Clone Of: 1120456
: 1138393 1139997 1140338 (view as bug list)
Environment:
Last Closed: 2015-05-14 17:26:57 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Comment 1 Anand Avati 2014-08-12 15:51:19 UTC
REVIEW: http://review.gluster.org/8462 (cluster/dht: Fix dht_access treating directory like files) posted (#2) for review on master by Shyamsundar Ranganathan (srangana@redhat.com)

Comment 2 Anand Avati 2014-08-12 17:51:14 UTC
REVIEW: http://review.gluster.org/8462 (cluster/dht: Fix dht_access treating directory like files) posted (#3) for review on master by Shyamsundar Ranganathan (srangana@redhat.com)

Comment 3 Susant Kumar Palai 2014-08-13 07:44:07 UTC
Analysis on the regression failure of the patch:

After adding another 2 subvolumes found the following messages:
[2014-08-13 03:13:40.757715] W [MSGID: 108008] [afr-read-txn.c:218:afr_read_txn] 0-patchy-replicate-2: Unreadable subvolume -1 found with event generation 2. (Possible split-brain)
[2014-08-13 03:13:40.758077] W [MSGID: 108008] [afr-read-txn.c:218:afr_read_txn] 0-patchy-replicate-3: Unreadable subvolume -1 found with event generation 2. (Possible split-brain)

++++ 
@messageid 108008
 * @diagnosis There is an inconsistency in the file's data/metadata/gfid
 * amongst the bricks of a replica set.
 * @recommendedaction Resolve the split brain by clearing the AFR changelog
 * attributes from the appropriate brick and trigger self-heal.
 */
#define AFR_MSG_SPLIT_BRAIN             (GLFS_COMP_BASE_AFR + 8)
+++++

And I could see repetitive logs related to ESTALE for opendir on client 4,5,6,7 which were added later to the volume. Possible that lookup was not received by the inodes and hence dht_selfheal has not created the parents and hence ESTALE was seen.

[2014-08-13 03:13:41.235328] W [client-rpc-fops.c:2677:client3_3_opendir_cbk] 0-patchy-client-4: remote operation failed: Stale file handle. Path: <gfid:d4497edf-eed6-4b1c-ac6f-039d1c92ccd6> (d4497edf-eed6-4b1c-ac6f-039d1c92ccd6)
[2014-08-13 03:13:41.235387] W [client-rpc-fops.c:2677:client3_3_opendir_cbk] 0-patchy-client-5: remote operation failed: Stale file handle. Path: <gfid:d4497edf-eed6-4b1c-ac6f-039d1c92ccd6> (d4497edf-eed6-4b1c-ac6f-039d1c92ccd6)
[2014-08-13 03:13:41.235435] W [client-rpc-fops.c:2677:client3_3_opendir_cbk] 0-patchy-client-6: remote operation failed: Stale file handle. Path: <gfid:d4497edf-eed6-4b1c-ac6f-039d1c92ccd6> (d4497edf-eed6-4b1c-ac6f-039d1c92ccd6)
[2014-08-13 03:13:41.235671] W [client-rpc-fops.c:2677:client3_3_opendir_cbk] 0-patchy-client-7: remote operation failed: Stale file handle. Path: <gfid:d4497edf-eed6-4b1c-ac6f-039d1c92ccd6> (d4497edf-eed6-4b1c-ac6f-039d1c92ccd6)


LOGS FOR ENOEMPTY
========================
[2014-08-13 03:13:54.131642] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: fd80ffa4: /dir0001 => -1 (Directory not empty)
[2014-08-13 03:13:54.377206] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 5a81ffa4: /dir0002 => -1 (Directory not empty)
[2014-08-13 03:13:54.630304] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: b781ffa4: /dir0003 => -1 (Directory not empty)
[2014-08-13 03:13:54.875241] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 1382ffa4: /dir0004 => -1 (Directory not empty)
[2014-08-13 03:13:55.130447] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 7882ffa4: /dir0005 => -1 (Directory not empty)
[2014-08-13 03:13:55.387722] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: d182ffa4: /dir0006 => -1 (Directory not empty)
[2014-08-13 03:13:55.641347] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 3183ffa4: /dir0007 => -1 (Directory not empty)
[2014-08-13 03:13:55.900819] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 9783ffa4: /dir0008 => -1 (Directory not empty)
[2014-08-13 03:13:56.157796] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: fa83ffa4: /dir0009 => -1 (Directory not empty)
[2014-08-13 03:13:56.397986] W [nfs3.c:3518:nfs3svc_rmdir_cbk] 0-nfs: 5984ffa4: /dir0010 => -1 (Directory not empty)
[2014-08-13 03:13:56.608139] W [socket.c:530:__socket_rwv] 0-patchy-client-7: readv on 23.253.200.127:49173 failed (No data available)
[2014-08-13 03:13:56.608218] I [client.c:2215:client_rpc_notify] 0-patchy-client-7: disconnected from patchy-client-7. C

I could not hit the issue on my vm. Hence, will rerun the regression test to get stat of the files left on the bricks(if finds) and add another rm -rf to the end to check whether the subsequent rm -rf works or not.

Comment 4 Susant Kumar Palai 2014-08-13 08:50:18 UTC
Got a machine on which the issue is reproduced consistently. Will debug and update. Hence, not sending any new test case.

Comment 5 Anand Avati 2014-08-13 14:32:19 UTC
REVIEW: http://review.gluster.org/8462 (cluster/dht: Fix dht_access treating directory like files) posted (#4) for review on master by Shyamsundar Ranganathan (srangana@redhat.com)

Comment 6 Anand Avati 2014-08-18 13:31:27 UTC
REVIEW: http://review.gluster.org/8462 (cluster/dht: Fix dht_access treating directory like files) posted (#5) for review on master by Shyamsundar Ranganathan (srangana@redhat.com)

Comment 7 Anand Avati 2014-08-18 17:28:22 UTC
COMMIT: http://review.gluster.org/8462 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 6630fff4812f4e8617336b98d8e3ac35976e5990
Author: Shyam <srangana@redhat.com>
Date:   Tue Aug 12 10:48:27 2014 -0400

    cluster/dht: Fix dht_access treating directory like files
    
    When the cluster topology changes due to add-brick, all sub
    volumes of DHT will not contain the directories till a rebalance
    is completed. Till the rebalance is run, if a caller bypasses
    lookup and calls access due to saved/cached inode information
    (like NFS server does) then, dht_access misreads the error
    (ESTALE/ENOENT) from the new subvolumes and incorrectly tries
    to handle the inode as a file. This results in the directories
    in memory state in DHT to be corrupted and not heal even post
    a rebalance.
    
    This commit fixes the problem in dht_access thereby preventing
    DHT from misrepresenting a directory as a file in the case
    presented above.
    
    Change-Id: Idcdaa3837db71c8fe0a40ec0084a6c3dbe27e772
    BUG: 1125824
    Signed-off-by: Shyam <srangana@redhat.com>
    Reviewed-on: http://review.gluster.org/8462
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Comment 8 Anand Avati 2014-09-23 09:31:30 UTC
REVIEW: http://review.gluster.org/8721 (cluster/dht: Fix dht_access treating directory like files) posted (#2) for review on release-3.5 by N Balachandran (nbalacha@redhat.com)

Comment 9 Niels de Vos 2015-05-14 17:26:57 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 10 Niels de Vos 2015-05-14 17:35:31 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 11 Niels de Vos 2015-05-14 17:37:53 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 12 Niels de Vos 2015-05-14 17:43:04 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.