+++ This bug was initially created as a clone of Bug #1305031 +++ Description of problem: AFR winds a few reads of a file in metadata split-brain. How reproducible: Always Steps to Reproduce: 1. cat a big file that is in metadata split-brain 2. It spews out a lot of data before eventually failing it with EIO. Actual results: readvs are sent to the brick Expected results: No readv should be wound to the brick. Additional info: --- Additional comment from Vijay Bellur on 2016-02-05 06:37:25 EST --- REVIEW: http://review.gluster.org/13389 (afr:Don't wind reads for files in metadata split-brain) posted (#1) for review on master by Ravishankar N (ravishankar) --- Additional comment from Vijay Bellur on 2016-03-01 08:03:45 EST --- REVIEW: http://review.gluster.org/13389 (afr:Don't wind reads for files in metadata split-brain) posted (#2) for review on master by Ravishankar N (ravishankar) --- Additional comment from Vijay Bellur on 2016-03-01 08:07:49 EST --- REVIEW: http://review.gluster.org/13389 (afr:Don't wind reads for files in metadata split-brain) posted (#3) for review on master by Ravishankar N (ravishankar) --- Additional comment from Vijay Bellur on 2016-03-01 23:07:13 EST --- REVIEW: http://review.gluster.org/13389 (afr:Don't wind reads for files in metadata split-brain) posted (#4) for review on master by Ravishankar N (ravishankar) --- Additional comment from Mike McCune on 2016-03-28 19:24:30 EDT --- This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions --- Additional comment from Vijay Bellur on 2016-06-23 05:20:56 EDT --- REVIEW: http://review.gluster.org/13389 (afr:Don't wind reads for files in metadata split-brain) posted (#5) for review on master by Ravishankar N (ravishankar) --- Additional comment from Vijay Bellur on 2016-06-24 07:26:05 EDT --- COMMIT: http://review.gluster.org/13389 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 7a1c1e2904701496968ed14b6d7479fb706c3188 Author: Ravishankar N <ravishankar> Date: Fri Feb 5 15:10:06 2016 +0530 afr:Don't wind reads for files in metadata split-brain Problem: For a read on a file in metadata split-brain: 1.lookup_done resets event_generation to zero. 2. readv is issued, goes to inode refresh due to mismatching event_gen. 3. After refresh is successful, we update event_generation, data and metdata readable. 3. We then call afr_read_txn_refresh_done() which in turn calls afr_inode_get_readable() but doesn't check for EIO. So afr_readv_wind is called with local->readable (which is populated with data_readable), thus winding the read to a brick. 4. Also, further parallel reads that come directly go to the wind path because there is no inode_refresh needed. Fix: 1.For any afr_read_txn(), readable must be an intersection of data and metadata readable. 2.Check for EIO in afr_read_txn_refresh_done(). Change-Id: I22dd221fdfaf96d7aced2f474e28ed1337d69f0e BUG: 1305031 Signed-off-by: Ravishankar N <ravishankar> Reviewed-on: http://review.gluster.org/13389 Reviewed-by: Ashish Pandey <aspandey> Smoke: Gluster Build System <jenkins.org> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu>
REVIEW: http://review.gluster.org/14791 (afr:Don't wind reads for files in metadata split-brain) posted (#1) for review on release-3.7 by Ravishankar N (ravishankar)
COMMIT: http://review.gluster.org/14791 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu) ------ commit e4ea25e9eea0f7259c11333f7a75049f3dccb7a7 Author: Ravishankar N <ravishankar> Date: Fri Feb 5 15:10:06 2016 +0530 afr:Don't wind reads for files in metadata split-brain Backport of http://review.gluster.org/#/c/13389/ Problem: For a read on a file in metadata split-brain: 1.lookup_done resets event_generation to zero. 2. readv is issued, goes to inode refresh due to mismatching event_gen. 3. After refresh is successful, we update event_generation, data and metdata readable. 3. We then call afr_read_txn_refresh_done() which in turn calls afr_inode_get_readable() but doesn't check for EIO. So afr_readv_wind is called with local->readable (which is populated with data_readable), thus winding the read to a brick. 4. Also, further parallel reads that come directly go to the wind path because there is no inode_refresh needed. Fix: 1.For any afr_read_txn(), readable must be an intersection of data and metadata readable. 2.Check for EIO in afr_read_txn_refresh_done(). Change-Id: I22dd221fdfaf96d7aced2f474e28ed1337d69f0e BUG: 1349881 Signed-off-by: Ravishankar N <ravishankar> (cherry picked from commit 7a1c1e2904701496968ed14b6d7479fb706c3188) Reviewed-on: http://review.gluster.org/14791 NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Tested-by: Gluster Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.13, please open a new bug report. glusterfs-3.7.13 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] https://www.gluster.org/pipermail/gluster-users/2016-July/027604.html [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user