Bug 1128721

Summary:	Tracker bug for afrv1 changelog support in afrv2
Product:	[Community] GlusterFS	Reporter:	Pranith Kumar K <pkarampu>
Component:	replicate	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	gluster-bugs, vbellur
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.7.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1136829 (view as bug list)		Environment:
Last Closed:	2015-05-14 17:27:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1117822, 1136829, 1155017

Description Pranith Kumar K 2014-08-11 12:15:52 UTC

Description of problem:
This bug tracks the changes need to be done for afrv1 changelog support in afrv2

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Anand Avati 2014-08-11 12:19:08 UTC

REVIEW: http://review.gluster.org/8456 (cluster/afr: s/healed_sinks/sinks in data self-heal) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 2 Anand Avati 2014-08-12 04:02:47 UTC

REVIEW: http://review.gluster.org/8456 (cluster/afr: Fix all locked_on bricks are sinks check is self-heals) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 3 Anand Avati 2014-08-12 04:03:33 UTC

REVIEW: http://review.gluster.org/8456 (cluster/afr: Fix all locked_on bricks are sinks check in self-heals) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 4 Anand Avati 2014-08-12 06:41:55 UTC

REVIEW: http://review.gluster.org/8456 (cluster/afr: Fix all locked_on bricks are sinks check in self-heals) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 5 Anand Avati 2014-08-25 13:11:47 UTC

REVIEW: http://review.gluster.org/8456 (cluster/afr: Fix all locked_on bricks are sinks check in self-heals) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 6 Anand Avati 2014-08-25 13:12:05 UTC

REVIEW: http://review.gluster.org/8536 (cluster/afr: Add afr-v1 xattr compatibility) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 7 Anand Avati 2014-08-25 13:12:08 UTC

REVIEW: http://review.gluster.org/8537 (cluster/afr: Fix synchronization issue with entry, name self-heals) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 8 Anand Avati 2014-08-25 13:12:11 UTC

REVIEW: http://review.gluster.org/8538 (tests: Data self-heal test cases) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 9 Anand Avati 2014-08-25 13:12:32 UTC

REVIEW: http://review.gluster.org/8539 (tests: entry self-heal test cases) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 10 Anand Avati 2014-08-26 05:52:59 UTC

REVIEW: http://review.gluster.org/8456 (cluster/afr: Fix all locked_on bricks are sinks check in self-heals) posted (#6) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 11 Anand Avati 2014-08-26 08:17:28 UTC

COMMIT: http://review.gluster.org/8456 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit dc450ab5e3798b47e89e90b64c49cb8e1f20efce
Author: Pranith Kumar K <pkarampu>
Date:   Fri Aug 8 11:52:17 2014 +0530

    cluster/afr: Fix all locked_on bricks are sinks check in self-heals
    
    Problem:
    Counts may give wrong results when the number of bricks is > 2. If the
    locks are acquired on one source and sink, but the source accuses even the
    down sink then there will be 2 sinks and lock is acquired on 2 bricks so
    even when there is a clear source and sink **_finalize_source functions think
    the file/directory is in split-brain.
    
    Fix:
    Check that all the bricks which are locked are sinks.
    
    Change-Id: Ia43790e8e1bfb5e72a3d0b56bcad94abd0dc58ab
    BUG: 1128721
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/8456
    Reviewed-by: Ravishankar N <ravishankar>
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Tested-by: Gluster Build System <jenkins.com>

Comment 12 Anand Avati 2014-08-26 12:26:01 UTC

REVIEW: http://review.gluster.org/8537 (cluster/afr: Fix synchronization issue with entry, name self-heals) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 13 Anand Avati 2014-08-26 12:26:04 UTC

REVIEW: http://review.gluster.org/8536 (cluster/afr: Add afr-v1 xattr compatibility) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 14 Anand Avati 2014-08-26 12:26:07 UTC

REVIEW: http://review.gluster.org/8539 (tests: entry self-heal test cases) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 15 Anand Avati 2014-08-26 12:26:10 UTC

REVIEW: http://review.gluster.org/8538 (tests: Data self-heal test cases) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 16 Anand Avati 2014-09-11 12:18:50 UTC

REVIEW: http://review.gluster.org/8537 (cluster/afr: Fix synchronization issue with entry, name self-heals) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 17 Anand Avati 2014-09-11 12:19:02 UTC

REVIEW: http://review.gluster.org/8536 (cluster/afr: Add afr-v1 xattr compatibility) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 18 Anand Avati 2014-09-11 12:19:07 UTC

REVIEW: http://review.gluster.org/8539 (tests: entry self-heal test cases) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 19 Anand Avati 2014-09-11 12:19:10 UTC

REVIEW: http://review.gluster.org/8538 (tests: Data self-heal test cases) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 20 Anand Avati 2014-09-11 12:19:13 UTC

REVIEW: http://review.gluster.org/8698 (cluster/afr: Don't start heal when lookup succeeds on < 2 children) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 21 Anand Avati 2014-09-14 06:57:28 UTC

REVIEW: http://review.gluster.org/8698 (cluster/afr: Don't start heal when lookup succeeds on < 2 children) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 22 Anand Avati 2014-09-14 06:57:31 UTC

REVIEW: http://review.gluster.org/8537 (cluster/afr: Fix synchronization issue with entry, name self-heals) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 23 Anand Avati 2014-09-14 06:57:34 UTC

REVIEW: http://review.gluster.org/8536 (cluster/afr: Add afr-v1 xattr compatibility) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 24 Anand Avati 2014-09-14 06:57:37 UTC

REVIEW: http://review.gluster.org/8539 (tests: entry self-heal test cases) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 25 Anand Avati 2014-09-14 06:57:40 UTC

REVIEW: http://review.gluster.org/8538 (tests: Data self-heal test cases) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 26 Anand Avati 2014-09-14 06:57:43 UTC

REVIEW: http://review.gluster.org/8709 (cluster/afr: Fix spurious metadata self-heals) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 27 Anand Avati 2014-09-14 09:05:17 UTC

REVIEW: http://review.gluster.org/8709 (cluster/afr: Fix spurious metadata self-heals) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 28 Anand Avati 2014-09-14 09:05:21 UTC

REVIEW: http://review.gluster.org/8698 (cluster/afr: Don't start heal when lookup succeeds on < 2 children) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 29 Anand Avati 2014-09-14 09:05:24 UTC

REVIEW: http://review.gluster.org/8537 (cluster/afr: Fix synchronization issue with entry, name self-heals) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 30 Anand Avati 2014-09-14 09:05:26 UTC

REVIEW: http://review.gluster.org/8536 (cluster/afr: Add afr-v1 xattr compatibility) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 31 Anand Avati 2014-09-14 09:05:29 UTC

REVIEW: http://review.gluster.org/8539 (tests: entry self-heal test cases) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 32 Anand Avati 2014-09-14 09:05:32 UTC

REVIEW: http://review.gluster.org/8538 (tests: Data self-heal test cases) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 33 Anand Avati 2014-09-23 07:26:22 UTC

REVIEW: http://review.gluster.org/8698 (cluster/afr: Don't start heal when lookup succeeds on < 2 children) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 34 Anand Avati 2014-09-23 14:21:53 UTC

COMMIT: http://review.gluster.org/8698 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit e149a051bf226e16c6b7f1a816f998dace85d33d
Author: Pranith Kumar K <pkarampu>
Date:   Tue Sep 23 12:43:02 2014 +0530

    cluster/afr: Don't start heal when lookup succeeds on < 2 children
    
    Problem:
    When self-heal code doesn't see at least 2 successes on looking up children,
    then self-heal can't be done. What is happening now is if all the lookups fail
    then the pending changelog is all zeros in xattrs so all the children are
    becoming sources and leading to crashes when the code paths further assume that
    some data structures are populated properly
    
    Fix:
    Don't proceed with self-heals when < 2 children succeed lookups.
    
    BUG: 1128721
    Change-Id: Iffdf0feebb6f98812d9d01cdd0cf97f3e19ba76f
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/8698
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Tested-by: Gluster Build System <jenkins.com>

Comment 35 Anand Avati 2014-09-23 14:47:43 UTC

REVIEW: http://review.gluster.org/8709 (cluster/afr: Fix spurious metadata self-heals) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 36 Anand Avati 2014-09-24 06:35:46 UTC

REVIEW: http://review.gluster.org/8709 (cluster/afr: Fix spurious metadata self-heals) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 37 Anand Avati 2014-09-24 08:51:39 UTC

COMMIT: http://review.gluster.org/8709 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 70d76f20ee127fe7e8e52b2d67e2362283a01f34
Author: Pranith Kumar K <pkarampu>
Date:   Sat Sep 13 12:08:56 2014 +0530

    cluster/afr: Fix spurious metadata self-heals
    
    - Added logging for metadata and data self-heals which helped
      in debugging this issue.
    - Added checks to skip self-heals when no sinks are available to heal
    
    Change-Id: I0d50dceb84cd9ad4fe00e0b749ddf7d4ff42348a
    BUG: 1128721
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/8709
    Reviewed-by: Krutika Dhananjay <kdhananj>
    Tested-by: Gluster Build System <jenkins.com>

Comment 38 Anand Avati 2014-09-24 10:08:11 UTC

REVIEW: http://review.gluster.org/8537 (cluster/afr: Fix synchronization issue with entry, name self-heals) posted (#6) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 39 Anand Avati 2014-09-24 11:13:33 UTC

REVIEW: http://review.gluster.org/8837 (cluster/afr: Fix locking issues in entry self-heal) posted (#1) for review on master by Krutika Dhananjay (kdhananj)

Comment 40 Anand Avati 2014-09-25 06:33:40 UTC

REVIEW: http://review.gluster.org/8537 (cluster/afr: Fix synchronization issue with entry, name self-heals) posted (#7) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 41 Anand Avati 2014-09-25 06:39:56 UTC

COMMIT: http://review.gluster.org/8837 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 2471aa9aa23960e2e5e3a4db1b6e3e8388e2476e
Author: Krutika Dhananjay <kdhananj>
Date:   Tue Sep 23 17:03:53 2014 +0530

    cluster/afr: Fix locking issues in entry self-heal
    
    Original reporter of the bug & designer of the solution:
            Pranith Kumar K <pkarampu>
    
    Change-Id: I9ed89aa92e4cd0f8049f5f6c7a3701e52989ae5e
    BUG: 1128721
    Signed-off-by: Krutika Dhananjay <kdhananj>
    Reviewed-on: http://review.gluster.org/8837
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Pranith Kumar Karampuri <pkarampu>

Comment 42 Anand Avati 2014-09-30 07:43:21 UTC

REVIEW: http://review.gluster.org/8536 (cluster/afr: Add afr-v1 xattr compatibility) posted (#6) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 43 Anand Avati 2014-09-30 07:43:25 UTC

REVIEW: http://review.gluster.org/8539 (tests: entry self-heal test cases) posted (#6) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 44 Anand Avati 2014-09-30 07:43:28 UTC

REVIEW: http://review.gluster.org/8538 (tests: Data self-heal test cases) posted (#6) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 45 Anand Avati 2014-10-01 05:39:23 UTC

REVIEW: http://review.gluster.org/8536 (cluster/afr: Add afr-v1 xattr compatibility) posted (#7) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 46 Anand Avati 2014-10-01 05:39:31 UTC

REVIEW: http://review.gluster.org/8539 (tests: entry self-heal test cases) posted (#7) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 47 Anand Avati 2014-10-01 05:39:34 UTC

REVIEW: http://review.gluster.org/8538 (tests: Data self-heal test cases) posted (#7) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 48 Anand Avati 2014-10-01 10:29:36 UTC

COMMIT: http://review.gluster.org/8536 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 443e48abf9b373bb1a8c56d016aad3b974554b80
Author: Pranith Kumar K <pkarampu>
Date:   Wed Aug 20 21:50:06 2014 +0530

    cluster/afr: Add afr-v1 xattr compatibility
    
    All the special cases v1 handles and also
    self-accusing pending changelog from v1 pre-op also is handled
    in this patch.
    
    Change-Id: Ie10f71633fb20276f01ecafbd728f20483e7029c
    BUG: 1128721
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/8536
    Reviewed-by: Ravishankar N <ravishankar>
    Tested-by: Gluster Build System <jenkins.com>

Comment 49 Anand Avati 2014-10-31 07:41:31 UTC

REVIEW: http://review.gluster.org/9020 (cluster/afr: Perform post-op in entry selfheal inside locks) posted (#1) for review on master by Krutika Dhananjay (kdhananj)

Comment 50 Anand Avati 2014-10-31 09:25:28 UTC

REVIEW: http://review.gluster.org/9020 (cluster/afr: Perform post-op in entry selfheal inside locks) posted (#2) for review on master by Krutika Dhananjay (kdhananj)

Comment 51 Anand Avati 2014-10-31 12:59:39 UTC

COMMIT: http://review.gluster.org/9020 committed in master by Vijay Bellur (vbellur) 
------
commit 927b9f3f48777dcb9e968f6b1bb3d5d450a8b7bc
Author: Krutika Dhananjay <kdhananj>
Date:   Fri Oct 31 12:51:15 2014 +0530

    cluster/afr: Perform post-op in entry selfheal inside locks
    
    Take entrylks in xlator domain before doing post-op (undo-pending) in
    entry self-heal. This is to prevent a parallel name self-heal on
    an entry under @fd->inode from reading pending xattrs while it is
    being modified by SHD after entry sh below, given that
    name self-heal takes locks ONLY in xlator domain and is free to read
    pending changelog in the absence of the following locking.
    
    Change-Id: Ie083ceab10155c460447f04bdce7688480f1ac4f
    BUG: 1128721
    Signed-off-by: Krutika Dhananjay <kdhananj>
    Reviewed-on: http://review.gluster.org/9020
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Pranith Kumar Karampuri <pkarampu>
    Tested-by: Gluster Build System <jenkins.com>

Comment 52 Anand Avati 2014-11-25 15:02:42 UTC

REVIEW: http://review.gluster.org/8539 (tests: entry self-heal test cases) posted (#8) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 53 Anand Avati 2014-11-27 08:09:35 UTC

COMMIT: http://review.gluster.org/8539 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 9ff09ed29cd6659c82f6dc6f9c3f17b570b97697
Author: Pranith Kumar K <pkarampu>
Date:   Wed Oct 1 10:58:19 2014 +0530

    tests: entry self-heal test cases
    
    Change-Id: I9390bc9274eb2485af541c4044bac12e48c44064
    BUG: 1128721
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/8539
    Reviewed-by: Ravishankar N <ravishankar>
    Tested-by: Gluster Build System <jenkins.com>

Comment 54 Anand Avati 2014-12-03 05:44:23 UTC

COMMIT: http://review.gluster.org/8538 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 85fa19eef36c166287ba87597684d1e0b532add0
Author: Pranith Kumar K <pkarampu>
Date:   Thu Aug 21 08:50:23 2014 +0530

    tests: Data self-heal test cases
    
    Change-Id: I74d08797b791ea6649d9aba585996e9ec680e3f8
    BUG: 1128721
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/8538
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Ravishankar N <ravishankar>

Comment 55 Niels de Vos 2015-05-14 17:27:05 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 56 Niels de Vos 2015-05-14 17:35:32 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 57 Niels de Vos 2015-05-14 17:37:54 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 58 Niels de Vos 2015-05-14 17:43:05 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user