Bug 1294051

Summary:	Though files are in split-brain able to perform writes to the file
Product:	[Community] GlusterFS	Reporter:	Pranith Kumar K <pkarampu>
Component:	replicate	Assignee:	Pranith Kumar K <pkarampu>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, ksubrahm, rmekala, storage-qa-internal
Target Milestone:	---	Keywords:	Reopened, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.13.0	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1277924
Clones:	1315140 1544635 (view as bug list)		Environment:
Last Closed:	2017-12-08 17:31:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1277924
Bug Blocks:	1315140, 1544635

Comment 1 Pranith Kumar K 2015-12-24 09:39:35 UTC

Description of problem:
==================
Though files are in split-brain able to perform writes to the file

How reproducible:
Always

Steps to Reproduce:
=============
1. Create 1x2 volume and mount it on client using fuse and disable self-heal-daemon
2. From the mount create few files and do continuous IO's to file and while IO is going on perfrom pkill gluster on node1 
3. After some time bring back the node1 and perfrom pkill on node2 (while IO is going on to the same files)
4. Now files are in split-brian and gluster vol heal info is shwoing the same 
5. From the mount do append to the file using echo "data" >> file (Don't give full file path )

Actual results:


Expected results:
===============
IO should fail and if i disable performance.write-behind getting IO error while writing to the file

Comment 2 Vijay Bellur 2015-12-24 09:52:21 UTC

REVIEW: http://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 3 Vijay Bellur 2015-12-24 09:52:24 UTC

REVIEW: http://review.gluster.org/13076 (performance/open-behind: Fix fdctx dump NULL dereference) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 4 Vijay Bellur 2016-01-06 04:55:38 UTC

REVIEW: http://review.gluster.org/13076 (performance/open-behind: Fix fdctx dump NULL dereference) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 5 Vijay Bellur 2016-01-06 05:03:16 UTC

REVIEW: http://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 6 Vijay Bellur 2016-01-06 09:37:33 UTC

REVIEW: http://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 7 Vijay Bellur 2016-01-06 09:42:48 UTC

REVIEW: http://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 8 Vijay Bellur 2016-01-13 12:00:11 UTC

REVIEW: http://review.gluster.org/13076 (performance/open-behind: Fix fdctx dump NULL dereference) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 9 Vijay Bellur 2016-01-27 09:18:29 UTC

REVIEW: http://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 10 Vijay Bellur 2016-01-27 09:53:03 UTC

REVIEW: http://review.gluster.org/13076 (performance/open-behind: Fix fdctx dump NULL dereference) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 11 Vijay Bellur 2016-01-29 06:41:52 UTC

REVIEW: http://review.gluster.org/13076 (performance/open-behind: Fix fdctx dump NULL dereference) posted (#5) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 12 Vijay Bellur 2016-02-08 14:05:41 UTC

REVIEW: http://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#6) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 13 Vijay Bellur 2016-02-09 05:25:01 UTC

COMMIT: http://review.gluster.org/13076 committed in master by Raghavendra G (rgowdapp) 
------
commit dd8306b2f2e38f5de1386461d7f33ac0b64f2886
Author: Pranith Kumar K <pkarampu>
Date:   Thu Dec 24 14:29:33 2015 +0530

    performance/open-behind: Fix fdctx dump NULL dereference
    
    Also printing flags correctly in statedump now
    
    Change-Id: Ibfdd74aab5643ecc47d0a88f109d5d1050685f5a
    BUG: 1294051
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/13076
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>

Comment 14 Mike McCune 2016-03-28 23:18:47 UTC

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 15 Niels de Vos 2016-06-16 13:52:16 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 16 Vijay Bellur 2016-06-24 11:44:43 UTC

REVIEW: http://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#7) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 17 Worker Ant 2017-09-04 12:45:58 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#8) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 18 Worker Ant 2017-09-06 12:09:46 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#9) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 19 Worker Ant 2017-09-06 12:48:59 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#10) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 20 Worker Ant 2017-09-06 13:39:44 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#11) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 21 Worker Ant 2017-10-17 11:47:12 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#12) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 22 Worker Ant 2017-10-17 12:12:19 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#13) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 23 Worker Ant 2017-10-17 12:26:04 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#14) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 24 Worker Ant 2017-10-17 12:49:46 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#15) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 25 Worker Ant 2017-10-17 16:44:04 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#16) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 26 Worker Ant 2017-10-19 08:04:40 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#17) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 27 Worker Ant 2017-10-24 13:56:56 UTC

REVIEW: https://review.gluster.org/13075 (cluster/afr: Fail open on split-brain) posted (#18) for review on master by Pranith Kumar Karampuri (pkarampu)

Comment 28 Worker Ant 2017-10-26 18:23:40 UTC

COMMIT: https://review.gluster.org/13075 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 786343abca3474ff01aa1017210112d97cbc4843
Author: Pranith Kumar K <pkarampu>
Date:   Mon Sep 4 16:57:25 2017 +0530

    cluster/afr: Fail open on split-brain
    
    Problem:
    Append on a file with split-brain succeeds. Open is intercepted by open-behind,
    when write comes on the file, open-behind does open+write. Open succeeds
    because afr doesn't fail it. Then write succeeds because write-behind
    intercepts it. Flush is also intercepted by write-behind, so the application
    never gets to know that the write failed.
    
    Fix:
    Fail open on split-brain, so that when open-behind does open+write open fails
    which leads to write failure. Application will know about this failure.
    
    Change-Id: I4bff1c747c97bb2925d6987f4ced5f1ce75dbc15
    BUG: 1294051
    Signed-off-by: Pranith Kumar K <pkarampu>

Comment 29 Shyamsundar 2017-12-08 17:31:43 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.13.0, please open a new bug report.

glusterfs-3.13.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-December/000087.html
[2] https://www.gluster.org/pipermail/gluster-users/