Bug 1484225

Summary:	[rpc]: EPOLLERR - disconnecting now messages every 3 secs after completing rebalance
Product:	[Community] GlusterFS	Reporter:	Milind Changire <mchangir>
Component:	rpc	Assignee:	Milind Changire <mchangir>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	amukherj, bugs, nbalacha, rhinduja, rhs-bugs
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.13.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1483956
Clones:	1484885 (view as bug list)		Environment:
Last Closed:	2017-12-08 17:39:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1483956, 1484885

Comment 1 Milind Changire 2017-08-23 04:48:18 UTC

Steps to Reproduce:
===================
1. Create 3x2 volume and write data to it
2. Remove brick start to make it 2x2
3. Once rebalance is completed, do commit. 
4. Monitor the glusterd log file

Actual results:
===============
EPOLLERR error message comes every 3 secs.

Comment 2 Worker Ant 2017-08-23 04:49:20 UTC

REVIEW: https://review.gluster.org/18093 (rpc: disable client on disconnection from rebalance) posted (#1) for review on master by Milind Changire (mchangir)

Comment 3 Worker Ant 2017-08-23 10:34:37 UTC

COMMIT: https://review.gluster.org/18093 committed in master by Raghavendra G (rgowdapp) 
------
commit 5b14c11d3cae38bc66006b02217ede485ae30dea
Author: Milind Changire <mchangir>
Date:   Wed Aug 23 10:10:13 2017 +0530

    rpc: disable client on disconnection from rebalance
    
    Problem:
    glusterd rpc code path attempts to reconnect to rebalance process
    via the reconnect timer even after the rebalance process disconnection
    
    Solution:
    Set the clnt->disabled flag to 1 to avoid reconnection and cause
    the clnt object to be unref'd
    
    Change-Id: I4e38eaef45d2fdea86d25e9dff9f1af0cd29cf66
    BUG: 1484225
    Signed-off-by: Milind Changire <mchangir>
    Reviewed-on: https://review.gluster.org/18093
    Smoke: Gluster Build System <jenkins.org>
    Tested-by: Raghavendra G <rgowdapp>
    Reviewed-by: Raghavendra G <rgowdapp>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 4 Worker Ant 2017-08-24 07:07:02 UTC

REVIEW: https://review.gluster.org/18113 (Revert "rpc: disable client on disconnection from rebalance") posted (#1) for review on master by Milind Changire (mchangir)

Comment 5 Worker Ant 2017-08-24 07:13:12 UTC

REVIEW: https://review.gluster.org/18114 (rpc: disable rpc_clnt_t after relalance process disconnection) posted (#1) for review on master by Milind Changire (mchangir)

Comment 6 Worker Ant 2017-08-24 07:42:04 UTC

REVIEW: https://review.gluster.org/18114 (glusterd: disable rpc_clnt_t after relalance process disconnection) posted (#2) for review on master by Atin Mukherjee (amukherj)

Comment 7 Worker Ant 2017-08-24 13:10:21 UTC

COMMIT: https://review.gluster.org/18114 committed in master by Atin Mukherjee (amukherj) 
------
commit a894d44427649e99d4344a241dc2f9d584a9a691
Author: Milind Changire <mchangir>
Date:   Thu Aug 24 12:39:47 2017 +0530

    glusterd: disable rpc_clnt_t after relalance process disconnection
    
    Problem:
    glusterd continues to connect to rebalance process even after
    the socket connection has disconnected.
    
    Solution:
    rpc_clnt_disable() disables the rpc_clnt_t object and disarms
    all relevant timers and drops refs to the rpc_clnt_t object
    and the transport as well.
    
    Change-Id: I981d6f1cc0087037f1927062c2770a4d5026a619
    BUG: 1484225
    Signed-off-by: Milind Changire <mchangir>
    Reviewed-on: https://review.gluster.org/18114
    Reviewed-by: MOHIT AGRAWAL <moagrawa>
    Tested-by: Atin Mukherjee <amukherj>
    Reviewed-by: Atin Mukherjee <amukherj>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 8 Worker Ant 2017-08-24 18:47:42 UTC

COMMIT: https://review.gluster.org/18113 committed in master by Atin Mukherjee (amukherj) 
------
commit ea48cae5df118d9b901e7d79cd8726b6f38d65a0
Author: Milind Changire <mchangir>
Date:   Thu Aug 24 12:35:03 2017 +0530

    Revert "rpc: disable client on disconnection from rebalance"
    
    This reverts commit 5b14c11d3cae38bc66006b02217ede485ae30dea.
    
    BUG: 1484225
    Change-Id: I3269d3fc64de3f3cc6f670ea564a87d7725e10fd
    Signed-off-by: Milind Changire <mchangir>
    Reviewed-on: https://review.gluster.org/18113
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Atin Mukherjee <amukherj>
    Tested-by: Atin Mukherjee <amukherj>
    Reviewed-by: Raghavendra G <rgowdapp>
    CentOS-regression: Gluster Build System <jenkins.org>

Comment 9 Shyamsundar 2017-12-08 17:39:07 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.13.0, please open a new bug report.

glusterfs-3.13.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2017-December/000087.html
[2] https://www.gluster.org/pipermail/gluster-users/