Bug 1500835

Summary: [geo-rep]: Status shows ACTIVE for most workers in EC before it becomes the PASSIVE
Product: [Community] GlusterFS Reporter: Kotresh HR <khiremat>
Component: geo-replicationAssignee: Kotresh HR <khiremat>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: low Docs Contact:
Priority: unspecified    
Version: 3.12CC: bugs, csaba, khiremat, rhinduja, rhs-bugs, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-glusterfs-3.12.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1500284 Environment:
Last Closed: 2017-10-13 12:47:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1460918, 1500284    
Bug Blocks:    

Description Kotresh HR 2017-10-11 15:02:13 UTC 
+++ This bug was initially created as a clone of Bug #1500284 +++

+++ This bug was initially created as a clone of Bug #1460918 +++

Description of problem:
=======================

My understanding is that all the workers will try to acquire lock and the one which gets becomes ACTIVE, rest all will be initializing and will go to PASSIVE. However with EC (Tried once), Observed all becoming ACTIVE and than it became PASSIVE. 

[root@dhcp37-150 scripts]# gluster volume geo-replication master 10.70.37.71::slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK       SLAVE USER    SLAVE                 SLAVE NODE      STATUS             CRAWL STATUS       LAST_SYNCED                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.37.150    master        /rhs/brick1/b1     root          10.70.37.71::slave    10.70.37.181    Active             Changelog Crawl    2017-06-13 06:51:59          
10.70.37.150    master        /rhs/brick2/b7     root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.171    master        /rhs/brick1/b2     root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.171    master        /rhs/brick2/b8     root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.105    master        /rhs/brick1/b3     root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.105    master        /rhs/brick2/b9     root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.194    master        /rhs/brick1/b4     root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.194    master        /rhs/brick2/b10    root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.42     master        /rhs/brick1/b5     root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.42     master        /rhs/brick2/b11    root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.190    master        /rhs/brick1/b6     root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
10.70.37.190    master        /rhs/brick2/b12    root          10.70.37.71::slave    N/A             Initializing...    N/A                N/A                          
[root@dhcp37-150 scripts]# 
[root@dhcp37-150 scripts]# 
[root@dhcp37-150 scripts]# gluster volume geo-replication master 10.70.37.71::slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK       SLAVE USER    SLAVE                 SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-----------------------------------------------------------------------------------------------------------------------------------------------------
10.70.37.150    master        /rhs/brick1/b1     root          10.70.37.71::slave    10.70.37.181    Active     Changelog Crawl    2017-06-13 06:52:01          
10.70.37.150    master        /rhs/brick2/b7     root          10.70.37.71::slave    10.70.37.181    Active     Changelog Crawl    2017-06-13 06:52:01          
10.70.37.42     master        /rhs/brick1/b5     root          10.70.37.71::slave    10.70.37.181    Active     N/A                N/A                          
10.70.37.42     master        /rhs/brick2/b11    root          10.70.37.71::slave    10.70.37.181    Active     N/A                N/A                          
10.70.37.190    master        /rhs/brick1/b6     root          10.70.37.71::slave    10.70.37.71     Passive    N/A                N/A                          
10.70.37.190    master        /rhs/brick2/b12    root          10.70.37.71::slave    10.70.37.71     Active     N/A                N/A                          
10.70.37.171    master        /rhs/brick1/b2     root          10.70.37.71::slave    10.70.37.71     Active     N/A                N/A                          
10.70.37.171    master        /rhs/brick2/b8     root          10.70.37.71::slave    10.70.37.71     Active     N/A                N/A                          
10.70.37.194    master        /rhs/brick1/b4     root          10.70.37.71::slave    10.70.37.71     Active     N/A                N/A                          
10.70.37.194    master        /rhs/brick2/b10    root          10.70.37.71::slave    10.70.37.71     Active     N/A                N/A                          
10.70.37.105    master        /rhs/brick1/b3     root          10.70.37.71::slave    10.70.37.181    Active     N/A                N/A                          
10.70.37.105    master        /rhs/brick2/b9     root          10.70.37.71::slave    10.70.37.181    Active     N/A                N/A                          
[root@dhcp37-150 scripts]# 


[root@dhcp37-150 scripts]# gluster volume geo-replication master 10.70.37.71::slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK       SLAVE USER    SLAVE                 SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-----------------------------------------------------------------------------------------------------------------------------------------------------
10.70.37.150    master        /rhs/brick1/b1     root          10.70.37.71::slave    10.70.37.181    Active     Changelog Crawl    2017-06-13 06:52:01          
10.70.37.150    master        /rhs/brick2/b7     root          10.70.37.71::slave    10.70.37.181    Active     Changelog Crawl    2017-06-13 06:52:01          
10.70.37.190    master        /rhs/brick1/b6     root          10.70.37.71::slave    10.70.37.71     Passive    N/A                N/A                          
10.70.37.190    master        /rhs/brick2/b12    root          10.70.37.71::slave    10.70.37.71     Passive    N/A                N/A                          
10.70.37.105    master        /rhs/brick1/b3     root          10.70.37.71::slave    10.70.37.181    Passive    N/A                N/A                          
10.70.37.105    master        /rhs/brick2/b9     root          10.70.37.71::slave    10.70.37.181    Passive    N/A                N/A                          
10.70.37.42     master        /rhs/brick1/b5     root          10.70.37.71::slave    10.70.37.181    Passive    N/A                N/A                          
10.70.37.42     master        /rhs/brick2/b11    root          10.70.37.71::slave    10.70.37.181    Passive    N/A                N/A                          
10.70.37.194    master        /rhs/brick1/b4     root          10.70.37.71::slave    10.70.37.71     Passive    N/A                N/A                          
10.70.37.194    master        /rhs/brick2/b10    root          10.70.37.71::slave    10.70.37.71     Passive    N/A                N/A                          
10.70.37.171    master        /rhs/brick1/b2     root          10.70.37.71::slave    10.70.37.71     Passive    N/A                N/A                          
10.70.37.171    master        /rhs/brick2/b8     root          10.70.37.71::slave    10.70.37.71     Passive    N/A                N/A                          
[root@dhcp37-150 scripts]# 


Version-Release number of selected component (if applicable):
=============================================================

mainline


How reproducible:
=================

Have seen once, and will try again


Steps to Reproduce:
===================
1. Create geo-replication with EC being Master
2. Monitor the status in a loop

--- Additional comment from Worker Ant on 2017-10-10 06:40:17 EDT ---

REVIEW: https://review.gluster.org/18464 (geo-rep: Fix status transition) posted (#1) for review on master by Kotresh HR (khiremat)

--- Additional comment from Worker Ant on 2017-10-11 06:13:39 EDT ---

COMMIT: https://review.gluster.org/18464 committed in master by Aravinda VK (avishwan) 
------
commit 3edf926a1bda43879c09694cf3904c214c94c9dc
Author: Kotresh HR <khiremat>
Date:   Tue Oct 10 05:54:04 2017 -0400

    geo-rep: Fix status transition
    
    The status transition is as below which is
    wrong.
    
    Created->Initializing->Active->Active/Passive->Stopped
    
    As soon as the monitor spawns the worker, the state
    is changed from 'Initializing' to 'Active' and then to
    'Active/Passive' based on whether worker gets the lock
    or not. This is wrong and it should directly tranistion
    as below.
    
    Created->Initializing->Active/Passive->Stopped
    
    Change-Id: Ibf5ca5c4fdf168c403c6da01db60b93f0604aae7
    BUG: 1500284
    Signed-off-by: Kotresh HR <khiremat>
Comment 1 Worker Ant 2017-10-11 15:04:18 UTC 
REVIEW: https://review.gluster.org/18491 (geo-rep: Fix status transition) posted (#1) for review on release-3.12 by Kotresh HR (khiremat)
Comment 2 Worker Ant 2017-10-12 18:46:32 UTC 
COMMIT: https://review.gluster.org/18491 committed in release-3.12 by jiffin tony Thottan (jthottan) 
------
commit d93c94d460698b9dd3a73c2ba399ff577349180c
Author: Kotresh HR <khiremat>
Date:   Tue Oct 10 05:54:04 2017 -0400

    geo-rep: Fix status transition
    
    The status transition is as below which is
    wrong.
    
    Created->Initializing->Active->Active/Passive->Stopped
    
    As soon as the monitor spawns the worker, the state
    is changed from 'Initializing' to 'Active' and then to
    'Active/Passive' based on whether worker gets the lock
    or not. This is wrong and it should directly tranistion
    as below.
    
    Created->Initializing->Active/Passive->Stopped
    
    > Change-Id: Ibf5ca5c4fdf168c403c6da01db60b93f0604aae7
    > BUG: 1500284
    > Signed-off-by: Kotresh HR <khiremat>
    (cherry picked from commit 3edf926a1bda43879c09694cf3904c214c94c9dc)
    
    Change-Id: Ibf5ca5c4fdf168c403c6da01db60b93f0604aae7
    BUG: 1500835
    Signed-off-by: Kotresh HR <khiremat>
Comment 3 Jiffin 2017-10-13 12:47:53 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-glusterfs-3.12.2, please open a new bug report.

glusterfs-glusterfs-3.12.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-October/032684.html
[2] https://www.gluster.org/pipermail/gluster-users/