Bug 1142960

Summary: dist-geo-rep: geo-rep status in one of rebooted node remains at "Stable(paused)" after session is resumed.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: M S Vishwanath Bhat <vbhat>
Component: geo-replicationAssignee: Kotresh HR <khiremat>
Status: CLOSED ERRATA QA Contact: M S Vishwanath Bhat <vbhat>
Severity: medium Docs Contact:
Priority: medium    
Version: rhgs-3.0CC: aavati, amainkar, avishwan, csaba, khiremat, mzywusko, nlevinki, nsathyan, sharne, smanjara, ssaha, ssamanta, surs
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.0.3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.6.0.31-1.el6rhs Doc Type: Bug Fix
Doc Text:
Previously, when geo-replication was paused and the node was rebooted, geo-replication status remained at "Stable(paused)" state even after session was resumed. The further geo-replication pause displayed "Geo-rep already paused" message. With this fix, there is no mismatch between status file and actual status of geo-replication processes and the geo-replication status in rebooted node remains intact after session is resumed.
 Story Points: ---
Clone Of:
: 1149982 (view as bug list) Environment:
Last Closed: 2015-01-15 13:39:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1149982, 1159195, 1162694    

Description M S Vishwanath Bhat 2014-09-17 16:22:59 UTC 
Description of problem:
When you pause the geo-rep session and reboot one of the passive node and resume the session after the session comes back up online, the status of the node is stuck at "Stable(paused)" even after long a long. All other machines have moved on to Active/Passive state except for the node which got rebooted.

Version-Release number of selected component (if applicable):
glusterfs-3.6.0.28-1.el6rhs.x86_64

How reproducible:
Hit only once. Haven't tried but seems like a easily reproducible issue.

Steps to Reproduce:
1. Create and start a geo-rep session between 2*2 dist-rep master and 282 dist-rep slave.
2. Now create some data and let it sync to slave.
3. pause the session using the geo-rep pause command. And check the status. It should be "Stable(paused)"
4. Now reboot one of the Passive node and wait for the node to come back online.
5. Check the status. All but the rebooted node should have "Stable(paused)" state. And rebooted node should have "faulty(paused)" state.
6. Now resume the session using the geo-rep resume command.
7. And check the status

Actual results:
Now the status of the rebooted node gets stuck at "Stable(paused)" while other node's state goes back to "Active/Passive"

MASTER NODE                 MASTER VOL    MASTER BRICK      SLAVE               STATUS            CHECKPOINT STATUS    CRAWL STATUS           
---------------------------------------------------------------------------------------------------------------------------------------
ccr.blr.redhat.com          master        /bricks/brick0    nirvana::slave      Active            N/A                  Changelog Crawl        
metallica.blr.redhat.com    master        /bricks/brick1    acdc::slave         Passive           N/A                  N/A                    
beatles.blr.redhat.com      master        /bricks/brick3    rammstein::slave    Stable(Paused)    N/A                  N/A                    
pinkfloyd.blr.redhat.com    master        /bricks/brick2    led::slave          Active            N/A                  Changelog Crawl        




Expected results:
All nodes should have the proper status updated after resume.


Additional info:
All the data got synced because the Active node was not paused (status was showing Ative). Not sure what happens when an Active node gets rebooted.
Comment 2 M S Vishwanath Bhat 2014-09-19 15:03:32 UTC 
I hit the same issue of one node in paused state and other node in Active/Passive state even without a reboot. It happens intermittently. Does not happen every time.

Resume fails following error. That geo-rep is not paused in following machines.

[root@pinkfloyd ~]# gluster v geo master acdc::slave resume
Staging failed on 10.70.43.127. Error: Geo-replication session between master and acdc::slave is not Paused.
Staging failed on beatles. Error: Geo-replication session between master and acdc::slave is not Paused.
Staging failed on metallica. Error: Geo-replication session between master and acdc::slave is not Paused.
geo-replication command failed


Then pause fails with following message.

[root@pinkfloyd ~]# gluster v geo master acdc::slave pause
Geo-replication session between master and acdc::slave already Paused.
geo-replication command failed


This does not happen evrytime without reboot. The only workaround I found was to stop and then restart geo-replication.
Comment 3 Kotresh HR 2014-10-16 05:24:55 UTC 
Upstream Patch (Status: Merged):
http://review.gluster.org/#/c/8911/

Downstream Patch:
https://code.engineering.redhat.com/gerrit/#/c/34691/
Comment 7 shilpa 2014-11-21 12:01:59 UTC 
Moving the bug to verified. Reference: c#5 and c#6
Comment 8 Shalaka 2015-01-09 09:33:54 UTC 
Please review and sign-off edited doc text.
Comment 9 Kotresh HR 2015-01-12 09:11:05 UTC 
Doc text looks fine to me.
Comment 11 errata-xmlrpc 2015-01-15 13:39:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0038.html