Bug 1341121

Summary: [geo-rep]: If the session is renamed, geo-rep configuration are not retained
Product: [Community] GlusterFS Reporter: Saravanakumar <sarumuga>
Component: geo-replicationAssignee: Saravanakumar <sarumuga>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.7.11CC: avishwan, bugs, csaba, khiremat, rcyriac, rhinduja, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.7.12 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1341108 Environment:
Last Closed: 2016-06-28 12:19:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1340383, 1340853, 1341108    
Bug Blocks: 1311817    

Description Saravanakumar 2016-05-31 10:10:55 UTC 
+++ This bug was initially created as a clone of Bug #1341108 +++

Description of problem:
=======================

With the recent changes, we support to rename the existing geo-rep session from one slave hot to another slave host. Expected is to rename only the session and retain all the configuration/status as of the previous session. 

But the older configurations are not retained which passively breaks the geo-rep functionality for this use case. 

Existing session between: baloo 10.70.37.88::bagheera

[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.88::bagheera config change_detector
xsync
[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.88::bagheera config ignore_deletes
true
[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.88::bagheera config use_meta_volume
true
[root@dhcp37-162 ~]#

New Session between: baloo 10.70.37.43::bagheera 

[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.43::bagheera config use_meta_volume
[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.43::bagheera config ignore_deletes
false
[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.43::bagheera config change_detector
changelog
[root@dhcp37-162 ~]# 



Version-Release number of selected component (if applicable):
=============================================================


How reproducible:
=================

Always


Steps to Reproduce:
===================
1. Create georep session between master, slavehost1 and slave
2. Update configs for this session
3. Stop existing session
4. Recreate session between master, slavehost2 and slave
5. Start the session
6. Verify for the configs setup at step 2

Actual results:
===============

Config options are reset


Expected results:
=================

Since it is a rename of a session and not the new session, all config options should be retained





--- Additional comment from Rahul Hinduja on 2016-05-27 04:56:14 EDT ---

[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.88::bagheera status
 
MASTER NODE     MASTER VOL    MASTER BRICK      SLAVE USER    SLAVE                    SLAVE NODE      STATUS     CRAWL STATUS    LAST_SYNCED                  
----------------------------------------------------------------------------------------------------------------------------------------------------
10.70.37.162    baloo         /rhs/brick1/b1    root          10.70.37.88::bagheera    10.70.37.213    Active     Hybrid Crawl    2016-05-26 14:41:23          
10.70.37.162    baloo         /rhs/brick2/b7    root          10.70.37.88::bagheera    10.70.37.88     Active     Hybrid Crawl    2016-05-26 14:41:23          
10.70.37.116    baloo         /rhs/brick1/b3    root          10.70.37.88::bagheera    10.70.37.43     Active     Hybrid Crawl    2016-05-26 14:41:23          
10.70.37.121    baloo         /rhs/brick1/b5    root          10.70.37.88::bagheera    10.70.37.200    Passive    N/A             N/A                          
10.70.37.190    baloo         /rhs/brick1/b6    root          10.70.37.88::bagheera    10.70.37.213    Active     Hybrid Crawl    2016-05-26 14:41:23          
10.70.37.189    baloo         /rhs/brick1/b4    root          10.70.37.88::bagheera    10.70.37.52     Passive    N/A             N/A                          
10.70.37.40     baloo         /rhs/brick1/b2    root          10.70.37.88::bagheera    10.70.37.88     Passive    N/A             N/A                          
10.70.37.40     baloo         /rhs/brick2/b8    root          10.70.37.88::bagheera    10.70.37.43     Passive    N/A             N/A                          
[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.88::bagheera config 
special_sync_mode: partial
session_owner: 11b99a73-649f-4439-abc4-1eac15943f0e
state_socket_unencoded: /var/lib/glusterd/geo-replication/baloo_10.70.37.88_bagheera/ssh%3A%2F%2Froot%4010.70.37.88%3Agluster%3A%2F%2F127.0.0.1%3Abagheera.socket
gluster_log_file: /var/log/glusterfs/geo-replication/baloo/ssh%3A%2F%2Froot%4010.70.37.88%3Agluster%3A%2F%2F127.0.0.1%3Abagheera.gluster.log
ssh_command: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem
ignore_deletes: true
change_detector: xsync
gluster_command_dir: /usr/sbin/
state_file: /var/lib/glusterd/geo-replication/baloo_10.70.37.88_bagheera/monitor.status
remote_gsyncd: /nonexistent/gsyncd
log_file: /var/log/glusterfs/geo-replication/baloo/ssh%3A%2F%2Froot%4010.70.37.88%3Agluster%3A%2F%2F127.0.0.1%3Abagheera.log
changelog_log_file: /var/log/glusterfs/geo-replication/baloo/ssh%3A%2F%2Froot%4010.70.37.88%3Agluster%3A%2F%2F127.0.0.1%3Abagheera-changes.log
socketdir: /var/run/gluster
working_dir: /var/lib/misc/glusterfsd/baloo/ssh%3A%2F%2Froot%4010.70.37.88%3Agluster%3A%2F%2F127.0.0.1%3Abagheera
state_detail_file: /var/lib/glusterd/geo-replication/baloo_10.70.37.88_bagheera/ssh%3A%2F%2Froot%4010.70.37.88%3Agluster%3A%2F%2F127.0.0.1%3Abagheera-detail.status
use_meta_volume: true
ssh_command_tar: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/tar_ssh.pem
pid_file: /var/lib/glusterd/geo-replication/baloo_10.70.37.88_bagheera/monitor.pid
georep_session_working_dir: /var/lib/glusterd/geo-replication/baloo_10.70.37.88_bagheera/
gluster_params: aux-gfid-mount acl
volume_id: 11b99a73-649f-4439-abc4-1eac15943f0e
[root@dhcp37-162 ~]#



[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.43::bagheera status
 
MASTER NODE     MASTER VOL    MASTER BRICK      SLAVE USER    SLAVE                    SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.37.162    baloo         /rhs/brick1/b1    root          10.70.37.43::bagheera    10.70.37.213    Active     Changelog Crawl    2016-05-26 14:41:29          
10.70.37.162    baloo         /rhs/brick2/b7    root          10.70.37.43::bagheera    10.70.37.88     Active     Changelog Crawl    2016-05-26 14:41:28          
10.70.37.116    baloo         /rhs/brick1/b3    root          10.70.37.43::bagheera    10.70.37.43     Active     Changelog Crawl    2016-05-26 14:41:37          
10.70.37.121    baloo         /rhs/brick1/b5    root          10.70.37.43::bagheera    10.70.37.200    Active     Changelog Crawl    2016-05-26 14:41:22          
10.70.37.190    baloo         /rhs/brick1/b6    root          10.70.37.43::bagheera    10.70.37.213    Passive    N/A                N/A                          
10.70.37.189    baloo         /rhs/brick1/b4    root          10.70.37.43::bagheera    10.70.37.52     Passive    N/A                N/A                          
10.70.37.40     baloo         /rhs/brick1/b2    root          10.70.37.43::bagheera    10.70.37.88     Passive    N/A                N/A                          
10.70.37.40     baloo         /rhs/brick2/b8    root          10.70.37.43::bagheera    10.70.37.43     Passive    N/A                N/A                          
[root@dhcp37-162 ~]# gluster volume geo-replication baloo 10.70.37.43::bagheera config
special_sync_mode: partial
state_socket_unencoded: /var/lib/glusterd/geo-replication/baloo_10.70.37.43_bagheera/ssh%3A%2F%2Froot%4010.70.37.43%3Agluster%3A%2F%2F127.0.0.1%3Abagheera.socket
gluster_log_file: /var/log/glusterfs/geo-replication/baloo/ssh%3A%2F%2Froot%4010.70.37.43%3Agluster%3A%2F%2F127.0.0.1%3Abagheera.gluster.log
ssh_command: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem
ignore_deletes: false
change_detector: changelog
gluster_command_dir: /usr/sbin/
state_file: /var/lib/glusterd/geo-replication/baloo_10.70.37.43_bagheera/monitor.status
remote_gsyncd: /nonexistent/gsyncd
log_file: /var/log/glusterfs/geo-replication/baloo/ssh%3A%2F%2Froot%4010.70.37.43%3Agluster%3A%2F%2F127.0.0.1%3Abagheera.log
changelog_log_file: /var/log/glusterfs/geo-replication/baloo/ssh%3A%2F%2Froot%4010.70.37.43%3Agluster%3A%2F%2F127.0.0.1%3Abagheera-changes.log
socketdir: /var/run/gluster
working_dir: /var/lib/misc/glusterfsd/baloo/ssh%3A%2F%2Froot%4010.70.37.43%3Agluster%3A%2F%2F127.0.0.1%3Abagheera
state_detail_file: /var/lib/glusterd/geo-replication/baloo_10.70.37.43_bagheera/ssh%3A%2F%2Froot%4010.70.37.43%3Agluster%3A%2F%2F127.0.0.1%3Abagheera-detail.status
session_owner: 11b99a73-649f-4439-abc4-1eac15943f0e
ssh_command_tar: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/tar_ssh.pem
pid_file: /var/lib/glusterd/geo-replication/baloo_10.70.37.43_bagheera/monitor.pid
georep_session_working_dir: /var/lib/glusterd/geo-replication/baloo_10.70.37.43_bagheera/
gluster_params: aux-gfid-mount acl
volume_id: 11b99a73-649f-4439-abc4-1eac15943f0e
[root@dhcp37-162 ~]#






RCA:
In gsyncd.conf, "peers" sections contains  old Slave Host details.

example:
peers gluster%3A%2F%2F127.0.0.1%3Atv1 ssh%3A2F%2Froot%40192.168.122.186%3Agluster%3A%2F%2F127.0.0.1%3Atv2

Where, 192.168.122.186 is old slave host IP address.

Once the geo-rep session is renamed, old host details are no longer valid.
So, with new host, it is NOT possible to get config details.

Solution:
Remove old host details from peers section.

Use only master volume and slave volume as part of peers section and remove slave host detail.
Comment 1 Vijay Bellur 2016-05-31 10:13:07 UTC 
REVIEW: http://review.gluster.org/14566 (geo-rep: update peers section in gsyncd conf) posted (#1) for review on release-3.7 by Saravanakumar Arumugam (sarumuga)
Comment 2 Vijay Bellur 2016-05-31 10:15:09 UTC 
REVIEW: http://review.gluster.org/14566 (geo-rep: update peers section in gsyncd conf) posted (#2) for review on release-3.7 by Saravanakumar Arumugam (sarumuga)
Comment 3 Vijay Bellur 2016-06-01 16:33:00 UTC 
COMMIT: http://review.gluster.org/14566 committed in release-3.7 by Aravinda VK (avishwan) 
------
commit 4d73205fee5bf99527c78ef6838b14e3a2fdd4e3
Author: Saravanakumar Arumugam <sarumuga>
Date:   Mon May 30 17:34:24 2016 +0530

    geo-rep: update peers section in gsyncd conf
    
    Problem:
    Once Slave volume uuid is involved as part of a geo-rep session, it is
    possible to create the same geo-rep session with different (slave)host.
    
    But, it reflects default values for geo-rep configuration values originally
    configured for old geo-rep session.
    Reason is, slave host is used while saving config options in gsyncd.conf.
    With new slave host, it is not possible to retrieve those config values.
    
    Solution:
    Remove slave host related information from gsyncd.conf and have only master
    volume and slave volume as part of peers section.
    
    Also, during upgrade from old geo-rep session, update peers section to
    reflect only master volume and slave volume.
    
    Change-Id: I7debf35a09a28d030b706b0c3e5d82c9b0467d0e
    BUG: 1341121
    Reviewed-on: http://review.gluster.org/#/c/14558/
    Signed-off-by: Saravanakumar Arumugam <sarumuga>
    Reviewed-on: http://review.gluster.org/14566
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Aravinda VK <avishwan>
Comment 4 Kaushal 2016-06-28 12:19:23 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.12, please open a new bug report.

glusterfs-3.7.12 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-devel/2016-June/049918.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user