Bug 1462251

Summary: [georep]Observing "transport end point error" in glustershd logs on slave EC nodes post a network disruption
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nag Pavan Chilakam <nchilaka>
Component: disperseAssignee: Sunil Kumar Acharya <sheggodu>
Status: CLOSED NOTABUG QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: low Docs Contact:
Priority: low    
Version: rhgs-3.3CC: amukherj, aspandey, jahernan, nchilaka, pkarampu, rhs-bugs, sheggodu, storage-qa-internal, ubansal
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-30 09:30:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nag Pavan Chilakam 2017-06-16 13:53:30 UTC
Description of problem:
==========================
I am seeing transport end point error in the glustershd logs of the nodes which were up after a network disruption on the victim node


I had created a geo-rep setup as below
master: 4node cluster: 2x2 volume
slave :3 node setup: 1x(4+2) ec volume (say s1,s2,s3)

All was good , but them I did ifdown on the first slave node s1 while IOs were going on, then did a ifup after some time,
I don't see any problem with the georep setup.
However, if i check the glustershd logs of the remaining 2 slave nodes ie s2 and s3 it shows below errors

e-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected]
[2017-06-16 13:36:07.000374] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-3: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected]
[2017-06-16 13:36:09.000326] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected]
[2017-06-16 13:37:07.000424] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-3: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected]
[2017-06-16 13:37:09.000280] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected]




The IOs I was creating was deep directory structure ie  a{1..10}/b{1..10}/c{1..10}/d{1..10}/e{1..50}

I stopped it once i observed these errors,and did a new file create on master to see if geo-replication is happening and to see if the file is getting written to  all the nodes or only s2 and s3


On Master mount:

[root@dhcp35-126 distrep-master]# dd if=/dev/urandom of=testagain bs=1055 count=10000
10000+0 records in
10000+0 records out
10550000 bytes (11 MB) copied, 1.53693 s, 6.9 MB/s
[root@dhcp35-126 distrep-master]# du -sh testagain 
11M	testagain


Also did an md5sum
[root@dhcp35-126 distrep-master]# md5sum testagain
3746f606d09af8683e4dc7c8d3661432  testagain


On Slave mount:
file was replicated and md5sum too matched
10.70.35.214:ec-slave on /mnt/ec-slave type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@dhcp37-53 ec-slave]# du -sh testagain 
11M	testagain
[root@dhcp37-53 ec-slave]# md5sum testagain
3746f606d09af8683e4dc7c8d3661432  testagain
[root@dhcp37-53 ec-slave]# 





Version-Release number of selected component (if applicable):
==========
3.8.4-28


How reproducible:


Steps to Reproduce:
1.create a georep setup with master as 2x2 and slave as EC 1+(4+2)
2.start populating IOs (was checking geo rep beahvior for about 3 hrs ) like pause, resume ,etc
3.started deep dir creates
4. now did an ifdown of s1
5. did a ifup of s1 after 5-10 min
6. also did a pause and resume of georep
7. Seeing s2 and s3 throw transport end point error(even after healing is complete)

client-0: Connected to ec-slave-client-0, attached to remote volume '/rhs/brick1/ec-slave'.
[2017-06-16 13:50:55.806632] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 0-ec-slave-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-06-16 13:50:55.806968] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-ec-slave-client-0: Server lk version = 1
[2017-06-16 13:50:34.000198] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected]
[2017-06-16 13:50:34.000256] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: glusterfs.xattrop_index_gfid [Transport endpoint is not connected]
[2017-06-16 13:50:34.000265] W [MSGID: 122032] [ec-heald.c:252:ec_shd_index_sweep] 0-ec-slave-disperse-0: unable to get index-dir on ec-slave-client-0

Comment 2 Nag Pavan Chilakam 2017-06-16 13:57:47 UTC
############# MASTER ##############
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status


 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 19:03:57          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Active     Changelog Crawl    2017-06-16 19:03:50          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Passive    N/A                N/A                          
[root@dhcp35-45 ~]# 

[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# gluster v status
Status of volume: distrep
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.45:/rhs/brick1/distrep       49152     0          Y       7632 
Brick 10.70.35.130:/rhs/brick1/distrep      49152     0          Y       1519 
Brick 10.70.35.122:/rhs/brick1/distrep      49152     0          Y       3955 
Brick 10.70.35.23:/rhs/brick1/distrep       49152     0          Y       16168
Self-heal Daemon on localhost               N/A       N/A        Y       10124
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       17979
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       4037 
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       5815 
 
Task Status of Volume distrep
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: gluster_shared_storage
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.130:/var/lib/glusterd/ss_bri
ck                                          49153     0          Y       4017 
Brick 10.70.35.122:/var/lib/glusterd/ss_bri
ck                                          49153     0          Y       5795 
Brick dhcp35-45.lab.eng.blr.redhat.com:/var
/lib/glusterd/ss_brick                      49153     0          Y       10104
Self-heal Daemon on localhost               N/A       N/A        Y       10124
Self-heal Daemon on 10.70.35.23             N/A       N/A        Y       17979
Self-heal Daemon on 10.70.35.130            N/A       N/A        Y       4037 
Self-heal Daemon on 10.70.35.122            N/A       N/A        Y       5815 
 
Task Status of Volume gluster_shared_storage
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-45 ~]# gluster v info
 
Volume Name: distrep
Type: Distributed-Replicate
Volume ID: c46bbf1f-8aae-4860-9b4e-2719c5c31b6d
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.70.35.45:/rhs/brick1/distrep
Brick2: 10.70.35.130:/rhs/brick1/distrep
Brick3: 10.70.35.122:/rhs/brick1/distrep
Brick4: 10.70.35.23:/rhs/brick1/distrep
Options Reconfigured:
changelog.changelog: on
geo-replication.ignore-pid-check: on
geo-replication.indexing: on
transport.address-family: inet
nfs.disable: on
cluster.enable-shared-storage: enable
 
Volume Name: gluster_shared_storage
Type: Replicate
Volume ID: f2688af7-f2ce-4ebe-803e-744ac3c82fbe
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.35.130:/var/lib/glusterd/ss_brick
Brick2: 10.70.35.122:/var/lib/glusterd/ss_brick
Brick3: dhcp35-45.lab.eng.blr.redhat.com:/var/lib/glusterd/ss_brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
cluster.enable-shared-storage: enable
[root@dhcp35-45 ~]# 





################# SLAVE ####################
[root@dhcp35-192 ~]# gluster v info
g 
Volume Name: ec-slave
Type: Disperse
Volume ID: 03a6765a-b493-4c72-82d3-945a8a040e2d
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.35.192:/rhs/brick1/ec-slave
Brick2: 10.70.35.214:/rhs/brick1/ec-slave
Brick3: 10.70.35.215:/rhs/brick1/ec-slave
Brick4: 10.70.35.192:/rhs/brick2/ec-slave
Brick5: 10.70.35.214:/rhs/brick2/ec-slave
Brick6: 10.70.35.215:/rhs/brick2/ec-slave
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[root@dhcp35-192 ~]# gluster v status
Status of volume: ec-slave
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.35.192:/rhs/brick1/ec-slave     49152     0          Y       21602
Brick 10.70.35.214:/rhs/brick1/ec-slave     49152     0          Y       2908 
Brick 10.70.35.215:/rhs/brick1/ec-slave     49152     0          Y       2954 
Brick 10.70.35.192:/rhs/brick2/ec-slave     49153     0          Y       21621
Brick 10.70.35.214:/rhs/brick2/ec-slave     49153     0          Y       2927 
Brick 10.70.35.215:/rhs/brick2/ec-slave     49153     0          Y       2973 
Self-heal Daemon on localhost               N/A       N/A        Y       21641
Self-heal Daemon on 10.70.35.215            N/A       N/A        Y       2993 
Self-heal Daemon on 10.70.35.214            N/A       N/A        Y       2947 
 
Task Status of Volume ec-slave
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp35-192 ~]# rpm -qa|grep gluster
gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64
glusterfs-fuse-3.8.4-28.el7rhgs.x86_64
python-gluster-3.8.4-24.el7rhgs.noarch
glusterfs-geo-replication-3.8.4-28.el7rhgs.x86_64
glusterfs-server-3.8.4-28.el7rhgs.x86_64
glusterfs-3.8.4-28.el7rhgs.x86_64
glusterfs-cli-3.8.4-28.el7rhgs.x86_64
glusterfs-debuginfo-3.8.4-28.el7rhgs.x86_64
gluster-nagios-common-0.2.4-1.el7rhgs.noarch
glusterfs-libs-3.8.4-28.el7rhgs.x86_64
glusterfs-rdma-3.8.4-28.el7rhgs.x86_64
glusterfs-api-3.8.4-28.el7rhgs.x86_64
glusterfs-client-xlators-3.8.4-28.el7rhgs.x86_64
[root@dhcp35-192 ~]#

Comment 3 Nag Pavan Chilakam 2017-06-16 13:58:44 UTC
o/p of georep status progressively(consume if it makes sense)
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:32:42    169      169     0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:32:42    169      169     0       0           N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:32:57    0        0       0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:32:57    169      169     0       0           N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:32:57    172      683     0       0           N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:33:57    0        0       0       0           N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:35:27    0        0       0       0           N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]# date
Fri Jun 16 18:35:48 IST 2017
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:37:42    0        0       0       0           N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# date
Fri Jun 16 18:38:01 IST 2017
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:37:57    0        0       0       0           N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]# date
Fri Jun 16 18:38:14 IST 2017
[root@dhcp35-45 ~]# ls /
bin  boot  dev  etc  gluster  home  lib  lib64  media  mnt  opt  proc  rhev  rhs  root  run  sbin  srv  sys  tmp  usr  var
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED            ENTRY    DATA    META    FAILURES    CHECKPOINT TIME    CHECKPOINT COMPLETED    CHECKPOINT COMPLETION TIME   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                    N/A      N/A     N/A     N/A         N/A                N/A                     N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:40:12    0        0       0       0           N/A                N/A                     N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43    169      170     0       0           N/A                N/A                     N/A                          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave pause
Pausing geo-replication session between distrep & 10.70.35.192::ec-slave has been successful
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED          
-----------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    N/A           Paused    N/A             N/A                  
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    N/A           Paused    N/A             N/A                  
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    N/A           Paused    N/A             N/A                  
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    N/A           Paused    N/A             N/A                  
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave resume
Resuming geo-replication session between distrep & 10.70.35.192::ec-slave has been successful
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:42:27          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:42:42          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:43:57          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Active     Changelog Crawl    2017-06-16 18:32:43          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:44:27          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Active     History Crawl      2017-06-16 15:34:07          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:44:27          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    N/A             Faulty     N/A                N/A                          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#  gluster volume geo-replication distrep 10.70.35.192::ec-slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK           SLAVE USER    SLAVE                     SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-------------------------------------------------------------------------------------------------------------------------------------------------------------
10.70.35.45     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Passive    N/A                N/A                          
10.70.35.23     distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.215    Active     Changelog Crawl    2017-06-16 18:50:05          
10.70.35.130    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.214    Active     Changelog Crawl    2017-06-16 18:49:57          
10.70.35.122    distrep       /rhs/brick1/distrep    root          10.70.35.192::ec-slave    10.70.35.192    Passive    N/A                N/A                          
[root@dhcp35-45 ~]# 
[root@dhcp35-45 ~]#

Comment 4 Nag Pavan Chilakam 2017-06-16 14:12:55 UTC
sosreports @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1462251/

Comment 6 Nag Pavan Chilakam 2017-06-19 11:59:21 UTC
I didnt see any functional impact, hence can be deferred

Comment 11 Nag Pavan Chilakam 2018-11-15 10:59:43 UTC
been quite some time, so unable to recollect.
Will retest and get back.
Upasana, can you look into this. Reach out to me for help