Description of problem: ========================== I am seeing transport end point error in the glustershd logs of the nodes which were up after a network disruption on the victim node I had created a geo-rep setup as below master: 4node cluster: 2x2 volume slave :3 node setup: 1x(4+2) ec volume (say s1,s2,s3) All was good , but them I did ifdown on the first slave node s1 while IOs were going on, then did a ifup after some time, I don't see any problem with the georep setup. However, if i check the glustershd logs of the remaining 2 slave nodes ie s2 and s3 it shows below errors e-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected] [2017-06-16 13:36:07.000374] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-3: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected] [2017-06-16 13:36:09.000326] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected] [2017-06-16 13:37:07.000424] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-3: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected] [2017-06-16 13:37:09.000280] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected] The IOs I was creating was deep directory structure ie a{1..10}/b{1..10}/c{1..10}/d{1..10}/e{1..50} I stopped it once i observed these errors,and did a new file create on master to see if geo-replication is happening and to see if the file is getting written to all the nodes or only s2 and s3 On Master mount: [root@dhcp35-126 distrep-master]# dd if=/dev/urandom of=testagain bs=1055 count=10000 10000+0 records in 10000+0 records out 10550000 bytes (11 MB) copied, 1.53693 s, 6.9 MB/s [root@dhcp35-126 distrep-master]# du -sh testagain 11M testagain Also did an md5sum [root@dhcp35-126 distrep-master]# md5sum testagain 3746f606d09af8683e4dc7c8d3661432 testagain On Slave mount: file was replicated and md5sum too matched 10.70.35.214:ec-slave on /mnt/ec-slave type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) [root@dhcp37-53 ec-slave]# du -sh testagain 11M testagain [root@dhcp37-53 ec-slave]# md5sum testagain 3746f606d09af8683e4dc7c8d3661432 testagain [root@dhcp37-53 ec-slave]# Version-Release number of selected component (if applicable): ========== 3.8.4-28 How reproducible: Steps to Reproduce: 1.create a georep setup with master as 2x2 and slave as EC 1+(4+2) 2.start populating IOs (was checking geo rep beahvior for about 3 hrs ) like pause, resume ,etc 3.started deep dir creates 4. now did an ifdown of s1 5. did a ifup of s1 after 5-10 min 6. also did a pause and resume of georep 7. Seeing s2 and s3 throw transport end point error(even after healing is complete) client-0: Connected to ec-slave-client-0, attached to remote volume '/rhs/brick1/ec-slave'. [2017-06-16 13:50:55.806632] I [MSGID: 114047] [client-handshake.c:1226:client_setvolume_cbk] 0-ec-slave-client-0: Server and Client lk-version numbers are not same, reopening the fds [2017-06-16 13:50:55.806968] I [MSGID: 114035] [client-handshake.c:201:client_set_lk_version_cbk] 0-ec-slave-client-0: Server lk version = 1 [2017-06-16 13:50:34.000198] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: trusted.glusterfs.pathinfo [Transport endpoint is not connected] [2017-06-16 13:50:34.000256] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-ec-slave-client-0: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001). Key: glusterfs.xattrop_index_gfid [Transport endpoint is not connected] [2017-06-16 13:50:34.000265] W [MSGID: 122032] [ec-heald.c:252:ec_shd_index_sweep] 0-ec-slave-disperse-0: unable to get index-dir on ec-slave-client-0
############# MASTER ############## [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 19:03:57 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Active Changelog Crawl 2017-06-16 19:03:50 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Passive N/A N/A [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster v status Status of volume: distrep Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.45:/rhs/brick1/distrep 49152 0 Y 7632 Brick 10.70.35.130:/rhs/brick1/distrep 49152 0 Y 1519 Brick 10.70.35.122:/rhs/brick1/distrep 49152 0 Y 3955 Brick 10.70.35.23:/rhs/brick1/distrep 49152 0 Y 16168 Self-heal Daemon on localhost N/A N/A Y 10124 Self-heal Daemon on 10.70.35.23 N/A N/A Y 17979 Self-heal Daemon on 10.70.35.130 N/A N/A Y 4037 Self-heal Daemon on 10.70.35.122 N/A N/A Y 5815 Task Status of Volume distrep ------------------------------------------------------------------------------ There are no active volume tasks Status of volume: gluster_shared_storage Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.130:/var/lib/glusterd/ss_bri ck 49153 0 Y 4017 Brick 10.70.35.122:/var/lib/glusterd/ss_bri ck 49153 0 Y 5795 Brick dhcp35-45.lab.eng.blr.redhat.com:/var /lib/glusterd/ss_brick 49153 0 Y 10104 Self-heal Daemon on localhost N/A N/A Y 10124 Self-heal Daemon on 10.70.35.23 N/A N/A Y 17979 Self-heal Daemon on 10.70.35.130 N/A N/A Y 4037 Self-heal Daemon on 10.70.35.122 N/A N/A Y 5815 Task Status of Volume gluster_shared_storage ------------------------------------------------------------------------------ There are no active volume tasks [root@dhcp35-45 ~]# gluster v info Volume Name: distrep Type: Distributed-Replicate Volume ID: c46bbf1f-8aae-4860-9b4e-2719c5c31b6d Status: Started Snapshot Count: 0 Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.70.35.45:/rhs/brick1/distrep Brick2: 10.70.35.130:/rhs/brick1/distrep Brick3: 10.70.35.122:/rhs/brick1/distrep Brick4: 10.70.35.23:/rhs/brick1/distrep Options Reconfigured: changelog.changelog: on geo-replication.ignore-pid-check: on geo-replication.indexing: on transport.address-family: inet nfs.disable: on cluster.enable-shared-storage: enable Volume Name: gluster_shared_storage Type: Replicate Volume ID: f2688af7-f2ce-4ebe-803e-744ac3c82fbe Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: 10.70.35.130:/var/lib/glusterd/ss_brick Brick2: 10.70.35.122:/var/lib/glusterd/ss_brick Brick3: dhcp35-45.lab.eng.blr.redhat.com:/var/lib/glusterd/ss_brick Options Reconfigured: transport.address-family: inet nfs.disable: on cluster.enable-shared-storage: enable [root@dhcp35-45 ~]# ################# SLAVE #################### [root@dhcp35-192 ~]# gluster v info g Volume Name: ec-slave Type: Disperse Volume ID: 03a6765a-b493-4c72-82d3-945a8a040e2d Status: Started Snapshot Count: 0 Number of Bricks: 1 x (4 + 2) = 6 Transport-type: tcp Bricks: Brick1: 10.70.35.192:/rhs/brick1/ec-slave Brick2: 10.70.35.214:/rhs/brick1/ec-slave Brick3: 10.70.35.215:/rhs/brick1/ec-slave Brick4: 10.70.35.192:/rhs/brick2/ec-slave Brick5: 10.70.35.214:/rhs/brick2/ec-slave Brick6: 10.70.35.215:/rhs/brick2/ec-slave Options Reconfigured: transport.address-family: inet nfs.disable: on [root@dhcp35-192 ~]# gluster v status Status of volume: ec-slave Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.192:/rhs/brick1/ec-slave 49152 0 Y 21602 Brick 10.70.35.214:/rhs/brick1/ec-slave 49152 0 Y 2908 Brick 10.70.35.215:/rhs/brick1/ec-slave 49152 0 Y 2954 Brick 10.70.35.192:/rhs/brick2/ec-slave 49153 0 Y 21621 Brick 10.70.35.214:/rhs/brick2/ec-slave 49153 0 Y 2927 Brick 10.70.35.215:/rhs/brick2/ec-slave 49153 0 Y 2973 Self-heal Daemon on localhost N/A N/A Y 21641 Self-heal Daemon on 10.70.35.215 N/A N/A Y 2993 Self-heal Daemon on 10.70.35.214 N/A N/A Y 2947 Task Status of Volume ec-slave ------------------------------------------------------------------------------ There are no active volume tasks [root@dhcp35-192 ~]# rpm -qa|grep gluster gluster-nagios-addons-0.2.8-1.el7rhgs.x86_64 glusterfs-fuse-3.8.4-28.el7rhgs.x86_64 python-gluster-3.8.4-24.el7rhgs.noarch glusterfs-geo-replication-3.8.4-28.el7rhgs.x86_64 glusterfs-server-3.8.4-28.el7rhgs.x86_64 glusterfs-3.8.4-28.el7rhgs.x86_64 glusterfs-cli-3.8.4-28.el7rhgs.x86_64 glusterfs-debuginfo-3.8.4-28.el7rhgs.x86_64 gluster-nagios-common-0.2.4-1.el7rhgs.noarch glusterfs-libs-3.8.4-28.el7rhgs.x86_64 glusterfs-rdma-3.8.4-28.el7rhgs.x86_64 glusterfs-api-3.8.4-28.el7rhgs.x86_64 glusterfs-client-xlators-3.8.4-28.el7rhgs.x86_64 [root@dhcp35-192 ~]#
o/p of georep status progressively(consume if it makes sense) [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:32:42 169 169 0 0 N/A N/A N/A [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:32:42 169 169 0 0 N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:32:57 0 0 0 0 N/A N/A N/A [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:32:57 169 169 0 0 N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:32:57 172 683 0 0 N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:33:57 0 0 0 0 N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:35:27 0 0 0 0 N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A [root@dhcp35-45 ~]# date Fri Jun 16 18:35:48 IST 2017 [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:37:42 0 0 0 0 N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# date Fri Jun 16 18:38:01 IST 2017 [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:37:57 0 0 0 0 N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A [root@dhcp35-45 ~]# date Fri Jun 16 18:38:14 IST 2017 [root@dhcp35-45 ~]# ls / bin boot dev etc gluster home lib lib64 media mnt opt proc rhev rhs root run sbin srv sys tmp usr var [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status detail MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT COMPLETED CHECKPOINT COMPLETION TIME -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A N/A N/A N/A N/A N/A N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:40:12 0 0 0 0 N/A N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 169 170 0 0 N/A N/A N/A [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave pause Pausing geo-replication session between distrep & 10.70.35.192::ec-slave has been successful [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ----------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave N/A Paused N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave N/A Paused N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave N/A Paused N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave N/A Paused N/A N/A [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave resume Resuming geo-replication session between distrep & 10.70.35.192::ec-slave has been successful [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:42:27 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:42:42 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:43:57 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Active Changelog Crawl 2017-06-16 18:32:43 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:44:27 [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Active History Crawl 2017-06-16 15:34:07 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:44:27 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave N/A Faulty N/A N/A [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# [root@dhcp35-45 ~]# gluster volume geo-replication distrep 10.70.35.192::ec-slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.35.45 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Passive N/A N/A 10.70.35.23 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.215 Active Changelog Crawl 2017-06-16 18:50:05 10.70.35.130 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.214 Active Changelog Crawl 2017-06-16 18:49:57 10.70.35.122 distrep /rhs/brick1/distrep root 10.70.35.192::ec-slave 10.70.35.192 Passive N/A N/A [root@dhcp35-45 ~]# [root@dhcp35-45 ~]#
sosreports @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1462251/
I didnt see any functional impact, hence can be deferred
been quite some time, so unable to recollect. Will retest and get back. Upasana, can you look into this. Reach out to me for help