Bug 1664524

Summary: Non-root geo-replication session goes to faulty state, when the session is started
Product: [Community] GlusterFS Reporter: Abhilash <amannath>
Component: geo-replicationAssignee: Sunny Kumar <sunkumar>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: high    
Version: 3.12CC: bugs, khiremat, sacharya
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-25 21:23:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Abhilash 2019-01-09 04:26:15 UTC 
Description of problem:
Non root geo-replication goes to faulty state when session is started. 

MASTER NODE        MASTER VOL    MASTER BRICK             SLAVE USER    SLAVE                                 SLAVE NODE    STATUS    CRAWL STATUS    LAST_SYNCED
----------------------------------------------------------------------------------------------------------------------------------------------------------------
server-114-74    gv100         /data/glusterfs/gv100    geoaccount    geoaccount@server-115-215::gv100    N/A           Faulty    N/A             N/A
server-115-37    gv100         /data/glusterfs/gv100    geoaccount    geoaccount@server-115-215::gv100    N/A           Faulty    N/A             N/A


Relevant Logs
--------------------
[2019-01-09 01:25:10.111979] I [gsyncd(monitor):715:main_i] <top>: Monitor Status: Created
[2019-01-09 01:28:17.722640] I [gsyncd(conf):689:main_i] <top>: Config Set: session-owner = 09b47d3f-553a-45f8-afd9-848d3f51a62f
[2019-01-09 01:28:18.276072] I [gsyncdstatus(monitor):241:set_worker_status] GeorepStatus: Worker Status: Initializing...
[2019-01-09 01:28:18.449407] I [monitor(monitor):275:monitor] Monitor: starting gsyncd worker(/data/glusterfs/gv100). Slave node: ssh://geoaccount@amannath-115-201:gluster://localhost:gv100
[2019-01-09 01:28:18.537137] I [changelogagent(/data/glusterfs/gv100):73:__init__] ChangelogAgent: Agent listining...
[2019-01-09 01:28:19.385097] E [syncdutils(/data/glusterfs/gv100):270:log_raise_exception] <top>: connection to peer is broken
[2019-01-09 01:28:19.385895] E [resource(/data/glusterfs/gv100):234:errlog] Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-u_GpUW/bd052dee486819c3bb070d3ed44cf6dc.sock geoaccount@amannath-115-201 /nonexistent/gsyncd --session-owner 09b47d3f-553a-45f8-afd9-848d3f51a62f --local-id .%2Fdata%2Fglusterfs%2Fgv100 --local-node amannath-114-74 -N --listen --timeout 120 gluster://localhost:gv100" returned with 1, saying:
[2019-01-09 01:28:19.386127] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.065573] I [cli.c:759:main] 0-cli: Started running /usr/sbin/gluster with version 3.10.12
[2019-01-09 01:28:19.386301] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.065685] I [cli.c:642:cli_rpc_init] 0-cli: Connecting to remote glusterd at localhost
[2019-01-09 01:28:19.386548] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.168915] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2019-01-09 01:28:19.386713] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.169141] I [socket.c:2426:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2019-01-09 01:28:19.386850] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.170204] I [cli-rpc-ops.c:7024:gf_cli_getwd_cbk] 0-cli: Received resp to getwd
[2019-01-09 01:28:19.387007] E [resource(/data/glusterfs/gv100):238:logerr] Popen: ssh> [2019-01-09 01:28:19.170325] I [input.c:31:cli_batch] 0-: Exiting with: 0
[2019-01-09 01:28:19.387537] I [syncdutils(/data/glusterfs/gv100):238:finalize] <top>: exiting.
[2019-01-09 01:28:19.389434] I [repce(/data/glusterfs/gv100):92:service_loop] RepceServer: terminating on reaching EOF.
[2019-01-09 01:28:19.389765] I [syncdutils(/data/glusterfs/gv100):238:finalize] <top>: exiting.
[2019-01-09 01:28:19.389770] I [monitor(monitor):347:monitor] Monitor: worker(/data/glusterfs/gv100) died before establishing connection
[2019-01-09 01:28:19.394583] I [gsyncdstatus(monitor):241:set_worker_status] GeorepStatus: Worker Status: Faulty



Version-Release number of selected component (if applicable):
glusterfs 3.10.12


How reproducible:
Always

Steps to Reproduce:
1.Follow the official documentation (https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/) to setup geo replication for non root user.
2.
3.

Actual results:
Geo replication session is faulty

Expected results:
Geo replication session should work properly and replicate the data


Additional info:
I had a look at the Bug 1222750, with similar symptoms, but it is marked as fixed in version 3.7.
Comment 1 Amar Tumballi 2019-06-17 05:16:59 UTC 
Hi Abhilash,

can you please upgrade to higher versions of glusterfs? We did fix multiple issues with glusterfs since 3.10.12.
Comment 2 Sunny Kumar 2019-11-25 21:23:22 UTC 
Upgrading the version will fix this problem. Closing this bug.