Bug 1696721

Summary:	geo-replication failing after upgrade from 5.5 to 6.0
Product:	[Community] GlusterFS	Reporter:	Chad Cropper <chad.cropper>
Component:	geo-replication	Assignee:	Sunny Kumar <sunkumar>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6	CC:	bugs, sunkumar
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-09-11 06:22:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chad Cropper 2019-04-05 13:50:20 UTC

Description of problem:
After upgrading Gluster from 5.5 to 6.0, geo-replication stays in initialized status permanently.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Stop all gluster vols/services
2. Upgrade RPMs form 5.5. to 6.0
3. Start all gluster vols/services
4. geo-repl will not start

Actual results:
Not moving past initializing state

Expected results:
geo-replication moves to active/changelog crawl

Additional info:
Last part of log
[2019-04-05 13:38:07.052740] I [socket.c:811:__socket_shutdown] 0-geovol-client-1: intentional socket shutdown(13)
[2019-04-05 13:38:07.053331] W [dict.c:986:str_to_data] (-->/usr/lib64/glusterfs/6.0/xlator/protocol/client.so(+0x40f3a) [0x7f42fb706f3a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f4309ed8bb6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f4309ed54d1] ) 0-dict: value is NULL [Invalid argument]
[2019-04-05 13:38:07.053385] I [MSGID: 114006] [client-handshake.c:1238:client_setvolume] 0-geovol-client-0: failed to set process-name in handshake msg
[2019-04-05 13:38:07.053463] W [dict.c:986:str_to_data] (-->/usr/lib64/glusterfs/6.0/xlator/protocol/client.so(+0x40f3a) [0x7f42fb706f3a] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f4309ed8bb6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f4309ed54d1] ) 0-dict: value is NULL [Invalid argument]
[2019-04-05 13:38:07.053493] I [MSGID: 114006] [client-handshake.c:1238:client_setvolume] 0-geovol-client-1: failed to set process-name in handshake msg
[2019-04-05 13:38:07.054023] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-geovol-client-0: Connected to geovol-client-0, attached to remote volume '/glusterfs/geovol_b1/brick'.
[2019-04-05 13:38:07.054314] I [MSGID: 114046] [client-handshake.c:1107:client_setvolume_cbk] 0-geovol-client-1: Connected to geovol-client-1, attached to remote volume '/glusterfs/geovol_b1/brick'.
[2019-04-05 13:38:07.056436] I [fuse-bridge.c:5142:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2019-04-05 13:38:07.056467] I [fuse-bridge.c:5753:fuse_graph_sync] 0-fuse: switched to graph 0
[2019-04-05 13:39:32.048849] C [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-geovol-client-1: server 192.168.xxx.xxx:49153 has not responded in the last 42 seconds, disconnecting.
[2019-04-05 13:39:32.048952] I [socket.c:811:__socket_shutdown] 0-geovol-client-1: intentional socket shutdown(10)

Comment 1 Sunny Kumar 2019-04-15 08:47:52 UTC

Hi,

Can you please check all brick status, it looks like brick/s is/are not up.
Please do a force gluster vol start it should work.

If it does not please share all logs form master and slave volumes.

Comment 2 Amar Tumballi 2019-06-18 09:33:32 UTC

Hi Chad, did you get a chance to try Sunny's suggestion?

Comment 3 Sunny Kumar 2019-09-11 06:22:24 UTC

Closing this bug as I am not seeing this behavior.