Bug 982471

Summary:	Dist-geo-rep: In geo rep status 'HEALTH' and 'UPTIME' is N/A and log gives error - 'Unable to fetch extra statusfor <master_vol> <slave_vol>'
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rachana Patel <racpatel>
Component:	geo-replication	Assignee:	Avra Sengupta <asengupt>
Status:	CLOSED ERRATA	QA Contact:	amainkar
Severity:	urgent	Docs Contact:
Priority:	medium
Version:	2.1	CC:	aavati, amarts, csaba, rhs-bugs, vbhat, vkoppad
Target Milestone:	---	Keywords:	Regression
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.4.0.23rhs-1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-09-23 22:29:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rachana Patel 2013-07-09 06:49:52 UTC

Description of problem:
 Dist-geo-rep: In geo rep status 'HEALTH' and 'UPTIME' is N/A and log gives error -  'Unable to fetch extra statusfor <master_vol> <slave_vol>'

Version-Release number of selected component (if applicable):
3.4.0.12rhs.beta3-1.el6rhs.x86_64

How reproducible:


Steps to Reproduce:
1. master was DHT volume(1x6), slave was DHT volume(1x3)
2. created data on master volume more than 80GB.
3. created geo rep session between master and slave, 
keep checking status

[root@hulk ~]#   gluster volume geo master1 10.70.43.147::slave1 status
NODE                        MASTER     SLAVE                   HEALTH    UPTIME       
--------------------------------------------------------------------------------------
ironman.blr.redhat.com      master1    10.70.43.147::slave1    Stable    00:26:44     
wall.blr.redhat.com         master1    10.70.43.147::slave1    Stable    01:54:20    


glusterd log says:-
[2013-07-08 14:59:41.815517] E [glusterd-geo-rep.c:2270:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master1(master), 10.70.43.147::slave1(slave)
[2013-07-08 15:00:27.981754] E [glusterd-geo-rep.c:2270:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master1(master), 10.70.43.147::slave1(slave)
[2013-07-08 15:00:36.049246] E [glusterd-geo-rep.c:2270:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master1(master), 10.70.43.147::slave1(slave)
[2013-07-08 15:00:41.752925] E [glusterd-geo-rep.c:2270:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master1(master), 10.70.43.147::slave1(slave)
[2013-07-08 15:01:26.045033] E [glusterd-geo-rep.c:2270:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master1(master), 10.70.43.147::slave1(slave)
[2013-07-08 15:01:49.681062] E [glusterd-geo-rep.c:2270:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master1(master), 10.70.43.147::slave1(slave)

Actual results:


Expected results:


Additional info:

Comment 2 Amar Tumballi 2013-08-01 10:29:31 UTC

not reproducible for us. Can you try with the latest build?

Comment 3 Rachana Patel 2013-08-13 09:00:20 UTC

not able to reproduce with latest build 3.4.0.18rhs-1.el6rhs.x86_64, hence moving to verified

Comment 4 Rachana Patel 2013-08-20 10:48:21 UTC

able to reproduce with 3.4.0.18rhs-1.el6rhs.x86_64
hence moving back to Assigned

log says
Unable to fetch extra statusfor master1(master)...

status file has status stable but status command shows N/A

Comment 5 M S Vishwanath Bhat 2013-08-20 11:13:21 UTC

I hit this with 20rhs-2 in few rather simple steps.

I rotated the logs using logrotate(8)

logrotate -f /etc/logrotate.d/glusterfs-georep

 
                                        MASTER: master  SLAVE: falcon::slave
 
NODE                         HEALTH    UPTIME      FILES SYNCD    FILES PENDING    BYTES PENDING    DELETES PENDING   
--------------------------------------------------------------------------------------------------------------------
spitfire.blr.redhat.com      Stable    01:22:50    1109           0                0Bytes           0                 
typhoon.blr.redhat.com       Stable    01:22:50    0              0                0Bytes           0                 
harrier.blr.redhat.com       Stable    01:22:50    1088           0                0Bytes           0                 
mustang.blr.redhat.com       N/A       N/A         N/A            N/A              N/A              N/A               
 
                                        MASTER: master  SLAVE: falcon::slave
 
NODE                         HEALTH    UPTIME      FILES SYNCD    FILES PENDING    BYTES PENDING    DELETES PENDING   
--------------------------------------------------------------------------------------------------------------------
spitfire.blr.redhat.com      Stable    01:22:55    1109           0                0Bytes           0                 
typhoon.blr.redhat.com       Stable    01:22:56    0              0                0Bytes           0                 
harrier.blr.redhat.com       Stable    01:22:56    1088           0                0Bytes           0                 
mustang.blr.redhat.com       N/A       N/A         N/A            N/A              N/A              N/A               


The status goes N/A on that node only. And I see lot of these error messages in the glusterd log.

[2013-08-20 10:44:05.870842] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)
[2013-08-20 10:44:09.827399] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)
[2013-08-20 10:44:13.836479] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)
[2013-08-20 10:44:17.764012] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)
[2013-08-20 10:44:21.738387] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)
[2013-08-20 10:44:25.686593] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)
[2013-08-20 10:44:29.688960] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)
[2013-08-20 10:44:33.609573] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)
[2013-08-20 10:44:37.586113] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)
[2013-08-20 10:44:41.485594] E [glusterd-geo-rep.c:2535:glusterd_read_status_file] 0-: Unable to fetch extra statusfor master(master), falcon::slave(slave)

Comment 6 Amar Tumballi 2013-08-26 08:15:03 UTC

https://code.engineering.redhat.com/gerrit/#/c/11927/

Comment 7 Rachana Patel 2013-09-08 13:48:47 UTC

unable to reproduce with 3.4.0.32rhs-1.el6_4.x86_64, hence marking it as verified

Comment 8 Scott Haines 2013-09-23 22:29:53 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html