Bug 1680324

Summary: geo-replication in "FAULTY" state after files are added to master volume: gsyncd worker crashed in syncdutils with "OSError: [Errno 22] Invalid argument
Product: [Community] GlusterFS Reporter: Boubacar Cisse <bcissea>
Component: geo-replicationAssignee: Sunny Kumar <sunkumar>
Status: CLOSED INSUFFICIENT_DATA QA Contact:
Severity: urgent Docs Contact:
Priority: medium    
Version: 5CC: bugs, csaba, khiremat, pasik, rhs-bugs, storage-qa-internal
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-31 12:16:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Gluster Config Info none

Description Boubacar Cisse 2019-02-23 22:14:53 UTC
Created attachment 1538011 [details]
Gluster Config Info

Description of problem:
geo-replication on slave in "Faulty" state after files are added to master server volume

How reproducible:
Always

Steps to Reproduce:
1/ Create and start a geo-rep relationship between master and slave.
	gluster volume geo-replication gfs1 geo-user@media03::gfs1-backup create ssh-port xxxx push-pem
	/usr/lib/x86_64-linux-gnu/glusterfs/set_geo_rep_pem_keys.sh geo-user gfs1 gfs1-backup
	gluster volume geo-replication gfs1 geo-user@media03::gfs1-backup config use-meta-volume false
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config ignore-deletes true
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config access_mount true
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config sync-acls true 
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config sync-xattrs true
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config log-rsync-performance true
	root@media01:/var/run/gluster# gluster volume geo-replication gfs1 geo-user@media03::gfs1 start
	Starting geo-replication session between gfs1 & geo-user@media03::gfs1 has been successful

2/ Geo rep status check shows crawl status as 
	gluster volume geo-replication status

3/ Create file on master using "touch /gfsMountPoint/test.txt"

4/ Geo rep status check now shows crawl status as "Initializing..." then "Faulty"
root@media01:/var/run/gluster# gluster volume geo-replication status

MASTER NODE    MASTER VOL    MASTER BRICK        SLAVE USER    SLAVE                           SLAVE NODE    STATUS             CRAWL STATUS    LAST_SYNCED
----------------------------------------------------------------------------------------------------------------------------------------------------------
media01        gfs1          /gfs1-data/brick    geo-user      ssh://geo-user@media03::gfs1    N/A           Initializing...    N/A             N/A
media01        gfs2          /gfs2-data/brick    geo-user      ssh://geo-user@media03::gfs2    N/A           Initializing...    N/A             N/A
media02        gfs1          /gfs1-data/brick    geo-user      ssh://geo-user@media03::gfs1    media03       Passive            N/A             N/A
media02        gfs2          /gfs2-data/brick    geo-user      ssh://geo-user@media03::gfs2    media03       Passive            N/A             N/A

root@media01:/var/run/gluster# gluster volume geo-replication status

MASTER NODE    MASTER VOL    MASTER BRICK        SLAVE USER    SLAVE                           SLAVE NODE    STATUS     CRAWL STATUS    LAST_SYNCED
--------------------------------------------------------------------------------------------------------------------------------------------------
media01        gfs1          /gfs1-data/brick    geo-user      ssh://geo-user@media03::gfs1    N/A           Faulty     N/A             N/A
media01        gfs2          /gfs2-data/brick    geo-user      ssh://geo-user@media03::gfs2    N/A           Faulty     N/A             N/A
media02        gfs1          /gfs1-data/brick    geo-user      ssh://geo-user@media03::gfs1    media03       Passive    N/A             N/A
media02        gfs2          /gfs2-data/brick    geo-user      ssh://geo-user@media03::gfs2    media03       Passive    N/A             N/A


Actual results:
geo replication in "Faulty" state

Expected results:
geo replication in "Active" state

Additional info:
See attachment for versions/config

Comment 2 Yaniv Kaul 2019-07-29 05:48:18 UTC
Status?

Comment 3 Kotresh HR 2019-07-29 06:47:04 UTC
This is an upstream bug mistakenly raised in downstream by a user.
I didn't know upstream user can raise a bug in downstream? 
Correcting the fields as per logs uploaded.

-----
6/ OS and Gluster Info
[MASTER OS INFO]
root@media01:/var/run/gluster# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic


[SLAVE OS INFO]
root@media03:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic


[MASTER GLUSTER VERSION]
root@media01:/var/run/gluster# glusterfs --version
glusterfs 5.3
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.


[SLAVE GLUSTER VERSION]
root@media03:~# glusterfs --version
glusterfs 5.3
--------------------------

Comment 4 Sunny Kumar 2019-11-25 21:42:20 UTC
Hi Boubaca,

Please revert back with below answers:

1. Session is created using root or non-root user.
2. Please share full log. log snippet has missing information.
 

It will help to root cause the issue.

** Alternatively, you can update to newer version of Gluster which is more more stable and contains more fixes.

/sunny

Comment 5 Sunny Kumar 2020-01-31 12:16:36 UTC
Closing this bug as no further information is provided to investigate the issue further.