Bug 1680324 - geo-replication in "FAULTY" state after files are added to master volume: gsyncd worker crashed in syncdutils with "OSError: [Errno 22] Invalid argument
Summary: geo-replication in "FAULTY" state after files are added to master volume: gsy...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: GlusterFS
Classification: Community
Component: geo-replication
Version: 5
Hardware: x86_64
OS: Linux
medium
urgent
Target Milestone: ---
Assignee: Sunny Kumar
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-23 22:14 UTC by Boubacar Cisse
Modified: 2020-01-31 12:16 UTC (History)
6 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2020-01-31 12:16:36 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)
Gluster Config Info (35.17 KB, text/plain)
2019-02-23 22:14 UTC, Boubacar Cisse
no flags Details

Description Boubacar Cisse 2019-02-23 22:14:53 UTC
Created attachment 1538011 [details]
Gluster Config Info

Description of problem:
geo-replication on slave in "Faulty" state after files are added to master server volume

How reproducible:
Always

Steps to Reproduce:
1/ Create and start a geo-rep relationship between master and slave.
	gluster volume geo-replication gfs1 geo-user@media03::gfs1-backup create ssh-port xxxx push-pem
	/usr/lib/x86_64-linux-gnu/glusterfs/set_geo_rep_pem_keys.sh geo-user gfs1 gfs1-backup
	gluster volume geo-replication gfs1 geo-user@media03::gfs1-backup config use-meta-volume false
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config ignore-deletes true
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config access_mount true
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config sync-acls true 
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config sync-xattrs true
	gluster vol geo-replication gfs1 geo-user@media03::gfs1-backup config log-rsync-performance true
	root@media01:/var/run/gluster# gluster volume geo-replication gfs1 geo-user@media03::gfs1 start
	Starting geo-replication session between gfs1 & geo-user@media03::gfs1 has been successful

2/ Geo rep status check shows crawl status as 
	gluster volume geo-replication status

3/ Create file on master using "touch /gfsMountPoint/test.txt"

4/ Geo rep status check now shows crawl status as "Initializing..." then "Faulty"
root@media01:/var/run/gluster# gluster volume geo-replication status

MASTER NODE    MASTER VOL    MASTER BRICK        SLAVE USER    SLAVE                           SLAVE NODE    STATUS             CRAWL STATUS    LAST_SYNCED
----------------------------------------------------------------------------------------------------------------------------------------------------------
media01        gfs1          /gfs1-data/brick    geo-user      ssh://geo-user@media03::gfs1    N/A           Initializing...    N/A             N/A
media01        gfs2          /gfs2-data/brick    geo-user      ssh://geo-user@media03::gfs2    N/A           Initializing...    N/A             N/A
media02        gfs1          /gfs1-data/brick    geo-user      ssh://geo-user@media03::gfs1    media03       Passive            N/A             N/A
media02        gfs2          /gfs2-data/brick    geo-user      ssh://geo-user@media03::gfs2    media03       Passive            N/A             N/A

root@media01:/var/run/gluster# gluster volume geo-replication status

MASTER NODE    MASTER VOL    MASTER BRICK        SLAVE USER    SLAVE                           SLAVE NODE    STATUS     CRAWL STATUS    LAST_SYNCED
--------------------------------------------------------------------------------------------------------------------------------------------------
media01        gfs1          /gfs1-data/brick    geo-user      ssh://geo-user@media03::gfs1    N/A           Faulty     N/A             N/A
media01        gfs2          /gfs2-data/brick    geo-user      ssh://geo-user@media03::gfs2    N/A           Faulty     N/A             N/A
media02        gfs1          /gfs1-data/brick    geo-user      ssh://geo-user@media03::gfs1    media03       Passive    N/A             N/A
media02        gfs2          /gfs2-data/brick    geo-user      ssh://geo-user@media03::gfs2    media03       Passive    N/A             N/A


Actual results:
geo replication in "Faulty" state

Expected results:
geo replication in "Active" state

Additional info:
See attachment for versions/config

Comment 2 Yaniv Kaul 2019-07-29 05:48:18 UTC
Status?

Comment 3 Kotresh HR 2019-07-29 06:47:04 UTC
This is an upstream bug mistakenly raised in downstream by a user.
I didn't know upstream user can raise a bug in downstream? 
Correcting the fields as per logs uploaded.

-----
6/ OS and Gluster Info
[MASTER OS INFO]
root@media01:/var/run/gluster# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic


[SLAVE OS INFO]
root@media03:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.2 LTS
Release:	18.04
Codename:	bionic


[MASTER GLUSTER VERSION]
root@media01:/var/run/gluster# glusterfs --version
glusterfs 5.3
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.


[SLAVE GLUSTER VERSION]
root@media03:~# glusterfs --version
glusterfs 5.3
--------------------------

Comment 4 Sunny Kumar 2019-11-25 21:42:20 UTC
Hi Boubaca,

Please revert back with below answers:

1. Session is created using root or non-root user.
2. Please share full log. log snippet has missing information.
 

It will help to root cause the issue.

** Alternatively, you can update to newer version of Gluster which is more more stable and contains more fixes.

/sunny

Comment 5 Sunny Kumar 2020-01-31 12:16:36 UTC
Closing this bug as no further information is provided to investigate the issue further.


Note You need to log in before you can comment on or make changes to this bug.