Bug 1222776

Summary:	[geo-rep]: With tarssh the file is created at slave but it doesnt get sync
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	geo-replication	Assignee:	Kotresh HR <khiremat>
Status:	CLOSED ERRATA	QA Contact:	storage-qa-internal <storage-qa-internal>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	rhgs-3.1	CC:	aavati, annair, asrivast, avishwan, csaba, khiremat, nlevinki
Target Milestone:	---	Keywords:	Regression
Target Release:	RHGS 3.1.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.0-3	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1223642 (view as bug list)		Environment:
Last Closed:	2015-07-29 04:43:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1223642, 1223644
Bug Blocks:	1202842, 1223636

Description Rahul Hinduja 2015-05-19 07:17:43 UTC

Description of problem:
=======================

When we are using sync option as tarssh "use-tarssh true". Whenever the file is created at master volume, the respective entry gets created at the slave but the actual sync doesn't happen.

At master volume:
=================

[root@wingo master]# pwd
/mnt/master
[root@wingo master]# ls
hosts  hosts.allow
[root@wingo master]# cat *
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
#
# hosts.allow	This file contains access rules which are used to
#		allow or deny connections to network services that
#		either use the tcp_wrappers library or that have been
#		started through a tcp_wrappers-enabled xinetd.
#
#		See 'man 5 hosts_options' and 'man 5 hosts_access'
#		for information on rule syntax.
#		See 'man tcpd' for information on tcp_wrappers
#
[root@wingo master]# 


At slave volume:
================

[root@wingo slave]# pwd
/mnt/slave
[root@wingo slave]# ls
hosts  hosts.allow
[root@wingo slave]# cat *
[root@wingo slave]# cat hosts
[root@wingo slave]# cat hosts.allow 
[root@wingo slave]# 

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.0-2.el6rhs.x86_64

How reproducible:
=================

Always


Steps to Reproduce:
===================
1. Create a master cluster
2. Create and Start the master volume 
3. Create a slave cluster
4. Create ans Start the slave volume
5. Create and start gluster_shared_storage volume
6. Mount gluster_shared_storage at /var/run/gluster/shared_storage on all the master nodes participating in master volume.
7. Create the geo-rep session
8. Set the config option use_meta_volume to true
9. Set the config option use_tarssh to true
10. Start the geo-rep session
11. Mount the master volume to client and create a file
12. Mount the slave volume
13. Check the files on master and slave volume 


Actual results:
===============

Files that are in master gets created at slave with no content. (The first phase of creation the entry is successful but the actual sync doesn't happen)


Expected results:
=================

The files should actually sync.

Comment 4 Kotresh HR 2015-05-19 09:02:32 UTC

The issue is with recent enhancement to sync extended attributes in geo-replication.

The --overwrite option being usind with tar command is not in effect when 
--xattrs option of tar command is used to sync xattrs. The consequence being if
file exists on destination, it does not overwrite, it tries to unlink (which fails in gfid-access translator as entries are banned on aux-gfid-mount) hence fails. 

NOTE: --overwrite option is used in tar over ssh mode to keep gfids intact. 

I am looking into the issue. And will update the same with further findings.

Comment 5 Aravinda VK 2015-05-19 15:31:56 UTC

As a workaround, set sync_xattrs to false using geo-rep config command.

gluster volume geo-replication <MASTERVOL> SLAVEHOST>::<SLAVEVOL> config sync_xattrs false

Comment 6 Kotresh HR 2015-05-21 07:19:09 UTC

There is nothing to do in gluster side to sync xattrs and acl using tar over ssh
as tar is not respecting --overwrite option when used with --xattrs option.
Hence decided to not support acls and xattr syncing with tar over ssh and this will be documented. But syncing xattrs and acls works fine with 'rsync' as sync engine.


Sent patch upstream:
http://review.gluster.org/#/c/10873/

Comment 7 Kotresh HR 2015-06-02 08:54:21 UTC

Patch already in downstream build from upstream 3.7.

Upstream Patch (master):
http://review.gluster.org/#/c/10873

Upstream Patch (3.7):
http://review.gluster.org/#/c/10899/

Comment 8 Rahul Hinduja 2015-06-05 08:39:37 UTC

Verified with build: glusterfs-3.7.0-3.el6rhs.x86_64

[root@georep1 scripts]# gluster volume geo-replication master 10.70.46.154::slave config use_tarssh
true
[root@georep1 scripts]# 


Master:
=======

[root@wingo master]# pwd
/mnt/master
[root@wingo master]# ls 
hosts  hosts.allow
[root@wingo master]# cat *
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
#
# hosts.allow	This file contains access rules which are used to
#		allow or deny connections to network services that
#		either use the tcp_wrappers library or that have been
#		started through a tcp_wrappers-enabled xinetd.
#
#		See 'man 5 hosts_options' and 'man 5 hosts_access'
#		for information on rule syntax.
#		See 'man tcpd' for information on tcp_wrappers
#
[root@wingo master]# 


Slave:
======

[root@wingo slave]# pwd
/mnt/slave
[root@wingo slave]# ls
hosts  hosts.allow
[root@wingo slave]# cat *
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
#
# hosts.allow	This file contains access rules which are used to
#		allow or deny connections to network services that
#		either use the tcp_wrappers library or that have been
#		started through a tcp_wrappers-enabled xinetd.
#
#		See 'man 5 hosts_options' and 'man 5 hosts_access'
#		for information on rule syntax.
#		See 'man tcpd' for information on tcp_wrappers
#
[root@wingo slave]# 



Arequal:
========

[root@wingo ~]# ./scripts/arequal-checksum -p /mnt/master

Entry counts
Regular files   : 2
Directories     : 3
Symbolic links  : 0
Other           : 0
Total           : 5

Metadata checksums
Regular files   : 3e9
Directories     : 24d74c
Symbolic links  : 3e9
Other           : 3e9

Checksums
Regular files   : 6b4cb7a6384c32b8811cf8591fc6145b
Directories     : 616328010401341a
Symbolic links  : 0
Other           : 0
Total           : 8b3367fe238b12f9
[root@wingo ~]# ./scripts/arequal-checksum -p /mnt/slave

Entry counts
Regular files   : 2
Directories     : 3
Symbolic links  : 0
Other           : 0
Total           : 5

Metadata checksums
Regular files   : 3e9
Directories     : 24d74c
Symbolic links  : 3e9
Other           : 3e9

Checksums
Regular files   : 6b4cb7a6384c32b8811cf8591fc6145b
Directories     : 616328010401341a
Symbolic links  : 0
Other           : 0
Total           : 8b3367fe238b12f9
[root@wingo ~]# 


Moving the bug to verified state

Comment 9 errata-xmlrpc 2015-07-29 04:43:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1495.html