Bug 1510994

Summary:	[GSS] Not all files synced using geo-replication
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Pan Ousley <pousley>
Component:	geo-replication	Assignee:	Aravinda VK <avishwan>
Status:	CLOSED UPSTREAM	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.2	CC:	amukherj, bugs, csaba, dimitri.ars, pousley, rhs-bugs, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1510342	Environment:
Last Closed:	2017-11-08 22:09:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1510342
Bug Blocks:

Description Pan Ousley 2017-11-08 13:50:41 UTC

+++ This bug was initially created as a clone of Bug #1510342 +++

Description of problem:
When using Sonatype Nexus3 as a docker repository on glusterfs and geo-replicating this volume, at least the .bytes files which contain the docker layer data do not get synced. The files are created on the geo-replicated site, but remain 0 bytes. Other files, like the .properties files are synced properly.
The moment you add a character to the .bytes file manually (echo >>), the .bytes file data does get synced...it seems like gluster doesn't detect writing data to the file in some cases, at least the way Nexus3 does it to those .bytes files. We suspect that there will more applications / files affected by this, resulting in a corrupt / incomplete data on the geo-replicated site.

Version-Release number of selected component (if applicable):
3.12.1-2

How reproducible:
100%

Steps to Reproduce:
1. Run Sonatype Nexus3 with a hosted docker repository and it's data (/nexus-data) on a glusterfs volume which is geo-replicated.
2. docker push an arbitrary image into this nexus docker repo
3. ls -laRf blobs/default/content | grep .bytes
   on both main site and geo-replicated site and see that the ones on the main site are non-0 bytes and on the geo-replicated site they're 0 bytes

Actual results:
main:
-rw-r--r--. 2 200 200  529 Nov  3 15:27 e39d9b3a-53e0-44bc-b4a2-31d145aeec81.bytes
-rw-r--r--. 1 200 200 1991435 Nov  3 19:00 613953bb-b542-4db7-ba18-01a331361994.bytes


geo-replicated:
-rw-r--r--. 0 root root    0 Nov  3 19:01 613953bb-b542-4db7-ba18-01a331361994.bytes
-rw-r--r--. 0 root root    0 Nov  3 15:28 e39d9b3a-53e0-44bc-b4a2-31d145aeec81.bytes

Expected results:
main:
-rw-r--r--. 2 200 200  529 Nov  3 15:27 e39d9b3a-53e0-44bc-b4a2-31d145aeec81.bytes
-rw-r--r--. 1 200 200 1991435 Nov  3 19:00 613953bb-b542-4db7-ba18-01a331361994.bytes


geo-replicated:
-rw-r--r--. 2 200 200  529 Nov  3 15:27 e39d9b3a-53e0-44bc-b4a2-31d145aeec81.bytes
-rw-r--r--. 1 200 200 1991435 Nov  3 19:00 613953bb-b542-4db7-ba18-01a331361994.bytes

Additional info:
Tried both rsync and use-tarssh, same issue.
Date/time is the same on main and geo-replicated site servers
An initial sync does correctly sync the .bytes files.
Maybe related to https://bugzilla.redhat.com/show_bug.cgi?id=1437244