Bug 1300682

Summary:	[georep+tiering]: Hardlink sync is broken if master volume is tiered
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rahul Hinduja <rhinduja>
Component:	geo-replication	Assignee:	Saravanakumar <sarumuga>
Status:	CLOSED ERRATA	QA Contact:	Rahul Hinduja <rhinduja>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.1	CC:	avishwan, byarlaga, chrisw, csaba, nlevinki, rcyriac, sankarshan
Target Milestone:	---	Keywords:	ZStream
Target Release:	RHGS 3.1.2
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.7.5-18	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1301032 (view as bug list)		Environment:
Last Closed:	2016-03-01 06:08:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1301032, 1302979

Description Rahul Hinduja 2016-01-21 12:58:07 UTC

Description of problem:
=======================

In a geo-replication setup where master is tiered and slave is non-tiered volume, some of the hardlinks are synced as T file while some are synced properly. 

For example:
============

Master volume hardlink files:

[root@dj scripts]# ls -l /mnt/vol0/thread0/level00/hardlink_to_files/
total 26
-rw-r--r--. 2 root root 10707 Jan 21 04:11 56a002ca%%IUUZUI3ZFV
-rw-r--r--. 2 root root 15094 Jan 21  2016 56a002ca%%VLS8G8PZ11
[root@dj scripts]# 

Synced to slave as:

[root@dj scripts]# ls -l /mnt/vol1/thread0/level00/hardlink_to_files/
total 11
-rw-r--r--. 1 root root 10707 Jan 21 04:11 56a002ca%%IUUZUI3ZFV
---------T. 1 root root     0 Jan 21  2016 56a002ca%%VLS8G8PZ11
[root@dj scripts]# 
[root@dj scripts]# cat /mnt/vol1/thread0/level00/hardlink_to_files/56a002ca%%VLS8G8PZ11
[root@dj scripts]# 


For the files which is synced properly, the changelogs records are as follows:
==============================================================================

Active Hot Tier changelog records in processed:
-----------------------------------------------

[root@dhcp37-165 .processed]# grep -i "03c926fd-4cc0-4d41-a8be-50efa70916a4" *
Binary file archive_201601.tar matches
CHANGELOG.1453329701:E 03c926fd-4cc0-4d41-a8be-50efa70916a4 CREATE 33188 0 0 84eae93e-208b-4890-aa09-1ee223d0f313%2F56a00268%25%25S5CGBPUMYP
CHANGELOG.1453329701:D 03c926fd-4cc0-4d41-a8be-50efa70916a4
CHANGELOG.1453329791:E 03c926fd-4cc0-4d41-a8be-50efa70916a4 LINK b8bb1da7-1d1b-4600-982d-d6243d6e952b%2F56a002ca%25%25IUUZUI3ZFV
[root@dhcp37-165 .processed]# 


Active Cold Tier changelog records in processed:
------------------------------------------------

[root@dhcp37-165 .processed]# grep -i "03c926fd-4cc0-4d41-a8be-50efa70916a4" *
Binary file archive_201601.tar matches
CHANGELOG.1453329693:E 03c926fd-4cc0-4d41-a8be-50efa70916a4 MKNOD 33280 0 0 84eae93e-208b-4890-aa09-1ee223d0f313%2F56a00268%25%25S5CGBPUMYP
CHANGELOG.1453329800:E 03c926fd-4cc0-4d41-a8be-50efa70916a4 MKNOD 33280 0 0 b8bb1da7-1d1b-4600-982d-d6243d6e952b%2F56a002ca%25%25IUUZUI3ZFV
[root@dhcp37-165 .processed]# 

For the files which is not synced properly, the changelogs records are as follows:
==============================================================================

Active Hot Tier changelog records in processed:
-----------------------------------------------

[root@dhcp37-165 .processed]# grep -i "85f08137-c61a-4b81-8264-23e9510a203a" *
Binary file archive_201601.tar matches
CHANGELOG.1453329701:E 85f08137-c61a-4b81-8264-23e9510a203a CREATE 33188 0 0 84eae93e-208b-4890-aa09-1ee223d0f313%2F56a00268%25%25S0KZN4J05C
CHANGELOG.1453329701:D 85f08137-c61a-4b81-8264-23e9510a203a
CHANGELOG.1453329791:E 85f08137-c61a-4b81-8264-23e9510a203a LINK b8bb1da7-1d1b-4600-982d-d6243d6e952b%2F56a002ca%25%25VLS8G8PZ11
[root@dhcp37-165 .processed]# 

Active Cold Tier changelog records in processed:
------------------------------------------------

[root@dhcp37-165 .processed]# grep -i "85f08137-c61a-4b81-8264-23e9510a203a" *
Binary file archive_201601.tar matches
CHANGELOG.1453329693:E 85f08137-c61a-4b81-8264-23e9510a203a MKNOD 33280 0 0 84eae93e-208b-4890-aa09-1ee223d0f313%2F56a00268%25%25S0KZN4J05C
[root@dhcp37-165 .processed]# 

Another Active Cold Tier changelog recorded followin in processed:
------------------------------------------------------------------

[root@dhcp37-155 .processed]# grep -i "85f08137-c61a-4b81-8264-23e9510a203a" *
Binary file archive_201601.tar matches
CHANGELOG.1453329789:E 85f08137-c61a-4b81-8264-23e9510a203a MKNOD 33280 0 0 b8bb1da7-1d1b-4600-982d-d6243d6e952b%2F56a002ca%25%25VLS8G8PZ11
[root@dhcp37-155 .processed]# 


Above test is done by disabling quick-read on slave to avoid client side crash.

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-16.el7rhgs.x86_64


How reproducible:
=================

Always


Steps Carried:
==============
1. Create master and slave cluster
2. Create master as tiered volume (HT: 2x2, CT: 2x(4+2)
3. Create slave volume (2x2) and disable quick-read
4. Create geo-replication session between master and slave
5. Mount master volume 
6. Create set of data on master volume, after a while promotion demotion should start
7. Create hardlinks of the files
8. Check for the hardlink files entry created on slave
9. Check the permissions of the hardlinks files synced to slave

Actual results:
===============

Many hardlink files are synced as T files to slave


Expected results:
=================

Files should be synced as regular link files

Comment 4 Aravinda VK 2016-01-22 09:23:18 UTC

RCA:

Since with Tiering internal MKNOD is recorded, hardlinks are also recorded as MKNOD. Geo-replication will create hardlinks as new files in Slave instead of hardlink. This new file will have the same GFID as original file but not hardlinked. Data may copied to original file alone when sync happens through rsync.

Comment 7 Aravinda VK 2016-01-28 14:44:49 UTC

Downstream patch sent
https://code.engineering.redhat.com/gerrit/#/c/66678/

Comment 9 Rahul Hinduja 2016-02-01 17:16:38 UTC

Verified with build: glusterfs-3.7.5-18.el7rhgs.x86_64

Note: quick-read is turned off on slave side. 

Create 10k files and another 10k hardlinks while promote/demotes were inprogress. 

All the hardlinks got sync to slave and arequal checksum matches as:

[root@dj scripts]# ./arequal-checksum -p /mnt/master/

Entry counts
Regular files   : 20099
Directories     : 2551
Symbolic links  : 270
Other           : 0
Total           : 22920

Metadata checksums
Regular files   : 486e85
Directories     : 24d481
Symbolic links  : 3e9
Other           : 3e9

Checksums
Regular files   : 3304cde9564cea9b7c58ed0b9b952814
Directories     : 521f19232d2d2244
Symbolic links  : 30611b563e514719
Other           : 0
Total           : 2d222297dea5a7d2
[root@dj scripts]# 


[root@dj scripts]# ./arequal-checksum -p /mnt/slave/

Entry counts
Regular files   : 20099
Directories     : 2551
Symbolic links  : 270
Other           : 0
Total           : 22920

Metadata checksums
Regular files   : 486e85
Directories     : 24d481
Symbolic links  : 3e9
Other           : 3e9

Checksums
Regular files   : 3304cde9564cea9b7c58ed0b9b952814
Directories     : 521f19232d2d2244
Symbolic links  : 30611b563e514719
Other           : 0
Total           : 2d222297dea5a7d2
[root@dj scripts]# 


Moving the bug to verified state. Will run the regression and update if the issue is reproducible.

Comment 11 errata-xmlrpc 2016-03-01 06:08:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html