Bug 1286637

Summary: [geo-rep+tiering]: symlinks are not getting synced to slave on tiered master setup
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rahul Hinduja <rhinduja>
Component: geo-replicationAssignee: Saravanakumar <sarumuga>
Status: CLOSED ERRATA QA Contact: Rahul Hinduja <rhinduja>
Severity: urgent Docs Contact:
Priority: high    
Version: rhgs-3.1CC: avishwan, byarlaga, chrisw, csaba, nchilaka, nlevinki, sankarshan, sarumuga
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.2   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: glusterfs-3.7.5-9 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1287519 (view as bug list) Environment:
Last Closed: 2016-03-01 05:59:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1260783, 1287519, 1288027    

Description Rahul Hinduja 2015-11-30 11:51:14 UTC
Description of problem:
=======================

Symlink creation on master volume which is tiered volume is not synced to slave volume. For example:

File {a,b,c} is created on master volume: 

[root@dj master]# touch a b c 
[root@dj master]# ls
a  b  c  etc.1  etc.2  etc.3  etc.4  etc.5
[root@dj master]#

Files get successfully synced to slave: 

[root@mia slave]# ls
a  b  c  etc.1  etc.2  etc.3  etc.4  etc.5
[root@mia slave]# 

Create a symlink from master volume:

[root@dj master]# ln -s a d
[root@dj master]# ls -la
total 243
drwxr-xr-x.  9 root root   650 Nov 30  2015 .
drwxr-xr-x.  4 root root    31 Nov 30 03:10 ..
-rw-r--r--.  1 root root     0 Nov 30 03:49 a
-rw-r--r--.  1 root root     0 Nov 30 03:49 b
-rw-r--r--.  1 root root     0 Nov 30 03:49 c
lrwxrwxrwx.  1 root root     1 Nov 30  2015 d -> a
drwxr-xr-x. 80 root root 49152 Nov 30  2015 etc.1
drwxr-xr-x. 80 root root 49152 Nov 30  2015 etc.2
drwxr-xr-x. 80 root root 49152 Nov 30  2015 etc.3
drwxr-xr-x. 80 root root 49152 Nov 30  2015 etc.4
drwxr-xr-x. 80 root root 49152 Nov 30  2015 etc.5
drwxr-xr-x.  3 root root   144 Nov 30  2015 .trashcan
[root@dj master]# 


On Slave it never gets synced:

[root@mia slave]# ls -la
total 168
drwxr-xr-x.  9 root root  8497 Nov 30  2015 .
drwxr-xr-x.  4 root root    31 Nov 30 08:40 ..
-rw-r--r--.  1 root root     0 Nov 30 03:49 a
-rw-r--r--.  1 root root     0 Nov 30 03:49 b
-rw-r--r--.  1 root root     0 Nov 30 03:49 c
drwxr-xr-x. 80 root root 32768 Nov 30  2015 etc.1
drwxr-xr-x. 80 root root 32768 Nov 30  2015 etc.2
drwxr-xr-x. 80 root root 32768 Nov 30  2015 etc.3
drwxr-xr-x. 80 root root 32768 Nov 30  2015 etc.4
drwxr-xr-x. 80 root root 32768 Nov 30  2015 etc.5
drwxr-xr-x.  3 root root    96 Nov 30  2015 .trashcan
[root@mia slave]# 


Hardlinks are synced properly:

On Master:
==========

[root@dj master]# touch {1..10}
[root@dj master]# for i in {1..10}; do ln $i hl.$i ; done 
[root@dj master]# ls
1   2  4  6  8  a  c  e      etc.2  etc.4  hl.1   hl.2  hl.4  hl.6  hl.8
10  3  5  7  9  b  d  etc.1  etc.3  etc.5  hl.10  hl.3  hl.5  hl.7  hl.9
[root@dj master]#


On Slave:
=========

[root@mia slave]# ls
1   2  4  6  8  a  c  etc.1  etc.3  etc.5  hl.10  hl.3  hl.5  hl.7  hl.9
10  3  5  7  9  b  e  etc.2  etc.4  hl.1   hl.2   hl.4  hl.6  hl.8
[root@mia slave]#

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-7.el7rhgs.x86_64


How reproducible:
=================

1/1

Steps to Reproduce:
===================
1. Create and Start Tier master volume
2. Create slave volume (DR)
3. Create and start geo-rep session between master and slave volume
4. Mount master and slave volume
5. Create a file on master 
6. Let it sync to slave
7. Create a symlink file on master
8. Check on slave

Actual results:
===============

Symlink never gets syncs to slave while the hardlinks, and new file created are synced


Expected results:
=================

Symlink should get synced too

Comment 5 Saravanakumar 2015-11-30 14:32:50 UTC
RCA:

Changes carried out in Geo-replication are under the assumption that HASH is HOT tier and Cache is COLD tier, so that all namespace operations are avoided in COLD tier.

This is no longer the case, (Currently changed as: HASH is COLD tier and CACHE is HOT tier).

SYMLINK operations are carried out only in COLD tier and as we avoid cold bricks(in current implementation), all symlink creation operation are not replayed at slave end.

Changes will be done such at all namespace operations will be captured only in COLD brick and avoided in HOT bricks.

Comment 7 Aravinda VK 2015-12-02 10:12:30 UTC
Upstream Patch: http://review.gluster.org/12844

Comment 8 Aravinda VK 2015-12-08 10:46:34 UTC
Downstream patch: https://code.engineering.redhat.com/gerrit/#/c/62896/

Comment 9 Rahul Hinduja 2015-12-08 12:38:31 UTC
Verified with build: glusterfs-3.7.5-10.el7rhgs.x86_64

Symlinks are synced to slave. Moving this bug to verified state.

Links on slave volume:
======================

[root@mia slave]# mount | grep slave
10.70.37.99:/slave on /mnt/slave type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@mia slave]# ls -l
total 0
-rw-r--r--. 1 root root 0 Dec  8 05:19 1
-rw-r--r--. 1 root root 0 Dec  8 05:19 2
-rw-r--r--. 1 root root 0 Dec  8 05:19 3
-rw-r--r--. 1 root root 0 Dec  8 05:19 4
-rw-r--r--. 1 root root 0 Dec  8 05:19 5
-rw-r--r--. 1 root root 0 Dec  8 05:19 6
-rw-r--r--. 1 root root 0 Dec  8 05:19 7
-rw-r--r--. 1 root root 0 Dec  8 05:19 8
-rw-r--r--. 1 root root 0 Dec  8 05:19 9
lrwxrwxrwx. 1 root root 1 Dec  8  2015 s.1 -> 1
lrwxrwxrwx. 1 root root 1 Dec  8  2015 s.2 -> 2
lrwxrwxrwx. 1 root root 1 Dec  8  2015 s.3 -> 3
lrwxrwxrwx. 1 root root 1 Dec  8  2015 s.4 -> 4
lrwxrwxrwx. 1 root root 1 Dec  8  2015 s.5 -> 5
lrwxrwxrwx. 1 root root 1 Dec  8  2015 s.6 -> 6
lrwxrwxrwx. 1 root root 1 Dec  8  2015 s.7 -> 7
lrwxrwxrwx. 1 root root 1 Dec  8  2015 s.8 -> 8
lrwxrwxrwx. 1 root root 1 Dec  8  2015 s.9 -> 9
[root@mia slave]#

Comment 12 errata-xmlrpc 2016-03-01 05:59:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html