Bug 1344908 - [geo-rep]: If the data is copied from .snaps directory to the master, it doesn't get sync to slave [First Copy]
Summary: [geo-rep]: If the data is copied from .snaps directory to the master, it does...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: geo-replication
Version: rhgs-3.1
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: RHGS 3.2.0
Assignee: Kotresh HR
QA Contact: Rahul Hinduja
URL:
Whiteboard:
Depends On:
Blocks: 1311843 1332080 1348904 1349271 1349274 1351522 1351530
TreeView+ depends on / blocked
 
Reported: 2016-06-12 12:03 UTC by Rahul Hinduja
Modified: 2017-03-23 05:36 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.8.4-1
Doc Type: Bug Fix
Doc Text:
Previously, enabling User Serviceable Snapshots (USS) caused a graph switch, which meant that when data was copied from the .snaps directory to the master volume, the first copy operation is not synchronized to the slave volume/s. The GFID access translator now correctly handles nameless lookups during the graph switch, and all data copied from the .snaps directory is correctly synced from the master volume to the slave volume/s.
Clone Of:
: 1348904 (view as bug list)
Environment:
Last Closed: 2017-03-23 05:36:09 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:0486 normal SHIPPED_LIVE Moderate: Red Hat Gluster Storage 3.2.0 security, bug fix, and enhancement update 2017-03-23 09:18:45 UTC

Description Rahul Hinduja 2016-06-12 12:03:21 UTC
Description of problem:
=======================

In a scenario, where data is copied from the snapshot from .snaps directory using (uss) to the master mount. The entries gets synced to slave but data remains 0 in size. 

Master:
=======

[root@tia master]# cp -rf /mnt/master/.snaps/geo_snap_multi_text_glusterfs_create_GMT-2016.06.12-07.26.06/thread0 .
[root@tia master]#

[root@tia master]# ls
thread0
[root@tia master]# 
[root@tia master]# cd thread0/
[root@tia thread0]# ls
level00  level01  level02
[root@tia thread0]# cd level00/
[root@tia level00]# ll
total 774
-rw-r--r--. 1 root root  81184 Jun 12  2016 575c5b2e%%179OT6LH9E
-rw-r--r--. 1 root root 189899 Jun 12  2016 575c5b2e%%8AKXW79FK8
-rw-r--r--. 1 root root 186345 Jun 12  2016 575c5b2e%%E4UB060CSR
-rw-r--r--. 1 root root 171280 Jun 12  2016 575c5b2e%%VGRDMPC2PL
-rw-r--r--. 1 root root 158768 Jun 12  2016 575c5b2e%%W7WO0PW4W9
drwxr-xr-x. 3 root root   4096 Jun 12  2016 level10
[root@tia level00]# 


Slave:
======

[root@tia slave]# ls
thread0
[root@tia slave]# cd thread0/
[root@tia thread0]# ls
level00  level01  level02
[root@tia thread0]# cd level00/
[root@tia level00]# ll
total 4
-rw-r--r--. 1 root root    0 Jun 12  2016 575c5b2e%%179OT6LH9E
-rw-r--r--. 1 root root    0 Jun 12  2016 575c5b2e%%8AKXW79FK8
-rw-r--r--. 1 root root    0 Jun 12  2016 575c5b2e%%E4UB060CSR
-rw-r--r--. 1 root root    0 Jun 12  2016 575c5b2e%%VGRDMPC2PL
-rw-r--r--. 1 root root    0 Jun 12  2016 575c5b2e%%W7WO0PW4W9
drwxr-xr-x. 3 root root 4096 Jun 12  2016 level10
[root@tia level00]# 


Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.9-10


How reproducible:
=================
Always


Steps to Reproduce:
==================
1. Create geo-rep session between master and slave
2. Create data on Master volume
3. It will sync to slave
4. Pause the geo-rep session
5. Take a snapshot of Master volume
6. Start the geo-rep session
7. Remove the data from Master
8. Data will be removed from Slave too
9. Enable uss on Master
10. Activate the snapshot 
11. Copy back the data from .snaps/<snapshot> to master root
12. Entries gets created at slave but data doesn't sync


Actual results:
===============
Entries gets created at slave but data doesn't sync

Expected results:
=================
Data along with entries should also get sync to slave

Comment 13 Kotresh HR 2016-06-15 09:52:05 UTC
RCA:
When uss is option is toggled to enable it and a file is copied from .snaps to mount the immediate stat is resulting in ENOENT. Hence geo-rep is failing to sync the data. 

We need to investigate further on why toggling of option is resulting in ENOENT for a very brief interval of time.

Comment 14 Kotresh HR 2016-06-22 09:47:01 UTC
Upstream Patch
http://review.gluster.org/14773 (master)

Comment 16 Atin Mukherjee 2016-09-17 13:34:13 UTC
Upstream mainline : http://review.gluster.org/14773
Upstream 3.8 : http://review.gluster.org/14776

And the fix is available in rhgs-3.2.0 as part of rebase to GlusterFS 3.8.4.

Comment 20 Rahul Hinduja 2017-02-05 17:04:45 UTC
Verified with the build: glusterfs-geo-replication-3.8.4-13.el7rhgs.x86_64

Entries copied from .snap folder after enabling uss gets synced to slave properly. Moving this bug to verified state.

Trial 1:
++++++++

Master:
=======

[root@dj .snaps]# cp -rf snap1_GMT-2017.02.05-16.15.32/thread0 /mnt/master/
[root@dj .snaps]# cd /mnt/master/
[root@dj master]# ls -l thread0/level00
total 20
-rw-r--r--. 1 root root 4291 Feb  5  2017 58969b38%%DAM0T9081Y
-rw-r--r--. 1 root root 4707 Feb  5  2017 58969b38%%FIJEL0B9YQ
-rw-r--r--. 1 root root 1760 Feb  5  2017 58969b38%%HKYMLOYL22
-rw-r--r--. 1 root root 2189 Feb  5  2017 58969b38%%MG8MVPGVB1
-rw-r--r--. 1 root root 1362 Feb  5  2017 58969b38%%YIRCIEM5M6
drwxr-xr-x. 3 root root 4096 Feb  5  2017 level10
[root@dj master]# 

Slave:
======

[root@dj ~]# cd /mnt/slave/
[root@dj slave]# 
[root@dj slave]# cd thread0/
[root@dj thread0]#
[root@dj thread0]# cd level00/
[root@dj level00]# ll
total 20
-rw-r--r--. 1 root root 4291 Feb  5  2017 58969b38%%DAM0T9081Y
-rw-r--r--. 1 root root 4707 Feb  5  2017 58969b38%%FIJEL0B9YQ
-rw-r--r--. 1 root root 1760 Feb  5  2017 58969b38%%HKYMLOYL22
-rw-r--r--. 1 root root 2189 Feb  5  2017 58969b38%%MG8MVPGVB1
-rw-r--r--. 1 root root 1362 Feb  5  2017 58969b38%%YIRCIEM5M6
drwxr-xr-x. 3 root root 4096 Feb  5  2017 level10
[root@dj level00]# 


Trial 2:
++++++++

Master:
=======

[root@dj level00]# rm -rf 58969b38%%2OK1946N7U
[root@dj level00]# pwd
/mnt/master/thread1/level00
[root@dj level00]# cp -rf /mnt/master/.snaps/snap2_GMT-2017.02.05-16.44.39/thread1/level00/58969b38%%2OK1946N7U .
[root@dj level00]# ls /mnt/master/thread1/level00/
58969b38%%2OK1946N7U  58969b38%%C7ZEJOEPYZ  58969b38%%KO4QYTCMV0  58969b38%%QC6XY94T26  58969b38%%VPC2ZIAKPD  level10
[root@dj level00]# ll /mnt/master/thread1/level00/
total 19
-rw-r--r--. 1 root root 3274 Feb  5  2017 58969b38%%2OK1946N7U
-rw-r--r--. 1 root root 1656 Feb  5  2017 58969b38%%C7ZEJOEPYZ
-rw-r--r--. 1 root root 3419 Feb  5  2017 58969b38%%KO4QYTCMV0
-rw-r--r--. 1 root root 4089 Feb  5  2017 58969b38%%QC6XY94T26
-rw-r--r--. 1 root root 1457 Feb  5  2017 58969b38%%VPC2ZIAKPD
drwxr-xr-x. 3 root root 4096 Feb  5  2017 level10
[root@dj level00]# 


Slave:
======

[root@dj level00]# ll
total 15
-rw-r--r--. 1 root root 1656 Feb  5  2017 58969b38%%C7ZEJOEPYZ
-rw-r--r--. 1 root root 3419 Feb  5  2017 58969b38%%KO4QYTCMV0
-rw-r--r--. 1 root root 4089 Feb  5  2017 58969b38%%QC6XY94T26
-rw-r--r--. 1 root root 1457 Feb  5  2017 58969b38%%VPC2ZIAKPD
drwxr-xr-x. 3 root root 4096 Feb  5  2017 level10
[root@dj level00]# ll
total 19
-rw-r--r--. 1 root root 3274 Feb  5  2017 58969b38%%2OK1946N7U
-rw-r--r--. 1 root root 1656 Feb  5  2017 58969b38%%C7ZEJOEPYZ
-rw-r--r--. 1 root root 3419 Feb  5  2017 58969b38%%KO4QYTCMV0
-rw-r--r--. 1 root root 4089 Feb  5  2017 58969b38%%QC6XY94T26
-rw-r--r--. 1 root root 1457 Feb  5  2017 58969b38%%VPC2ZIAKPD
drwxr-xr-x. 3 root root 4096 Feb  5  2017 level10
[root@dj level00]#

Comment 24 errata-xmlrpc 2017-03-23 05:36:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0486.html


Note You need to log in before you can comment on or make changes to this bug.