Description of problem: ======================= In a scenario, where data is copied from the snapshot from .snaps directory using (uss) to the master mount. The entries gets synced to slave but data remains 0 in size. Master: ======= [root@tia master]# cp -rf /mnt/master/.snaps/geo_snap_multi_text_glusterfs_create_GMT-2016.06.12-07.26.06/thread0 . [root@tia master]# [root@tia master]# ls thread0 [root@tia master]# [root@tia master]# cd thread0/ [root@tia thread0]# ls level00 level01 level02 [root@tia thread0]# cd level00/ [root@tia level00]# ll total 774 -rw-r--r--. 1 root root 81184 Jun 12 2016 575c5b2e%%179OT6LH9E -rw-r--r--. 1 root root 189899 Jun 12 2016 575c5b2e%%8AKXW79FK8 -rw-r--r--. 1 root root 186345 Jun 12 2016 575c5b2e%%E4UB060CSR -rw-r--r--. 1 root root 171280 Jun 12 2016 575c5b2e%%VGRDMPC2PL -rw-r--r--. 1 root root 158768 Jun 12 2016 575c5b2e%%W7WO0PW4W9 drwxr-xr-x. 3 root root 4096 Jun 12 2016 level10 [root@tia level00]# Slave: ====== [root@tia slave]# ls thread0 [root@tia slave]# cd thread0/ [root@tia thread0]# ls level00 level01 level02 [root@tia thread0]# cd level00/ [root@tia level00]# ll total 4 -rw-r--r--. 1 root root 0 Jun 12 2016 575c5b2e%%179OT6LH9E -rw-r--r--. 1 root root 0 Jun 12 2016 575c5b2e%%8AKXW79FK8 -rw-r--r--. 1 root root 0 Jun 12 2016 575c5b2e%%E4UB060CSR -rw-r--r--. 1 root root 0 Jun 12 2016 575c5b2e%%VGRDMPC2PL -rw-r--r--. 1 root root 0 Jun 12 2016 575c5b2e%%W7WO0PW4W9 drwxr-xr-x. 3 root root 4096 Jun 12 2016 level10 [root@tia level00]# Version-Release number of selected component (if applicable): ============================================================= glusterfs-3.7.9-10 How reproducible: ================= Always Steps to Reproduce: ================== 1. Create geo-rep session between master and slave 2. Create data on Master volume 3. It will sync to slave 4. Pause the geo-rep session 5. Take a snapshot of Master volume 6. Start the geo-rep session 7. Remove the data from Master 8. Data will be removed from Slave too 9. Enable uss on Master 10. Activate the snapshot 11. Copy back the data from .snaps/<snapshot> to master root 12. Entries gets created at slave but data doesn't sync Actual results: =============== Entries gets created at slave but data doesn't sync Expected results: ================= Data along with entries should also get sync to slave
RCA: When uss is option is toggled to enable it and a file is copied from .snaps to mount the immediate stat is resulting in ENOENT. Hence geo-rep is failing to sync the data. We need to investigate further on why toggling of option is resulting in ENOENT for a very brief interval of time.
Upstream Patch http://review.gluster.org/14773 (master)
Upstream mainline : http://review.gluster.org/14773 Upstream 3.8 : http://review.gluster.org/14776 And the fix is available in rhgs-3.2.0 as part of rebase to GlusterFS 3.8.4.
Verified with the build: glusterfs-geo-replication-3.8.4-13.el7rhgs.x86_64 Entries copied from .snap folder after enabling uss gets synced to slave properly. Moving this bug to verified state. Trial 1: ++++++++ Master: ======= [root@dj .snaps]# cp -rf snap1_GMT-2017.02.05-16.15.32/thread0 /mnt/master/ [root@dj .snaps]# cd /mnt/master/ [root@dj master]# ls -l thread0/level00 total 20 -rw-r--r--. 1 root root 4291 Feb 5 2017 58969b38%%DAM0T9081Y -rw-r--r--. 1 root root 4707 Feb 5 2017 58969b38%%FIJEL0B9YQ -rw-r--r--. 1 root root 1760 Feb 5 2017 58969b38%%HKYMLOYL22 -rw-r--r--. 1 root root 2189 Feb 5 2017 58969b38%%MG8MVPGVB1 -rw-r--r--. 1 root root 1362 Feb 5 2017 58969b38%%YIRCIEM5M6 drwxr-xr-x. 3 root root 4096 Feb 5 2017 level10 [root@dj master]# Slave: ====== [root@dj ~]# cd /mnt/slave/ [root@dj slave]# [root@dj slave]# cd thread0/ [root@dj thread0]# [root@dj thread0]# cd level00/ [root@dj level00]# ll total 20 -rw-r--r--. 1 root root 4291 Feb 5 2017 58969b38%%DAM0T9081Y -rw-r--r--. 1 root root 4707 Feb 5 2017 58969b38%%FIJEL0B9YQ -rw-r--r--. 1 root root 1760 Feb 5 2017 58969b38%%HKYMLOYL22 -rw-r--r--. 1 root root 2189 Feb 5 2017 58969b38%%MG8MVPGVB1 -rw-r--r--. 1 root root 1362 Feb 5 2017 58969b38%%YIRCIEM5M6 drwxr-xr-x. 3 root root 4096 Feb 5 2017 level10 [root@dj level00]# Trial 2: ++++++++ Master: ======= [root@dj level00]# rm -rf 58969b38%%2OK1946N7U [root@dj level00]# pwd /mnt/master/thread1/level00 [root@dj level00]# cp -rf /mnt/master/.snaps/snap2_GMT-2017.02.05-16.44.39/thread1/level00/58969b38%%2OK1946N7U . [root@dj level00]# ls /mnt/master/thread1/level00/ 58969b38%%2OK1946N7U 58969b38%%C7ZEJOEPYZ 58969b38%%KO4QYTCMV0 58969b38%%QC6XY94T26 58969b38%%VPC2ZIAKPD level10 [root@dj level00]# ll /mnt/master/thread1/level00/ total 19 -rw-r--r--. 1 root root 3274 Feb 5 2017 58969b38%%2OK1946N7U -rw-r--r--. 1 root root 1656 Feb 5 2017 58969b38%%C7ZEJOEPYZ -rw-r--r--. 1 root root 3419 Feb 5 2017 58969b38%%KO4QYTCMV0 -rw-r--r--. 1 root root 4089 Feb 5 2017 58969b38%%QC6XY94T26 -rw-r--r--. 1 root root 1457 Feb 5 2017 58969b38%%VPC2ZIAKPD drwxr-xr-x. 3 root root 4096 Feb 5 2017 level10 [root@dj level00]# Slave: ====== [root@dj level00]# ll total 15 -rw-r--r--. 1 root root 1656 Feb 5 2017 58969b38%%C7ZEJOEPYZ -rw-r--r--. 1 root root 3419 Feb 5 2017 58969b38%%KO4QYTCMV0 -rw-r--r--. 1 root root 4089 Feb 5 2017 58969b38%%QC6XY94T26 -rw-r--r--. 1 root root 1457 Feb 5 2017 58969b38%%VPC2ZIAKPD drwxr-xr-x. 3 root root 4096 Feb 5 2017 level10 [root@dj level00]# ll total 19 -rw-r--r--. 1 root root 3274 Feb 5 2017 58969b38%%2OK1946N7U -rw-r--r--. 1 root root 1656 Feb 5 2017 58969b38%%C7ZEJOEPYZ -rw-r--r--. 1 root root 3419 Feb 5 2017 58969b38%%KO4QYTCMV0 -rw-r--r--. 1 root root 4089 Feb 5 2017 58969b38%%QC6XY94T26 -rw-r--r--. 1 root root 1457 Feb 5 2017 58969b38%%VPC2ZIAKPD drwxr-xr-x. 3 root root 4096 Feb 5 2017 level10 [root@dj level00]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html