Bug 1003580 - Dist-geo-rep : geo-rep initial xsync crawl failed to sync few symlinks to slave in a cascaded-fanout setup.
Dist-geo-rep : geo-rep initial xsync crawl failed to sync few symlinks to sla...
Status: CLOSED EOL
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: geo-replication (Show other bugs)
2.1
x86_64 Linux
medium Severity high
: ---
: ---
Assigned To: Bug Updates Notification Mailing List
amainkar
fanout
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-02 08:22 EDT by Vijaykumar Koppad
Modified: 2015-11-25 03:51 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-25 03:49:39 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Vijaykumar Koppad 2013-09-02 08:22:49 EDT
Description of problem: geo-rep initial xsync crawl failed to get the entry in the XSYNC-CHANGELOG for few symlinks to files. 

The symlinks which were missed, had no entry in the XSYNC-CHANGELOG.


Version-Release number of selected component (if applicable):glusterfs-3.4.0.30rhs-2.el6rhs.x86_64


How reproducible: Didn't try to reproduce it. 


Steps to Reproduce:
1.create and start a geo-rep cascaded-fanout  setup ( 1-4-4)( totally 21 volumes involved , 1 master , 4imasters , and 16slaves)   
2.stop all the geo-rep sessions . 
3.create data on the master. After the completion of the data creation , start geo-rep between master and imasters. let them sync. 
4. After the completion of sync, start geo-rep sessions between imasters and slaves and let files sync slaves.  
5. Now stop all the geo-rep sessions.
6. Create symlinks to all the files.
7. start the geo-rep sessions between master and imasters. 
8. Check if all the symlinks get synced to all he imasters. 

Actual results:It fails to sync few symlinks to one imaster and reason being no entry for the missing files in XSYNC-CHANGELOG 


Expected results:It should sync all symlinks to imaster and consequently all the slaves. 


Additional info:
Comment 2 Vijaykumar Koppad 2013-09-03 06:57:19 EDT
This happened in plain fanout setup too in the build glusterfs-3.4.0.30rhs-2.el6rhs.x86_64, it is consistently happening.
Comment 3 Amar Tumballi 2013-09-04 08:37:05 EDT
Fix for bug 1001684 @ https://code.engineering.redhat.com/gerrit/#/c/12111/ should be fixing it. Will run some tests as the patch is now in for fixing that bug, and move the status of the accordingly
Comment 4 Amar Tumballi 2013-09-05 08:46:23 EDT
considering the patch made it to glusterfs-3.4.0.31rhs-1, can we get it tested again once more?
Comment 5 Vijaykumar Koppad 2013-09-09 07:27:33 EDT
The issue described is about not getting the entry in XSYNC-CHANGELOG in the first xsync crawl. I might have done mistake in investigating it properly, because after this bug filed, issue related to symlink syncing in xsync crawl as well as in changelog syncing was hit with only fanout and cascaded setups. 

I was able to hit it in the build glusterfs-3.4.0.31rhs-1, in a fanout setup , and while syncing symlinks in first xsync crawl.

steps I followed
1. create and start a geo-rep fanout setup from a master to  4 slaves.
2. stop all geo-rep sessions and create files on master.
3. start all the geo-rep sessions and let files to sync to all the slaves.
4. Now stop all geo-rep sessions, and create symlinks to all those files. 
5. Start the geo-rep session.
6. First xsync crawl fails to sync around 16 to 20 files out of 2000 files,  to all the slaves.

on master missing file
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[root@Ray ~]# getfattr -h -n glusterfs.gfid.string /mnt/master/level05/level15/symlink_to_files/5229cf3e~~4CA6FC77NJ
getfattr: Removing leading '/' from absolute path names
# file: mnt/master/level05/level15/symlink_to_files/5229cf3e~~4CA6FC77NJ
glusterfs.gfid.string="0854f9ef-aa93-462a-aa02-10e9d510f34d"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

On master, the missing file had entries in the XSYNC-CHANGELOG
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[root@Ray bd42ad17ef8864d51407b1c6478f5dc6]# grep 0854f9ef-aa93-462a-aa02-10e9d510f34d xsync/*
xsync/XSYNC-CHANGELOG.1378471766:E 0854f9ef-aa93-462a-aa02-10e9d510f34d SYMLINK fddd8985-0185-4655-a6fd-565b7c5a54ea%2F5229cf3e%7E%7E4CA6FC77NJ
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

on the slave for same GFID, 

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[2013-09-06 12:49:35.192857] W [dht-layout.c:179:dht_layout_search] 0-imaster1-dht:
 no subvolume for hash (value) = 3927056761
[2013-09-06 12:49:35.192898] I [fuse-bridge.c:3515:fuse_auxgfid_newentry_cbk] 0-fuse-aux-gfid-mount: failed to create the entry <gfid:fddd8985-0185-4655-a6fd-565b7c5a54ea>/5229cf3e~~4CA6FC77NJ with gfid (0854f9ef-aa93-462a-aa02-10e9d510f34d): No such file or directory
[2013-09-06 12:49:35.192930] W [fuse-bridge.c:1627:fuse_err_cbk] 0-glusterfs-fuse: 726: SYMLINK() <gfid:fddd8985-0185-4655-a6fd-565b7c5a54ea>/5229cf3e~~4CA6FC77NJ => -1 (No such file or directory)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>


Similar kind of issue was hit cascaded setup though changelog syncing too in the build glusterfs-3.4.0.32rhs-1.el6rhs.x86_64 . But fortunately or unfortunately it was never hit in plain single master and single slave setup.
Comment 7 Gowrishankar Rajaiyan 2013-10-08 04:41:50 EDT
Fixed in version please.
Comment 8 Vijaykumar Koppad 2013-10-18 04:33:36 EDT
I tried on the build glusterfs-3.4.0.35rhs,

I have explained most of the things related to this issue here https://bugzilla.redhat.com/show_bug.cgi?id=1020352#c2, 

I tried the same setup, this time files got synced properly, but the few symlinks failed to sync to 2 imaster ( intermediate master) in first xsync crawl.
Comment 13 Aravinda VK 2014-12-24 02:46:13 EST
Moving priority to medium as it is a cascaded fan out setup.
Comment 15 Aravinda VK 2015-11-25 03:49:39 EST
Closing this bug since RHGS 2.1 release reached EOL. Required bugs are cloned to RHGS 3.1. Please re-open this issue if found again.
Comment 16 Aravinda VK 2015-11-25 03:51:18 EST
Closing this bug since RHGS 2.1 release reached EOL. Required bugs are cloned to RHGS 3.1. Please re-open this issue if found again.

Note You need to log in before you can comment on or make changes to this bug.