Red Hat Bugzilla – Bug 1003580
Dist-geo-rep : geo-rep initial xsync crawl failed to sync few symlinks to slave in a cascaded-fanout setup.
Last modified: 2015-11-25 03:51:18 EST
Description of problem: geo-rep initial xsync crawl failed to get the entry in the XSYNC-CHANGELOG for few symlinks to files. The symlinks which were missed, had no entry in the XSYNC-CHANGELOG. Version-Release number of selected component (if applicable):glusterfs-3.4.0.30rhs-2.el6rhs.x86_64 How reproducible: Didn't try to reproduce it. Steps to Reproduce: 1.create and start a geo-rep cascaded-fanout setup ( 1-4-4)( totally 21 volumes involved , 1 master , 4imasters , and 16slaves) 2.stop all the geo-rep sessions . 3.create data on the master. After the completion of the data creation , start geo-rep between master and imasters. let them sync. 4. After the completion of sync, start geo-rep sessions between imasters and slaves and let files sync slaves. 5. Now stop all the geo-rep sessions. 6. Create symlinks to all the files. 7. start the geo-rep sessions between master and imasters. 8. Check if all the symlinks get synced to all he imasters. Actual results:It fails to sync few symlinks to one imaster and reason being no entry for the missing files in XSYNC-CHANGELOG Expected results:It should sync all symlinks to imaster and consequently all the slaves. Additional info:
This happened in plain fanout setup too in the build glusterfs-3.4.0.30rhs-2.el6rhs.x86_64, it is consistently happening.
Fix for bug 1001684 @ https://code.engineering.redhat.com/gerrit/#/c/12111/ should be fixing it. Will run some tests as the patch is now in for fixing that bug, and move the status of the accordingly
considering the patch made it to glusterfs-3.4.0.31rhs-1, can we get it tested again once more?
The issue described is about not getting the entry in XSYNC-CHANGELOG in the first xsync crawl. I might have done mistake in investigating it properly, because after this bug filed, issue related to symlink syncing in xsync crawl as well as in changelog syncing was hit with only fanout and cascaded setups. I was able to hit it in the build glusterfs-3.4.0.31rhs-1, in a fanout setup , and while syncing symlinks in first xsync crawl. steps I followed 1. create and start a geo-rep fanout setup from a master to 4 slaves. 2. stop all geo-rep sessions and create files on master. 3. start all the geo-rep sessions and let files to sync to all the slaves. 4. Now stop all geo-rep sessions, and create symlinks to all those files. 5. Start the geo-rep session. 6. First xsync crawl fails to sync around 16 to 20 files out of 2000 files, to all the slaves. on master missing file >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [root@Ray ~]# getfattr -h -n glusterfs.gfid.string /mnt/master/level05/level15/symlink_to_files/5229cf3e~~4CA6FC77NJ getfattr: Removing leading '/' from absolute path names # file: mnt/master/level05/level15/symlink_to_files/5229cf3e~~4CA6FC77NJ glusterfs.gfid.string="0854f9ef-aa93-462a-aa02-10e9d510f34d" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On master, the missing file had entries in the XSYNC-CHANGELOG >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [root@Ray bd42ad17ef8864d51407b1c6478f5dc6]# grep 0854f9ef-aa93-462a-aa02-10e9d510f34d xsync/* xsync/XSYNC-CHANGELOG.1378471766:E 0854f9ef-aa93-462a-aa02-10e9d510f34d SYMLINK fddd8985-0185-4655-a6fd-565b7c5a54ea%2F5229cf3e%7E%7E4CA6FC77NJ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the slave for same GFID, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2013-09-06 12:49:35.192857] W [dht-layout.c:179:dht_layout_search] 0-imaster1-dht: no subvolume for hash (value) = 3927056761 [2013-09-06 12:49:35.192898] I [fuse-bridge.c:3515:fuse_auxgfid_newentry_cbk] 0-fuse-aux-gfid-mount: failed to create the entry <gfid:fddd8985-0185-4655-a6fd-565b7c5a54ea>/5229cf3e~~4CA6FC77NJ with gfid (0854f9ef-aa93-462a-aa02-10e9d510f34d): No such file or directory [2013-09-06 12:49:35.192930] W [fuse-bridge.c:1627:fuse_err_cbk] 0-glusterfs-fuse: 726: SYMLINK() <gfid:fddd8985-0185-4655-a6fd-565b7c5a54ea>/5229cf3e~~4CA6FC77NJ => -1 (No such file or directory) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Similar kind of issue was hit cascaded setup though changelog syncing too in the build glusterfs-3.4.0.32rhs-1.el6rhs.x86_64 . But fortunately or unfortunately it was never hit in plain single master and single slave setup.
https://code.engineering.redhat.com/gerrit/#/c/12602/
Fixed in version please.
I tried on the build glusterfs-3.4.0.35rhs, I have explained most of the things related to this issue here https://bugzilla.redhat.com/show_bug.cgi?id=1020352#c2, I tried the same setup, this time files got synced properly, but the few symlinks failed to sync to 2 imaster ( intermediate master) in first xsync crawl.
Moving priority to medium as it is a cascaded fan out setup.
Closing this bug since RHGS 2.1 release reached EOL. Required bugs are cloned to RHGS 3.1. Please re-open this issue if found again.