Bug 1003580
| Summary: | Dist-geo-rep : geo-rep initial xsync crawl failed to sync few symlinks to slave in a cascaded-fanout setup. | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Vijaykumar Koppad <vkoppad> |
| Component: | geo-replication | Assignee: | Bug Updates Notification Mailing List <rhs-bugs> |
| Status: | CLOSED EOL | QA Contact: | amainkar |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 2.1 | CC: | avishwan, chrisw, csaba, david.macdonald, grajaiya, nsathyan, rhs-bugs, rwheeler, sdharane, vagarwal |
| Target Milestone: | --- | Keywords: | ZStream |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | fanout | ||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-11-25 08:49:39 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Vijaykumar Koppad
2013-09-02 12:22:49 UTC
This happened in plain fanout setup too in the build glusterfs-3.4.0.30rhs-2.el6rhs.x86_64, it is consistently happening. Fix for bug 1001684 @ https://code.engineering.redhat.com/gerrit/#/c/12111/ should be fixing it. Will run some tests as the patch is now in for fixing that bug, and move the status of the accordingly considering the patch made it to glusterfs-3.4.0.31rhs-1, can we get it tested again once more? The issue described is about not getting the entry in XSYNC-CHANGELOG in the first xsync crawl. I might have done mistake in investigating it properly, because after this bug filed, issue related to symlink syncing in xsync crawl as well as in changelog syncing was hit with only fanout and cascaded setups. I was able to hit it in the build glusterfs-3.4.0.31rhs-1, in a fanout setup , and while syncing symlinks in first xsync crawl. steps I followed 1. create and start a geo-rep fanout setup from a master to 4 slaves. 2. stop all geo-rep sessions and create files on master. 3. start all the geo-rep sessions and let files to sync to all the slaves. 4. Now stop all geo-rep sessions, and create symlinks to all those files. 5. Start the geo-rep session. 6. First xsync crawl fails to sync around 16 to 20 files out of 2000 files, to all the slaves. on master missing file >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [root@Ray ~]# getfattr -h -n glusterfs.gfid.string /mnt/master/level05/level15/symlink_to_files/5229cf3e~~4CA6FC77NJ getfattr: Removing leading '/' from absolute path names # file: mnt/master/level05/level15/symlink_to_files/5229cf3e~~4CA6FC77NJ glusterfs.gfid.string="0854f9ef-aa93-462a-aa02-10e9d510f34d" >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On master, the missing file had entries in the XSYNC-CHANGELOG >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [root@Ray bd42ad17ef8864d51407b1c6478f5dc6]# grep 0854f9ef-aa93-462a-aa02-10e9d510f34d xsync/* xsync/XSYNC-CHANGELOG.1378471766:E 0854f9ef-aa93-462a-aa02-10e9d510f34d SYMLINK fddd8985-0185-4655-a6fd-565b7c5a54ea%2F5229cf3e%7E%7E4CA6FC77NJ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on the slave for same GFID, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2013-09-06 12:49:35.192857] W [dht-layout.c:179:dht_layout_search] 0-imaster1-dht: no subvolume for hash (value) = 3927056761 [2013-09-06 12:49:35.192898] I [fuse-bridge.c:3515:fuse_auxgfid_newentry_cbk] 0-fuse-aux-gfid-mount: failed to create the entry <gfid:fddd8985-0185-4655-a6fd-565b7c5a54ea>/5229cf3e~~4CA6FC77NJ with gfid (0854f9ef-aa93-462a-aa02-10e9d510f34d): No such file or directory [2013-09-06 12:49:35.192930] W [fuse-bridge.c:1627:fuse_err_cbk] 0-glusterfs-fuse: 726: SYMLINK() <gfid:fddd8985-0185-4655-a6fd-565b7c5a54ea>/5229cf3e~~4CA6FC77NJ => -1 (No such file or directory) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Similar kind of issue was hit cascaded setup though changelog syncing too in the build glusterfs-3.4.0.32rhs-1.el6rhs.x86_64 . But fortunately or unfortunately it was never hit in plain single master and single slave setup. Fixed in version please. I tried on the build glusterfs-3.4.0.35rhs, I have explained most of the things related to this issue here https://bugzilla.redhat.com/show_bug.cgi?id=1020352#c2, I tried the same setup, this time files got synced properly, but the few symlinks failed to sync to 2 imaster ( intermediate master) in first xsync crawl. Moving priority to medium as it is a cascaded fan out setup. Closing this bug since RHGS 2.1 release reached EOL. Required bugs are cloned to RHGS 3.1. Please re-open this issue if found again. Closing this bug since RHGS 2.1 release reached EOL. Required bugs are cloned to RHGS 3.1. Please re-open this issue if found again. |