+++ This bug was initially created as a clone of Bug #860208 +++ Description of problem: In Self heal , if there is some gfid mismatch or entrylk couldn't be acquired or etc , it is returning NULL gfid in struct iatt of lookup_cbk along with op_ret = 0 Client logs where errors occured. ########################################## ----/var/log/glusterfs/mnt-client4.log---- [2012-09-10 16:53:09.182425] I [afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-m\ aster-replicate-0: Non blocking entrylks failed. [2012-09-10 16:53:09.182463] E [afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-master-replica\ te-0: background meta-data data entry missing-entry gfid self-heal failed on /linux5/linux-2.6.39/arch/cr\ is/include/asm/uaccess.h [2012-09-10 16:53:09.182678] W [inode.c:914:inode_lookup] (-->/usr/lib64/glusterfs/3.3.0rhs/xlator/debug/i\ o-stats.so(io_stats_lookup_cbk+0xff) [0x7f438a408a9f] (-->/usr/lib64/glusterfs/3.3.0rhs/xlator/mount/fuse.\ so(+0xf098) [0x7f438e399098] (-->/usr/lib64/glusterfs/3.3.0rhs/xlator/mount/fuse.so(+0xeefb) [0x7f438e398e\ fb]))) 0-fuse: inode not found ################################################ ################################################ ----/var/log/glusterfs/mnt/client-5.log---- [2012-09-13 10:19:55.691810] W [afr-common.c:1196:afr_detect_self_heal_by_iatt] 0-master-replicate-0: /lin\ ux2/linux-2.6.39/drivers/media/video/ivtv/ivtv-firmware.h: gfid different on subvolume [2012-09-13 10:19:55.692045] I [afr-self-heal-common.c:1970:afr_sh_post_nb_entrylk_gfid_sh_cbk] 0-master-r\ eplicate-0: Non blocking entrylks failed. [2012-09-13 10:19:55.692245] W [inode.c:914:inode_lookup] (-->/usr/lib64/glusterfs/3.3.0rhs/xlator/debug/i\ o-stats.so(io_stats_lookup_cbk+0xff) [0x7f9cb6caca9f] (-->/usr/lib64/glusterfs/3.3.0rhs/xlator/mount/fuse.\ so(+0xf098) [0x7f9cbed71098] (-->/usr/lib64/glusterfs/3.3.0rhs/xlator/mount/fuse.so(+0xeefb) [0x7f9cbed70e\ fb]))) 0-fuse: inode not found ----end---- ##################################### Version-Release number of selected component (if applicable):RHS-2.0 How reproducible:Consistent. Steps to Reproduce: 1.Create dist-rep volume 2.Run kernel untar on some clients parallely, Eg: while : ; do mkdir client1 ; tar -xvf linux-2.6.39.tar.gz -C client1 ; rm -rf client1; done Like this run on multiple fuse clients parallely. 3. On one of the other client run fs_mark to create file with -F option in loop. Eg: while :; do rm -rvf $MNT_PT/*; ./fs_mark -s 100000 -D 1000 -N 1000 -L 10 -n 10 -t 50 -F -d $MNT_PT; rm -rvf $MNT_PT/*; done 4: These might hit the problem if we run it for some 7 days or more. Actual results: Expected results: Additional info:
http://review.gluster.org/4625 http://review.gluster.org/4626 After the above patches are taken in we need to handle gfid self-heal failure because of failure in entrylk
REVIEW: http://review.gluster.org/5173 (cluster/afr: detect in-progress creation in lookup and return ENOENT) posted (#1) for review on release-3.3 by Pranith Kumar Karampuri (pkarampu)
COMMIT: http://review.gluster.org/5173 committed in release-3.3 by Vijay Bellur (vbellur) ------ commit 3521020078e822d74eed235074dc978981fb56ed Author: Pranith Kumar K <pkarampu> Date: Thu Jun 6 11:25:07 2013 +0530 cluster/afr: detect in-progress creation in lookup and return ENOENT Port of http://review.gluster.org/4625 if any subvol returned ENOENT while parent entrylk lock was held, yield and return ENOENT for the entire lookup. This is how the issue happens: Multiple clients A, B and C are attempting 'mkdir -p /mnt/a/b/c' 1 Client A is in the middle of mkdir(/a). It has acquired lock. It has performed mkdir(/a) on one subvol, and second one is still in progress 2 Client B performs a lookup, sees directory /a on one, ENOENT on the other, succeeds lookup. 3 Client B performs lookup on /a/b on both subvols, both return ENOENT (one subvol because /a/b does not exist, another because /a itself does not exist) 4 Client B proceeds to mkdir /a/b. It obtains entrylk on inode=/a with basename=b on one subvol, but fails on other subvol as /a is yet to be created by Client A. 5 Client A finishes mkdir of /a on other subvol 6 Client C also attempts to create /a/b, lookup returns ENOENT on both subvols. 7 Client C tries to obtain entrylk on on inode=/a with basename=b, obtains on one subvol (where B had failed), and waits for B to unlock on other subvol. 8 Client B finishes mkdir() on one subvol with GFID-1 and completes transaction and unlocks 9 Client C gets the lock on the second subvol, At this stage second subvol already has /a/b created from Client B, but Client C does not check that in the middle of mkdir transaction 10 Client C attempts mkdir /a/b on both subvols. It succeeds on ONLY ONE (where Client B could not get lock because of missing parent /a dir) with GFID-2, and gets EEXIST from ONE subvol. This way we have /a/b in GFID mismatch. One subvol got GFID-1 because Client B performed transaction on only one subvol (because entrylk() could not be obtained on second subvol because of missing parent dir -- caused by premature/speculative succeeding of lookup() on /a when locks are detected). Other subvol gets GFID-2 from Client C because while it was waiting for entrylk() on both subvols, Client B was in the middle of creating mkdir() on only one subvol, and Client C does not "expect" this when it is between lock() and pre-op()/op() phase of the transaction. Change-Id: I40107d4638ffdcb7b1ff4748c8e5ea92e62697e8 BUG: 860210 Signed-off-by: Pranith Kumar K <pkarampu> Reviewed-on: http://review.gluster.org/5173 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>