Bug 860210 - In self heal , if gfid mismatch , it return null gfid with op_ret=0.
In self heal , if gfid mismatch , it return null gfid with op_ret=0.
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: replicate (Show other bugs)
3.3.0
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Pranith Kumar K
:
Depends On: 860208
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-25 05:25 EDT by Vijaykumar Koppad
Modified: 2014-08-24 20:49 EDT (History)
7 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 860208
Environment:
Last Closed: 2013-07-24 13:20:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Vijaykumar Koppad 2012-09-25 05:25:01 EDT
+++ This bug was initially created as a clone of Bug #860208 +++

Description of problem:
In Self heal , if there is some gfid mismatch or entrylk couldn't be acquired or etc , it is returning NULL gfid in struct iatt of lookup_cbk along with 
op_ret = 0

Client logs where errors occured.

##########################################
----/var/log/glusterfs/mnt-client4.log----
[2012-09-10 16:53:09.182425] I 
[afr-self-heal-common.c:1941:afr_sh_post_nb_entrylk_conflicting_sh_cbk] 0-m\
aster-replicate-0: Non blocking entrylks failed.
[2012-09-10 16:53:09.182463] E 
[afr-self-heal-common.c:2160:afr_self_heal_completion_cbk] 0-master-replica\
te-0: background  meta-data data entry missing-entry gfid self-heal 
failed on /linux5/linux-2.6.39/arch/cr\
is/include/asm/uaccess.h
[2012-09-10 16:53:09.182678] W [inode.c:914:inode_lookup] 
(-->/usr/lib64/glusterfs/3.3.0rhs/xlator/debug/i\
o-stats.so(io_stats_lookup_cbk+0xff) [0x7f438a408a9f] 
(-->/usr/lib64/glusterfs/3.3.0rhs/xlator/mount/fuse.\
so(+0xf098) [0x7f438e399098] 
(-->/usr/lib64/glusterfs/3.3.0rhs/xlator/mount/fuse.so(+0xeefb) 
[0x7f438e398e\
fb]))) 0-fuse: inode not found
################################################

################################################
----/var/log/glusterfs/mnt/client-5.log----
[2012-09-13 10:19:55.691810] W 
[afr-common.c:1196:afr_detect_self_heal_by_iatt] 0-master-replicate-0: /lin\
ux2/linux-2.6.39/drivers/media/video/ivtv/ivtv-firmware.h: gfid 
different on subvolume
[2012-09-13 10:19:55.692045] I 
[afr-self-heal-common.c:1970:afr_sh_post_nb_entrylk_gfid_sh_cbk] 0-master-r\
eplicate-0: Non blocking entrylks failed.
[2012-09-13 10:19:55.692245] W [inode.c:914:inode_lookup] 
(-->/usr/lib64/glusterfs/3.3.0rhs/xlator/debug/i\
o-stats.so(io_stats_lookup_cbk+0xff) [0x7f9cb6caca9f] 
(-->/usr/lib64/glusterfs/3.3.0rhs/xlator/mount/fuse.\
so(+0xf098) [0x7f9cbed71098] 
(-->/usr/lib64/glusterfs/3.3.0rhs/xlator/mount/fuse.so(+0xeefb) 
[0x7f9cbed70e\
fb]))) 0-fuse: inode not found
----end----
#####################################


Version-Release number of selected component (if applicable):RHS-2.0 


How reproducible:Consistent. 


Steps to Reproduce:
1.Create dist-rep volume 
2.Run kernel untar on some clients parallely, 
   Eg: while : ; do mkdir client1 ; tar -xvf linux-2.6.39.tar.gz -C client1 ; rm -rf client1; done 
 Like this run on multiple fuse clients parallely. 
3. On one of the other client run fs_mark to create file with -F option in loop. 
   Eg: while :; do rm -rvf $MNT_PT/*; ./fs_mark -s 100000 -D 1000 -N 1000 -L 10        -n 10 -t 50 -F -d $MNT_PT; rm -rvf $MNT_PT/*; done 

4: These might hit the problem if we run it for some 7 days or more.
  
Actual results: 


Expected results:


Additional info:
Comment 1 Pranith Kumar K 2013-03-12 05:51:50 EDT
http://review.gluster.org/4625
http://review.gluster.org/4626

After the above patches are taken in we need to handle gfid self-heal failure because of failure in entrylk
Comment 2 Anand Avati 2013-06-06 02:29:08 EDT
REVIEW: http://review.gluster.org/5173 (cluster/afr: detect in-progress creation in lookup and return ENOENT) posted (#1) for review on release-3.3 by Pranith Kumar Karampuri (pkarampu@redhat.com)
Comment 3 Anand Avati 2013-06-19 01:49:37 EDT
COMMIT: http://review.gluster.org/5173 committed in release-3.3 by Vijay Bellur (vbellur@redhat.com) 
------
commit 3521020078e822d74eed235074dc978981fb56ed
Author: Pranith Kumar K <pkarampu@redhat.com>
Date:   Thu Jun 6 11:25:07 2013 +0530

    cluster/afr: detect in-progress creation in lookup and return ENOENT
    
            Port of http://review.gluster.org/4625
    
    if any subvol returned ENOENT while parent entrylk lock was held,
    yield and return ENOENT for the entire lookup.
    
    This is how the issue happens:
    
    Multiple clients A, B and C are attempting 'mkdir -p /mnt/a/b/c'
    
    1 Client A is in the middle of mkdir(/a). It has acquired lock.
      It has performed mkdir(/a) on one subvol, and second one is still
      in progress
    2 Client B performs a lookup, sees directory /a on one,
      ENOENT on the other, succeeds lookup.
    3 Client B performs lookup on /a/b on both subvols, both return ENOENT
      (one subvol because /a/b does not exist, another because /a
      itself does not exist)
    4 Client B proceeds to mkdir /a/b. It obtains entrylk on inode=/a with
      basename=b on one subvol, but fails on other subvol as /a is yet to
      be created by Client A.
    5 Client A finishes mkdir of /a on other subvol
    6 Client C also attempts to create /a/b, lookup returns ENOENT on
      both subvols.
    7 Client C tries to obtain entrylk on on inode=/a with basename=b,
      obtains on one subvol (where B had failed), and waits for B to unlock
      on other subvol.
    8 Client B finishes mkdir() on one subvol with GFID-1 and completes
      transaction and unlocks
    9 Client C gets the lock on the second subvol, At this stage second
      subvol already has /a/b created from Client B, but Client C does not
      check that in the middle of mkdir transaction
    10 Client C attempts mkdir /a/b on both subvols. It succeeds on
       ONLY ONE (where Client B could not get lock because of
       missing parent /a dir) with GFID-2, and gets EEXIST from ONE subvol.
    This way we have /a/b in GFID mismatch. One subvol got GFID-1 because
    Client B performed transaction on only one subvol (because entrylk()
    could not be obtained on second subvol because of missing parent dir --
    caused by premature/speculative succeeding of lookup() on /a when locks
    are detected). Other subvol gets GFID-2 from Client C because while
    it was waiting for entrylk() on both subvols, Client B was in the
    middle of creating mkdir() on only one subvol, and Client C does not
    "expect" this when it is between lock() and pre-op()/op() phase of the
    transaction.
    
    Change-Id: I40107d4638ffdcb7b1ff4748c8e5ea92e62697e8
    BUG: 860210
    Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
    Reviewed-on: http://review.gluster.org/5173
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Note You need to log in before you can comment on or make changes to this bug.