Bug 1337394 - DHT : few Files are not accessible and not listed on mount + more than one Directory have same gfid + (sometimes) attributes has ?? in ls output after renaming Directories from multiple client at same time
Summary: DHT : few Files are not accessible and not listed on mount + more than one Di...
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: distribute
Version: 3.8.0
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Sakshi
QA Contact:
URL:
Whiteboard: dht-gfid-dir,dht-file-access
Keywords:
Depends On: 1118762 1118770 1336698
Blocks: 1299184 1337022
TreeView+ depends on / blocked
 
Reported: 2016-05-19 06:11 UTC by Sakshi
Modified: 2016-08-01 01:22 UTC (History)
11 users (show)

(edit)
Clone Of: 1336698
(edit)
Last Closed: 2016-06-16 14:07:31 UTC


Attachments (Terms of Use)

Description Sakshi 2016-05-19 06:11:06 UTC
+++ This bug was initially created as a clone of Bug #1336698 +++

+++ This bug was initially created as a clone of Bug #1118762 +++

Description of problem:
=======================
try to rename Directories from multiple client at same time. e.g. from one mount give mv <src> <dest> and from another mv <src> <dest1> (where dest and dest1 is not hashing to same) or simply from one mount point mv <src> <dest> and from another mount point mv <src> <dest> .

If they are not done one after another, then due to race after rename is finished :-
- same gfid for different Directories (at same level and/or diffrent level)
- files inside those directories are not listed on mount and sometime even Directory is not listed (so unable to access data)
- sometimes ls output shows question mark in attribute filed.


Version-Release number :
=========================
3.6.0.24-1.el6rhs.x86_64


How reproducible:
=================
Intermittent 


Steps to Reproduce:
====================
1. create and mount distributed volume. (mount on multiple client)
2. create few files and Directories on mount point.
[root@OVM5 race]# mkdir src dest 
[root@OVM5 race]# touch src/f{1..5}

[to reproduce race, we are putting breakpoint at dht_rename_hashed_dir_cbk for both rename operation]

3. from one mount point execute 'mv src dest' and from other mpount point execute 'mv dest src'
Now continue execution from break point from both mount point.

4. verify data from another mount and bricks also

mount:-

[root@OVM5 race]# ls -l 
total 0 
drwxr-xr-x 3 root root 69 Jul 10 12:32 src    <---only src present

[root@OVM5 race]# mkdir dest 
mkdir: cannot create directory `dest': File exists    <--------
[root@OVM5 race]# ls -l                              <------- dest is not shown
total 0 
drwxr-xr-x 3 root root 69 Jul 10 12:32 src 
[root@OVM5 race]# touch src/f1 
touch: cannot touch `src/f1': Input/output error 
[root@OVM5 race]# 
[root@OVM5 race]# ls -l src     <-------f1, f2 and f5 are not listed
total 0 
drwxr-xr-x 3 root root 38 Jul 10 12:32 dest   <--- directory having same name
drwxr-xr-x 3 root root 38 Jul 10 12:32 dest 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f3 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f4 


another mount point:-
[root@OVM5 race]# ls -lR 
.: 
total 0 
drwxr-xr-x 3 root root 69 Jul 10 12:32 src 

./src: 
total 0 
drwxr-xr-x 3 root root 70 Jul 10 12:32 dest 
drwxr-xr-x 3 root root 70 Jul 10 12:32 dest 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f3 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f4 

./src/dest: 
ls: cannot access ./src/dest/src: No such file or directory 
ls: cannot access ./src/dest/src: No such file or directory 
total 0 
?????????? ? ? ? ?            ? src 
?????????? ? ? ? ?            ? src 

./src/dest: 
ls: cannot access ./src/dest/src: No such file or directory 
ls: cannot access ./src/dest/src: No such file or directory 
total 0 
?????????? ? ? ? ?            ? src        <------------------ ?? in attribute
?????????? ? ? ? ?            ? src 

bricks :-

[root@OVM5 race]# tree /brick*/race/ 
/brick1/race/ 
├── dest 
│   └── src 
│       ├── f1 
│       ├── f2 
│       └── f5 
└── src 
    └── dest 
/brick2/race/ 
└── src 
    ├── dest 
    │   └── src 
    └── f4 
/brick3/race/ 
├── dest 
└── src 
    ├── dest 
    │   └── src 
    └── f3 

11 directories, 5 files 


Actual results:
===============
- same gfid for different Directories (at same level and/or different level)
- files inside those directories are not listed on mount and sometime even Directory is not listed (so unable to access data)
- sometimes ls output shows question mark in attribute filed.


Expected results:
=================
- no two directory should have same gfid
- if both renames are fail/successful or one succeed and other fail; regardless of all these - all files inside those Directories should be accessible from mount point

--- Additional comment from Rachana Patel on 2014-07-11 09:27:34 EDT ---

[root@OVM3 ~]# gluster v info race
 
Volume Name: race
Type: Distribute
Volume ID: 30f7ff59-a90b-44e3-a991-223c81c15d67
Status: Started
Snap Volume: no
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: 10.70.35.172:/brick1/race
Brick2: 10.70.35.172:/brick2/race
Brick3: 10.70.35.172:/brick3/race
Options Reconfigured:
performance.readdir-ahead: on
snap-max-hard-limit: 256
snap-max-soft-limit: 90
auto-delete: disable

--- Additional comment from Vijay Bellur on 2016-05-17 04:52:04 EDT ---

REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent directory if destination exists) posted (#2) for review on master by Sakshi Bansal

--- Additional comment from Vijay Bellur on 2016-05-17 05:32:25 EDT ---

REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent directory if destination exists) posted (#3) for review on master by Sakshi Bansal

--- Additional comment from Vijay Bellur on 2016-05-18 05:59:28 EDT ---

COMMIT: http://review.gluster.org/14371 committed in master by Raghavendra G (rgowdapp@redhat.com) 
------
commit 6a6b953d4f999b6db11cc6b62880e810aee9a4c7
Author: Sakshi Bansal <sabansal@redhat.com>
Date:   Tue May 17 13:23:28 2016 +0530

    dht: rename takes lock on parent directory if destination exists
    
    For directory rename if destination exists the source directory
    is created as a child of the given destination directory. Since
    the new child directory does not exist take lock on parent of the
    child directory.
    
    Change-Id: I24a34605a2cd65984910643ff5462f35e8fc7e71
    BUG: 1336698
    Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    Reviewed-on: http://review.gluster.org/14371
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>

Comment 1 Vijay Bellur 2016-05-19 06:16:16 UTC
REVIEW: http://review.gluster.org/14417 (dht: rename takes lock on parent directory if destination exists) posted (#1) for review on release-3.8 by Sakshi Bansal

Comment 2 Vijay Bellur 2016-05-20 10:54:59 UTC
COMMIT: http://review.gluster.org/14417 committed in release-3.8 by Raghavendra G (rgowdapp@redhat.com) 
------
commit b82fefed59cd3964c4d667b1645e70825be8e984
Author: Sakshi Bansal <sabansal@redhat.com>
Date:   Tue May 17 13:23:28 2016 +0530

    dht: rename takes lock on parent directory if destination exists
    
    For directory rename if destination exists the source directory
    is created as a child of the given destination directory. Since
    the new child directory does not exist take lock on parent of the
    child directory.
    
    Backport of http://review.gluster.org/#/c/14371/
    
    > Change-Id: I24a34605a2cd65984910643ff5462f35e8fc7e71
    > BUG: 1336698
    > Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    
    Change-Id: I24a34605a2cd65984910643ff5462f35e8fc7e71
    BUG: 1337394
    Signed-off-by: Sakshi Bansal <sabansal@redhat.com>
    Reviewed-on: http://review.gluster.org/14417
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    Reviewed-by: Raghavendra G <rgowdapp@redhat.com>

Comment 3 Niels de Vos 2016-06-16 14:07:31 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.