Bug 1337394

+++ This bug was initially created as a clone of Bug #1336698 +++

+++ This bug was initially created as a clone of Bug #1118762 +++

Description of problem:
=======================
try to rename Directories from multiple client at same time. e.g. from one mount give mv <src> <dest> and from another mv <src> <dest1> (where dest and dest1 is not hashing to same) or simply from one mount point mv <src> <dest> and from another mount point mv <src> <dest> .

If they are not done one after another, then due to race after rename is finished :-
- same gfid for different Directories (at same level and/or diffrent level)
- files inside those directories are not listed on mount and sometime even Directory is not listed (so unable to access data)
- sometimes ls output shows question mark in attribute filed.


Version-Release number :
=========================
3.6.0.24-1.el6rhs.x86_64


How reproducible:
=================
Intermittent 


Steps to Reproduce:
====================
1. create and mount distributed volume. (mount on multiple client)
2. create few files and Directories on mount point.
[root@OVM5 race]# mkdir src dest 
[root@OVM5 race]# touch src/f{1..5}

[to reproduce race, we are putting breakpoint at dht_rename_hashed_dir_cbk for both rename operation]

3. from one mount point execute 'mv src dest' and from other mpount point execute 'mv dest src'
Now continue execution from break point from both mount point.

4. verify data from another mount and bricks also

mount:-

[root@OVM5 race]# ls -l 
total 0 
drwxr-xr-x 3 root root 69 Jul 10 12:32 src    <---only src present

[root@OVM5 race]# mkdir dest 
mkdir: cannot create directory `dest': File exists    <--------
[root@OVM5 race]# ls -l                              <------- dest is not shown
total 0 
drwxr-xr-x 3 root root 69 Jul 10 12:32 src 
[root@OVM5 race]# touch src/f1 
touch: cannot touch `src/f1': Input/output error 
[root@OVM5 race]# 
[root@OVM5 race]# ls -l src     <-------f1, f2 and f5 are not listed
total 0 
drwxr-xr-x 3 root root 38 Jul 10 12:32 dest   <--- directory having same name
drwxr-xr-x 3 root root 38 Jul 10 12:32 dest 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f3 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f4 


another mount point:-
[root@OVM5 race]# ls -lR 
.: 
total 0 
drwxr-xr-x 3 root root 69 Jul 10 12:32 src 

./src: 
total 0 
drwxr-xr-x 3 root root 70 Jul 10 12:32 dest 
drwxr-xr-x 3 root root 70 Jul 10 12:32 dest 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f3 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f4 

./src/dest: 
ls: cannot access ./src/dest/src: No such file or directory 
ls: cannot access ./src/dest/src: No such file or directory 
total 0 
?????????? ? ? ? ?            ? src 
?????????? ? ? ? ?            ? src 

./src/dest: 
ls: cannot access ./src/dest/src: No such file or directory 
ls: cannot access ./src/dest/src: No such file or directory 
total 0 
?????????? ? ? ? ?            ? src        <------------------ ?? in attribute
?????????? ? ? ? ?            ? src 

bricks :-

[root@OVM5 race]# tree /brick*/race/ 
/brick1/race/ 
├── dest 
│   └── src 
│       ├── f1 
│       ├── f2 
│       └── f5 
└── src 
    └── dest 
/brick2/race/ 
└── src 
    ├── dest 
    │   └── src 
    └── f4 
/brick3/race/ 
├── dest 
└── src 
    ├── dest 
    │   └── src 
    └── f3 

11 directories, 5 files 


Actual results:
===============
- same gfid for different Directories (at same level and/or different level)
- files inside those directories are not listed on mount and sometime even Directory is not listed (so unable to access data)
- sometimes ls output shows question mark in attribute filed.


Expected results:
=================
- no two directory should have same gfid
- if both renames are fail/successful or one succeed and other fail; regardless of all these - all files inside those Directories should be accessible from mount point

--- Additional comment from Rachana Patel on 2014-07-11 09:27:34 EDT ---

[root@OVM3 ~]# gluster v info race
 
Volume Name: race
Type: Distribute
Volume ID: 30f7ff59-a90b-44e3-a991-223c81c15d67
Status: Started
Snap Volume: no
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: 10.70.35.172:/brick1/race
Brick2: 10.70.35.172:/brick2/race
Brick3: 10.70.35.172:/brick3/race
Options Reconfigured:
performance.readdir-ahead: on
snap-max-hard-limit: 256
snap-max-soft-limit: 90
auto-delete: disable

--- Additional comment from Vijay Bellur on 2016-05-17 04:52:04 EDT ---

REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent directory if destination exists) posted (#2) for review on master by Sakshi Bansal

--- Additional comment from Vijay Bellur on 2016-05-17 05:32:25 EDT ---

REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent directory if destination exists) posted (#3) for review on master by Sakshi Bansal

--- Additional comment from Vijay Bellur on 2016-05-18 05:59:28 EDT ---

COMMIT: http://review.gluster.org/14371 committed in master by Raghavendra G (rgowdapp) 
------
commit 6a6b953d4f999b6db11cc6b62880e810aee9a4c7
Author: Sakshi Bansal <sabansal>
Date:   Tue May 17 13:23:28 2016 +0530

    dht: rename takes lock on parent directory if destination exists
    
    For directory rename if destination exists the source directory
    is created as a child of the given destination directory. Since
    the new child directory does not exist take lock on parent of the
    child directory.
    
    Change-Id: I24a34605a2cd65984910643ff5462f35e8fc7e71
    BUG: 1336698
    Signed-off-by: Sakshi Bansal <sabansal>
    Reviewed-on: http://review.gluster.org/14371
    Smoke: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>