Bug 1337022

+++ This bug was initially created as a clone of Bug #1336698 +++

+++ This bug was initially created as a clone of Bug #1118762 +++

Description of problem:
=======================
try to rename Directories from multiple client at same time. e.g. from one mount give mv <src> <dest> and from another mv <src> <dest1> (where dest and dest1 is not hashing to same) or simply from one mount point mv <src> <dest> and from another mount point mv <src> <dest> .

If they are not done one after another, then due to race after rename is finished :-
- same gfid for different Directories (at same level and/or diffrent level)
- files inside those directories are not listed on mount and sometime even Directory is not listed (so unable to access data)
- sometimes ls output shows question mark in attribute filed.


Version-Release number :
=========================
3.6.0.24-1.el6rhs.x86_64


How reproducible:
=================
Intermittent 


Steps to Reproduce:
====================
1. create and mount distributed volume. (mount on multiple client)
2. create few files and Directories on mount point.
[root@OVM5 race]# mkdir src dest 
[root@OVM5 race]# touch src/f{1..5}

[to reproduce race, we are putting breakpoint at dht_rename_hashed_dir_cbk for both rename operation]

3. from one mount point execute 'mv src dest' and from other mpount point execute 'mv dest src'
Now continue execution from break point from both mount point.

4. verify data from another mount and bricks also

mount:-

[root@OVM5 race]# ls -l 
total 0 
drwxr-xr-x 3 root root 69 Jul 10 12:32 src    <---only src present

[root@OVM5 race]# mkdir dest 
mkdir: cannot create directory `dest': File exists    <--------
[root@OVM5 race]# ls -l                              <------- dest is not shown
total 0 
drwxr-xr-x 3 root root 69 Jul 10 12:32 src 
[root@OVM5 race]# touch src/f1 
touch: cannot touch `src/f1': Input/output error 
[root@OVM5 race]# 
[root@OVM5 race]# ls -l src     <-------f1, f2 and f5 are not listed
total 0 
drwxr-xr-x 3 root root 38 Jul 10 12:32 dest   <--- directory having same name
drwxr-xr-x 3 root root 38 Jul 10 12:32 dest 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f3 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f4 


another mount point:-
[root@OVM5 race]# ls -lR 
.: 
total 0 
drwxr-xr-x 3 root root 69 Jul 10 12:32 src 

./src: 
total 0 
drwxr-xr-x 3 root root 70 Jul 10 12:32 dest 
drwxr-xr-x 3 root root 70 Jul 10 12:32 dest 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f3 
-rw-r--r-- 1 root root  0 Jul 10 12:32 f4 

./src/dest: 
ls: cannot access ./src/dest/src: No such file or directory 
ls: cannot access ./src/dest/src: No such file or directory 
total 0 
?????????? ? ? ? ?            ? src 
?????????? ? ? ? ?            ? src 

./src/dest: 
ls: cannot access ./src/dest/src: No such file or directory 
ls: cannot access ./src/dest/src: No such file or directory 
total 0 
?????????? ? ? ? ?            ? src        <------------------ ?? in attribute
?????????? ? ? ? ?            ? src 

bricks :-

[root@OVM5 race]# tree /brick*/race/ 
/brick1/race/ 
├── dest 
│   └── src 
│       ├── f1 
│       ├── f2 
│       └── f5 
└── src 
    └── dest 
/brick2/race/ 
└── src 
    ├── dest 
    │   └── src 
    └── f4 
/brick3/race/ 
├── dest 
└── src 
    ├── dest 
    │   └── src 
    └── f3 

11 directories, 5 files 


Actual results:
===============
- same gfid for different Directories (at same level and/or different level)
- files inside those directories are not listed on mount and sometime even Directory is not listed (so unable to access data)
- sometimes ls output shows question mark in attribute filed.


Expected results:
=================
- no two directory should have same gfid
- if both renames are fail/successful or one succeed and other fail; regardless of all these - all files inside those Directories should be accessible from mount point

--- Additional comment from Rachana Patel on 2014-07-11 09:27:34 EDT ---

[root@OVM3 ~]# gluster v info race
 
Volume Name: race
Type: Distribute
Volume ID: 30f7ff59-a90b-44e3-a991-223c81c15d67
Status: Started
Snap Volume: no
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: 10.70.35.172:/brick1/race
Brick2: 10.70.35.172:/brick2/race
Brick3: 10.70.35.172:/brick3/race
Options Reconfigured:
performance.readdir-ahead: on
snap-max-hard-limit: 256
snap-max-soft-limit: 90
auto-delete: disable

--- Additional comment from Vijay Bellur on 2016-05-17 04:52:04 EDT ---

REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent directory if destination exists) posted (#2) for review on master by Sakshi Bansal

--- Additional comment from Vijay Bellur on 2016-05-17 05:32:25 EDT ---

REVIEW: http://review.gluster.org/14371 (dht: rename takes lock on parent directory if destination exists) posted (#3) for review on master by Sakshi Bansal