Description of problem:

We are considering a distribute of 3 bricks - b1, b2, b3.

Case 1:
=======

Operation: rename (src, dst) - dst does not exist

T0: rename successful on Hashed subvol but not on other bricks
T1: Snapshot on b1, b2, b3

Result: After snapshot is restored and healing is complete on src, dst we end up with two directories src and dst having gfid of src

Case 2:
=======

Operation: Two parallel rename (src, dst) and rename (dst, src). Both src and dst exist and hash to b1 and b2 respectively

T0: rename (src, dst) successful on b1
T1: rename (dst, src) successful on b2
T3: Snapshot on b1, b2, b3

Result:
After restore, if lookup happens on src and is healed to b1 from b2, gfids of src on each brick will be,
b1 - (src, dst-gfid)
b2 - (src, dst-gfid)
b3 - (src, src-gfid)

Case 3:
=======

Operation: Parallel rename and two mkdirs. Only src exists. Both hash to same brick b1.

T0: two lookups triggered as part of application mkdir1 and mkdir2 complete with ENOENT.
T1: mkdir2 goes ahead and creates directory with gfid, gfid1
T2: rename (src, dst) on b1
T3: mkdir1 (src) on b1
T4: snapshot on b1, b2 and b3

Result:
After restore and healing of src and dst, we end up with,
b1 - (src, gfid2) and (dst, gfid1)
b2 - (src, gfid1) and (dst, gfid1)
b3 - (src, gfid1) and (dst, gfid1)

Another reason for this inconsistency is that dht don't consider mkdir failures with EEXIST on subvols as failures. More details can be found in [2].

Case 4:
=======


Operation: Parallel rename (src, dst) and rmdir (src). Both src and dst exist with gfids gfid1 and gfid2 respectively

T0: rename (src, dst) on b1
T1: rmdir (src) on b2 and b3
T2: snapshot on b1, b2 and b3

Result: After restore and healing,
b1 - (dst, gfid1)
b2 - (dst, gfid2)
b3 - (dst, gfid2)

case 5:
=======

This bug was hit and fix being reviewed at [1]

Operation: Parallel two rmdir and two mkdirs. Directory dir does not exist to start with.

T0: two lookups triggered as part of application mkdir1 and mkdir2 complete with ENOENT.
T1: mkdir2 goes ahead and creates directory with gfid, gfid1
T2: rmdir1 (dir) on b1
T3: lookup (dir) triggered as part of rmdir2 (or any name based opeartion), heals dir on b1 with gfid, gfid2
T4: mkdir1 (dir, gfid2) on b2 and b3
T5: snapshots on b1, b2 and b3

Result:
b1 - (dir, gfid1)
b2 - (dir, gfid2)
b3 - (dir, gfid2)

Considering all these issues, following set of fixes have been proposed:

1. in posix, if we receive mkdir (dir1) on an existing gfid (with name dir2), posix will convert mkdir (dir1) into rename (dir1, dir2). This solves case 1

2. in case of rename (src, dst), if dst already exists, rmdir (dst), so that we don't bring in inconsistency into dst gfid space. This solves all the cases of inconsistencies in dst gfid with rename failing.

3. hold entrylks in directory heal (part of lookup) and rmdir. This solves consistency issues because of races b/w mkdir and rmdir [1].

[1] http://review.gluster.org/#/c/4846/
[2] http://review.gluster.org/4459


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: