Bug 1161311

+++ This bug was initially created as a clone of Bug #1136714 +++

Description of problem:
=======================
Hard-links are missing after rebalance if hard-link is created while file migration is in progress

How reproducible:
=================
always

Steps to Reproduce:
==================

1. Created a 1GB file on the mount point.
2. Started rebalance force after adding brick [so that file will be migrated]
3. Created multiple hard links after this.
[root@vm100 mnt]# ll -h
total 954M
-rw-r--r--. 1 root root 954M Sep  1 23:39 file
[root@vm100 mnt]# ln file link
[root@vm100 mnt]# ls
file  link
[root@vm100 mnt]# ln file link2
[root@vm100 mnt]# ls
file  link  link2
[root@vm100 mnt]# ln file link3
[root@vm100 mnt]#


5. Waited for rebalance to complete.
[root@vm100 ~]# gluster v rebalance test1 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                1       953.7MB             1             0             0            completed              44.00
6. Checked file status on mount point.
[root@vm100 mnt]# ll -h
total 954M
-rw-r--r--. 1 root root 954M Sep  1 23:39 file
[root@vm100 mnt]#


Actual results:
===============
The hard links are gone as mentioned earlier. 


Expected results:
=================
hard link should be present

Additional info:
================

--- Additional comment from Rachana Patel on 2014-09-04 05:59:50 EDT ---

AS mentioned in Description above, If we create hard-link while file is in migration, hard-link will be deleted after migration.

With same steps mentioned above, if we unlink original file after creating hard link (while file is in migration), we will loose both files and after migration we will not have data on backend or mount.

Steps to Reproduce:
==================

1. Created a  file on the mount point.
2. Started rebalance force after adding brick [so that file will be migrated]
3. While file is in migration, Created multiple hard links after this.
[root@vm100 mnt]# ll -h
total 954M
-rw-r--r--. 1 root root 954M Sep  1 23:39 file
[root@vm100 mnt]# ln file link
[root@vm100 mnt]# ls
file  link
[root@vm100 mnt]# ln file link2
[root@vm100 mnt]# ls
file  link  link2
[root@vm100 mnt]# ln file link3
[root@vm100 mnt]# ls 
file  link  link2 link3

4. while file is in migration unlink original file as we have multiple hard link for the same.

[root@vm100 mnt]# unlink file

5.  Waited for rebalance to complete. check data on mount and bricks
[root@vm100 mnt]# ls
[root@vm100 mnt]# 


Actual results:
===============
No files are present on mount or bricks


Expected results:
=================
There should not be any data/file loss If user has deleted orginal file after creating hard link for the same

--- Additional comment from Shyamsundar on 2014-11-06 16:07:01 EST ---

Reasons for this happening:
---------------------------

1) dht_link would create a link to the cached file and a linkto at the hashed location

2) When file is under migration, if (1) happens, then we create a hard link to the cached (which is under migration) and create a linkto on the subvol that the new name hashes to.

3) If new name hashes to the same subvol as the old name, then the file survives, as the linkto file, for new name, on the hashed is a hard link to the linkto file for old name

4) If new name hashed to a different subvol, then the file does not survive. As the cached is on the subvol that is migrating and hence when migration is over that file is truncated and made 0, so it retains the P2 state old file (as the file is migrated) which has linkto information and sticky bit. So in all we lose the file.

The resolution for this is, to redirect the link to the new cached subvol for a file under migration. So when we get a dht_link on the postop, we need to send a link to the real cached subvol that the file is being migrated to, IOW follow the linkto and link the file there as well.

The above is a first cut RCA and resolution thought process.

Here is some FS data on the same, to help corelate with the comments above,

< State of the brick before rebalance starts on FILE1>
# ls -l /d/backends/patchy*
/d/backends/patchy1:
total 5242880
-rw-r--r--. 2 root root 5368709120 Nov  6 14:49 FILE1

/d/backends/patchy2:
total 0

/d/backends/patchy3:
total 0

< State of the brick as soon as rebalance starts on FILE1 >
[root@marvin ~]# ls -l /d/backends/patchy*
/d/backends/patchy1:
total 5242884
-rw-r-Sr-T. 2 root root 5368709120 Nov  6 14:49 FILE1

/d/backends/patchy2:
total 0

/d/backends/patchy3:
total 52868
---------T. 2 root root 5368709120 Nov  6 14:50 FILE1

<Create hard link FILE2 which hashes to second subvol
 and hard link FILE5 which hashes to third subvol >
[root@marvin ~]# ls -l /d/backends/patchy*
/d/backends/patchy1:
total 20971536
-rw-r-Sr-T. 5 root root 5368709120 Nov  6 14:49 FILE1
-rw-r-Sr-T. 5 root root 5368709120 Nov  6 14:49 FILE2 (this is the hard link as cached for FILE1 is subvol1)
-rw-r-Sr-T. 5 root root 5368709120 Nov  6 14:49 FILE5

/d/backends/patchy2:
total 4
---------T. 2 root root 0 Nov  6 14:50 FILE2 (this is the linkto for FILE2)

/d/backends/patchy3:
total 5164812
---------T. 4 root root 5368709120 Nov  6 14:50 FILE1
---------T. 4 root root 5368709120 Nov  6 14:50 FILE5 (this is the linkto for FILE5 but is a hardlink to FILE1 as GFID is the same (check stat information))

< End of rebalance of FILE1, so on subvol1 we have the P1 file left, which is a linkto file>
[root@marvin ~]# ls -l /d/backends/patchy*
/d/backends/patchy:
total 0

/d/backends/patchy1:
total 12
---------T. 4 root root 0 Nov  6 14:50 FILE2 (bad FILE2, as FILE1 post migration was truncated, and made a linkto file and then unlinked, so the hard links survive with the P2 file, and linkto as subvol3, where FILE2 does not exist, IF it existed then it would be a double linkto, which again a lookup everywhere may cleanup)
---------T. 4 root root 0 Nov  6 14:50 FILE5 (good FILE5, but sort of useless as it is stale linkto, will get cleaned up later)

/d/backends/patchy2:
total 4
---------T. 2 root root 0 Nov  6 14:50 FILE2 (Hashed subvol of FILE2 pointing to NULL file in subvol1)

/d/backends/patchy3:
total 15728640
-rw-r--r--. 4 root root 5368709120 Nov  6 14:49 FILE1 (FILE1 is now hashed/cached here and good)
-rw-r--r--. 4 root root 5368709120 Nov  6 14:49 FILE5 (FILE5 was created as a hard link to the linkto for FILE1 during rebalance, so now automatically becomes a good file)

So bottom line is, we should create a hardlink in dht_link following a file under migration to its _new_ destination to resolve the issue.

Even if the original file was unlinked during migration, I assume the target would survive as there are hard links to it, this is a test case that needs repetition once we fix the issue in dht_link.