Bug 1437477

+++ This bug was initially created as a clone of Bug #1434066 +++

Description of problem:
Files are not appear in .trashcan when they were located not in root volume dir and when glusterfs volume's type is distributed replicated.

Steps to Reproduce:
1. Create distributed replicated volume:
sudo gluster volume create test-volume replica 2 transport tcp srv1:/raw_gluster_dir srv2:/raw_gluster_dir srv3:/raw_gluster_dir srv4:/raw_gluster_dir
sudo gluster volume start test-volume

2. Enable trash feature:
user@srv1 ~ $ sudo gluster volume set test-volume features.trash on
volume set: success
user@srv1 ~ $ sudo gluster volume get test-volume all | grep trash
features.trash                          on                                      
features.trash-dir                      .trashcan                               
features.trash-eliminate-path           (null)                                  
features.trash-max-filesize             5MB                                     
features.trash-internal-op              off 

3. mount glusterfs volume:
user@srv1 ~ $ sudo mount -t glusterfs srv1:/test-volume /mnt/test-glusterfs-volume/

user@srv1 ~ $ ls -al /mnt/test-glusterfs-volume/
total 12
drwxr-xr-x  4 root root 4096 Mar 20 18:34 .
drwxr-xr-x. 4 root root 4096 Mar 20 18:36 ..
drwxr-xr-x  3 root root 4096 Mar 20 18:34 .trashcan

4. Try to delete file in root dir:
user@srv1 ~ $ sudo touch /mnt/test-glusterfs-volume/file_in_root                                                                               
user@srv1 ~ $ ls -l /mnt/test-glusterfs-volume/
total 0                                                                                                                                              
-rw-r--r-- 1 root root 0 Mar 20 18:38 file_in_root                                                                                                   
user@srv1 ~ $ sudo rm /mnt/test-glusterfs-volume/file_in_root 

# it's all fine at this step
user@srv1 ~ $ sudo ls -l /mnt/test-glusterfs-volume/.trashcan/
total 4                                                                                                                                              
-rw-r--r-- 1 root root    0 Mar 20 18:38 file_in_root_2017-03-20_153859                                                                              
drwxr-xr-x 2 root root 4096 Mar 20 18:34 internal_op                                                                                                 

5. Try to delete files in subdirs:
user@srv1 ~ $ sudo mkdir -p /mnt/test-glusterfs-volume/aa/bb
user@srv1 ~ $ sudo touch /mnt/test-glusterfs-volume/aa/bb/test_file1                                                                           
# When there is a single file in subdir all work fine too
# Issue occurs when there are both deleted and undeleted files in subdirs
# That's why we are creating test_file2
user@srv1 ~ $ sudo touch /mnt/test-glusterfs-volume/aa/bb/test_file2

user@srv1 ~ $ sudo rm -f /mnt/test-glusterfs-volume/aa/bb/test_file1                                                                           

# test_file1 should be appeared in /mnt/test-glusterfs-volume/.trashcan/aa/bb/test_file1_*
# but there is no files in mounted volume:
user@srv1 ~ $ sudo ls -l /mnt/test-glusterfs-volume/.trashcan/aa/
total 0

# nevertheless file appeared in 'raw' dir:
user@srv1 ~ $ ls -l /raw_gluster_dir/.trashcan/aa/bb/
total 4                                                                                                                                              
-rw-r--r-- 2 root root 0 Mar 20 18:41 test_file1_2017-03-20_154233                                                                                   


Actual results:
Deleted file is not visible in .trashcan/

Expected results:
Deleted file is available in .trashcan/

--- Additional comment from Niels de Vos on 2017-03-28 08:15:24 EDT ---

Jiffin, Anoop, has this been fixed in more recent versions yet? Please see to cloning this bug, or backporting the fix(es).

Thanks!

--- Additional comment from Jiffin on 2017-03-30 07:46:37 EDT ---

This is known issue for trash from day one. In case of distribute volumes for deleted files at server side trash xlator will create entire directory hierarchy inside trash directory if destination path is missing. This happens only in bricks where orginal file exists, in all other bricks this path will be missing which results in above behavior. Only way to bypass this behaviour perform a forced lookup from client side which heals the entire hierarchy on other bricks