Description of problem: Files are not appear in .trashcan when they were located not in root volume dir and when glusterfs volume's type is distributed replicated. Steps to Reproduce: 1. Create distributed replicated volume: sudo gluster volume create test-volume replica 2 transport tcp srv1:/raw_gluster_dir srv2:/raw_gluster_dir srv3:/raw_gluster_dir srv4:/raw_gluster_dir sudo gluster volume start test-volume 2. Enable trash feature: user@srv1 ~ $ sudo gluster volume set test-volume features.trash on volume set: success user@srv1 ~ $ sudo gluster volume get test-volume all | grep trash features.trash on features.trash-dir .trashcan features.trash-eliminate-path (null) features.trash-max-filesize 5MB features.trash-internal-op off 3. mount glusterfs volume: user@srv1 ~ $ sudo mount -t glusterfs srv1:/test-volume /mnt/test-glusterfs-volume/ user@srv1 ~ $ ls -al /mnt/test-glusterfs-volume/ total 12 drwxr-xr-x 4 root root 4096 Mar 20 18:34 . drwxr-xr-x. 4 root root 4096 Mar 20 18:36 .. drwxr-xr-x 3 root root 4096 Mar 20 18:34 .trashcan 4. Try to delete file in root dir: user@srv1 ~ $ sudo touch /mnt/test-glusterfs-volume/file_in_root user@srv1 ~ $ ls -l /mnt/test-glusterfs-volume/ total 0 -rw-r--r-- 1 root root 0 Mar 20 18:38 file_in_root user@srv1 ~ $ sudo rm /mnt/test-glusterfs-volume/file_in_root # it's all fine at this step user@srv1 ~ $ sudo ls -l /mnt/test-glusterfs-volume/.trashcan/ total 4 -rw-r--r-- 1 root root 0 Mar 20 18:38 file_in_root_2017-03-20_153859 drwxr-xr-x 2 root root 4096 Mar 20 18:34 internal_op 5. Try to delete files in subdirs: user@srv1 ~ $ sudo mkdir -p /mnt/test-glusterfs-volume/aa/bb user@srv1 ~ $ sudo touch /mnt/test-glusterfs-volume/aa/bb/test_file1 # When there is a single file in subdir all work fine too # Issue occurs when there are both deleted and undeleted files in subdirs # That's why we are creating test_file2 user@srv1 ~ $ sudo touch /mnt/test-glusterfs-volume/aa/bb/test_file2 user@srv1 ~ $ sudo rm -f /mnt/test-glusterfs-volume/aa/bb/test_file1 # test_file1 should be appeared in /mnt/test-glusterfs-volume/.trashcan/aa/bb/test_file1_* # but there is no files in mounted volume: user@srv1 ~ $ sudo ls -l /mnt/test-glusterfs-volume/.trashcan/aa/ total 0 # nevertheless file appeared in 'raw' dir: user@srv1 ~ $ ls -l /raw_gluster_dir/.trashcan/aa/bb/ total 4 -rw-r--r-- 2 root root 0 Mar 20 18:41 test_file1_2017-03-20_154233 Actual results: Deleted file is not visible in .trashcan/ Expected results: Deleted file is available in .trashcan/
Jiffin, Anoop, has this been fixed in more recent versions yet? Please see to cloning this bug, or backporting the fix(es). Thanks!
This is known issue for trash from day one. In case of distribute volumes for deleted files at server side trash xlator will create entire directory hierarchy inside trash directory if destination path is missing. This happens only in bricks where orginal file exists, in all other bricks this path will be missing which results in above behavior. Only way to bypass this behaviour perform a forced lookup from client side which heals the entire hierarchy on other bricks
Jiffin, Did I understand correctly that client should lookup exactly deleted file in .trashcan/ ? Seems that heal doesn't perform if client do listing (which calling stat() and should initiate healing) in root of .trashcan/. If client should lookup exactly deleted file, he should get list of deleted files at first. But how he should get this list?
(In reply to Znamensky Pavel from comment #3) > Jiffin, > Did I understand correctly that client should lookup exactly deleted file in > .trashcan/ ? For distributed volume , yes it is a known issue. We have a couple of ways to solve it, but didn't find enough time to fix it. If you are interested I can help you with it > Seems that heal doesn't perform if client do listing (which calling stat() > and should initiate healing) in root of .trashcan/. > If client should lookup exactly deleted file, he should get list of deleted > files at first. > But how he should get this list? Only feasible solution to my mind atm is that after deleting a file, application should perform a lookup on deleted path at .trashcan
Jiffin, Thanks for the explanations. Unfortunately this is important feature for us. And we use glusterfs as a storage for a lots of applications. We can't change behavior of most of this apps in order to perform lookup at .trashcan/. I know two bad workarounds to create list of deleted files: - run inotify/auditd on all clients or an all glusterfs nodes and watch for deleted files - find files on all glusterfs nodes at "raw" .trashcan/ dirs
(In reply to Znamensky Pavel from comment #5) > Jiffin, > Thanks for the explanations. > > Unfortunately this is important feature for us. > And we use glusterfs as a storage for a lots of applications. We can't > change behavior of most of this apps in order to perform lookup at > .trashcan/. > > I know two bad workarounds to create list of deleted files: > - run inotify/auditd on all clients or an all glusterfs nodes and watch for > deleted files This should work > - find files on all glusterfs nodes at "raw" .trashcan/ dirs I didn't understand that. If u are interested we can actually solved it in gluster for 3.11, it is not difficult to do. I can help u out
> - find files on all glusterfs nodes at "raw" .trashcan/ dirs I mean find files at brick's export directory at .trashcan/ subdir. >If u are interested we can actually solved it in gluster for 3.11, it is not difficult to do. I can help u out It would be great! Also it would be great if it would be possible to backport the fix to stable versions of glusterfs.
(In reply to Znamensky Pavel from comment #7) > > - find files on all glusterfs nodes at "raw" .trashcan/ dirs > I mean find files at brick's export directory at .trashcan/ subdir. This is also okay, if brick count is very less > > >If u are interested we can actually solved it in gluster for 3.11, it is not difficult to do. I can help u out > It would be great! Solution in my mind is that, currently path creation are happened on the brick directly. Instead if we can send it from client then it will works fine. Currently this logic resides in brick stack at trash translator. We need to move this logic from server to client(Currently 3.11 feature freeze is target on April 30th) > Also it would be great if it would be possible to backport the fix to stable > versions of glusterfs. Even though it looks like bug fix, The changes looks like new enhancement. So usually gluster don't back port huge changes to stable branch
Jiffin, It would be great even if this enhancement/fix will not be back ported to the current stable branch. Someday unstable branch will become a new stable branch.
Hi, Is there any update on this? We are still hoping that .trashcan/ will works without workarounds.
This bug reported is against a version of Gluster that is no longer maintained (or has been EOL'd). See https://www.gluster.org/release-schedule/ for the versions currently maintained. As a result this bug is being closed. If the bug persists on a maintained version of gluster or against the mainline gluster repository, request that it be reopened and the Version field be marked appropriately.