Bug 1434066

Summary:	.trashcan doesn't show deleted files in distributed replicated cluster
Product:	[Community] GlusterFS	Reporter:	Pavel Znamensky <kompastver>
Component:	trash-xlator	Assignee:	bugs <bugs>
Status:	CLOSED EOL	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.10	CC:	anoopcs, bugs, jthottan, kompastver
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1437477 (view as bug list)		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1437477, 1437478

Description Pavel Znamensky 2017-03-20 16:35:03 UTC

Description of problem:
Files are not appear in .trashcan when they were located not in root volume dir and when glusterfs volume's type is distributed replicated.

Steps to Reproduce:
1. Create distributed replicated volume:
sudo gluster volume create test-volume replica 2 transport tcp srv1:/raw_gluster_dir srv2:/raw_gluster_dir srv3:/raw_gluster_dir srv4:/raw_gluster_dir
sudo gluster volume start test-volume

2. Enable trash feature:
user@srv1 ~ $ sudo gluster volume set test-volume features.trash on
volume set: success
user@srv1 ~ $ sudo gluster volume get test-volume all | grep trash
features.trash                          on                                      
features.trash-dir                      .trashcan                               
features.trash-eliminate-path           (null)                                  
features.trash-max-filesize             5MB                                     
features.trash-internal-op              off 

3. mount glusterfs volume:
user@srv1 ~ $ sudo mount -t glusterfs srv1:/test-volume /mnt/test-glusterfs-volume/

user@srv1 ~ $ ls -al /mnt/test-glusterfs-volume/
total 12
drwxr-xr-x  4 root root 4096 Mar 20 18:34 .
drwxr-xr-x. 4 root root 4096 Mar 20 18:36 ..
drwxr-xr-x  3 root root 4096 Mar 20 18:34 .trashcan

4. Try to delete file in root dir:
user@srv1 ~ $ sudo touch /mnt/test-glusterfs-volume/file_in_root                                                                               
user@srv1 ~ $ ls -l /mnt/test-glusterfs-volume/
total 0                                                                                                                                              
-rw-r--r-- 1 root root 0 Mar 20 18:38 file_in_root                                                                                                   
user@srv1 ~ $ sudo rm /mnt/test-glusterfs-volume/file_in_root 

# it's all fine at this step
user@srv1 ~ $ sudo ls -l /mnt/test-glusterfs-volume/.trashcan/
total 4                                                                                                                                              
-rw-r--r-- 1 root root    0 Mar 20 18:38 file_in_root_2017-03-20_153859                                                                              
drwxr-xr-x 2 root root 4096 Mar 20 18:34 internal_op                                                                                                 

5. Try to delete files in subdirs:
user@srv1 ~ $ sudo mkdir -p /mnt/test-glusterfs-volume/aa/bb
user@srv1 ~ $ sudo touch /mnt/test-glusterfs-volume/aa/bb/test_file1                                                                           
# When there is a single file in subdir all work fine too
# Issue occurs when there are both deleted and undeleted files in subdirs
# That's why we are creating test_file2
user@srv1 ~ $ sudo touch /mnt/test-glusterfs-volume/aa/bb/test_file2

user@srv1 ~ $ sudo rm -f /mnt/test-glusterfs-volume/aa/bb/test_file1                                                                           

# test_file1 should be appeared in /mnt/test-glusterfs-volume/.trashcan/aa/bb/test_file1_*
# but there is no files in mounted volume:
user@srv1 ~ $ sudo ls -l /mnt/test-glusterfs-volume/.trashcan/aa/
total 0

# nevertheless file appeared in 'raw' dir:
user@srv1 ~ $ ls -l /raw_gluster_dir/.trashcan/aa/bb/
total 4                                                                                                                                              
-rw-r--r-- 2 root root 0 Mar 20 18:41 test_file1_2017-03-20_154233                                                                                   


Actual results:
Deleted file is not visible in .trashcan/

Expected results:
Deleted file is available in .trashcan/

Comment 1 Niels de Vos 2017-03-28 12:15:24 UTC

Jiffin, Anoop, has this been fixed in more recent versions yet? Please see to cloning this bug, or backporting the fix(es).

Thanks!

Comment 2 Jiffin 2017-03-30 11:46:37 UTC

This is known issue for trash from day one. In case of distribute volumes for deleted files at server side trash xlator will create entire directory hierarchy inside trash directory if destination path is missing. This happens only in bricks where orginal file exists, in all other bricks this path will be missing which results in above behavior. Only way to bypass this behaviour perform a forced lookup from client side which heals the entire hierarchy on other bricks

Comment 3 Pavel Znamensky 2017-03-30 12:43:15 UTC

Jiffin,
Did I understand correctly that client should lookup exactly deleted file in .trashcan/ ?
Seems that heal doesn't perform if client do listing (which calling stat() and should initiate healing) in root of .trashcan/.
If client should lookup exactly deleted file, he should get list of deleted files at first.
But how he should get this list?

Comment 4 Jiffin 2017-04-03 06:39:49 UTC

(In reply to Znamensky Pavel from comment #3)
> Jiffin,
> Did I understand correctly that client should lookup exactly deleted file in
> .trashcan/ ?

For distributed volume , yes it is a known issue. We have a couple of ways to solve it, but didn't find enough time to fix it. If you are interested I can help you with it

> Seems that heal doesn't perform if client do listing (which calling stat()
> and should initiate healing) in root of .trashcan/.
> If client should lookup exactly deleted file, he should get list of deleted
> files at first.
> But how he should get this list?
Only feasible solution to my mind atm is that after deleting a file, application should perform a lookup on deleted path at .trashcan

Comment 5 Pavel Znamensky 2017-04-03 09:44:18 UTC

Jiffin,
Thanks for the explanations.

Unfortunately this is important feature for us. 
And we use glusterfs as a storage for a lots of applications. We can't change behavior of most of this apps in order to perform lookup at .trashcan/.

I know two bad workarounds to create list of deleted files:
- run inotify/auditd on all clients or an all glusterfs nodes and watch for deleted files
- find files on all glusterfs nodes at "raw" .trashcan/ dirs

Comment 6 Jiffin 2017-04-04 10:02:06 UTC

(In reply to Znamensky Pavel from comment #5)
> Jiffin,
> Thanks for the explanations.
> 
> Unfortunately this is important feature for us. 
> And we use glusterfs as a storage for a lots of applications. We can't
> change behavior of most of this apps in order to perform lookup at
> .trashcan/.
> 
> I know two bad workarounds to create list of deleted files:
> - run inotify/auditd on all clients or an all glusterfs nodes and watch for
> deleted files

This should work

> - find files on all glusterfs nodes at "raw" .trashcan/ dirs
I didn't understand that.

If u are interested we can actually solved it in gluster for 3.11, it is not difficult to do. I can help u out

Comment 7 Pavel Znamensky 2017-04-04 10:42:38 UTC

> - find files on all glusterfs nodes at "raw" .trashcan/ dirs
I mean find files at brick's export directory at .trashcan/ subdir.

>If u are interested we can actually solved it in gluster for 3.11, it is not difficult to do. I can help u out
It would be great!
Also it would be great if it would be possible to backport the fix to stable versions of glusterfs.

Comment 8 Jiffin 2017-04-06 10:37:16 UTC

(In reply to Znamensky Pavel from comment #7)
> > - find files on all glusterfs nodes at "raw" .trashcan/ dirs
> I mean find files at brick's export directory at .trashcan/ subdir.
This is also okay, if brick count is very less

> 
> >If u are interested we can actually solved it in gluster for 3.11, it is not difficult to do. I can help u out
> It would be great!

Solution in my mind is that, currently path creation are happened on the brick directly. Instead if we can send it from client then it will works fine. Currently this logic resides in brick stack at trash translator. We need to move this logic from server to client(Currently 3.11 feature freeze is target on April 30th)

> Also it would be great if it would be possible to backport the fix to stable
> versions of glusterfs.
Even though it looks like bug fix, The changes looks like new enhancement. So usually gluster don't back port huge changes to stable branch

Comment 9 Pavel Znamensky 2017-04-07 08:39:20 UTC

Jiffin,
It would be great even if this enhancement/fix will not be back ported to the current stable branch. Someday unstable branch will become a new stable branch.

Comment 10 Pavel Znamensky 2017-06-28 13:25:31 UTC

Hi,
Is there any update on this?
We are still hoping that .trashcan/ will works without workarounds.

Comment 11 Shyamsundar 2018-06-20 18:27:33 UTC

This bug reported is against a version of Gluster that is no longer maintained (or has been EOL'd). See https://www.gluster.org/release-schedule/ for the versions currently maintained.

As a result this bug is being closed.

If the bug persists on a maintained version of gluster or against the mainline gluster repository, request that it be reopened and the Version field be marked appropriately.

Comment 12 Shyamsundar 2018-06-20 18:27:55 UTC

This bug reported is against a version of Gluster that is no longer maintained
(or has been EOL'd). See https://www.gluster.org/release-schedule/ for the
versions currently maintained.

As a result this bug is being closed.

If the bug persists on a maintained version of gluster or against the mainline
gluster repository, request that it be reopened and the Version field be marked
appropriately.