Bug 1258782

Summary:	DHT: After deletion of all files, still "ls" shows few files
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	RajeshReddy <rmekala>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	storage-qa-internal <storage-qa-internal>
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	rhgs-3.1	CC:	mzywusko, nbalacha, rgowdapp, smohan
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	triaged, rebalance-readdir-race
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-02-23 07:12:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description RajeshReddy 2015-09-01 09:38:02 UTC

Document URL: 
==================
DHT: After deletion of all files, still "ls" shows few files 

Steps:
=============
1. Create distributed volume (3 bricks) and mount it on client using FUSE 
2. On the client create 50k files 
3. Remove one of the brick, and as part of the removal of brick, re-balance triggered and while files are migrating run rm -rf * on mount though command exit without any error but ls on the same direcotry shows files 

Expected Result:
====================
After removal of files, all files needs to be deleted 

Info
======
[root@rhs-client9 dht7]# gluster vol status dht7
Status of volume: dht7
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick rhs-client9.lab.eng.blr.redhat.com:/r
hs/brick4/dht7                              49173     0          Y       11879
Brick rhs-client4.lab.eng.blr.redhat.com:/r
hs/brick4/dht7                              49172     0          Y       5051 
Brick rhs-client39.lab.eng.blr.redhat.com:/
rhs/brick4/dht7                             49162     0          Y       28520
NFS Server on localhost                     2049      0          Y       12217
NFS Server on rhs-client4.lab.eng.blr.redha
t.com                                       2049      0          Y       5564 
NFS Server on rhs-client39.lab.eng.blr.redh
at.com                                      2049      0          Y       28636
 
Task Status of Volume dht7
------------------------------------------------------------------------------
Task                 : Remove brick        
ID                   : 714849d5-ee98-4833-952e-af4509d4198e
Removed bricks:     
rhs-client4.lab.eng.blr.redhat.com:/rhs/brick4/dht7
Status               : completed           

On the client
=================

[root@scarface data]# ls | wc -l
1641

Attaching the sos reports

Comment 2 Sakshi 2015-09-23 05:25:46 UTC

It is possible that the files which are remaining are the ones that were migrated from the brick that you are removing. However the files were migrated only after the rm -rf had completed reading from the remaining subvols. Hence some of the migrated files may not be a part of the readdirp that happened as a part of rm -rf and hence did not get deleted.