Bug 1128737

Summary:	rebalance :- rm -rf failed with with error 'No such file or directory' for few files and directory while rebalance is in progress
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Rachana Patel <racpatel>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED DEFERRED	QA Contact:
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	2.1	CC:	mzywusko, nbalacha, sabansal, spalai
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:	dht-rm-rf
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1286042 (view as bug list)		Environment:
Last Closed:	2015-11-27 10:10:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1115367, 1245065, 1257894
Bug Blocks:	1286042

Description Rachana Patel 2014-08-11 12:54:56 UTC

Description of problem:
=======================
after adding brick start rebalance and start removing data from mount using rm -rf. 
For few files and Directories it failed with error 'No such file or directory'.


Version-Release number of selected component (if applicable):
=============================================================
3.4.0.59rhs-1.2.toyota.hotfix.el6rhs.x86_64


How reproducible:
=================
always

Steps to Reproduce:
===================
1. create and mount Dist-rep(18x2) volume 
2. create files and Directories on it.
3. add 12 new bricks in that volume.
4. start rebalance with start force option
5. from mount point start deleting data using rm -rf


[root@localhost in]# cd linux-2.6.32.63/
[root@localhost linux-2.6.32.63]# ls -l
total 4
drwxrwxr-x 42 root root 4096 Aug 11  2014 drivers
drwxrwxr-x  3 root root   17 Aug 11  2014 include
drwxrwxr-x  3 root root   20 Aug 11  2014 samples
drwxrwxr-x  3 root root   20 Aug 11  2014 scripts
drwxrwxr-x  3 root root   22 Aug 11  2014 security
drwxrwxr-x  3 root root   19 Aug 11  2014 sound
drwxrwxr-x  3 root root   17 Aug 11  2014 tools
drwxrwxr-x  3 root root   16 Aug 11  2014 virt
[root@localhost linux-2.6.32.63]# rm -rf *
rm: cannot remove `drivers': No such file or directory
rm: cannot remove `include/acpi/platform': No such file or directory
rm: cannot remove `tools/perf/util/include/asm': No such file or directory
[root@localhost linux-2.6.32.63]# ls
drivers  include  tools


Actual results:
===============
File/Directory deletion is failing with error 'No such file or Directory'


Expected results:
================
rm -rf should not fail with such error if all sub-volumes were up.


Additional info:
================

log snippet :-
[2014-08-11 04:45:10.771372] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.773968] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.775313] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.775472] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.776247] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.777131] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.852869] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-41: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.852931] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-40: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860235] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-37: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860565] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-36: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860722] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-38: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860757] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-39: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860790] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-42: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860818] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-43: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860850] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-44: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860878] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-45: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860910] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-46: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
...
...
[2014-08-11 04:45:11.510973] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-37: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511417] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-36: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511730] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-38: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511769] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-39: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511812] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-40: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511859] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-41: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511892] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-43: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511919] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-42: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511967] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-44: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.512006] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-45: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.512039] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-46: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.512081] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-47: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.585285] W [nfs3.c:3514:nfs3svc_rmdir_cbk] 0-nfs: ef918778: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 => -1 (No such file or directory)
...

Comment 3 Nithya Balachandran 2014-08-12 08:45:40 UTC

As there was no data loss in the scenario, this will not be fixed for Denali. Moving this to RHS 3.0.z

Comment 6 Susant Kumar Palai 2015-11-27 10:10:16 UTC

Cloning this bug in 3.1. To be fixed in future release.