Bug 1128737 - rebalance :- rm -rf failed with with error 'No such file or directory' for few files and directory while rebalance is in progress
Summary: rebalance :- rm -rf failed with with error 'No such file or directory' for fe...
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute
Version: 2.1
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Nithya Balachandran
QA Contact:
URL:
Whiteboard: dht-rm-rf
Depends On: 1115367 1245065 1257894
Blocks: 1286042
TreeView+ depends on / blocked
 
Reported: 2014-08-11 12:54 UTC by Rachana Patel
Modified: 2015-11-27 10:11 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1286042 (view as bug list)
Environment:
Last Closed: 2015-11-27 10:10:16 UTC


Attachments (Terms of Use)

Description Rachana Patel 2014-08-11 12:54:56 UTC
Description of problem:
=======================
after adding brick start rebalance and start removing data from mount using rm -rf. 
For few files and Directories it failed with error 'No such file or directory'.


Version-Release number of selected component (if applicable):
=============================================================
3.4.0.59rhs-1.2.toyota.hotfix.el6rhs.x86_64


How reproducible:
=================
always

Steps to Reproduce:
===================
1. create and mount Dist-rep(18x2) volume 
2. create files and Directories on it.
3. add 12 new bricks in that volume.
4. start rebalance with start force option
5. from mount point start deleting data using rm -rf


[root@localhost in]# cd linux-2.6.32.63/
[root@localhost linux-2.6.32.63]# ls -l
total 4
drwxrwxr-x 42 root root 4096 Aug 11  2014 drivers
drwxrwxr-x  3 root root   17 Aug 11  2014 include
drwxrwxr-x  3 root root   20 Aug 11  2014 samples
drwxrwxr-x  3 root root   20 Aug 11  2014 scripts
drwxrwxr-x  3 root root   22 Aug 11  2014 security
drwxrwxr-x  3 root root   19 Aug 11  2014 sound
drwxrwxr-x  3 root root   17 Aug 11  2014 tools
drwxrwxr-x  3 root root   16 Aug 11  2014 virt
[root@localhost linux-2.6.32.63]# rm -rf *
rm: cannot remove `drivers': No such file or directory
rm: cannot remove `include/acpi/platform': No such file or directory
rm: cannot remove `tools/perf/util/include/asm': No such file or directory
[root@localhost linux-2.6.32.63]# ls
drivers  include  tools


Actual results:
===============
File/Directory deletion is failing with error 'No such file or Directory'


Expected results:
================
rm -rf should not fail with such error if all sub-volumes were up.


Additional info:
================

log snippet :-
[2014-08-11 04:45:10.771372] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.773968] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.775313] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.775472] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.776247] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.777131] W [dht-selfheal.c:427:dht_selfheal_dir_mkdir_cbk] 0-mio-dht: selfhealing directory <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform failed: No such file or directory
[2014-08-11 04:45:10.852869] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-41: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.852931] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-40: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860235] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-37: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860565] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-36: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860722] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-38: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860757] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-39: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860790] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-42: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860818] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-43: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860850] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-44: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860878] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-45: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
[2014-08-11 04:45:10.860910] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-46: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform (9112ea8b-64e4-4051-8104-30860b0fbad5)
...
...
[2014-08-11 04:45:11.510973] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-37: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511417] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-36: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511730] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-38: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511769] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-39: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511812] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-40: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511859] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-41: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511892] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-43: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511919] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-42: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.511967] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-44: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.512006] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-45: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.512039] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-46: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.512081] W [client-rpc-fops.c:2523:client3_3_opendir_cbk] 0-mio-client-47: remote operation failed: No such file or directory. Path: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 (8d20420c-d6bf-45d9-98cc-a4fe0d18d256)
[2014-08-11 04:45:11.585285] W [nfs3.c:3514:nfs3svc_rmdir_cbk] 0-nfs: ef918778: <gfid:cff3a67d-ee90-435f-aaf9-5e811a145fa9>/platform/x86 => -1 (No such file or directory)
...

Comment 3 Nithya Balachandran 2014-08-12 08:45:40 UTC
As there was no data loss in the scenario, this will not be fixed for Denali. Moving this to RHS 3.0.z

Comment 6 Susant Kumar Palai 2015-11-27 10:10:16 UTC
Cloning this bug in 3.1. To be fixed in future release.


Note You need to log in before you can comment on or make changes to this bug.