Bug 969297 - DHT- dist-rep volume - rm -rf is failing and giving error 'rm: cannot remove `<invalid argument>': Invalid argument (all bricks are up and peers are connected)
DHT- dist-rep volume - rm -rf is failing and giving error 'rm: cannot remove ...
Status: CLOSED DEFERRED
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute (Show other bugs)
2.0
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Nithya Balachandran
Matt Zywusko
dht-rm-rf
:
Depends On:
Blocks: 1286054
  Show dependency treegraph
 
Reported: 2013-05-31 03:04 EDT by Rachana Patel
Modified: 2015-11-27 05:22 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1286054 (view as bug list)
Environment:
Last Closed: 2015-11-27 05:21:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rachana Patel 2013-05-31 03:04:07 EDT
Description of problem:
DHT- dist-rep volume - rm -rf is failing and giving error 'rm: cannot remove `<invalid argument>': Invalid argument (all bricks are up and peers are connected)

Version-Release number of selected component (if applicable):
3.3.0.10rhs-1.el6.x86_64


Steps to Reproduce:
1.had a cluster of 3 peers and dist-rep volume - 3X2, mounted as FUSE mount having some data


[root@rhsauto031 ~]# gluster v info dist-rep
 
Volume Name: dist-rep
Type: Distributed-Replicate
Volume ID: 9dee64a6-9f86-4463-bacc-47d97c750803
Status: Started
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: rhsauto018.lab.eng.blr.redhat.com:/rhs/brick4/2
Brick2: rhsauto038.lab.eng.blr.redhat.com:/rhs/brick4/2
Brick3: rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/1
Brick4: rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/2
Brick5: rhsauto018.lab.eng.blr.redhat.com:/rhs/brick4/3
Brick6: rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/3

mount point:-
[root@localhost system_light]# mount | grep rahul
glusterfs#rhsauto018.lab.eng.blr.redhat.com:/dist-rep on /mnt/rahulsanity type fuse (rw,default_permissions,allow_other,max_read=131072)

[root@localhost system_light]# ls /mnt/rahulsanity/
d27  d36  d44  d52  d60  d69  d77  d85  d93     file200  file29  file37  file45  file53  file61  file69  file77  file85  file93
d29  d37  d45  d53  d61  d7   d78  d86  d94     file21   file3   file38  file46  file54  file61  file7   file78  file86  file94
d3   d38  d46  d54  d62  d70  d79  d87  d95     file22   file30  file39  file47  file55  file62  file70  file79  file87  file95
d30  d39  d47  d55  d63  d71  d8   d88  d96     file23   file31  file4   file48  file56  file63  file71  file8   file88  file96
d31  d4   d48  d56  d64  d72  d80  d89  d97     file24   file32  file40  file49  file57  file64  file72  file80  file89  file97
d32  d40  d49  d57  d65  d73  d81  d9   d98     file25   file33  file41  file5   file58  file65  file73  file81  file9   file98
d33  d41  d5   d58  d66  d74  d82  d90  d99     file26   file34  file42  file50  file59  file66  file74  file82  file90  file99
d34  d42  d50  d59  d67  d75  d83  d91  file2   file27   file35  file43  file51  file6   file67  file75  file83  file91  run32490
d35  d43  d51  d6   d68  d76  d84  d92  file20  file28   file36  file44  file52  file60  file68  file76  file84  file92

2. added bricks to make it 6X2
[root@rhsauto031 ~]# gluster volume add-brick dist-rep rhsauto018.lab.eng.blr.redhat.com:/rhs/brick3/8 rhsauto038.lab.eng.blr.redhat.com:/rhs/brick3/8 rhsauto031.lab.eng.blr.redhat.com:/rhs/brick3/8  rhsauto031.lab.eng.blr.redhat.com:/rhs/brick3/6 rhsauto018.lab.eng.blr.redhat.com:/rhs/brick3/6 rhsauto038.lab.eng.blr.redhat.com:/rhs/brick3/6
Add Brick successful

3. start a rebalance with start force option

4. while rebalance is in progress. remove data from mount point using rm -rf *


[root@rhsauto031 ~]# gluster volume rebalance dist-rep status
                                    Node Rebalanced-files          size       scanned      failures         status
                               ---------      -----------   -----------   -----------   -----------   ------------
                               localhost               27     28311552           62            0    in progress
       rhsauto018.lab.eng.blr.redhat.com               32     26214400          167            0    in progress
                             10.70.37.13                0            0          254            0    in progress




Actual results:
[root@localhost system_light]# rm -rf  /mnt/rahulsanity/*
rm: cannot remove `/mnt/rahulsanity/d36/file30': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d40/f99': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d40/f100': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d40/f16': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d40/f54': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d40/file24': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d41/f1': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d41/f51': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d41/f83': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d41/f79': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d41/f66': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d42/f83': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d42/f94': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d42/file26': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d42/file39': Invalid argument
rm: cannot remove `/mnt/rahulsanity/d42/f41': Invalid argument
...
....

Expected results:
rm -rf * should not fail with invalid argument when files are present and no bricks are down

[root@rhsauto038 ~]# gluster v status dist-rep
Status of volume: dist-rep
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick rhsauto018.lab.eng.blr.redhat.com:/rhs/brick4/2	24014	Y13998
Brick rhsauto038.lab.eng.blr.redhat.com:/rhs/brick4/2	24014	Y13906
Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/1	24016	Y1948
Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/2	24017	Y1954
Brick rhsauto018.lab.eng.blr.redhat.com:/rhs/brick4/3	24021	Y17653
Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/3	24020	Y1960
Brick rhsauto018.lab.eng.blr.redhat.com:/rhs/brick3/8	24026	Y20061
Brick rhsauto038.lab.eng.blr.redhat.com:/rhs/brick3/8	24019	Y19293
Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick3/8	24025	Y12986
Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick3/6	24026	Y12992
Brick rhsauto018.lab.eng.blr.redhat.com:/rhs/brick3/6	24027	Y20066
Brick rhsauto038.lab.eng.blr.redhat.com:/rhs/brick3/6	24020	Y19298
NFS Server on localhost					38467	Y19305
Self-heal Daemon on localhost				N/A	Y19311
NFS Server on rhsauto018.lab.eng.blr.redhat.com		38467	Y20073
Self-heal Daemon on rhsauto018.lab.eng.blr.redhat.com	N/A	Y20079
NFS Server on rhsauto031.lab.eng.blr.redhat.com		38467	Y12998
Self-heal Daemon on rhsauto031.lab.eng.blr.redhat.com	N/A	Y13004



Additional info:
Comment 8 Susant Kumar Palai 2015-11-27 05:21:31 EST
Cloning this bug in 3.1. To be fixed in future release.

Note You need to log in before you can comment on or make changes to this bug.