Description of problem: DHT- dist-rep volume - rm -rf is failing and giving error 'rm: cannot remove `<invalid argument>': Invalid argument (all bricks are up and peers are connected) Version-Release number of selected component (if applicable): 3.3.0.10rhs-1.el6.x86_64 Steps to Reproduce: 1.had a cluster of 3 peers and dist-rep volume - 3X2, mounted as FUSE mount having some data [root@rhsauto031 ~]# gluster v info dist-rep Volume Name: dist-rep Type: Distributed-Replicate Volume ID: 9dee64a6-9f86-4463-bacc-47d97c750803 Status: Started Number of Bricks: 3 x 2 = 6 Transport-type: tcp Bricks: Brick1: rhsauto018.lab.eng.blr.redhat.com:/rhs/brick4/2 Brick2: rhsauto038.lab.eng.blr.redhat.com:/rhs/brick4/2 Brick3: rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/1 Brick4: rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/2 Brick5: rhsauto018.lab.eng.blr.redhat.com:/rhs/brick4/3 Brick6: rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/3 mount point:- [root@localhost system_light]# mount | grep rahul glusterfs#rhsauto018.lab.eng.blr.redhat.com:/dist-rep on /mnt/rahulsanity type fuse (rw,default_permissions,allow_other,max_read=131072) [root@localhost system_light]# ls /mnt/rahulsanity/ d27 d36 d44 d52 d60 d69 d77 d85 d93 file200 file29 file37 file45 file53 file61 file69 file77 file85 file93 d29 d37 d45 d53 d61 d7 d78 d86 d94 file21 file3 file38 file46 file54 file61 file7 file78 file86 file94 d3 d38 d46 d54 d62 d70 d79 d87 d95 file22 file30 file39 file47 file55 file62 file70 file79 file87 file95 d30 d39 d47 d55 d63 d71 d8 d88 d96 file23 file31 file4 file48 file56 file63 file71 file8 file88 file96 d31 d4 d48 d56 d64 d72 d80 d89 d97 file24 file32 file40 file49 file57 file64 file72 file80 file89 file97 d32 d40 d49 d57 d65 d73 d81 d9 d98 file25 file33 file41 file5 file58 file65 file73 file81 file9 file98 d33 d41 d5 d58 d66 d74 d82 d90 d99 file26 file34 file42 file50 file59 file66 file74 file82 file90 file99 d34 d42 d50 d59 d67 d75 d83 d91 file2 file27 file35 file43 file51 file6 file67 file75 file83 file91 run32490 d35 d43 d51 d6 d68 d76 d84 d92 file20 file28 file36 file44 file52 file60 file68 file76 file84 file92 2. added bricks to make it 6X2 [root@rhsauto031 ~]# gluster volume add-brick dist-rep rhsauto018.lab.eng.blr.redhat.com:/rhs/brick3/8 rhsauto038.lab.eng.blr.redhat.com:/rhs/brick3/8 rhsauto031.lab.eng.blr.redhat.com:/rhs/brick3/8 rhsauto031.lab.eng.blr.redhat.com:/rhs/brick3/6 rhsauto018.lab.eng.blr.redhat.com:/rhs/brick3/6 rhsauto038.lab.eng.blr.redhat.com:/rhs/brick3/6 Add Brick successful 3. start a rebalance with start force option 4. while rebalance is in progress. remove data from mount point using rm -rf * [root@rhsauto031 ~]# gluster volume rebalance dist-rep status Node Rebalanced-files size scanned failures status --------- ----------- ----------- ----------- ----------- ------------ localhost 27 28311552 62 0 in progress rhsauto018.lab.eng.blr.redhat.com 32 26214400 167 0 in progress 10.70.37.13 0 0 254 0 in progress Actual results: [root@localhost system_light]# rm -rf /mnt/rahulsanity/* rm: cannot remove `/mnt/rahulsanity/d36/file30': Invalid argument rm: cannot remove `/mnt/rahulsanity/d40/f99': Invalid argument rm: cannot remove `/mnt/rahulsanity/d40/f100': Invalid argument rm: cannot remove `/mnt/rahulsanity/d40/f16': Invalid argument rm: cannot remove `/mnt/rahulsanity/d40/f54': Invalid argument rm: cannot remove `/mnt/rahulsanity/d40/file24': Invalid argument rm: cannot remove `/mnt/rahulsanity/d41/f1': Invalid argument rm: cannot remove `/mnt/rahulsanity/d41/f51': Invalid argument rm: cannot remove `/mnt/rahulsanity/d41/f83': Invalid argument rm: cannot remove `/mnt/rahulsanity/d41/f79': Invalid argument rm: cannot remove `/mnt/rahulsanity/d41/f66': Invalid argument rm: cannot remove `/mnt/rahulsanity/d42/f83': Invalid argument rm: cannot remove `/mnt/rahulsanity/d42/f94': Invalid argument rm: cannot remove `/mnt/rahulsanity/d42/file26': Invalid argument rm: cannot remove `/mnt/rahulsanity/d42/file39': Invalid argument rm: cannot remove `/mnt/rahulsanity/d42/f41': Invalid argument ... .... Expected results: rm -rf * should not fail with invalid argument when files are present and no bricks are down [root@rhsauto038 ~]# gluster v status dist-rep Status of volume: dist-rep Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick rhsauto018.lab.eng.blr.redhat.com:/rhs/brick4/2 24014 Y13998 Brick rhsauto038.lab.eng.blr.redhat.com:/rhs/brick4/2 24014 Y13906 Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/1 24016 Y1948 Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/2 24017 Y1954 Brick rhsauto018.lab.eng.blr.redhat.com:/rhs/brick4/3 24021 Y17653 Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick4/3 24020 Y1960 Brick rhsauto018.lab.eng.blr.redhat.com:/rhs/brick3/8 24026 Y20061 Brick rhsauto038.lab.eng.blr.redhat.com:/rhs/brick3/8 24019 Y19293 Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick3/8 24025 Y12986 Brick rhsauto031.lab.eng.blr.redhat.com:/rhs/brick3/6 24026 Y12992 Brick rhsauto018.lab.eng.blr.redhat.com:/rhs/brick3/6 24027 Y20066 Brick rhsauto038.lab.eng.blr.redhat.com:/rhs/brick3/6 24020 Y19298 NFS Server on localhost 38467 Y19305 Self-heal Daemon on localhost N/A Y19311 NFS Server on rhsauto018.lab.eng.blr.redhat.com 38467 Y20073 Self-heal Daemon on rhsauto018.lab.eng.blr.redhat.com N/A Y20079 NFS Server on rhsauto031.lab.eng.blr.redhat.com 38467 Y12998 Self-heal Daemon on rhsauto031.lab.eng.blr.redhat.com N/A Y13004 Additional info:
Cloning this bug in 3.1. To be fixed in future release.