Description of problem: I tried to do add-brick while files of 1MB were getting created on the nfs mount-point. And started getting "No such file or directory" on the client. This quota limit-set was 1GB on the root of the volume. Version-Release number of selected component (if applicable): glusterfs3.4.0.35rhs How reproducible: happen to be seen this time Steps to Reproduce: 1. create a volume, start it 2. enable quota and set 1GB limit on the root of the volume 3. mount it over nfs 4. create one directory inside the mount-point 5. inside this directory start creating 1MB file in a loop. 6. while I/O is going trigger add-brick 7. trigger rebalance Actual results: Around step6 and step7 I found that the client is throwing "No such file or directory" errors dd: opening `f.1049': No such file or directory real 0m0.004s user 0m0.000s sys 0m0.001s 1049 dd: opening `f.1050': No such file or directory real 0m0.004s user 0m0.000s sys 0m0.000s even after a delay dd was not successful [root@konsoul dir]# time dd if=/dev/urandom of=f.n bs=1024 count=1024 dd: opening `f.n': No such file or directory real 0m0.013s user 0m0.000s sys 0m0.002s in the nfs.log, I can see messages of this kind. [2013-10-22 18:58:44.145811] W [dht-layout.c:179:dht_layout_search] 0-dist-rep2-dht: no subvolume for hash (value) = 3856949515 [2013-10-22 18:58:44.150914] W [dht-layout.c:179:dht_layout_search] 0-dist-rep2-dht: no subvolume for hash (value) = 3856949515 [2013-10-22 18:58:44.151052] E [fd.c:536:fd_unref] (-->/usr/lib64/glusterfs/3.4.0.35rhs/xlator/cluster/distribute.so(dht_create+0x37b) [0x7fc924bbce2b] (-->/usr/lib64/glusterfs/3.4.0.35rhs/xlator/debug/io-stats.so(io_stats_create_cbk+0x260) [0x7fc9247842f0] (-->/usr/lib64/glusterfs/3.4.0.35rhs/xlator/nfs/server.so(nfs_fop_create_cbk+0x99) [0x7fc92453a379]))) 0-fd: fd is NULL [2013-10-22 18:58:44.151080] W [nfs3.c:2360:nfs3svc_create_cbk] 0-nfs: 419d317b: /dir/f.n => -1 (No such file or directory) [2013-10-22 18:58:44.151299] W [nfs3-helpers.c:3460:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 419d317b, CREATE: NFS: 2(No such file or directory), POSIX: 2(No such file or directory), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 also, [root@quota1 ~]# time gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 100.0MB 924.0MB Expected results: add-brick should pass without any errors Additional info:
rm -rf results in rm: cannot remove `dir/f.85': Invalid argument rm: cannot remove `dir/f.21': Invalid argument rm: cannot remove `dir/f.19': Invalid argument rm: cannot remove `dir/f.25': Invalid argument rm: cannot remove `dir/f.29': Invalid argument rm: cannot remove `dir/f.62': Invalid argument [root@konsoul nfs-test]# ls dir Status of volume: dist-rep2 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.42.186:/rhs/brick1/d1r12 49183 Y 567 Brick 10.70.43.181:/rhs/brick1/d1r22 49155 Y 31450 Brick 10.70.43.18:/rhs/brick1/d2r12 49167 Y 21726 Brick 10.70.43.22:/rhs/brick1/d2r22 49155 Y 29694 Brick 10.70.42.186:/rhs/brick1/d3r12 49184 Y 578 Brick 10.70.43.181:/rhs/brick1/d3r22 49156 Y 31461 Brick 10.70.43.18:/rhs/brick1/d4r12 49168 Y 21737 Brick 10.70.43.22:/rhs/brick1/d4r22 49156 Y 29705 Brick 10.70.42.186:/rhs/brick1/d5r12 49185 Y 589 Brick 10.70.43.181:/rhs/brick1/d5r22 49157 Y 31472 Brick 10.70.43.18:/rhs/brick1/d6r12 49169 Y 21748 Brick 10.70.43.22:/rhs/brick1/d6r22 49157 Y 29716 Brick 10.70.42.186:/rhs/brick1/d1r12-add 49186 Y 3590 Brick 10.70.43.181:/rhs/brick1/d1r22-add 49158 Y 1378 Brick 10.70.42.186:/rhs/brick1/d1r12-addd 49187 Y 3707 Brick 10.70.43.181:/rhs/brick1/d1r22-addd 49159 Y 1448 NFS Server on localhost 2049 Y 1466 Self-heal Daemon on localhost N/A Y 1473 Quota Daemon on localhost N/A Y 1480 NFS Server on 10.70.43.18 2049 Y 24168 Self-heal Daemon on 10.70.43.18 N/A Y 24175 Quota Daemon on 10.70.43.18 N/A Y 24182 NFS Server on 10.70.43.22 2049 Y 32127 Self-heal Daemon on 10.70.43.22 N/A Y 32134 Quota Daemon on 10.70.43.22 N/A Y 32141 NFS Server on 10.70.42.186 2049 Y 3719 Self-heal Daemon on 10.70.42.186 N/A Y 3726 Quota Daemon on 10.70.42.186 N/A Y 3733 Task ID Status ---- -- ------ Rebalance 1342e5b4-157c-4754-9590-1bd5fa05a38c 3
Here is the update for bug with quota disabled. [root@vm1 BIGBEND]# gvi Volume Name: test1 Type: Distribute Volume ID: 1cb54af6-3ec5-4a5d-bd20-79c167576302 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 192.168.122.240:/brick2/x Brick2: 192.168.122.240:/brick2/z Options Reconfigured: features.quota: off [root@vm1 BIGBEND]# mount -t nfs -o vers=3 192.168.122.240:/test1 /home/ [root@vm1 BIGBEND]# Output after add-brick is triggered in the middle of I/O. dd: opening `file2082': No such file or directory dd: opening `file2083': No such file or directory dd: opening `file2084': No such file or directory dd: opening `file2085': No such file or directory dd: opening `file2086': No such file or directory dd: opening `file2087': No such file or directory dd: opening `file2088': No such file or directory dd: opening `file2089': No such file or directory dd: opening `file2090': No such file or directory dd: opening `file2091': No such file or directory dd: opening `file2092': No such file or directory dd: opening `file2093': No such file or directory dd: opening `file2094': No such file or directory dd: opening `file2095': No such file or directory dd: opening `file2096': No such file or directory dd: opening `file2097': No such file or directory dd: opening `file2098': No such file or directory dd: opening `file2099': No such file or directory dd: opening `file2100': No such file or directory [root@vm1 dir]# ls file1 file13 file17 file20 file24 file28 file31 file35 file39 file42 file46 file6 file10 file14 file18 file21 file25 file29 file32 file36 file4 file43 file47 file7 file11 file15 file19 file22 file26 file3 file33 file37 file40 file44 file48 file8 file12 file16 file2 file23 file27 file30 file34 file38 file41 file45 file5 file9 [root@vm1 dir]# rm -rf file1 rm: cannot remove `file1': Invalid argument [root@vm1 dir]# rm -rf file20 rm: cannot remove `file20': Invalid argument [root@vm1 ~]# gvi Volume Name: test1 Type: Distribute Volume ID: 1cb54af6-3ec5-4a5d-bd20-79c167576302 Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 192.168.122.240:/brick2/x Brick2: 192.168.122.240:/brick2/z Brick3: 192.168.122.240:/brick2/new Options Reconfigured: features.quota: off
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days