Bug 1022031
| Summary: | quota: add-brick causes "No such file or directory" | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Saurabh <saujain> | |
| Component: | distribute | Assignee: | Susant Kumar Palai <spalai> | |
| Status: | CLOSED DEFERRED | QA Contact: | storage-qa-internal <storage-qa-internal> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 2.1 | CC: | mzywusko, saujain, spalai, vbellur, vmallika | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1286195 (view as bug list) | Environment: | ||
| Last Closed: | 2015-11-27 12:23:10 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1286195 | |||
rm -rf results in
rm: cannot remove `dir/f.85': Invalid argument
rm: cannot remove `dir/f.21': Invalid argument
rm: cannot remove `dir/f.19': Invalid argument
rm: cannot remove `dir/f.25': Invalid argument
rm: cannot remove `dir/f.29': Invalid argument
rm: cannot remove `dir/f.62': Invalid argument
[root@konsoul nfs-test]# ls
dir
Status of volume: dist-rep2
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick 10.70.42.186:/rhs/brick1/d1r12 49183 Y 567
Brick 10.70.43.181:/rhs/brick1/d1r22 49155 Y 31450
Brick 10.70.43.18:/rhs/brick1/d2r12 49167 Y 21726
Brick 10.70.43.22:/rhs/brick1/d2r22 49155 Y 29694
Brick 10.70.42.186:/rhs/brick1/d3r12 49184 Y 578
Brick 10.70.43.181:/rhs/brick1/d3r22 49156 Y 31461
Brick 10.70.43.18:/rhs/brick1/d4r12 49168 Y 21737
Brick 10.70.43.22:/rhs/brick1/d4r22 49156 Y 29705
Brick 10.70.42.186:/rhs/brick1/d5r12 49185 Y 589
Brick 10.70.43.181:/rhs/brick1/d5r22 49157 Y 31472
Brick 10.70.43.18:/rhs/brick1/d6r12 49169 Y 21748
Brick 10.70.43.22:/rhs/brick1/d6r22 49157 Y 29716
Brick 10.70.42.186:/rhs/brick1/d1r12-add 49186 Y 3590
Brick 10.70.43.181:/rhs/brick1/d1r22-add 49158 Y 1378
Brick 10.70.42.186:/rhs/brick1/d1r12-addd 49187 Y 3707
Brick 10.70.43.181:/rhs/brick1/d1r22-addd 49159 Y 1448
NFS Server on localhost 2049 Y 1466
Self-heal Daemon on localhost N/A Y 1473
Quota Daemon on localhost N/A Y 1480
NFS Server on 10.70.43.18 2049 Y 24168
Self-heal Daemon on 10.70.43.18 N/A Y 24175
Quota Daemon on 10.70.43.18 N/A Y 24182
NFS Server on 10.70.43.22 2049 Y 32127
Self-heal Daemon on 10.70.43.22 N/A Y 32134
Quota Daemon on 10.70.43.22 N/A Y 32141
NFS Server on 10.70.42.186 2049 Y 3719
Self-heal Daemon on 10.70.42.186 N/A Y 3726
Quota Daemon on 10.70.42.186 N/A Y 3733
Task ID Status
---- -- ------
Rebalance 1342e5b4-157c-4754-9590-1bd5fa05a38c 3
Here is the update for bug with quota disabled. [root@vm1 BIGBEND]# gvi Volume Name: test1 Type: Distribute Volume ID: 1cb54af6-3ec5-4a5d-bd20-79c167576302 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 192.168.122.240:/brick2/x Brick2: 192.168.122.240:/brick2/z Options Reconfigured: features.quota: off [root@vm1 BIGBEND]# mount -t nfs -o vers=3 192.168.122.240:/test1 /home/ [root@vm1 BIGBEND]# Output after add-brick is triggered in the middle of I/O. dd: opening `file2082': No such file or directory dd: opening `file2083': No such file or directory dd: opening `file2084': No such file or directory dd: opening `file2085': No such file or directory dd: opening `file2086': No such file or directory dd: opening `file2087': No such file or directory dd: opening `file2088': No such file or directory dd: opening `file2089': No such file or directory dd: opening `file2090': No such file or directory dd: opening `file2091': No such file or directory dd: opening `file2092': No such file or directory dd: opening `file2093': No such file or directory dd: opening `file2094': No such file or directory dd: opening `file2095': No such file or directory dd: opening `file2096': No such file or directory dd: opening `file2097': No such file or directory dd: opening `file2098': No such file or directory dd: opening `file2099': No such file or directory dd: opening `file2100': No such file or directory [root@vm1 dir]# ls file1 file13 file17 file20 file24 file28 file31 file35 file39 file42 file46 file6 file10 file14 file18 file21 file25 file29 file32 file36 file4 file43 file47 file7 file11 file15 file19 file22 file26 file3 file33 file37 file40 file44 file48 file8 file12 file16 file2 file23 file27 file30 file34 file38 file41 file45 file5 file9 [root@vm1 dir]# rm -rf file1 rm: cannot remove `file1': Invalid argument [root@vm1 dir]# rm -rf file20 rm: cannot remove `file20': Invalid argument [root@vm1 ~]# gvi Volume Name: test1 Type: Distribute Volume ID: 1cb54af6-3ec5-4a5d-bd20-79c167576302 Status: Started Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: 192.168.122.240:/brick2/x Brick2: 192.168.122.240:/brick2/z Brick3: 192.168.122.240:/brick2/new Options Reconfigured: features.quota: off The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: I tried to do add-brick while files of 1MB were getting created on the nfs mount-point. And started getting "No such file or directory" on the client. This quota limit-set was 1GB on the root of the volume. Version-Release number of selected component (if applicable): glusterfs3.4.0.35rhs How reproducible: happen to be seen this time Steps to Reproduce: 1. create a volume, start it 2. enable quota and set 1GB limit on the root of the volume 3. mount it over nfs 4. create one directory inside the mount-point 5. inside this directory start creating 1MB file in a loop. 6. while I/O is going trigger add-brick 7. trigger rebalance Actual results: Around step6 and step7 I found that the client is throwing "No such file or directory" errors dd: opening `f.1049': No such file or directory real 0m0.004s user 0m0.000s sys 0m0.001s 1049 dd: opening `f.1050': No such file or directory real 0m0.004s user 0m0.000s sys 0m0.000s even after a delay dd was not successful [root@konsoul dir]# time dd if=/dev/urandom of=f.n bs=1024 count=1024 dd: opening `f.n': No such file or directory real 0m0.013s user 0m0.000s sys 0m0.002s in the nfs.log, I can see messages of this kind. [2013-10-22 18:58:44.145811] W [dht-layout.c:179:dht_layout_search] 0-dist-rep2-dht: no subvolume for hash (value) = 3856949515 [2013-10-22 18:58:44.150914] W [dht-layout.c:179:dht_layout_search] 0-dist-rep2-dht: no subvolume for hash (value) = 3856949515 [2013-10-22 18:58:44.151052] E [fd.c:536:fd_unref] (-->/usr/lib64/glusterfs/3.4.0.35rhs/xlator/cluster/distribute.so(dht_create+0x37b) [0x7fc924bbce2b] (-->/usr/lib64/glusterfs/3.4.0.35rhs/xlator/debug/io-stats.so(io_stats_create_cbk+0x260) [0x7fc9247842f0] (-->/usr/lib64/glusterfs/3.4.0.35rhs/xlator/nfs/server.so(nfs_fop_create_cbk+0x99) [0x7fc92453a379]))) 0-fd: fd is NULL [2013-10-22 18:58:44.151080] W [nfs3.c:2360:nfs3svc_create_cbk] 0-nfs: 419d317b: /dir/f.n => -1 (No such file or directory) [2013-10-22 18:58:44.151299] W [nfs3-helpers.c:3460:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 419d317b, CREATE: NFS: 2(No such file or directory), POSIX: 2(No such file or directory), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 also, [root@quota1 ~]# time gluster volume quota dist-rep2 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0GB 80% 100.0MB 924.0MB Expected results: add-brick should pass without any errors Additional info: