Bug 812230
| Summary: | Quota: quota show wrong value and log full of "quota context not set in inode" | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Anthony <sokar6012> | |
| Component: | quota | Assignee: | vpshastry <vshastry> | |
| Status: | CLOSED DUPLICATE | QA Contact: | ||
| Severity: | low | Docs Contact: | ||
| Priority: | medium | |||
| Version: | 3.2.5 | CC: | gluster-bugs, ndevos, nsathyan, rwheeler, sokar6012 | |
| Target Milestone: | --- | Keywords: | Triaged | |
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 848250 (view as bug list) | Environment: | ||
| Last Closed: | 2014-07-11 15:08:58 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 848250 | |||
Don't know if it's related but I got plenty of file descriptor staying open : lsof -n -P | grep deleted ksh 1184 u347750 3u REG 253,6 28 13 /tmp/ast23.u8m (deleted) ksh 11950 cdui 2w REG 253,10 952 190 /users/mmr00/log/0905mmr.log (deleted) glusterfs 28600 root 11u REG 253,6 0 25 /tmp/tmpfW7uOgu (deleted) glusterfs 28606 root 10u REG 253,6 0 28 /tmp/tmpfy6HZgu (deleted) glusterfs 28614 root 10u REG 253,6 0 43 /tmp/tmpfCSySeu (deleted) glusterfs 28614 root 22u REG 253,13 13499473920 109 /users3/poolsave/yval9000/test/tmp/27-04-16-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 23u REG 253,13 13510993920 1329 /users3/poolsave/yval9000/test/tmp/28-04-03-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 24u REG 253,13 13511403520 1341 /users3/poolsave/yval9000/test/tmp/28-04-04-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 25u REG 253,13 13513953280 1346 /users3/poolsave/yval9000/test/tmp/28-04-05-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 26u REG 253,13 13481103360 1351 /users3/poolsave/yval9000/test/tmp/28-04-12-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 27u REG 253,13 13501521920 1356 /users3/poolsave/yval9000/test/tmp/28-04-18-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 28u REG 253,13 13530583040 1361 /users3/poolsave/yval9000/test/tmp/29-04-00-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 29u REG 253,13 13490493440 1366 /users3/poolsave/yval9000/test/tmp/29-04-01-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 30u REG 253,13 13486848000 1371 /users3/poolsave/yval9000/test/tmp/29-04-06-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 31u REG 253,13 13500446720 1376 /users3/poolsave/yval9000/test/tmp/29-04-17-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 32u REG 253,13 13510891520 1381 /users3/poolsave/yval9000/test/tmp/29-04-20-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 33u REG 253,13 13448273920 1386 /users3/poolsave/yval9000/test/tmp/30-04-03-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 34u REG 253,13 13451304960 1391 /users3/poolsave/yval9000/test/tmp/30-04-04-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 35u REG 253,13 13454643200 1396 /users3/poolsave/yval9000/test/tmp/30-04-05-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 36u REG 253,13 13487892480 1401 /users3/poolsave/yval9000/test/tmp/30-04-14-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 37u REG 253,13 13472409600 1406 /users3/poolsave/yval9000/test/tmp/30-04-17-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 38u REG 253,13 13505894400 1411 /users3/poolsave/yval9000/test/tmp/01-05-00-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 39u REG 253,13 13469276160 1420 /users3/poolsave/yval9000/test/tmp/01-05-01-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 40u REG 253,13 13481236480 1425 /users3/poolsave/yval9000/test/tmp/01-05-03-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 41u REG 253,13 13504051200 1442 /users3/poolsave/yval9000/test/tmp/01-05-04-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 42u REG 253,13 13566986240 1447 /users3/poolsave/yval9000/test/tmp/01-05-08-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 43u REG 253,13 13586114560 1452 /users3/poolsave/yval9000/test/tmp/01-05-09-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 44u REG 253,13 13597675520 1457 /users3/poolsave/yval9000/test/tmp/01-05-11-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 45u REG 253,13 13618135040 1462 /users3/poolsave/yval9000/test/tmp/01-05-13-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 46u REG 253,13 13642618880 1467 /users3/poolsave/yval9000/test/tmp/01-05-15-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 47u REG 253,13 13683333120 1472 /users3/poolsave/yval9000/test/tmp/01-05-18-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 48u REG 253,13 13694607360 1477 /users3/poolsave/yval9000/test/tmp/01-05-22-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 49u REG 253,13 13653626880 1482 /users3/poolsave/yval9000/test/tmp/02-05-06-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 50u REG 253,13 13684602880 1487 /users3/poolsave/yval9000/test/tmp/02-05-15-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 51u REG 253,13 13697372160 1512 /users3/poolsave/yval9000/test/tmp/03-05-09-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 52u REG 253,13 13707724800 27 /users3/poolsave/yval9000/test/tmp/03-05-13-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 53u REG 253,13 13725112320 32 /users3/poolsave/yval9000/test/tmp/03-05-16-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 54u REG 253,13 13704785920 37 /users3/poolsave/yval9000/test/tmp/03-05-21-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 55u REG 253,13 13707335680 42 /users3/poolsave/yval9000/test/tmp/03-05-22-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 56u REG 253,13 13721405440 47 /users3/poolsave/yval9000/test/tmp/04-05-00-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 57u REG 253,13 13652213760 52 /users3/poolsave/yval9000/test/tmp/04-05-01-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 58u REG 253,13 13658071040 57 /users3/poolsave/yval9000/test/tmp/04-05-02-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 59u REG 253,13 13659729920 62 /users3/poolsave/yval9000/test/tmp/04-05-04-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 60u REG 253,13 13670010880 67 /users3/poolsave/yval9000/test/tmp/04-05-06-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 61u REG 253,13 13679452160 72 /users3/poolsave/yval9000/test/tmp/04-05-10-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 62u REG 253,13 13667594240 77 /users3/poolsave/yval9000/test/tmp/04-05-13-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 63u REG 253,13 13689886720 86 /users3/poolsave/yval9000/test/tmp/04-05-18-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 64u REG 253,13 13673891840 1331 /users3/poolsave/yval9000/test/tmp/05-05-14-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 65u REG 253,13 13694126080 1336 /users3/poolsave/yval9000/test/tmp/05-05-19-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 66u REG 253,13 13702410240 1525 /users3/poolsave/yval9000/test/gluster/05-05-21-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 67u REG 253,13 13705379840 1718 /users3/poolsave/yval9000/test/gluster/05-05-22-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 68u REG 253,13 13663406080 1732 /users3/poolsave/yval9000/test/gluster/06-05-05-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 69u REG 253,13 13681684480 1746 /users3/poolsave/yval9000/test/tmp/06-05-09-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 70u REG 253,13 13683435520 1759 /users3/poolsave/yval9000/test/gluster/06-05-10-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 71u REG 253,13 13687132160 1773 /users3/poolsave/yval9000/test/tmp/06-05-11-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 72u REG 253,13 13695488000 1786 /users3/poolsave/yval9000/test/gluster/06-05-14-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 73u REG 253,13 13702615040 1800 /users3/poolsave/yval9000/test/gluster/06-05-15-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 74u REG 253,13 13705953280 1814 /users3/poolsave/yval9000/test/tmp/06-05-16-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 75u REG 253,13 13707745280 1823 /users3/poolsave/yval9000/test/gluster/06-05-17-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 76u REG 253,13 13698611200 1837 /users3/poolsave/yval9000/test/gluster/06-05-21-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 77u REG 253,13 13705666560 1851 /users3/poolsave/yval9000/test/gluster/06-05-22-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 78u REG 253,13 13666662400 1865 /users3/poolsave/yval9000/test/tmp/07-05-01-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 79u REG 253,13 13675653120 1878 /users3/poolsave/yval9000/test/gluster/07-05-02-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 80u REG 253,13 13683128320 1892 /users3/poolsave/yval9000/test/gluster/07-05-09-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 81u REG 253,13 13682421760 1906 /users3/poolsave/yval9000/test/tmp/07-05-10-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 82u REG 253,13 13686323200 1919 /users3/poolsave/yval9000/test/tmp/07-05-11-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 83u REG 253,13 13693808640 1928 /users3/poolsave/yval9000/test/tmp/07-05-12-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 84u REG 253,13 13717125120 1941 /users3/poolsave/yval9000/test/tmp/07-05-20-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 85u REG 253,13 13730764800 1954 /users3/poolsave/yval9000/test/tmp/07-05-22-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 86u REG 253,13 13671157760 1963 /users3/poolsave/yval9000/test/tmp/08-05-13-00_test_glusterfs_save.tar (deleted) glusterfs 28614 root 87u REG 253,13 13277577216 1976 /users3/poolsave/yval9000/test/tmp/08-05-14-00_test_glusterfs_save.tar (deleted) Furthermore when the file is moved from folder tmp to gluster, the file is to big to be moved in real time so a link is created(0 byte file) (see http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/) what happen if the file is deleted before it was moved on the proper brick ? ylal3510:/users3/poolsave/yval9000/test # ls -lR .: total 0 drwxr-xr-x 2 root root 104 May 9 15:13 gluster drwxr-xr-x 2 root root 48 May 9 15:13 tmp ./gluster: total 13376170 -rw-r--r-- 1 root root 13683834880 May 9 15:13 09-05-15-00_test_glusterfs_save.tar ylal3530:/users3/poolsave/yval9000/test # ls -lR .: total 0 drwxr-xr-x 2 root root 104 May 9 15:13 gluster drwxr-xr-x 2 root root 48 May 9 15:13 tmp ./gluster: total 13376170 -rw-r--r-- 1 root root 13683834880 May 9 15:13 09-05-15-00_test_glusterfs_save.tar ylal3520:/users3/poolsave/yval9000/test # ls -lR .: total 0 drwxr-xr-x 2 root root 104 May 9 15:13 gluster drwxr-xr-x 2 root root 48 May 9 15:13 tmp ./gluster: total 0 ---------T 1 root root 0 May 9 15:13 09-05-15-00_test_glusterfs_save.tar ylal3540:/users3/poolsave/yval9000/test # ls -lR .: total 0 drwxr-xr-x 2 root root 104 May 9 15:13 gluster drwxr-xr-x 2 root root 48 May 9 15:13 tmp ./gluster: total 0 ---------T 1 root root 0 May 9 15:13 09-05-15-00_test_glusterfs_save.tar This is because of the NFS's FH based access to the files. We construct the inode context in the lookup calls. Absence of the lookup calls in NFS mount leads to NULL context. Change http://review.gluster.org/#/c/5951 tries to solve this issue by building the path given a gfid. *** This bug has been marked as a duplicate of bug 990028 *** |
Configuration : Volume Name: poolsave Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: ylal3510:/users3/poolsave Brick2: ylal3530:/users3/poolsave Brick3: ylal3520:/users3/poolsave Brick4: ylal3540:/users3/poolsave Options Reconfigured: features.limit-usage: /yval9000:1TB diagnostics.count-fop-hits: on diagnostics.latency-measurement: on performance.io-thread-count: 64 nfs.port: 2049 performance.cache-refresh-timeout: 2 performance.cache-max-file-size: 4GB performance.cache-min-file-size: 1KB network.ping-timeout: 60 performance.cache-size: 6GB features.quota: on features.quota-timeout: 30 nfs.enable-ino32: off Client : yval9000 mount -t nfs -o rw,nolock,hard,bg,timeo=600,actimeo=0,nointr,rsize=32768,wsize=32768,proto=tcp,vers=3 ylal3510:/poolsave/yval9000 /users98 Description of problem: Quota show wrong value # gluster volume quota poolsave list path limit_set size ---------------------------------------------------------------------------------- /yval9000 100GB 100.0GB But in reality du show me : 6.4G ./yval9000 Moreover when I access the folder I got plenty of : [2012-04-13 09:33:58.766025] W [quota.c:2165:quota_fstat_cbk] 0-poolsave-quota: quota context not set in inode (ino:-2969972712, gfid:9f59f832-ee26-455f-8230-a42e254c43de) [2012-04-13 09:33:58.768169] W [quota.c:2165:quota_fstat_cbk] 0-poolsave-quota: quota context not set in inode (ino:-2969972712, gfid:9f59f832-ee26-455f-8230-a42e254c43de) It seems that quota continuously get added to the previous value Here is xattr on each brick ylal3510 : getfattr -h -e hex -d -m . /users3/poolsave/yval9000/ getfattr: Removing leading '/' from absolute path names # file: users3/poolsave/yval9000/ trusted.gfid=0xec0c049e78b94972b14c2fa60f503e29 trusted.glusterfs-quota-du=0x36343839343337323634 trusted.glusterfs.dht=0x0000000100000000000000007ffffffe trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000003662a0c00 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x00000003662a0c00 ylal3520 : getfattr -h -e hex -d -m . /users3/poolsave/yval9000/ getfattr: Removing leading '/' from absolute path names # file: users3/poolsave/yval9000/ trusted.gfid=0xec0c049e78b94972b14c2fa60f503e29 trusted.glusterfs-quota-du=0x36343839343337323634 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000001599da1e00 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x0000001599da1e00 ylal3530 : getfattr -h -e hex -d -m . /users3/poolsave/yval9000/ getfattr: Removing leading '/' from absolute path names # file: users3/poolsave/yval9000/ trusted.gfid=0xec0c049e78b94972b14c2fa60f503e29 trusted.glusterfs-quota-du=0x36343839343337323634 trusted.glusterfs.dht=0x0000000100000000000000007ffffffe trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000003662a0c00 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x00000003662a0c00 ylal3540 : getfattr -h -e hex -d -m . /users3/poolsave/yval9000/ getfattr: Removing leading '/' from absolute path names # file: users3/poolsave/yval9000/ trusted.gfid=0xec0c049e78b94972b14c2fa60f503e29 trusted.glusterfs-quota-du=0x36343839343337323634 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000001599d71e00 trusted.glusterfs.quota.dirty=0x3000 trusted.glusterfs.quota.size=0x0000001599d71e00 How reproducible: I'm using 2 scripts running continuously : test_save.sh is executed each hour,it takes a bunch of data to compress (dest : REP_SAVE_TEMP) and then move it in a folder (REP_SAVE) that the netback.sh script will scan each 30 min #!/usr/bin/ksh # ________________________________________________________________________ # | # Nom test_save.sh # ____________|___________________________________________________________ # | # Description | test GlusterFS # ____________|___________________________________________________________ UNIXSAVE=/users98/test REP_SAVE_TEMP=${UNIXSAVE}/tmp REP_SAVE=${UNIXSAVE}/gluster LOG=/users/glusterfs_test f_tar_mv() { echo "\n" ARCHNAME=${REP_SAVE_TEMP}/`date +%d-%m-%H-%M`_${SUBNAME}.tar tar -cpvf ${ARCHNAME} ${REPERTOIRE} echo "creation of ${ARCHNAME}" # mv ${REP_SAVE_TEMP}/*_${SUBNAME}.tar ${REP_SAVE} mv ${REP_SAVE_TEMP}/* ${REP_SAVE} echo "Moving archive in ${REP_SAVE} " echo "\n" return $? } REPERTOIRE="/users2/" SUBNAME="test_glusterfs_save" f_tar_mv >$LOG/save_`date +%d-%m-%Y-%H-%M`.log 2>&1 #!/usr/bin/ksh # ________________________________________________________________________ # | # Nom netback.sh # ____________|___________________________________________________________ # | # Description | Sauvegarde test GlusterFS # ____________|___________________________________________________________ UNIXSAVE=/users98/test REP_SAVE_TEMP=${UNIXSAVE}/tmp REP_SAVE=${UNIXSAVE}/gluster LOG=/users/glusterfs_test f_net_back() { if [[ `find ${REP_SAVE} -type f | wc -l` -eq 0 ]] then echo "nothing to save"; else echo "Simulation netbackup, tar in /dev/null" tar -cpvf /dev/null ${REP_SAVE}/* echo "deletion archive" rm ${REP_SAVE}/* fi return $? } f_net_back >${LOG}/netback_`date +%d-%m-%H-%M`.log 2>&1 Actual results: Quota grows indefinitely