Bug 812230 - Quota: quota show wrong value and log full of "quota context not set in inode"
Quota: quota show wrong value and log full of "quota context not set in inode"
Status: CLOSED DUPLICATE of bug 990028
Product: GlusterFS
Classification: Community
Component: quota (Show other bugs)
3.2.5
x86_64 Linux
medium Severity low
: ---
: ---
Assigned To: vpshastry
: Triaged
Depends On:
Blocks: 848250
  Show dependency treegraph
 
Reported: 2012-04-13 03:57 EDT by Anthony
Modified: 2014-08-11 19:22 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 848250 (view as bug list)
Environment:
Last Closed: 2014-07-11 11:08:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Anthony 2012-04-13 03:57:54 EDT
Configuration : 

Volume Name: poolsave
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: ylal3510:/users3/poolsave
Brick2: ylal3530:/users3/poolsave
Brick3: ylal3520:/users3/poolsave
Brick4: ylal3540:/users3/poolsave
Options Reconfigured:
features.limit-usage: /yval9000:1TB
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
performance.io-thread-count: 64
nfs.port: 2049
performance.cache-refresh-timeout: 2
performance.cache-max-file-size: 4GB
performance.cache-min-file-size: 1KB
network.ping-timeout: 60
performance.cache-size: 6GB
features.quota: on
features.quota-timeout: 30
nfs.enable-ino32: off

Client : yval9000
mount -t nfs -o rw,nolock,hard,bg,timeo=600,actimeo=0,nointr,rsize=32768,wsize=32768,proto=tcp,vers=3 ylal3510:/poolsave/yval9000 /users98


Description of problem:
Quota show wrong value

# gluster volume quota poolsave list
        path              limit_set          size
----------------------------------------------------------------------------------
/yval9000                 100GB              100.0GB

But in reality du show me :

6.4G ./yval9000 

Moreover when I access the folder I got plenty of :

[2012-04-13 09:33:58.766025] W [quota.c:2165:quota_fstat_cbk] 0-poolsave-quota: quota context not set in inode (ino:-2969972712, gfid:9f59f832-ee26-455f-8230-a42e254c43de)
[2012-04-13 09:33:58.768169] W [quota.c:2165:quota_fstat_cbk] 0-poolsave-quota: quota context not set in inode (ino:-2969972712, gfid:9f59f832-ee26-455f-8230-a42e254c43de)

It seems that quota continuously get added to the previous value

Here is xattr on each brick
ylal3510 : 
getfattr -h -e hex -d -m . /users3/poolsave/yval9000/
getfattr: Removing leading '/' from absolute path names
# file: users3/poolsave/yval9000/
trusted.gfid=0xec0c049e78b94972b14c2fa60f503e29
trusted.glusterfs-quota-du=0x36343839343337323634
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000003662a0c00
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x00000003662a0c00

ylal3520 : 
getfattr -h -e hex -d -m . /users3/poolsave/yval9000/
getfattr: Removing leading '/' from absolute path names
# file: users3/poolsave/yval9000/
trusted.gfid=0xec0c049e78b94972b14c2fa60f503e29
trusted.glusterfs-quota-du=0x36343839343337323634
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000001599da1e00
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x0000001599da1e00

ylal3530 : 
getfattr -h -e hex -d -m . /users3/poolsave/yval9000/
getfattr: Removing leading '/' from absolute path names
# file: users3/poolsave/yval9000/
trusted.gfid=0xec0c049e78b94972b14c2fa60f503e29
trusted.glusterfs-quota-du=0x36343839343337323634
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x00000003662a0c00
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x00000003662a0c00

ylal3540 : 
getfattr -h -e hex -d -m . /users3/poolsave/yval9000/
getfattr: Removing leading '/' from absolute path names
# file: users3/poolsave/yval9000/
trusted.gfid=0xec0c049e78b94972b14c2fa60f503e29
trusted.glusterfs-quota-du=0x36343839343337323634
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000001599d71e00
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x0000001599d71e00





How reproducible:
I'm using 2 scripts running continuously  :

test_save.sh is executed each hour,it takes a bunch of data to compress (dest : REP_SAVE_TEMP) and then move it in a folder (REP_SAVE) that the netback.sh script will scan each 30 min

#!/usr/bin/ksh
# ________________________________________________________________________
#             |
# Nom         test_save.sh
# ____________|___________________________________________________________
#             |
# Description | test GlusterFS
# ____________|___________________________________________________________

UNIXSAVE=/users98/test
REP_SAVE_TEMP=${UNIXSAVE}/tmp
REP_SAVE=${UNIXSAVE}/gluster
LOG=/users/glusterfs_test


f_tar_mv()
{
  echo "\n"
ARCHNAME=${REP_SAVE_TEMP}/`date +%d-%m-%H-%M`_${SUBNAME}.tar

  tar -cpvf ${ARCHNAME} ${REPERTOIRE}

  echo "creation of ${ARCHNAME}"


  # mv ${REP_SAVE_TEMP}/*_${SUBNAME}.tar ${REP_SAVE}
  mv  ${REP_SAVE_TEMP}/* ${REP_SAVE}
  echo "Moving archive in ${REP_SAVE} "
  echo "\n"

  return $?
}

REPERTOIRE="/users2/"
SUBNAME="test_glusterfs_save"
f_tar_mv >$LOG/save_`date +%d-%m-%Y-%H-%M`.log 2>&1


#!/usr/bin/ksh
# ________________________________________________________________________
#             |
# Nom         netback.sh
# ____________|___________________________________________________________
#             |
# Description | Sauvegarde test GlusterFS
# ____________|___________________________________________________________

UNIXSAVE=/users98/test
REP_SAVE_TEMP=${UNIXSAVE}/tmp
REP_SAVE=${UNIXSAVE}/gluster
LOG=/users/glusterfs_test

f_net_back()
{
if [[ `find ${REP_SAVE} -type f | wc -l` -eq 0 ]]
then
echo "nothing to save";
else
echo "Simulation netbackup, tar in /dev/null"
tar -cpvf /dev/null ${REP_SAVE}/*
echo "deletion archive"
rm ${REP_SAVE}/*

fi
return $?
}

f_net_back >${LOG}/netback_`date +%d-%m-%H-%M`.log 2>&1


  
Actual results:
Quota grows indefinitely
Comment 1 Anthony 2012-05-09 09:21:15 EDT
Don't know if it's related but I got plenty of file descriptor staying open : 

lsof -n -P | grep deleted
ksh        1184    u347750    3u      REG              253,6          28         13 /tmp/ast23.u8m (deleted)
ksh       11950       cdui    2w      REG             253,10         952        190 /users/mmr00/log/0905mmr.log (deleted)
glusterfs 28600       root   11u      REG              253,6           0         25 /tmp/tmpfW7uOgu (deleted)
glusterfs 28606       root   10u      REG              253,6           0         28 /tmp/tmpfy6HZgu (deleted)
glusterfs 28614       root   10u      REG              253,6           0         43 /tmp/tmpfCSySeu (deleted)
glusterfs 28614       root   22u      REG             253,13 13499473920        109 /users3/poolsave/yval9000/test/tmp/27-04-16-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   23u      REG             253,13 13510993920       1329 /users3/poolsave/yval9000/test/tmp/28-04-03-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   24u      REG             253,13 13511403520       1341 /users3/poolsave/yval9000/test/tmp/28-04-04-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   25u      REG             253,13 13513953280       1346 /users3/poolsave/yval9000/test/tmp/28-04-05-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   26u      REG             253,13 13481103360       1351 /users3/poolsave/yval9000/test/tmp/28-04-12-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   27u      REG             253,13 13501521920       1356 /users3/poolsave/yval9000/test/tmp/28-04-18-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   28u      REG             253,13 13530583040       1361 /users3/poolsave/yval9000/test/tmp/29-04-00-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   29u      REG             253,13 13490493440       1366 /users3/poolsave/yval9000/test/tmp/29-04-01-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   30u      REG             253,13 13486848000       1371 /users3/poolsave/yval9000/test/tmp/29-04-06-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   31u      REG             253,13 13500446720       1376 /users3/poolsave/yval9000/test/tmp/29-04-17-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   32u      REG             253,13 13510891520       1381 /users3/poolsave/yval9000/test/tmp/29-04-20-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   33u      REG             253,13 13448273920       1386 /users3/poolsave/yval9000/test/tmp/30-04-03-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   34u      REG             253,13 13451304960       1391 /users3/poolsave/yval9000/test/tmp/30-04-04-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   35u      REG             253,13 13454643200       1396 /users3/poolsave/yval9000/test/tmp/30-04-05-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   36u      REG             253,13 13487892480       1401 /users3/poolsave/yval9000/test/tmp/30-04-14-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   37u      REG             253,13 13472409600       1406 /users3/poolsave/yval9000/test/tmp/30-04-17-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   38u      REG             253,13 13505894400       1411 /users3/poolsave/yval9000/test/tmp/01-05-00-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   39u      REG             253,13 13469276160       1420 /users3/poolsave/yval9000/test/tmp/01-05-01-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   40u      REG             253,13 13481236480       1425 /users3/poolsave/yval9000/test/tmp/01-05-03-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   41u      REG             253,13 13504051200       1442 /users3/poolsave/yval9000/test/tmp/01-05-04-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   42u      REG             253,13 13566986240       1447 /users3/poolsave/yval9000/test/tmp/01-05-08-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   43u      REG             253,13 13586114560       1452 /users3/poolsave/yval9000/test/tmp/01-05-09-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   44u      REG             253,13 13597675520       1457 /users3/poolsave/yval9000/test/tmp/01-05-11-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   45u      REG             253,13 13618135040       1462 /users3/poolsave/yval9000/test/tmp/01-05-13-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   46u      REG             253,13 13642618880       1467 /users3/poolsave/yval9000/test/tmp/01-05-15-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   47u      REG             253,13 13683333120       1472 /users3/poolsave/yval9000/test/tmp/01-05-18-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   48u      REG             253,13 13694607360       1477 /users3/poolsave/yval9000/test/tmp/01-05-22-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   49u      REG             253,13 13653626880       1482 /users3/poolsave/yval9000/test/tmp/02-05-06-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   50u      REG             253,13 13684602880       1487 /users3/poolsave/yval9000/test/tmp/02-05-15-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   51u      REG             253,13 13697372160       1512 /users3/poolsave/yval9000/test/tmp/03-05-09-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   52u      REG             253,13 13707724800         27 /users3/poolsave/yval9000/test/tmp/03-05-13-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   53u      REG             253,13 13725112320         32 /users3/poolsave/yval9000/test/tmp/03-05-16-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   54u      REG             253,13 13704785920         37 /users3/poolsave/yval9000/test/tmp/03-05-21-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   55u      REG             253,13 13707335680         42 /users3/poolsave/yval9000/test/tmp/03-05-22-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   56u      REG             253,13 13721405440         47 /users3/poolsave/yval9000/test/tmp/04-05-00-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   57u      REG             253,13 13652213760         52 /users3/poolsave/yval9000/test/tmp/04-05-01-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   58u      REG             253,13 13658071040         57 /users3/poolsave/yval9000/test/tmp/04-05-02-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   59u      REG             253,13 13659729920         62 /users3/poolsave/yval9000/test/tmp/04-05-04-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   60u      REG             253,13 13670010880         67 /users3/poolsave/yval9000/test/tmp/04-05-06-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   61u      REG             253,13 13679452160         72 /users3/poolsave/yval9000/test/tmp/04-05-10-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   62u      REG             253,13 13667594240         77 /users3/poolsave/yval9000/test/tmp/04-05-13-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   63u      REG             253,13 13689886720         86 /users3/poolsave/yval9000/test/tmp/04-05-18-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   64u      REG             253,13 13673891840       1331 /users3/poolsave/yval9000/test/tmp/05-05-14-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   65u      REG             253,13 13694126080       1336 /users3/poolsave/yval9000/test/tmp/05-05-19-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   66u      REG             253,13 13702410240       1525 /users3/poolsave/yval9000/test/gluster/05-05-21-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   67u      REG             253,13 13705379840       1718 /users3/poolsave/yval9000/test/gluster/05-05-22-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   68u      REG             253,13 13663406080       1732 /users3/poolsave/yval9000/test/gluster/06-05-05-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   69u      REG             253,13 13681684480       1746 /users3/poolsave/yval9000/test/tmp/06-05-09-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   70u      REG             253,13 13683435520       1759 /users3/poolsave/yval9000/test/gluster/06-05-10-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   71u      REG             253,13 13687132160       1773 /users3/poolsave/yval9000/test/tmp/06-05-11-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   72u      REG             253,13 13695488000       1786 /users3/poolsave/yval9000/test/gluster/06-05-14-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   73u      REG             253,13 13702615040       1800 /users3/poolsave/yval9000/test/gluster/06-05-15-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   74u      REG             253,13 13705953280       1814 /users3/poolsave/yval9000/test/tmp/06-05-16-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   75u      REG             253,13 13707745280       1823 /users3/poolsave/yval9000/test/gluster/06-05-17-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   76u      REG             253,13 13698611200       1837 /users3/poolsave/yval9000/test/gluster/06-05-21-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   77u      REG             253,13 13705666560       1851 /users3/poolsave/yval9000/test/gluster/06-05-22-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   78u      REG             253,13 13666662400       1865 /users3/poolsave/yval9000/test/tmp/07-05-01-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   79u      REG             253,13 13675653120       1878 /users3/poolsave/yval9000/test/gluster/07-05-02-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   80u      REG             253,13 13683128320       1892 /users3/poolsave/yval9000/test/gluster/07-05-09-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   81u      REG             253,13 13682421760       1906 /users3/poolsave/yval9000/test/tmp/07-05-10-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   82u      REG             253,13 13686323200       1919 /users3/poolsave/yval9000/test/tmp/07-05-11-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   83u      REG             253,13 13693808640       1928 /users3/poolsave/yval9000/test/tmp/07-05-12-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   84u      REG             253,13 13717125120       1941 /users3/poolsave/yval9000/test/tmp/07-05-20-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   85u      REG             253,13 13730764800       1954 /users3/poolsave/yval9000/test/tmp/07-05-22-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   86u      REG             253,13 13671157760       1963 /users3/poolsave/yval9000/test/tmp/08-05-13-00_test_glusterfs_save.tar (deleted)
glusterfs 28614       root   87u      REG             253,13 13277577216       1976 /users3/poolsave/yval9000/test/tmp/08-05-14-00_test_glusterfs_save.tar (deleted)


Furthermore  when the file is moved from folder tmp to gluster, the file is to big to be moved in real time so a link is created(0 byte file)
(see http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/)
what happen if the file is deleted before it was moved on the proper brick ?  

ylal3510:/users3/poolsave/yval9000/test # ls -lR
.:
total 0
drwxr-xr-x 2 root root 104 May  9 15:13 gluster
drwxr-xr-x 2 root root  48 May  9 15:13 tmp

./gluster:
total 13376170
-rw-r--r-- 1 root root 13683834880 May  9 15:13 09-05-15-00_test_glusterfs_save.tar

ylal3530:/users3/poolsave/yval9000/test # ls -lR
.:
total 0
drwxr-xr-x 2 root root 104 May  9 15:13 gluster
drwxr-xr-x 2 root root  48 May  9 15:13 tmp

./gluster:
total 13376170
-rw-r--r-- 1 root root 13683834880 May  9 15:13 09-05-15-00_test_glusterfs_save.tar

ylal3520:/users3/poolsave/yval9000/test # ls -lR
.:
total 0
drwxr-xr-x 2 root root 104 May  9 15:13 gluster
drwxr-xr-x 2 root root  48 May  9 15:13 tmp

./gluster:
total 0
---------T 1 root root 0 May  9 15:13 09-05-15-00_test_glusterfs_save.tar


ylal3540:/users3/poolsave/yval9000/test # ls -lR
.:
total 0
drwxr-xr-x 2 root root 104 May  9 15:13 gluster
drwxr-xr-x 2 root root  48 May  9 15:13 tmp

./gluster:
total 0
---------T 1 root root 0 May  9 15:13 09-05-15-00_test_glusterfs_save.tar
Comment 2 vpshastry 2013-09-20 01:33:46 EDT
This is because of the NFS's FH based access to the files. We construct the inode context in the lookup calls. Absence of the lookup calls in NFS mount leads to NULL context.

Change http://review.gluster.org/#/c/5951 tries to solve this issue by building the path given a gfid.
Comment 3 Niels de Vos 2014-07-11 11:08:58 EDT

*** This bug has been marked as a duplicate of bug 990028 ***

Note You need to log in before you can comment on or make changes to this bug.