Bug 1023149 - Quota limit set to higher value doesn't comes to effect after remove-brick [NEEDINFO]
Quota limit set to higher value doesn't comes to effect after remove-brick
Status: CLOSED WORKSFORME
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: quota (Show other bugs)
2.1
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Vijaikumar Mallikarjuna
storage-qa-internal@redhat.com
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-24 13:39 EDT by shylesh
Modified: 2016-09-17 08:42 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-17 03:50:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
spalai: needinfo? (shmohan)


Attachments (Terms of Use)

  None (edit)
Description shylesh 2013-10-24 13:39:01 EDT
Description of problem:
After starting the remove-brick if we change the quota limit to some higher value doesn't comes into effect and old quota limit will be still imposed.

Version-Release number of selected component (if applicable):
3.4.0.36rhs-1.el6rhs.x86_64

How reproducible:
always

Steps to Reproduce:

1.I had a 16x2 distributed-replicate volume

2.creates sub directories root and test

3. set the quota limit on both directories as 1GB
[root@rhs-client4 test]# gluster v quota s1 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/root                                      1.0GB       80%    1005.2MB  18.8MB
/test                                      1.0GB       80%       1.0GB  0MB

4. created a subdirectory inside test and filled up with files till i get "Disk quota exceeded"

5. started removing one of the pair using remove-brick start option
gluster v remove-brick s1 rhs-client9.lab.eng.blr.redhat.com:/home/s130 rhs-client39.lab.eng.blr.redhat.com:/home/s131 start

6. once the status is completed , without committing changed the quota limit on test directory to a higher value

[root@rhs-client4 test]# gluster v quota s1 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/root                                      1.0GB       80%    1005.2MB  18.8MB
/test                                      2.0GB       80%       1.0GB  1.0GB


7. now create another directory inside /test say test/dir1

8. try to create files inside this directory 

9. some files got created and some files not , they failed with 
"Disk quota exceeded"

Actual results:



Expected results:
New quota value should take effect and data creation should be possible since we have increased the quota limit

Additional info:


cluster info
------------
rhs-client4.lab.eng.blr.redhat.com
rhs-client40.lab.eng.blr.redhat.com
rhs-client39.lab.eng.blr.redhat.com
rhs-client9.lab.eng.blr.redhat.com

mount point
=============
rhs-client4.lab.eng.blr.redhat.com:/mnt

mount type
----------
NFS


Volume Name: s1
Type: Distributed-Replicate
Volume ID: 57f48f8a-d3d3-4e7c-8ea1-0240a17e37f0
Status: Started
Number of Bricks: 15 x 2 = 30
Transport-type: tcp
Bricks:
Brick1: rhs-client4.lab.eng.blr.redhat.com:/home/s10
Brick2: rhs-client9.lab.eng.blr.redhat.com:/home/s11
Brick3: rhs-client39.lab.eng.blr.redhat.com:/home/s12
Brick4: rhs-client4.lab.eng.blr.redhat.com:/home/s13
Brick5: rhs-client9.lab.eng.blr.redhat.com:/home/s14
Brick6: rhs-client39.lab.eng.blr.redhat.com:/home/s15
Brick7: rhs-client4.lab.eng.blr.redhat.com:/home/s16
Brick8: rhs-client9.lab.eng.blr.redhat.com:/home/s17
Brick9: rhs-client39.lab.eng.blr.redhat.com:/home/s18
Brick10: rhs-client4.lab.eng.blr.redhat.com:/home/s19
Brick11: rhs-client9.lab.eng.blr.redhat.com:/home/s110
Brick12: rhs-client39.lab.eng.blr.redhat.com:/home/s111
Brick13: rhs-client4.lab.eng.blr.redhat.com:/home/s112
Brick14: rhs-client9.lab.eng.blr.redhat.com:/home/s113
Brick15: rhs-client9.lab.eng.blr.redhat.com:/home/s114
Brick16: rhs-client39.lab.eng.blr.redhat.com:/home/s115
Brick17: rhs-client4.lab.eng.blr.redhat.com:/home/s116
Brick18: rhs-client9.lab.eng.blr.redhat.com:/home/s117
Brick19: rhs-client39.lab.eng.blr.redhat.com:/home/s118
Brick20: rhs-client4.lab.eng.blr.redhat.com:/home/s119
Brick21: rhs-client4.lab.eng.blr.redhat.com:/home/s120
Brick22: rhs-client9.lab.eng.blr.redhat.com:/home/s121
Brick23: rhs-client9.lab.eng.blr.redhat.com:/home/s122
Brick24: rhs-client39.lab.eng.blr.redhat.com:/home/s123
Brick25: rhs-client39.lab.eng.blr.redhat.com:/home/s124
Brick26: rhs-client4.lab.eng.blr.redhat.com:/home/s125
Brick27: rhs-client39.lab.eng.blr.redhat.com:/home/s126
Brick28: rhs-client4.lab.eng.blr.redhat.com:/home/s127
Brick29: rhs-client4.lab.eng.blr.redhat.com:/home/s128
Brick30: rhs-client9.lab.eng.blr.redhat.com:/home/s129
Options Reconfigured:
features.quota: on



[root@rhs-client4 an]# gluster v quota s1 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/root                                      1.0GB       80%    1005.2MB  18.8MB
/test                                      2.0GB       80%       1.1GB 945.2MB


though /test has 1more GB to run out of quota , data creation still fails
1048576 bytes (1.0 MB) copied, 0.208439 s, 5.0 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.207601 s, 5.1 MB/s
dd: opening `97': Disk quota exceeded
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.209932 s, 5.0 MB/s
1+0 records in
1+0 records out


2013-10-24 17:06:11.251030] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-20: remote operation failed: No such file or directory
[2013-10-24 17:06:11.251843] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-20: remote operation failed: No such file or directory
[2013-10-24 17:06:11.252447] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-20: remote operation failed: No such file or directory
[2013-10-24 17:06:11.252617] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-20: remote operation failed: No such file or directory
[2013-10-24 17:06:11.253111] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-20: remote operation failed: No such file or directory
[2013-10-24 17:06:11.255517] W [client-rpc-fops.c:928:client3_3_flush_cbk] 0-s1-client-20: remote operation failed: No such file or directory
[2013-10-24 17:06:11.261413] W [client-rpc-fops.c:2058:client3_3_create_cbk] 0-s1-client-25: remote operation failed: Disk quota exceeded. Path: <gfid:57d9
3e36-2c38-4029-83e4-960de5779bd0>/an/86
[2013-10-24 17:06:11.261487] W [client-rpc-fops.c:2058:client3_3_create_cbk] 0-s1-client-24: remote operation failed: Disk quota exceeded. Path: <gfid:57d9
3e36-2c38-4029-83e4-960de5779bd0>/an/86
[2013-10-24 17:06:11.261515] W [nfs3.c:2360:nfs3svc_create_cbk] 0-nfs: 88ab8c3b: <gfid:57d93e36-2c38-4029-83e4-960de5779bd0>/an/86 => -1 (Disk quota exceed
ed)
[2013-10-24 17:06:11.267027] W [client-rpc-fops.c:2058:client3_3_create_cbk] 0-s1-client-11: remote operation failed: Disk quota exceeded. Path: <gfid:57d9
3e36-2c38-4029-83e4-960de5779bd0>/an/87
[2013-10-24 17:06:11.432273] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.432905] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.433680] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.434305] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.434695] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.435108] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.435571] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.436091] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.436445] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.436715] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.436847] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.436963] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.437074] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.437214] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.437598] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.438225] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.438360] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such file or directory
[2013-10-24 17:06:11.440340] W [client-rpc-fops.c:1579:client3_3_finodelk_cbk] 0-s1-client-11: remote operation failed: No such



attached the sosreports
Comment 3 Susant Kumar Palai 2014-01-13 01:39:59 EST
Tried to reproduce the issue.
[root@vm1 dir2]# glusterfs --version
glusterfs 3.4.0.56rhs built on Jan 10 2014 22:58:18

[root@vm1 ~]# gvi
 
Volume Name: test1
Type: Distributed-Replicate
Volume ID: 7c8fa6e4-3e43-4fe0-a30f-6e8c3b5e87b4
Status: Created
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 192.168.122.240:/brick1/1
Brick2: 192.168.122.240:/brick2/2
Brick3: 192.168.122.240:/brick1/3
Brick4: 192.168.122.240:/brick2/4
Brick5: 192.168.122.240:/brick1/5
Brick6: 192.168.122.240:/brick2/6
Brick7: 192.168.122.240:/brick1/7
Brick8: 192.168.122.240:/brick2/8

[root@vm1 test]# gluster volume quota test1 list 
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/test                                    500.0MB       80%      0Bytes 500.0MB              No                   No
/root                                    500.0MB       80%      0Bytes 500.0MB              No                   No

Used the following script to create files

[root@vm1 dir1]# for i in {1..600} ; do dd if=/dev/zero of=file$i bs=1MB count=1 ; done


[root@vm1 dir1]# gluster volume quota test1 list 
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/test                                    500.0MB       80%     572.5MB  0Bytes             Yes                  Yes
/root                                    500.0MB       80%      0Bytes 500.0MB              No                   No



[root@vm1 dir1]# gluster volume remove-brick test1 192.168.122.240:/brick1/7 192.168.122.240:/brick2/8 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost              298       284.2MB           600             0             0            completed              62.00



 * set limit of test dir to 1GB

[root@vm1 dir1]# gluster volume quota test1 list 
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/test                                      1.0GB       80%     572.5MB 451.5MB              No                   No
/root                                    500.0MB       80%      0Bytes 500.0MB              No                   No
[root@vm1 dir1]# 


Created a directory /test/dir2 and created files under it. But it filled the directory quota limit completely without any error.

[root@vm1 dir2]# gluster volume quota test1 list 
                  Path                   Hard-limit Soft-limit   Used  Available  Soft-limit exceeded? Hard-limit exceeded?
---------------------------------------------------------------------------------------------------------------------------
/test                                      1.0GB       80%       1.0GB  0Bytes             Yes                  Yes
/root                                    500.0MB       80%      0Bytes 500.0MB              No                   No
[root@vm1 dir2]# 



Shylesh can you verify the bug in latest build.
Comment 4 Vijaikumar Mallikarjuna 2015-11-17 03:50:03 EST
Not able to re-create the problem as per comment# 3.
Also 2.1 is EOL'ed, so closing the bug

Note You need to log in before you can comment on or make changes to this bug.