Bug 802910

Summary: quota: after rebalance quota list keeps displaying different value
Product: [Community] GlusterFS Reporter: Saurabh <saujain>
Component: quotaAssignee: vpshastry <vshastry>
Status: CLOSED WONTFIX QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: pre-releaseCC: amarts, gluster-bugs, mzywusko, nsathyan
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 848248 (view as bug list) Environment:
Last Closed: 2013-02-04 10:58:53 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 848248    

Description Saurabh 2012-03-13 18:20:09 UTC
Description of problem:
simple rebalance
then quota list several time and it keeps displaying different values,
meanwhile on the point the find . | xargs stat was also executed to collect stat info.

Version-Release number of selected component (if applicable):


How reproducible:
happened several timest
1. set quota limits on volume and a dir
2. put some data in them
3. start rebalance after adding data
4. now execute the quota list command
  
Actual results:
the list command does show the result as it was earlier

Expected results:
the list command is suppose to show same result, if their is no addition of data in between subsequent command executions.


Additional info:

Comment 1 Raghavendra G 2012-04-08 14:10:40 UTC
I am assuming that you are running rebalance after an add brick (otherwise rebalance would move no data)

I did a series of add-brick and rebalance on gluster compiled with commit 1222e2ae2d42cf12569 as HEAD. I was not able to reproduce the issue.

gluster> volume quota dist list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                          10GB                1.6GB
/dir                       10GB              748.2MB

gluster> volume rebalance dist start
Starting rebalance on volume dist has been successful

gluster> volume rebalance dist status
                                    Node Rebalanced-files          size       scanned         status
                               ---------      -----------   -----------   -----------   ------------
                               localhost                0            0         3042     completed

gluster> volume quota dist list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                          10GB                3.2GB
/dir                       10GB                2.3GB
gluster> volume add-brick dist booradley:/home/export/dist-4
Add Brick successful
gluster> volume info dist
 
Volume Name: dist
Type: Distribute
Volume ID: 0a1fe7a8-70d5-4b39-a081-dd732d5487ea
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: booradley:/home/export/dist-1
Brick2: booradley:/export/dist-2
Brick3: booradley:/home/export/dist-3
Brick4: booradley:/home/export/dist-4
Options Reconfigured:
features.limit-usage: /:10GB,/dir:10GB
features.quota: on
gluster> volume quota dist list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                          10GB                3.2GB
/dir                       10GB                2.3GB

gluster> volume quota dist list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                          10GB                3.2GB
/dir                       10GB                2.3GB
gluster> volume rebalance dist start
Starting rebalance on volume dist has been successful

gluster> volume rebalance dist status
                                    Node Rebalanced-files          size       scanned         status
                               ---------      -----------   -----------   -----------   ------------
                               localhost             1791   1368838849         5061     completed

gluster> volume quota dist list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                          10GB                3.2GB
/dir                       10GB                2.3GB

after adding some more data,
gluster> volume quota dist list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                          10GB                5.9GB
/dir                       10GB                2.3GB
gluster> volume info dist
 
Volume Name: dist
Type: Distribute
Volume ID: 0a1fe7a8-70d5-4b39-a081-dd732d5487ea
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: booradley:/home/export/dist-1
Brick2: booradley:/export/dist-2
Brick3: booradley:/home/export/dist-3
Brick4: booradley:/home/export/dist-4
Options Reconfigured:
features.limit-usage: /:10GB,/dir:10GB
features.quota: on
gluster> volume  add-brick dist booradley:/home/export/dist-5
Add Brick successful
gluster> volume quota dist list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                          10GB                5.9GB
/dir                       10GB                2.3GB

gluster> volume rebalance dist start
Starting rebalance on volume dist has been successful

gluster> volume rebalance dist status
                                    Node Rebalanced-files          size       scanned         status
                               ---------      -----------   -----------   -----------   ------------
                               localhost             3314   2256386068        13859     completed

gluster> volume quota dist list
	path		  limit_set	     size
----------------------------------------------------------------------------------
/                          10GB                5.9GB
/dir                       10GB                2.3GB

regards,
Raghavendra.

Comment 2 Saurabh 2012-04-09 09:23:19 UTC
Rebalance status without adding any brick,

steps:-
1. create large of files, inside a directory with quota limit set on that directory.

2. get quota list information.

3. start and finish rebalance

4. get quota list again,

Result, there is a slight change in information of space consumed before and after rebalance, though subsequent quota list commands didn't different results.


[root@RHS2 ~]# gluster volume quota quota_dist_rep list
    path          limit_set         size
----------------------------------------------------------------------------------
/                          10GB                3.1GB
/dir                       10GB                3.1GB
[root@RHS2 ~]#



[root@RHS2 ~]# gluster volume rebalance quota_dist_rep status
                                    Node Rebalanced-files          size       scanned         status
                               ---------      -----------   -----------   -----------   ------------
                               localhost                0            0         3116     completed
                           172.17.251.71                0            0         3153     completed
                           172.17.251.74                0            0         2952     completed
                           172.17.251.73                0            0         3037     completed

[root@RHS2 ~]# gluster volume quota quota_dist_rep list
    path          limit_set         size
----------------------------------------------------------------------------------
/                          10GB                2.9GB
/dir                       10GB                2.9GB






Add-brick and then rebalance,

steps,
1. take the quota list results,
2. add a brick
3. start the rebalance
4. after rebalance finishes, get quota list command results

Result:- This time it has not shown any spurious results,


[root@RHS2 ~]# gluster volume add-brick quota_dist_rep 172.17.251.71:/export/add-quota_dist_rep.1333512079  172.17.251.72:/export/add-quota_dist_rep.1333512079
Add Brick successful


[root@RHS2 ~]# gluster volume rebalance quota_dist_rep status
                                    Node Rebalanced-files          size       scanned         status
                               ---------      -----------   -----------   -----------   ------------
                               localhost                0            0          761     completed
                           172.17.251.71              576    413728768         1694     completed
                           172.17.251.73                0            0          761     completed
                           172.17.251.74                0            0          760     completed

[root@RHS2 ~]#
[root@RHS2 ~]#
[root@RHS2 ~]# gluster volume quota quota_dist_rep list
    path          limit_set         size
----------------------------------------------------------------------------------
/                          10GB                2.9GB
/dir                       10GB                2.9GB
[root@RHS2 ~]# gluster volume quota quota_dist_rep list
    path          limit_set         size
----------------------------------------------------------------------------------
/                          10GB                2.9GB
/dir                       10GB                2.9GB
[root@RHS2 ~]#

Comment 3 Amar Tumballi 2012-05-11 10:59:40 UTC
A small offset in quota sizes will happen after a rebalance operation as internally lot of data migration happens (below the hood of quota), which bypasses the quota layer.

Taking it out of beta blocker list as this is currently known issue, we will RCA it down the line (after 3.3.0 release)

Comment 4 Amar Tumballi 2013-02-04 10:58:53 UTC
I would recommend keeping in mind below numbers in mind while testing quota.
Minimum value set on the directory should be 1GB, and we should have 1-3%
margin for exceeding limit when we do any of the distributed volume operations.