Hide Forgot
Description of problem: simple rebalance then quota list several time and it keeps displaying different values, meanwhile on the point the find . | xargs stat was also executed to collect stat info. Version-Release number of selected component (if applicable): How reproducible: happened several timest 1. set quota limits on volume and a dir 2. put some data in them 3. start rebalance after adding data 4. now execute the quota list command Actual results: the list command does show the result as it was earlier Expected results: the list command is suppose to show same result, if their is no addition of data in between subsequent command executions. Additional info:
I am assuming that you are running rebalance after an add brick (otherwise rebalance would move no data) I did a series of add-brick and rebalance on gluster compiled with commit 1222e2ae2d42cf12569 as HEAD. I was not able to reproduce the issue. gluster> volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 10GB 1.6GB /dir 10GB 748.2MB gluster> volume rebalance dist start Starting rebalance on volume dist has been successful gluster> volume rebalance dist status Node Rebalanced-files size scanned status --------- ----------- ----------- ----------- ------------ localhost 0 0 3042 completed gluster> volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 10GB 3.2GB /dir 10GB 2.3GB gluster> volume add-brick dist booradley:/home/export/dist-4 Add Brick successful gluster> volume info dist Volume Name: dist Type: Distribute Volume ID: 0a1fe7a8-70d5-4b39-a081-dd732d5487ea Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: booradley:/home/export/dist-1 Brick2: booradley:/export/dist-2 Brick3: booradley:/home/export/dist-3 Brick4: booradley:/home/export/dist-4 Options Reconfigured: features.limit-usage: /:10GB,/dir:10GB features.quota: on gluster> volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 10GB 3.2GB /dir 10GB 2.3GB gluster> volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 10GB 3.2GB /dir 10GB 2.3GB gluster> volume rebalance dist start Starting rebalance on volume dist has been successful gluster> volume rebalance dist status Node Rebalanced-files size scanned status --------- ----------- ----------- ----------- ------------ localhost 1791 1368838849 5061 completed gluster> volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 10GB 3.2GB /dir 10GB 2.3GB after adding some more data, gluster> volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 10GB 5.9GB /dir 10GB 2.3GB gluster> volume info dist Volume Name: dist Type: Distribute Volume ID: 0a1fe7a8-70d5-4b39-a081-dd732d5487ea Status: Started Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: booradley:/home/export/dist-1 Brick2: booradley:/export/dist-2 Brick3: booradley:/home/export/dist-3 Brick4: booradley:/home/export/dist-4 Options Reconfigured: features.limit-usage: /:10GB,/dir:10GB features.quota: on gluster> volume add-brick dist booradley:/home/export/dist-5 Add Brick successful gluster> volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 10GB 5.9GB /dir 10GB 2.3GB gluster> volume rebalance dist start Starting rebalance on volume dist has been successful gluster> volume rebalance dist status Node Rebalanced-files size scanned status --------- ----------- ----------- ----------- ------------ localhost 3314 2256386068 13859 completed gluster> volume quota dist list path limit_set size ---------------------------------------------------------------------------------- / 10GB 5.9GB /dir 10GB 2.3GB regards, Raghavendra.
Rebalance status without adding any brick, steps:- 1. create large of files, inside a directory with quota limit set on that directory. 2. get quota list information. 3. start and finish rebalance 4. get quota list again, Result, there is a slight change in information of space consumed before and after rebalance, though subsequent quota list commands didn't different results. [root@RHS2 ~]# gluster volume quota quota_dist_rep list path limit_set size ---------------------------------------------------------------------------------- / 10GB 3.1GB /dir 10GB 3.1GB [root@RHS2 ~]# [root@RHS2 ~]# gluster volume rebalance quota_dist_rep status Node Rebalanced-files size scanned status --------- ----------- ----------- ----------- ------------ localhost 0 0 3116 completed 172.17.251.71 0 0 3153 completed 172.17.251.74 0 0 2952 completed 172.17.251.73 0 0 3037 completed [root@RHS2 ~]# gluster volume quota quota_dist_rep list path limit_set size ---------------------------------------------------------------------------------- / 10GB 2.9GB /dir 10GB 2.9GB Add-brick and then rebalance, steps, 1. take the quota list results, 2. add a brick 3. start the rebalance 4. after rebalance finishes, get quota list command results Result:- This time it has not shown any spurious results, [root@RHS2 ~]# gluster volume add-brick quota_dist_rep 172.17.251.71:/export/add-quota_dist_rep.1333512079 172.17.251.72:/export/add-quota_dist_rep.1333512079 Add Brick successful [root@RHS2 ~]# gluster volume rebalance quota_dist_rep status Node Rebalanced-files size scanned status --------- ----------- ----------- ----------- ------------ localhost 0 0 761 completed 172.17.251.71 576 413728768 1694 completed 172.17.251.73 0 0 761 completed 172.17.251.74 0 0 760 completed [root@RHS2 ~]# [root@RHS2 ~]# [root@RHS2 ~]# gluster volume quota quota_dist_rep list path limit_set size ---------------------------------------------------------------------------------- / 10GB 2.9GB /dir 10GB 2.9GB [root@RHS2 ~]# gluster volume quota quota_dist_rep list path limit_set size ---------------------------------------------------------------------------------- / 10GB 2.9GB /dir 10GB 2.9GB [root@RHS2 ~]#
A small offset in quota sizes will happen after a rebalance operation as internally lot of data migration happens (below the hood of quota), which bypasses the quota layer. Taking it out of beta blocker list as this is currently known issue, we will RCA it down the line (after 3.3.0 release)
I would recommend keeping in mind below numbers in mind while testing quota. Minimum value set on the directory should be 1GB, and we should have 1-3% margin for exceeding limit when we do any of the distributed volume operations.