Bug 1026371 - quota: Soft limit alert not triggered after increasing quota size and even after modifying soft-limit %age
Summary: quota: Soft limit alert not triggered after increasing quota size and even af...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: quota
Version: 2.1
Hardware: x86_64
OS: All
low
high
Target Milestone: ---
: ---
Assignee: krishnan parthasarathi
QA Contact: Ben Turner
URL:
Whiteboard:
Depends On: 1182890 1182921
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-11-04 14:16 UTC by Ben Turner
Modified: 2016-09-17 12:43 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
Expected behavior of quota: If the rate of I/O is more than the value of hard-timeout and soft-timeout, there is possibility of quota limit being exceeded For example: If the rate of IO is 1GB/sec If hard-timeout is set to 5sec (default value). If soft-timeout is set to 60sec (default value). Then we may exceed quota limit by ~30GB - 60GB In order to attain a strict checking of quota limit, then you need to lower the value of hard-timeout and soft-timeout Command to set timeout: gluster volume quota <volume-name> soft-timeout 0 gluster volume quota <volume-name> hard-timeout 0
Clone Of:
Environment:
Last Closed: 2015-01-16 08:07:24 UTC
Embargoed:


Attachments (Terms of Use)

Description Ben Turner 2013-11-04 14:16:05 UTC
Description of problem:

When I hit the soft limit, increase the size of my quota, and add more data I don't see a second soft limit message in the logs.

Version-Release number of selected component (if applicable):

glusterfs-server-3.4.0.38

How reproducible:

I have ran this test 3 times.

Steps to Reproduce:
1. create a volume, start it.
2. gluster volume quota <vol-name> enable
3. gluster volume quota <vol-name> limit-usage / 1GB
4. gluster volume quota <vol-name> list
5. mount -t glusterfs <server-ip>:<vol-name> <mount-point>
6. create a single file that is 10% larger than the soft limit
7. gluster volume quota <vol-name> list
8. check log for soft alert message
9. gluster volume quota <vol-name> limit-usage / 5GB
10. rewrite the same file as above to 10% over the new soft limit
11. gluster volume quota <vol-name> list
12. check logs

Actual results:

The soft limit message is not seen after I rewrite the file.

Expected results:

A new soft limit message is logged.

Additional info:

I am not seeing this problem if I create a new file, I am only seeing this problem when DD over an existing file.

Comment 2 Saurabh 2013-11-07 13:42:08 UTC
I tried a similar test case and changed the soft-limit percentage for that partiuclar directory, but failed to find any alert-message after changes,

directory in consideration /qa2, limit set is 200GB 

[root@quota5 ~]# time gluster volume quota dist-rep1 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0TB       80%     477.5GB 546.5GB
/qa1                                     200.0GB       80%     224.1GB  0Bytes
/qa2                                     200.0GB       80%     200.1GB  0Bytes
/qa3/Downloads                            50.0GB       80%      50.0GB  0Bytes


increase the limit to 300GB
[root@quota5 ~]# time gluster volume quota dist-rep1 limit-usage /qa2 300GB 75%
volume quota : success


after data creation,

[root@quota5 ~]# time gluster volume quota dist-rep1 list
                  Path                   Hard-limit Soft-limit   Used  Available
--------------------------------------------------------------------------------
/                                          1.0TB       80%     577.4GB 446.6GB
/qa1                                     200.0GB       80%     224.1GB  0Bytes
/qa2                                     300.0GB       75%     300.0GB  0Bytes
/qa3/Downloads                            50.0GB       80%      50.0GB  0Bytes


brick logs from different nodes,

[root@quota5 ~]# less /var/log/glusterfs/bricks/rhs-brick2-d*r11.log | grep "\sA\s" | grep "qa2"
[2013-11-06 14:13:38.644196] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 85.3GB used by /qa2/
[2013-11-06 14:13:00.383524] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.6GB used by /qa2/
[2013-11-06 14:13:13.882093] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 83.5GB used by /qa2/


[root@quota6 dist-rep1]# less /var/log/glusterfs/bricks/rhs-brick2-d*r21.log | grep "\sA\s" | grep "qa2"
[2013-11-06 14:13:39.903090] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 85.3GB used by /qa2/
[2013-11-06 14:13:01.642696] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.6GB used by /qa2/
[2013-11-06 14:13:15.140320] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 83.5GB used by /qa2/


[root@quota7 ~]# less /var/log/glusterfs/bricks/rhs-brick2-d*r11.log | grep "\sA\s" | grep "qa2"
[2013-11-06 14:12:58.003291] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.4GB used by /qa2/
[2013-11-06 15:06:04.016517] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/
[2013-11-07 11:33:56.264442] A [quota.c:3664:quota_log_usage] 0-dist-rep1-quota: Usage crossed soft limit: 160.0GB used by /qa2
[2013-11-06 14:12:30.662733] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 80.4GB used by /qa2/
[2013-11-06 14:58:31.291197] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2
[2013-11-07 11:34:07.352905] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 163.1GB used by /qa2
[2013-11-06 14:12:45.663201] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 81.5GB used by /qa2/
[2013-11-06 15:06:04.017637] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/
[2013-11-07 11:33:49.596430] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 162.0GB used by /qa2



[root@quota8 ~]# less /var/log/glusterfs/bricks/rhs-brick2-d*r21.log | grep "\sA\s" | grep "qa2"
[2013-11-06 14:12:58.004548] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.4GB used by /qa2/
[2013-11-06 15:06:04.152727] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/
[2013-11-07 11:33:56.386715] A [quota.c:3664:quota_log_usage] 0-dist-rep1-quota: Usage crossed soft limit: 160.0GB used by /qa2
[2013-11-06 14:12:30.691168] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 80.4GB used by /qa2/
[2013-11-06 14:58:31.427640] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2
[2013-11-07 11:34:07.489171] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 163.1GB used by /qa2
[2013-11-06 14:12:45.799641] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 81.5GB used by /qa2/
[2013-11-06 15:06:04.153322] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/
[2013-11-07 11:33:46.008949] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 161.8GB used by /qa2


expected was to get "alert message" after 225GB is crossed, or on the basis of some write

now at this time I miss both string "Usage crossed soft-limit" and "Usage above soft-limit"

Comment 3 Vijaikumar Mallikarjuna 2015-01-12 13:35:17 UTC
This needs to be documented:

When the quota hard-timeout is set to default value of 30,
the quota limit is checked once in 30 seconds and during
that 30 second time window there is possibility of quota
hard-limit being exceeded. In order to attain a strict
checking of quota limit it is recommended to set the
quota soft-timeout and hard-timeout to lower value so
that quota limit is checked frequently, and possibility of
quota hard-limit being exceeded is reduced.


Note You need to log in before you can comment on or make changes to this bug.