| Summary: | quota: Soft limit alert not triggered after increasing quota size and even after modifying soft-limit %age | ||
|---|---|---|---|
| Product: | Red Hat Gluster Storage | Reporter: | Ben Turner <bturner> |
| Component: | quota | Assignee: | krishnan parthasarathi <kparthas> |
| Status: | CLOSED WONTFIX | QA Contact: | Ben Turner <bturner> |
| Severity: | high | Docs Contact: | |
| Priority: | low | ||
| Version: | 2.1 | CC: | nsathyan, rhs-bugs, saujain, storage-qa-internal, vbellur, vmallika |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Known Issue | |
| Doc Text: |
Expected behavior of quota:
If the rate of I/O is more than the value of hard-timeout and soft-timeout, there is possibility of quota limit being exceeded
For example:
If the rate of IO is 1GB/sec
If hard-timeout is set to 5sec (default value).
If soft-timeout is set to 60sec (default value).
Then we may exceed quota limit by ~30GB - 60GB
In order to attain a strict checking of quota limit, then you need to lower the value of hard-timeout and soft-timeout
Command to set timeout:
gluster volume quota <volume-name> soft-timeout 0
gluster volume quota <volume-name> hard-timeout 0
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-01-16 08:07:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Bug Depends On: | 1182890, 1182921 | ||
| Bug Blocks: | |||
|
Description
Ben Turner
2013-11-04 14:16:05 UTC
I tried a similar test case and changed the soft-limit percentage for that partiuclar directory, but failed to find any alert-message after changes,
directory in consideration /qa2, limit set is 200GB
[root@quota5 ~]# time gluster volume quota dist-rep1 list
Path Hard-limit Soft-limit Used Available
--------------------------------------------------------------------------------
/ 1.0TB 80% 477.5GB 546.5GB
/qa1 200.0GB 80% 224.1GB 0Bytes
/qa2 200.0GB 80% 200.1GB 0Bytes
/qa3/Downloads 50.0GB 80% 50.0GB 0Bytes
increase the limit to 300GB
[root@quota5 ~]# time gluster volume quota dist-rep1 limit-usage /qa2 300GB 75%
volume quota : success
after data creation,
[root@quota5 ~]# time gluster volume quota dist-rep1 list
Path Hard-limit Soft-limit Used Available
--------------------------------------------------------------------------------
/ 1.0TB 80% 577.4GB 446.6GB
/qa1 200.0GB 80% 224.1GB 0Bytes
/qa2 300.0GB 75% 300.0GB 0Bytes
/qa3/Downloads 50.0GB 80% 50.0GB 0Bytes
brick logs from different nodes,
[root@quota5 ~]# less /var/log/glusterfs/bricks/rhs-brick2-d*r11.log | grep "\sA\s" | grep "qa2"
[2013-11-06 14:13:38.644196] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 85.3GB used by /qa2/
[2013-11-06 14:13:00.383524] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.6GB used by /qa2/
[2013-11-06 14:13:13.882093] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 83.5GB used by /qa2/
[root@quota6 dist-rep1]# less /var/log/glusterfs/bricks/rhs-brick2-d*r21.log | grep "\sA\s" | grep "qa2"
[2013-11-06 14:13:39.903090] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 85.3GB used by /qa2/
[2013-11-06 14:13:01.642696] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.6GB used by /qa2/
[2013-11-06 14:13:15.140320] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 83.5GB used by /qa2/
[root@quota7 ~]# less /var/log/glusterfs/bricks/rhs-brick2-d*r11.log | grep "\sA\s" | grep "qa2"
[2013-11-06 14:12:58.003291] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.4GB used by /qa2/
[2013-11-06 15:06:04.016517] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/
[2013-11-07 11:33:56.264442] A [quota.c:3664:quota_log_usage] 0-dist-rep1-quota: Usage crossed soft limit: 160.0GB used by /qa2
[2013-11-06 14:12:30.662733] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 80.4GB used by /qa2/
[2013-11-06 14:58:31.291197] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2
[2013-11-07 11:34:07.352905] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 163.1GB used by /qa2
[2013-11-06 14:12:45.663201] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 81.5GB used by /qa2/
[2013-11-06 15:06:04.017637] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/
[2013-11-07 11:33:49.596430] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 162.0GB used by /qa2
[root@quota8 ~]# less /var/log/glusterfs/bricks/rhs-brick2-d*r21.log | grep "\sA\s" | grep "qa2"
[2013-11-06 14:12:58.004548] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.4GB used by /qa2/
[2013-11-06 15:06:04.152727] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/
[2013-11-07 11:33:56.386715] A [quota.c:3664:quota_log_usage] 0-dist-rep1-quota: Usage crossed soft limit: 160.0GB used by /qa2
[2013-11-06 14:12:30.691168] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 80.4GB used by /qa2/
[2013-11-06 14:58:31.427640] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2
[2013-11-07 11:34:07.489171] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 163.1GB used by /qa2
[2013-11-06 14:12:45.799641] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 81.5GB used by /qa2/
[2013-11-06 15:06:04.153322] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/
[2013-11-07 11:33:46.008949] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 161.8GB used by /qa2
expected was to get "alert message" after 225GB is crossed, or on the basis of some write
now at this time I miss both string "Usage crossed soft-limit" and "Usage above soft-limit"
This needs to be documented: When the quota hard-timeout is set to default value of 30, the quota limit is checked once in 30 seconds and during that 30 second time window there is possibility of quota hard-limit being exceeded. In order to attain a strict checking of quota limit it is recommended to set the quota soft-timeout and hard-timeout to lower value so that quota limit is checked frequently, and possibility of quota hard-limit being exceeded is reduced. |