Description of problem: When I hit the soft limit, increase the size of my quota, and add more data I don't see a second soft limit message in the logs. Version-Release number of selected component (if applicable): glusterfs-server-3.4.0.38 How reproducible: I have ran this test 3 times. Steps to Reproduce: 1. create a volume, start it. 2. gluster volume quota <vol-name> enable 3. gluster volume quota <vol-name> limit-usage / 1GB 4. gluster volume quota <vol-name> list 5. mount -t glusterfs <server-ip>:<vol-name> <mount-point> 6. create a single file that is 10% larger than the soft limit 7. gluster volume quota <vol-name> list 8. check log for soft alert message 9. gluster volume quota <vol-name> limit-usage / 5GB 10. rewrite the same file as above to 10% over the new soft limit 11. gluster volume quota <vol-name> list 12. check logs Actual results: The soft limit message is not seen after I rewrite the file. Expected results: A new soft limit message is logged. Additional info: I am not seeing this problem if I create a new file, I am only seeing this problem when DD over an existing file.
I tried a similar test case and changed the soft-limit percentage for that partiuclar directory, but failed to find any alert-message after changes, directory in consideration /qa2, limit set is 200GB [root@quota5 ~]# time gluster volume quota dist-rep1 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0TB 80% 477.5GB 546.5GB /qa1 200.0GB 80% 224.1GB 0Bytes /qa2 200.0GB 80% 200.1GB 0Bytes /qa3/Downloads 50.0GB 80% 50.0GB 0Bytes increase the limit to 300GB [root@quota5 ~]# time gluster volume quota dist-rep1 limit-usage /qa2 300GB 75% volume quota : success after data creation, [root@quota5 ~]# time gluster volume quota dist-rep1 list Path Hard-limit Soft-limit Used Available -------------------------------------------------------------------------------- / 1.0TB 80% 577.4GB 446.6GB /qa1 200.0GB 80% 224.1GB 0Bytes /qa2 300.0GB 75% 300.0GB 0Bytes /qa3/Downloads 50.0GB 80% 50.0GB 0Bytes brick logs from different nodes, [root@quota5 ~]# less /var/log/glusterfs/bricks/rhs-brick2-d*r11.log | grep "\sA\s" | grep "qa2" [2013-11-06 14:13:38.644196] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 85.3GB used by /qa2/ [2013-11-06 14:13:00.383524] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.6GB used by /qa2/ [2013-11-06 14:13:13.882093] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 83.5GB used by /qa2/ [root@quota6 dist-rep1]# less /var/log/glusterfs/bricks/rhs-brick2-d*r21.log | grep "\sA\s" | grep "qa2" [2013-11-06 14:13:39.903090] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 85.3GB used by /qa2/ [2013-11-06 14:13:01.642696] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.6GB used by /qa2/ [2013-11-06 14:13:15.140320] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 83.5GB used by /qa2/ [root@quota7 ~]# less /var/log/glusterfs/bricks/rhs-brick2-d*r11.log | grep "\sA\s" | grep "qa2" [2013-11-06 14:12:58.003291] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.4GB used by /qa2/ [2013-11-06 15:06:04.016517] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/ [2013-11-07 11:33:56.264442] A [quota.c:3664:quota_log_usage] 0-dist-rep1-quota: Usage crossed soft limit: 160.0GB used by /qa2 [2013-11-06 14:12:30.662733] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 80.4GB used by /qa2/ [2013-11-06 14:58:31.291197] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2 [2013-11-07 11:34:07.352905] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 163.1GB used by /qa2 [2013-11-06 14:12:45.663201] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 81.5GB used by /qa2/ [2013-11-06 15:06:04.017637] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/ [2013-11-07 11:33:49.596430] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 162.0GB used by /qa2 [root@quota8 ~]# less /var/log/glusterfs/bricks/rhs-brick2-d*r21.log | grep "\sA\s" | grep "qa2" [2013-11-06 14:12:58.004548] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 82.4GB used by /qa2/ [2013-11-06 15:06:04.152727] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/ [2013-11-07 11:33:56.386715] A [quota.c:3664:quota_log_usage] 0-dist-rep1-quota: Usage crossed soft limit: 160.0GB used by /qa2 [2013-11-06 14:12:30.691168] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 80.4GB used by /qa2/ [2013-11-06 14:58:31.427640] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2 [2013-11-07 11:34:07.489171] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 163.1GB used by /qa2 [2013-11-06 14:12:45.799641] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 81.5GB used by /qa2/ [2013-11-06 15:06:04.153322] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 100.1GB used by /qa2/ [2013-11-07 11:33:46.008949] A [quota.c:3670:quota_log_usage] 0-dist-rep1-quota: Usage is above soft limit: 161.8GB used by /qa2 expected was to get "alert message" after 225GB is crossed, or on the basis of some write now at this time I miss both string "Usage crossed soft-limit" and "Usage above soft-limit"
This needs to be documented: When the quota hard-timeout is set to default value of 30, the quota limit is checked once in 30 seconds and during that 30 second time window there is possibility of quota hard-limit being exceeded. In order to attain a strict checking of quota limit it is recommended to set the quota soft-timeout and hard-timeout to lower value so that quota limit is checked frequently, and possibility of quota hard-limit being exceeded is reduced.