Bug 1077159
| Summary: | Quota hard limit not being enforced consistently | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Justin Clift <jclift> | ||||
| Component: | quota | Assignee: | Nagaprasad Sathyanarayana <nsathyan> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | |||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.5.0 | CC: | bugs, gluster-bugs, kwade, ndevos, smohan | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | glusterfs-3.6.0beta1 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-11-11 08:28:47 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Justin Clift
2014-03-17 11:39:09 UTC
Created attachment 875447 [details]
quota.t from master branch, modified to show the output from each iteration of the hard limit testing loop
Confirmed in email from Varun: *************************************************************** Hey Justin, I found the cause that why it happens. The problem lies in DHT/rebalance code. Cause: trusted.glusterfs.quota.limit-set is the quota configuration xattrs which stores this on the directories with quota limits. When there is an add-brick (as same as in my test case quota.t) its DHT's responsibility to copy the xattrs to the directory created in the newly added brick. But DHT/rebalance fails to heal for some reason which I suspect on md-cache or readdirp for lack of lookups on the directory. I need to contract DHT guys for further query/solution on the same. Cause for intermittent Failures and successes: Which ever file hashes to the newly added brick which doesn't have the configuration in the xattrs allows the writes and the old bricks doesn't. - Varun Shastry PS: I tried the same without the add-brick for around 10 times and its working fine. *************************************************************** As a data point, this problem doesn't seem to exist in 3.4. Tested the quota.t checking code in a loop, and no instances of the problem showed up. Comment #2 is based on the assumption that DHT in its fop call path heals the xattrs. After reading the code I came to know that only rebalance code heals the xattrs. So, in the quota.t test script the rebalance succeeds before the further writes only few times. Whenever it fails to heal the xattrs before the next write instructions in the script, we see the quota exceeding the limit. The ideal solution is to wait for the rebalance to complete before start further writing. So, I'm modifying the script to wait for the rebalance process to complete. REVIEW: http://review.gluster.org/7380 (tests/quota: Wait till the rebalance is complete) posted (#1) for review on master by Varun Shastry (vshastry) COMMIT: http://review.gluster.org/7380 committed in master by Vijay Bellur (vbellur) ------ commit 1c1b8269d994c0885d753c8f0da8d5154876c7ae Author: Varun Shastry <vshastry> Date: Tue Mar 25 09:36:45 2014 +0530 tests/quota: Wait till the rebalance is complete Change-Id: Ia6f0c81fb1542ce1de965a69a61535691df056c3 BUG: 1077159 Signed-off-by: Varun Shastry <vshastry> Reviewed-on: http://review.gluster.org/7380 Reviewed-by: Raghavendra G <rgowdapp> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Justin Clift <justin> Tested-by: Justin Clift <justin> Reviewed-by: Vijay Bellur <vbellur> We do not seem to have a backport to the release-3.5 branch yet? Moving to glusterfs-3.5.1 and setting to ASSIGNED. Justin, I do not think this issue applies to 3.5. The quota.t test in 3.5 does not seem have the add-brick scenario. If you really have this problem with quota.t on 3.5, please file a separate bug, this one is now only for the master branch. Thanks! A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html [2] http://supercolony.gluster.org/pipermail/gluster-users/ This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report. glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html [2] http://supercolony.gluster.org/mailman/listinfo/gluster-users |