Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1099890

Summary:	DHT: cluster.min-free-disk not having an effect on new files when the volume has quota-deem-statfs enabled
Product:	[Community] GlusterFS	Reporter:	Krutika Dhananjay <kdhananj>
Component:	distribute	Assignee:	Krutika Dhananjay <kdhananj>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	high
Version:	mainline	CC:	bugs, gluster-bugs
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.6.0beta1	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-11-11 08:32:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	970813, 1110651, 1122920

Description Krutika Dhananjay 2014-05-21 12:14:52 UTC

Description of problem:

cluster.min-free-disk does not have any effect on keeping the volume's disk usage under balance as far as new file creation is concerned, when quota-deem-statfs in enabled on the volume.

What this means is that after a certain point, all new file creation gets routed to the first subvolume of DHT, causing the associated bricks to eventually become 100% full, even when there is ample space on the remaining subvol(s).

Version-Release number of selected component (if applicable):

How reproducible:
ALWAYS

Steps to Reproduce:
1. Create a plain distribute volume. Start it.
2. Enable quota on it.
3. Set quota limit on the root of the volume.
4. Set quota-deem-statfs to "on" on the volume.
5. Set min-free-disk to some x%.
6. Mount the volume. Perform new file creation (say using dd) in a way that file names hash to both sub-volumes until the volume's total disk usage has reached (volume's total size - min-free-disk) bytes, although individual sub-volumes of DHT have not crossed min-free-disk yet.
7. At this point, start creating more files (again using dd) such that their names hash to both sub-volumes.
8. Watch what happens to disk usage on individual bricks.

Actual results:
All files in step 7 end up getting creating on brick 0 (including the ones whose names hash to other bricks) eventually making it 100% full.

Expected results:

At step 7 still, files should get routed to their respective hashed sub-volumes until any one of the bricks is left with less than x% of its total capacity available, in which case DHT should start creating 0-byte link files on this relatively full subvol and cache their data in the subvol with maximum space available.

Additional info:

Will attach a script to create/test the issue, once ready.

Comment 1 Anand Avati 2014-05-22 12:31:33 UTC

REVIEW: http://review.gluster.org/7845 (cluster/dht: Fix min-free-disk calculations when quota-deem-statfs is on) posted (#1) for review on master by Krutika Dhananjay (kdhananj)

Comment 2 Krutika Dhananjay 2014-05-22 12:33:32 UTC

 http://review.gluster.org/7845

Comment 3 Anand Avati 2014-05-22 14:12:26 UTC

REVIEW: http://review.gluster.org/7845 (cluster/dht: Fix min-free-disk calculations when quota-deem-statfs is on) posted (#2) for review on master by Krutika Dhananjay (kdhananj)

Comment 4 Anand Avati 2014-05-23 07:56:42 UTC

REVIEW: http://review.gluster.org/7845 (cluster/dht: Fix min-free-disk calculations when quota-deem-statfs is on) posted (#3) for review on master by Krutika Dhananjay (kdhananj)

Comment 5 Anand Avati 2014-05-27 11:08:00 UTC

REVIEW: http://review.gluster.org/7845 (cluster/dht: Fix min-free-disk calculations when quota-deem-statfs is on) posted (#4) for review on master by Krutika Dhananjay (kdhananj)

Comment 6 Anand Avati 2014-05-27 12:33:49 UTC

REVIEW: http://review.gluster.org/7845 (cluster/dht: Fix min-free-disk calculations when quota-deem-statfs is on) posted (#5) for review on master by Krutika Dhananjay (kdhananj)

Comment 7 Anand Avati 2014-05-28 05:51:50 UTC

REVIEW: http://review.gluster.org/7845 (cluster/dht: Fix min-free-disk calculations when quota-deem-statfs is on) posted (#6) for review on master by Krutika Dhananjay (kdhananj)

Comment 8 Anand Avati 2014-05-28 11:31:25 UTC

REVIEW: http://review.gluster.org/7845 (cluster/dht: Fix min-free-disk calculations when quota-deem-statfs is on) posted (#7) for review on master by Krutika Dhananjay (kdhananj)

Comment 9 Anand Avati 2014-05-29 05:49:56 UTC

REVIEW: http://review.gluster.org/7845 (cluster/dht: Fix min-free-disk calculations when quota-deem-statfs is on) posted (#8) for review on master by Krutika Dhananjay (kdhananj)

Comment 10 Anand Avati 2014-06-02 10:56:53 UTC

COMMIT: http://review.gluster.org/7845 committed in master by Vijay Bellur (vbellur) 
------
commit db022ef7ecca77cbecbcc4c046b6d3aafd2cb86f
Author: Krutika Dhananjay <kdhananj>
Date:   Wed May 21 17:47:03 2014 +0530

    cluster/dht: Fix min-free-disk calculations when quota-deem-statfs is on
    
    PROBLEM:
    
    As part of file creation, DHT sends a statfs call to all of its
    sub-volumes and expects in return the local space consumption and
    availability on each one of them. This information is used by DHT to
    ensure that atleast min-free-disk amount of space is left on each
    sub-volume in the event that there ARE other sub-volumes with more
    space available.
    But when quota-deem-statfs is enabled, quota xlator on every brick
    unwinds the statfs call with volume-wide consumption of disk space.
    This leads to miscalculation in min-free-disk algo, thereby misleading
    DHT at some point, into thinking all sub-volumes have equal available
    space, in which case DHT keeps sending new file creates to subvol-0,
    causing it to become 100% full at some point although there ARE other
    subvols with ample space available.
    
    FIX:
    
    The fix is to make quota_statfs() behave as if quota xlator weren't
    enabled, thereby making every brick return only its local consumption
    and disk space availability.
    
    Change-Id: I211371a1eddb220037bd36a128973938ea8124c2
    BUG: 1099890
    Signed-off-by: Krutika Dhananjay <kdhananj>
    Reviewed-on: http://review.gluster.org/7845
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra G <rgowdapp>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 11 Niels de Vos 2014-09-22 12:40:45 UTC

A beta release for GlusterFS 3.6.0 has been released. Please verify if the release solves this bug report for you. In case the glusterfs-3.6.0beta1 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-September/018836.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 12 Niels de Vos 2014-11-11 08:32:54 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.6.1, please reopen this bug report.

glusterfs-3.6.1 has been announced [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019410.html
[2] http://supercolony.gluster.org/mailman/listinfo/gluster-users