Bug 1360402

Summary:	Clients can starve under heavy load
Product:	[Community] GlusterFS	Reporter:	Jeff Darcy <jdarcy>
Component:	io-threads	Assignee:	Jeff Darcy <jdarcy>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, rgowdapp
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-03-09 20:34:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jeff Darcy 2016-07-26 15:36:23 UTC

A user has reported an instance of $subject.

http://www.gluster.org/pipermail/gluster-users/2016-July/027355.html

We have long known this could be a problem, and the long-term solution is a proper QoS implementation.  In the interim, we should at least implement one of the measures described in a followup:

> We might be able to tweak io-threads (which already runs on the
> bricks and already has a global queue) to schedule requests in a
> fairer way across clients.  Right now it executes them in the
> same order that they were read from the network.  That tends to
> be a bit "unfair" and that should be fixed in the network code,
> but that's a much harder task.

Comment 1 Vijay Bellur 2016-07-26 15:37:57 UTC

REVIEW: http://review.gluster.org/14989 (io-threads: remove least-rate-limit option and code) posted (#4) for review on master by Jeff Darcy (jdarcy)

Comment 2 Vijay Bellur 2016-07-26 15:38:00 UTC

REVIEW: http://review.gluster.org/14904 (io-threads: distribute work fairly among clients) posted (#6) for review on master by Jeff Darcy (jdarcy)

Comment 3 Raghavendra G 2016-07-27 10:55:43 UTC

*** Bug 1360689 has been marked as a duplicate of this bug. ***

Comment 4 Vijay Bellur 2016-07-28 14:50:56 UTC

COMMIT: http://review.gluster.org/14989 committed in master by Jeff Darcy (jdarcy) 
------
commit 25d1bdc094a8408afc074817ec6d63c812c571f9
Author: Jeff Darcy <jdarcy>
Date:   Thu Jul 21 14:49:55 2016 -0400

    io-threads: remove least-rate-limit option and code
    
    This will be unnecessary, and mostly in the way, as real fairness
    guarantees are implemented.
    
    Change-Id: Ic61ec1c9e9add58385f1a4eafcfe2cc554ceefc8
    BUG: 1360402
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: http://review.gluster.org/14989
    Smoke: Gluster Build System <jenkins.org>
    Tested-by: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    NetBSD-regression: Gluster Build System <jenkins.org>

Comment 5 Vijay Bellur 2016-07-28 14:53:12 UTC

REVIEW: http://review.gluster.org/14904 (io-threads: distribute work fairly among clients) posted (#7) for review on master by Jeff Darcy (jdarcy)

Comment 6 Vijay Bellur 2016-07-28 18:52:10 UTC

COMMIT: http://review.gluster.org/14904 committed in master by Jeff Darcy (jdarcy) 
------
commit 1faf452818fe51fae1f07d17afab910a5e0da0cd
Author: Jeff Darcy <jdarcy>
Date:   Fri Jul 22 13:11:15 2016 -0400

    io-threads: distribute work fairly among clients
    
    This is the full "queue of queues" approach where each client gets its
    own queue (per priority) and we round-robin among them.
    
    Change-Id: I73955d1b9bb93f2ff781b48dfe2509009c519ec6
    BUG: 1360402
    Signed-off-by: Jeff Darcy <jdarcy>
    Reviewed-on: http://review.gluster.org/14904
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>

Comment 7 Jeff Darcy 2017-03-09 20:34:43 UTC

This has been in for a while, long enough to be part of 3.10, so it's done.