Bug 1428052

Summary:	performance/io-threads: Eliminate spinlock contention via fops-per-thread-ratio
Product:	[Community] GlusterFS	Reporter:	Vijay Bellur <vbellur>
Component:	io-threads	Assignee:	Vijay Bellur <vbellur>
Status:	CLOSED DEFERRED	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	atumball, bugs
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-05-10 12:55:32 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Vijay Bellur 2017-03-01 19:04:53 UTC

performance/io-threads: Eliminate spinlock contention via fops-per-thread-ratio

Summary:
- Background: Frequently spinlock is observed on busy GFS clusters,
  which wastes CPU and destroys the performance of the cluster.
  Current solutions to this problem involve under-provisioning the thread
  pool, but this is problematic as during busy periods there may not be
  enough threads to service the queue.
- This patch introduces a technique to avoid the stampeding herd problem with
  the io-threads workers.  This is done by dynamically tuning the
  threads by a ratio of threads to queue depth, there-by keeping
  already running threads sufficiently busy by a tunable FOP to thread
  ratio.  Ratio is controllable by the
  performanace.io-threads-fops-per-threads-ratio option.
- More detailed reading on this approach can be found here:
  https://h21007.www2.hp.com/portal/download/files/unprot/hpux/MakingConditionVariablesPerform.pdf
- Cherry-pick of D2530504 for 3.8

Test Plan:
- Stress teston my dev server
- shadow testing

Reviewed By: moox, sshreyas

Signed-off-by: Shreyas Siravara <sshreyas>

Change-Id: I771ae783aa4ca5a6fd0449db64e07d1f4bff0d04
Reviewed-on: http://review.gluster.org/16080
CentOS-regression: Gluster Build System <jenkins.org>
Tested-by: Shreyas Siravara <sshreyas>
NetBSD-regression: NetBSD Build System <jenkins.org>
Smoke: Gluster Build System <jenkins.org>
Reviewed-by: Kevin Vigor <kvigor>

Comment 1 Worker Ant 2017-03-02 20:32:50 UTC

REVIEW: https://review.gluster.org/16832 (performance/io-threads: Eliminate spinlock contention via fops-per-thread-ratio) posted (#1) for review on master by Vijay Bellur (vbellur)

Comment 2 Yaniv Kaul 2019-04-17 12:26:51 UTC

Status?

Comment 3 Vijay Bellur 2019-04-17 17:32:42 UTC

The patch did not progress as we failed to get an update from Facebook about the nature of their performance testing. In our performance tests, we were not able to observe significant gains. I will revisit this  patch to see if we can get any gains in performance now. Thanks!