Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1695480

Summary:	Global Thread Pool
Product:	[Community] GlusterFS	Reporter:	Xavi Hernandez <jahernan>
Component:	core	Assignee:	Xavi Hernandez <jahernan>
Status:	CLOSED UPSTREAM	QA Contact:
Severity:	high	Docs Contact:
Priority:	high
Version:	mainline	CC:	bugs, pasik
Target Milestone:	---	Keywords:	Improvement, Performance
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-12 13:00:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Xavi Hernandez 2019-04-03 08:09:22 UTC

Description of problem:

The Global Thread Pool provides lower contention and increased performance in some cases, but it has been observed that sometimes there's a huge increment in the number of requests going to the disks in parallel which seems to be causing a performance degradation.

Actually, it seems that sending the same amount of requests but from fewer threads is giving higher performance.

The current implementation already does some dynamic adjustment of the number of active threads based on the current number of requests, but it doesn't consider the load on the back-end file systems. This means that as long as more requests come, the number of threads is scaled accordingly, which could have a negative impact if the back-end is already saturated.

The way to control that in current version is to manually adjust the maximum number of threads that can be used, which effectively limits the load on back-end file systems even if more requests are coming, but this is only useful for volumes whose workload is homogeneous and constant.

To make it more versatile, the maximum number of threads need to be automatically self-adjusted to adapt dynamically to the current load so that it can be useful in a general case.

Version-Release number of selected component (if applicable): mainline


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Xavi Hernandez 2019-06-25 18:16:55 UTC

The autoscaling implementation was not giving the expected results. I implemented a new approach that works much better automatically. However it doesn't beats the current implementation in all cases.

Results can be found here: https://docs.google.com/spreadsheets/d/19JqvuFKZxKifgrhLF-5-bgemYj8XKldUox1QwsmGj2k/edit?usp=sharing

One thing that I've observed is that both the old and new implementations have a huge variation for some tests. I cannot explain why right now. My plan is to add instrumentation to gluster code to be able to analyze these cases in more detail and see where the delays come from.

Comment 2 Worker Ant 2020-03-12 13:00:31 UTC

This bug is moved to https://github.com/gluster/glusterfs/issues/983, and will be tracked there from now on. Visit GitHub issues URL for further details