performance/io-threads: Eliminate spinlock contention via fops-per-thread-ratio Summary: - Background: Frequently spinlock is observed on busy GFS clusters, which wastes CPU and destroys the performance of the cluster. Current solutions to this problem involve under-provisioning the thread pool, but this is problematic as during busy periods there may not be enough threads to service the queue. - This patch introduces a technique to avoid the stampeding herd problem with the io-threads workers. This is done by dynamically tuning the threads by a ratio of threads to queue depth, there-by keeping already running threads sufficiently busy by a tunable FOP to thread ratio. Ratio is controllable by the performanace.io-threads-fops-per-threads-ratio option. - More detailed reading on this approach can be found here: https://h21007.www2.hp.com/portal/download/files/unprot/hpux/MakingConditionVariablesPerform.pdf - Cherry-pick of D2530504 for 3.8 Test Plan: - Stress teston my dev server - shadow testing Reviewed By: moox, sshreyas Signed-off-by: Shreyas Siravara <sshreyas> Change-Id: I771ae783aa4ca5a6fd0449db64e07d1f4bff0d04 Reviewed-on: http://review.gluster.org/16080 CentOS-regression: Gluster Build System <jenkins.org> Tested-by: Shreyas Siravara <sshreyas> NetBSD-regression: NetBSD Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Kevin Vigor <kvigor>
REVIEW: https://review.gluster.org/16832 (performance/io-threads: Eliminate spinlock contention via fops-per-thread-ratio) posted (#1) for review on master by Vijay Bellur (vbellur)
Status?
The patch did not progress as we failed to get an update from Facebook about the nature of their performance testing. In our performance tests, we were not able to observe significant gains. I will revisit this patch to see if we can get any gains in performance now. Thanks!