Red Hat Bugzilla – Bug 656939
GFS2: [RFE] glock scalability patches
Last modified: 2011-05-19 08:42:18 EDT
This follows on from the upstream inode scalability work, although it doesn't need all of that to go upstream before we do this in GFS2. We do want to see the hlist_bl_rcu work upstream though. As a result we are quite likely, but not 100% certain to be able to do this before the 6.1 deadline. This is basically a performance change. The idea is to use rcu for lookups of glocks in the hash table (hence the need for hlist_bl_rcu) but at the same time it also reduces the code size (ignoring the new hlist_bl_rcu header) since the locking is also simplified as well. The first version of the upstream patch (using normal hlist_rcu) has already been posted upstream, so the changes required are now fairly small.
Upstream post of current patch: https://www.redhat.com/archives/cluster-devel/2010-November/msg00051.html
This request was evaluated by Red Hat Product Management for inclusion in the current release of Red Hat Enterprise Linux. Because the affected component is not scheduled to be updated in the current release, Red Hat is unfortunately unable to address this request at this time. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux. If you would like it considered as an exception in the current release, please ask your support representative.
It looks like the upstream component of this will be merged in the currently open merge window.
The upstream pre-requisites for this patch have been merged in the current merge window. That opens the way to push this patch into the -nmw tree very shortly, so there should be no problems with having this ready for rhel 6.1.
Created attachment 473669 [details] RHEL6 port of the patch
Created attachment 474310 [details] RHEL6 port of the patch - Fixed one-line error (glock_free -> gfs2_glock_free)
A RHEL6 build with the above patch in comment #7 and another one-liner patch from bz 669877 is here: http://porkchop.devel.redhat.com/brewroot/packages/kernel/2.6.32/99.el6.gfs2abhi.001/
Posted patch in comment #7 to rhkernel-list
Is there a benchmark we can run that specifically tests glock scalability?
Anything which has lots of threads doing things simultaneously. Barry is running such a test at the moment, results expected on Monday.
Patch(es) available on kernel-2.6.32-112.el6
SPECsfs testing: Ive finally started testing this feature based off the -106 kernel which Steve Whitehouse stated had the bits. Due to another BZ, I had to build a custom -106 kernel, and also created a -105 for reference. Preliminary results looked good, until about 15% workload throughput. Up to this point, operations such as getattr, lookup, read performance was improved, in some cases more than 15%. After that things started to go south. Removes were the first to start regressing badly, then writes and reads ... eventually everything (except writes) started falling over. I will post the results when they are done. Question ... Is it worth doing some profiling on these two runs ? barry
What do you mean by "falling over"? Just that things were slow? It would be interesting to have a comparison with one other filesystem, say ext3, just in case there is a generic factor at work. Also it would help to have some handle on what "regressing badly" means. Can you give a rough idea in terms of the % regression in those cases? It might be worth doing some profiling if the above doesn't reveal the problem.
Created attachment 477463 [details] profiling on BIGI with gfs2 using the -106 kernel. There is also some SPECsfs reporting as well. Interesting places include 4500 Ops/sec where write response surges and then at 5000 Ops/sec removes too. barry
Created attachment 477645 [details] profiling on BIGI with gfs2 using the -105 kernel Note that these are snapshots of 5000 ops and 10000 ops. Barry
Created attachment 478051 [details] capturing gfs2 log flush counts using tracepoints Note what happens at 5000 Ops/sec here barry
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0542.html