Bug 656939 - GFS2: [RFE] glock scalability patches
Summary: GFS2: [RFE] glock scalability patches
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.1
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: rc
: ---
Assignee: Steve Whitehouse
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-11-24 14:59 UTC by Steve Whitehouse
Modified: 2011-05-19 12:42 UTC (History)
6 users (show)

Fixed In Version: kernel-2.6.32-112.el6
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-05-19 12:42:18 UTC
Target Upstream Version:


Attachments (Terms of Use)
RHEL6 port of the patch (49.61 KB, patch)
2011-01-15 20:22 UTC, Steve Whitehouse
no flags Details | Diff
RHEL6 port of the patch - Fixed one-line error (glock_free -> gfs2_glock_free) (48.05 KB, patch)
2011-01-19 16:30 UTC, Abhijith Das
no flags Details | Diff
profiling on BIGI with gfs2 using the -106 kernel. (20.17 KB, text/x-log)
2011-02-07 17:27 UTC, Barry Marson
no flags Details
profiling on BIGI with gfs2 using the -105 kernel (8.20 KB, text/x-log)
2011-02-08 16:32 UTC, Barry Marson
no flags Details
capturing gfs2 log flush counts using tracepoints (2.37 KB, text/x-log)
2011-02-10 13:28 UTC, Barry Marson
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2011:0542 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 6.1 kernel security, bug fix and enhancement update 2011-05-19 11:58:07 UTC

Description Steve Whitehouse 2010-11-24 14:59:45 UTC
This follows on from the upstream inode scalability work, although it doesn't need all of that to go upstream before we do this in GFS2. We do want to see the hlist_bl_rcu work upstream though. As a result we are quite likely, but not 100% certain to be able to do this before the 6.1 deadline.

This is basically a performance change. The idea is to use rcu for lookups of glocks in the hash table (hence the need for hlist_bl_rcu) but at the same time it also reduces the code size (ignoring the new hlist_bl_rcu header) since the locking is also simplified as well.

The first version of the upstream patch (using normal hlist_rcu) has already been posted upstream, so the changes required are now fairly small.

Comment 1 Steve Whitehouse 2010-12-01 11:24:29 UTC
Upstream post of current patch:

https://www.redhat.com/archives/cluster-devel/2010-November/msg00051.html

Comment 3 RHEL Program Management 2011-01-07 03:51:13 UTC
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.

Comment 4 Steve Whitehouse 2011-01-07 10:20:46 UTC
It looks like the upstream component of this will be merged in the currently open merge window.

Comment 5 Steve Whitehouse 2011-01-10 09:44:39 UTC
The upstream pre-requisites for this patch have been merged in the current merge window. That opens the way to push this patch into the -nmw tree very shortly, so there should be no problems with having this ready for rhel 6.1.

Comment 6 Steve Whitehouse 2011-01-15 20:22:07 UTC
Created attachment 473669 [details]
RHEL6 port of the patch

Comment 7 Abhijith Das 2011-01-19 16:30:02 UTC
Created attachment 474310 [details]
RHEL6 port of the patch - Fixed one-line error (glock_free -> gfs2_glock_free)

Comment 8 Abhijith Das 2011-01-19 16:34:08 UTC
A RHEL6 build with the above patch in comment #7 and another one-liner patch from bz 669877 is here:
http://porkchop.devel.redhat.com/brewroot/packages/kernel/2.6.32/99.el6.gfs2abhi.001/

Comment 9 Abhijith Das 2011-01-21 20:45:34 UTC
Posted patch in comment #7 to rhkernel-list

Comment 10 Nate Straz 2011-01-21 21:24:45 UTC
Is there a benchmark we can run that specifically tests glock scalability?

Comment 11 Steve Whitehouse 2011-01-21 21:58:36 UTC
Anything which has lots of threads doing things simultaneously. Barry is running such a test at the moment, results expected on Monday.

Comment 12 Aristeu Rozanski 2011-02-03 16:38:29 UTC
Patch(es) available on kernel-2.6.32-112.el6

Comment 14 Barry Marson 2011-02-03 21:33:40 UTC
SPECsfs testing:

Ive finally started testing this feature based off the -106 kernel which Steve Whitehouse stated had the bits.  Due to another BZ, I had to build a custom -106 kernel, and also created a -105 for reference.

Preliminary results looked good, until about 15% workload throughput.  Up to this point, operations such as getattr, lookup, read performance was improved, in some cases more than 15%.  After that things started to go south.  Removes were the first to start regressing badly, then writes and reads ... eventually everything (except writes) started falling over.  I will post the results when they are done.

Question ... Is it worth doing some profiling on these two runs ?

barry

Comment 15 Steve Whitehouse 2011-02-04 10:04:54 UTC
What do you mean by "falling over"? Just that things were slow?

It would be interesting to have a comparison with one other filesystem, say ext3, just in case there is a generic factor at work.

Also it would help to have some handle on what "regressing badly" means. Can you give a rough idea in terms of the % regression in those cases?

It might be worth doing some profiling if the above doesn't reveal the problem.

Comment 16 Barry Marson 2011-02-07 17:27:03 UTC
Created attachment 477463 [details]
profiling on BIGI with gfs2 using the -106 kernel.

There is also some SPECsfs reporting as well.  Interesting places include 4500 Ops/sec where write response surges and then at 5000 Ops/sec removes too.

barry

Comment 17 Barry Marson 2011-02-08 16:32:12 UTC
Created attachment 477645 [details]
profiling on BIGI with gfs2 using the -105 kernel

Note that these are snapshots of 5000 ops and 10000 ops.  

Barry

Comment 19 Barry Marson 2011-02-10 13:28:12 UTC
Created attachment 478051 [details]
capturing gfs2 log flush counts using tracepoints

Note what happens at 5000 Ops/sec here

barry

Comment 22 errata-xmlrpc 2011-05-19 12:42:18 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0542.html


Note You need to log in before you can comment on or make changes to this bug.