RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 656939 - GFS2: [RFE] glock scalability patches
Summary: GFS2: [RFE] glock scalability patches
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.1
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: rc
: ---
Assignee: Steve Whitehouse
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-11-24 14:59 UTC by Steve Whitehouse
Modified: 2011-05-19 12:42 UTC (History)
6 users (show)

Fixed In Version: kernel-2.6.32-112.el6
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-05-19 12:42:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
RHEL6 port of the patch (49.61 KB, patch)
2011-01-15 20:22 UTC, Steve Whitehouse 		no flags Details | Diff
RHEL6 port of the patch - Fixed one-line error (glock_free -> gfs2_glock_free) (48.05 KB, patch)
2011-01-19 16:30 UTC, Abhijith Das 		no flags Details | Diff
profiling on BIGI with gfs2 using the -106 kernel. (20.17 KB, text/x-log)
2011-02-07 17:27 UTC, Barry Marson 		no flags Details
profiling on BIGI with gfs2 using the -105 kernel (8.20 KB, text/x-log)
2011-02-08 16:32 UTC, Barry Marson 		no flags Details
capturing gfs2 log flush counts using tracepoints (2.37 KB, text/x-log)
2011-02-10 13:28 UTC, Barry Marson 		no flags Details
Show Obsolete (1) View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2011:0542 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 6.1 kernel security, bug fix and enhancement update 2011-05-19 11:58:07 UTC

Description Steve Whitehouse 2010-11-24 14:59:45 UTC 
This follows on from the upstream inode scalability work, although it doesn't need all of that to go upstream before we do this in GFS2. We do want to see the hlist_bl_rcu work upstream though. As a result we are quite likely, but not 100% certain to be able to do this before the 6.1 deadline.

This is basically a performance change. The idea is to use rcu for lookups of glocks in the hash table (hence the need for hlist_bl_rcu) but at the same time it also reduces the code size (ignoring the new hlist_bl_rcu header) since the locking is also simplified as well.

The first version of the upstream patch (using normal hlist_rcu) has already been posted upstream, so the changes required are now fairly small.
Comment 1 Steve Whitehouse 2010-12-01 11:24:29 UTC 
Upstream post of current patch:

https://www.redhat.com/archives/cluster-devel/2010-November/msg00051.html
Comment 3 RHEL Program Management 2011-01-07 03:51:13 UTC 
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.
Comment 4 Steve Whitehouse 2011-01-07 10:20:46 UTC 
It looks like the upstream component of this will be merged in the currently open merge window.
Comment 5 Steve Whitehouse 2011-01-10 09:44:39 UTC 
The upstream pre-requisites for this patch have been merged in the current merge window. That opens the way to push this patch into the -nmw tree very shortly, so there should be no problems with having this ready for rhel 6.1.
Comment 6 Steve Whitehouse 2011-01-15 20:22:07 UTC 
Created attachment 473669 [details]
RHEL6 port of the patch
Comment 7 Abhijith Das 2011-01-19 16:30:02 UTC 
Created attachment 474310 [details]
RHEL6 port of the patch - Fixed one-line error (glock_free -> gfs2_glock_free)
Comment 8 Abhijith Das 2011-01-19 16:34:08 UTC 
A RHEL6 build with the above patch in comment #7 and another one-liner patch from bz 669877 is here:
http://porkchop.devel.redhat.com/brewroot/packages/kernel/2.6.32/99.el6.gfs2abhi.001/
Comment 9 Abhijith Das 2011-01-21 20:45:34 UTC 
Posted patch in comment #7 to rhkernel-list
Comment 10 Nate Straz 2011-01-21 21:24:45 UTC 
Is there a benchmark we can run that specifically tests glock scalability?
Comment 11 Steve Whitehouse 2011-01-21 21:58:36 UTC 
Anything which has lots of threads doing things simultaneously. Barry is running such a test at the moment, results expected on Monday.
Comment 12 Aristeu Rozanski 2011-02-03 16:38:29 UTC 
Patch(es) available on kernel-2.6.32-112.el6
Comment 14 Barry Marson 2011-02-03 21:33:40 UTC 
SPECsfs testing:

Ive finally started testing this feature based off the -106 kernel which Steve Whitehouse stated had the bits.  Due to another BZ, I had to build a custom -106 kernel, and also created a -105 for reference.

Preliminary results looked good, until about 15% workload throughput.  Up to this point, operations such as getattr, lookup, read performance was improved, in some cases more than 15%.  After that things started to go south.  Removes were the first to start regressing badly, then writes and reads ... eventually everything (except writes) started falling over.  I will post the results when they are done.

Question ... Is it worth doing some profiling on these two runs ?

barry
Comment 15 Steve Whitehouse 2011-02-04 10:04:54 UTC 
What do you mean by "falling over"? Just that things were slow?

It would be interesting to have a comparison with one other filesystem, say ext3, just in case there is a generic factor at work.

Also it would help to have some handle on what "regressing badly" means. Can you give a rough idea in terms of the % regression in those cases?

It might be worth doing some profiling if the above doesn't reveal the problem.
Comment 16 Barry Marson 2011-02-07 17:27:03 UTC 
Created attachment 477463 [details]
profiling on BIGI with gfs2 using the -106 kernel.

There is also some SPECsfs reporting as well.  Interesting places include 4500 Ops/sec where write response surges and then at 5000 Ops/sec removes too.

barry
Comment 17 Barry Marson 2011-02-08 16:32:12 UTC 
Created attachment 477645 [details]
profiling on BIGI with gfs2 using the -105 kernel

Note that these are snapshots of 5000 ops and 10000 ops.  

Barry
Comment 19 Barry Marson 2011-02-10 13:28:12 UTC 
Created attachment 478051 [details]
capturing gfs2 log flush counts using tracepoints

Note what happens at 5000 Ops/sec here

barry
Comment 22 errata-xmlrpc 2011-05-19 12:42:18 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0542.html
Note You need to log in before you can comment on or make changes to this bug.