RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 656042 - [RFE] Include autogroup patch to aid in automatic creation of cgroups
Summary: [RFE] Include autogroup patch to aid in automatic creation of cgroups
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Vivek Goyal
QA Contact: Jan Tluka
URL:
Whiteboard:
Depends On:
Blocks: 662543
TreeView+ depends on / blocked
 
Reported: 2010-11-22 22:17 UTC by Jeremy West
Modified: 2018-11-14 16:23 UTC (History)
4 users (show)

Fixed In Version: kernel-2.6.32-117.el6
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-05-23 20:29:33 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2011:0542 0 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 6.1 kernel security, bug fix and enhancement update 2011-05-19 11:58:07 UTC

Description Jeremy West 2010-11-22 22:17:44 UTC 
1. Customer Name
DreamWorks Animation

2. What is the nature and description of the request?
Customer would like the following recent upstream patch regarding the automatic creation of task groups for improved scheduling

Original kernel thread:

http://marc.info/?l=linux-kernel&m=128978361700898&w=2

A recurring complaint from CFS users is that parallel kbuild has a negative
impact on desktop interactivity. This patch implements an idea from Linus,
to automatically create task groups. This patch only implements Linus' per
tty task group suggestion, and only for fair class tasks, but leaves the way
open for enhancement.

Implementation: each task's signal struct contains an inherited pointer to a
refcounted autogroup struct containing a task group pointer, the default for
all tasks pointing to the init_task_group. When a task calls __proc_set_tty(),
the process wide reference to the default group is dropped, a new task group is
created, and the process is moved into the new task group. Children thereafter
inherit this task group, and increase it's refcount. On exit, a reference to the
current task group is dropped when the last reference to each signal struct is
dropped. The task group is destroyed when the last signal struct referencing
it is freed. At runqueue selection time, IFF a task has no cgroup assignment,
it's current autogroup is used.

The feature is enabled from boot by default if CONFIG_SCHED_AUTOGROUP is
selected, but can be disabled via the boot option noautogroup, and can be
also be turned on/off on the fly via..
   echo [01] > /proc/sys/kernel/sched_autogroup_enabled.
..which will automatically move tasks to/from the root task group.

Some numbers.

A 100% hog overhead measurement proggy pinned to the same CPU as a make -j10

About measurement proggy:
  pert/sec = perturbations/sec
  min/max/avg = scheduler service latencies in usecs
  sum/s = time accrued by the competition per sample period (1 sec here)
  overhead = %CPU received by the competition per sample period

pert/s:       31 >40475.37us:        3 min:  0.37 max:48103.60 avg:29573.74 \
                sum/s:916786us overhead:90.24%
pert/s:       23 >41237.70us:       12 min:  0.36 max:56010.39 avg:40187.01 \
                sum/s:924301us overhead:91.99%
pert/s:       24 >42150.22us:       12 min:  8.86 max:61265.91 avg:39459.91 \
                sum/s:947038us overhead:92.20%
pert/s:       26 >42344.91us:       11 min:  3.83 max:52029.60 avg:36164.70 \
                sum/s:940282us overhead:91.12%
pert/s:       24 >44262.90us:       14 min:  5.05 max:82735.15 avg:40314.33 \
sum/s:967544us overhead:92.22%

Same load with this patch applied.

pert/s:      229 >5484.43us:       41 min:  0.15 max:12069.42 avg:2193.81 \
                sum/s:502382us overhead:50.24%
pert/s:      222 >5652.28us:       43 min:  0.46 max:12077.31 avg:2248.56 \
                sum/s:499181us overhead:49.92%
pert/s:      211 >5809.38us:       43 min:  0.16 max:12064.78 avg:2381.70 \
                sum/s:502538us overhead:50.25%
pert/s:      223 >6147.92us:       43 min:  0.15 max:16107.46 avg:2282.17 \
                sum/s:508925us overhead:50.49%
pert/s:      218 >6252.64us:       43 min:  0.16 max:12066.13 avg:2324.11 \
sum/s:506656us overhead:50.27%

Average service latency is an order of magnitude better with autogroup.
(Imagine that pert were Xorg or whatnot instead)

3. Why does the customer need this? (List the business requirements here)
Customer has not conducted any testing yet, but in reading and analyzing the patch they believe that it could greatly improve desktop interactivity for the way that their apps are written and work in RHEL6 w/ CFS

4. How would the customer like to achieve this? (List the functional requirements here)
patched kernel

5. For each functional requirement listed in question 4, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.
Desktop interactivity average service latency is reduced

6. Is there already an existing RFE upstream or in Red Hat bugzilla?
No

7. How quickly does this need resolved? (desired target release)
6.1

8. Does this request meet the RHEL Inclusion criteria (please review)
Yes

9. List the affected packages
kernel
Comment 5 RHEL Program Management 2011-01-07 04:47:37 UTC 
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.
Comment 6 Suzanne Logcher 2011-01-07 16:04:18 UTC 
This request was erroneously denied for the current release of Red Hat
Enterprise Linux.  The error has been fixed and this request has been
re-proposed for the current release.
Comment 13 RHEL Program Management 2011-02-01 05:29:06 UTC 
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.
Comment 14 RHEL Program Management 2011-02-01 18:53:32 UTC 
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.
Comment 17 Aristeu Rozanski 2011-02-03 15:01:41 UTC 
Patch(es) available on kernel-2.6.32-112.el6
Comment 21 Tom Tracy 2011-02-14 16:30:07 UTC 
Built fresh RHEl6.1-20110207.3 systems on Intel Westmere. When I started
testing, I noticed the performance had dropped significantly using the
.113 kernel. So I went and tested the .94 kernel and the results were
what I reported in January.  Next I tested various kernels to pinpoint
which kernel started this regression. Found that starting with the .112
kernel and further and regression is apparent.   Below is an example

MRG 1.3  with 2.6.32-111.el6.x86_64 kernel


# of Pubs # of Subs   Transfers/Sec   MB/Sec

70354.1    67794.1    1.56402e+06    11.9325
69272.4    66420.6    1.55439e+06    11.859
79369.8    70189.8    1.08535e+06    8.28059
69252      66698.1    1.5352e+06    11.7127
68242.8    65861.3    1.53814e+06    11.7351
68892.4    66442.1    1.55164e+06    11.838
69170.6    66748.6    1.59185e+06    12.1448
68935.4    66392.8    1.54703e+06    11.8029
68947.6    66228.2    1.54834e+06    11.8129
69211.3    66919.8    1.5436e+06    11.7768
68625.4    65933.9    1.54062e+06    11.754

MRG1.3 with 2.6.32-112.el6.x86_64 kernel

# of Pubs # of Subs   Transfers/Sec   MB/Sec

53283.1    51093      1.21434e+06    9.26468
51543      49941.3    1.18038e+06    9.00561
51819.4    50174.1    1.18401e+06    9.03328
51279.9    49856.3    1.17822e+06    8.98911
51442.6    49824.5    1.17183e+06    8.94037
51422.7    50125.5    1.18574e+06    9.04649
51865.6    50458.5    1.20652e+06    9.20499
51363.1    50099.6    1.17458e+06    8.96132
51314.9    49555.3    1.17232e+06    8.94411
51769.2    50297.4    1.18347e+06    9.02915
51551.5    49878.2    1.17719e+06    8.98127
51727.1    50104.6    1.18235e+06    9.02063
51378      49870.1    1.18098e+06    9.01018

As you can see I was averaging ~1.55 million Transfers/Sec at
11.5MB/Sec.  Now with the 112 and higher kernels the throughput has
dropped to 1.17 million Transfer/Sec at 9MB/Sec

The changelog indicated on the 112 autogroup was installed and I tried
disabling it to no avail.  Here is what else has changed in the 112 kernel

I have taken oprofile results running on the .111 and .112 kernels.
There are also perf top logs for each kernel


With the 111 kernel, here are the top routines

samples  %        image name               app name
symbol name
4846706  28.8400  libqpidbroker.so.4.0.0   qpidd
/usr/lib64/libqpidbroker.so.4.0.0
3976948  23.6645  libqpidcommon.so.4.0.0   qpidd
/usr/lib64/libqpidcommon.so.4.0.0
641869    3.8194  libc-2.12.so             qpidd
_int_malloc
594161    3.5355  libstdc++.so.6.0.13      qpidd
/usr/lib64/libstdc++.so.6.0.13
563315    3.3520  libpthread-2.12.so       qpidd
pthread_mutex_lock
488250    2.9053  vmlinux                  vmlinux
futex_wake
453144    2.6964  libpthread-2.12.so       qpidd
pthread_mutex_unlock
419855    2.4983  libc-2.12.so             qpidd
_int_free
332677    1.9796  libc-2.12.so             qpidd                    malloc
299754    1.7837  vmlinux                  vmlinux
futex_wait_setup

With the .112 kernel you see futex wait climb from 2.9% to 12.6% along
with futex_wait_setup and tg_shares_up

samples  %        image name               app name
symbol name
3982579  22.0255  libqpidbroker.so.4.0.0   qpidd
/usr/lib64/libqpidbroker.so.4.0.0
3317650  18.3481  libqpidcommon.so.4.0.0   qpidd
/usr/lib64/libqpidcommon.so.4.0.0
2295122  12.6931  vmlinux                  vmlinux
futex_wake
1759122   9.7287  vmlinux                  vmlinux
futex_wait_setup
833752    4.6110  vmlinux                  vmlinux
tg_shares_up
525718    2.9075  libc-2.12.so             qpidd
_int_malloc
486086    2.6883  libstdc++.so.6.0.13      qpidd
/usr/lib64/libstdc++.so.6.0.13
449337    2.4850  libpthread-2.12.so       qpidd
pthread_mutex_lock
368629    2.0387  libpthread-2.12.so       qpidd
pthread_mutex_unlock
290155    1.6047  libc-2.12.so             qpidd
_int_free
257811    1.4258  libc-2.12.so             qpidd                    malloc
Comment 22 Tom Tracy 2011-02-14 16:37:44 UTC 
Talking with Vivek, he built a test kernel with disables  the autogroup unless it is requested. With the test kernel and autogroup disabled, performance returned to previous kernels. Oprofile results show similar characteristics of the 111 kernel.

With kernel-2.6.32-115.el6.vgoyal.autogroupfix.v1.x86_64 testing Broadcom 1Gb network card


Autogroup enabled in 112 kernel  

8 Bytes
                       Transfers/Sec   MB/Sec
53283.1    51093      1.21434e+06    9.26468
51543      49941.3    1.18038e+06    9.00561
51819.4    50174.1    1.18401e+06    9.03328
51279.9    49856.3    1.17822e+06    8.98911
51442.6    49824.5    1.17183e+06    8.94037
51422.7    50125.5    1.18574e+06    9.04649
51865.6    50458.5    1.20652e+06    9.20499


Autogroup Diabled in kernel  
8 Bytes	MB/Sec (samall sample)
1.58E+006	12.07
1.60E+006	12.17
1.13E+006	8.63
1.59E+006	12.09
1.60E+006	12.22
1.59E+006	12.15
1.60E+006	12.21
1.58E+006	12.07
Comment 24 Aristeu Rozanski 2011-02-18 22:09:15 UTC 
Patch(es) available on kernel-2.6.32-117.el6
Comment 29 errata-xmlrpc 2011-05-23 20:29:33 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0542.html
Note You need to log in before you can comment on or make changes to this bug.