Tuned is an optional package. If we want settings by default, configure the kernel that way.

Closing as 'WONTFIX' for initscripts; this is not the right place for this.

Hi Bill,
actually the approach to use tuned/ktune and a profile for s390x sysctl defaults was agreed with Red Hat (Phil Knirsch). Since configuring the kernel or /etc/sysctl.conf was decided by RH no to be the preferred way.

In order to be effective per default the performance optimizations for every System z customer, we need to have the package installed per default (tracked in RH Bug 633452), to have the tuned/ktune service started per default (tracked in this RH Bug) and to have the s390x optimized profile active (tracked in RH Bug 633453).

------- Comment From rsisk.com 2010-10-04 10:42 EDT-------
Code Upstream Status: No Code Required

As discussed with Bill we're going to fix this via a simple change in the systcl.conf file for s390x instead using the recommended values from bugzilla #619377. I'm therefore reopening this bug so we can track it for initscripts for 6.1. Here the associated technical note:


System z Performance
Some of the default tunables in Red Hat Enterprise Linux 6 are currently not optimally configured for System z workloads. Under most circumstances, System z machines will perform better using the following recommendations.
Dirty Ratio
It is recommended that the dirty ratio be set to 40 (Red Hat Enterprise Linux 6 default 20) Changing this tunable tells the system to not spend as much process time too early to write out dirty pages. Add the following line to /etc/sysctl.conf to set this tunable:

vm.dirty_ratio = 40

Scheduler
To increase the average time a process runs continuously and also improve the cache utilization and server style workload throughput at minor latency cost it is recommended to set the following higher values in /etc/sysctl.conf.

kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000
kernel.sched_tunable_scaling = 0
kernel.sched_latency_ns = 80000000

Additionally, deactivating the Fair-Sleepers feature improves performance on a System z machine. To achieve this, set the following value in /etc/sysctl.conf

kernel.sched_features = 15834234

False positive hung task reports
It is recommended to prevent false positive hung task reports (which are rare, but might occur under very heavy overcommitment ratios). This feature can be used, but to improve performance, deactivate it by default by setting the following parameter in /etc/sysctl.conf:

kernel.hung_task_timeout_secs = 0

Thanks & regards, Phil

About the specific settings...

> machines will perform better using the following recommendations.
> Dirty Ratio
> It is recommended that the dirty ratio be set to 40 (Red Hat Enterprise Linux 6
> default 20) Changing this tunable tells the system to not spend as much process
> time too early to write out dirty pages. Add the following line to
> /etc/sysctl.conf to set this tunable:
> 
> vm.dirty_ratio = 40

Why is this s390-specific? What is different about s390 that makes this appropriate?

> Scheduler
> To increase the average time a process runs continuously and also improve the
> cache utilization and server style workload throughput at minor latency cost it
> is recommended to set the following higher values in /etc/sysctl.conf.
> 
> kernel.sched_min_granularity_ns = 10000000
> kernel.sched_wakeup_granularity_ns = 15000000
> kernel.sched_tunable_scaling = 0
> kernel.sched_latency_ns = 80000000

Why would this not be appropriate on other architectures?

> Additionally, deactivating the Fair-Sleepers feature improves performance on a
> System z machine. To achieve this, set the following value in /etc/sysctl.conf
> 
> kernel.sched_features = 15834234

This sounds like a bug, not a configuration change. Please file this against the kernel.

> False positive hung task reports
> It is recommended to prevent false positive hung task reports (which are rare,
> but might occur under very heavy overcommitment ratios). This feature can be
> used, but to improve performance, deactivate it by default by setting the
> following parameter in /etc/sysctl.conf:
> 
> kernel.hung_task_timeout_secs = 0

This is not a performance setting and will not be changed in a s390-specific file. As stated in another bug... if they feel this is an inappropiate setting (too much debugging info, etc), then it should be changed globally, not locally on s390.

(In reply to comment #8)
Hi Bill

> About the specific settings...
> 
> > machines will perform better using the following recommendations.
> > Dirty Ratio
> > It is recommended that the dirty ratio be set to 40 (Red Hat Enterprise Linux 6
> > default 20) Changing this tunable tells the system to not spend as much process
> > time too early to write out dirty pages. Add the following line to
> > /etc/sysctl.conf to set this tunable:
> > 
> > vm.dirty_ratio = 40
> 
> Why is this s390-specific? What is different about s390 that makes this
> appropriate?
For IBM System z is important to have it per default for RHEL 6 on System z. IBM System z neither can nor want to claim that we know how this affects other architectures, but we can do this sure for System z - that is the reason why we want it arch specific, to "avoid cross arch discussions, as it is too hardware related anyway".
Please read past discussion:
Bug 589609 - kernel: Suboptimal vm dirty_ratio default value
Bug 604046 - Profile for tuned missing

> > Scheduler
> > To increase the average time a process runs continuously and also improve the
> > cache utilization and server style workload throughput at minor latency cost it
> > is recommended to set the following higher values in /etc/sysctl.conf.
> > 
> > kernel.sched_min_granularity_ns = 10000000
> > kernel.sched_wakeup_granularity_ns = 15000000
> > kernel.sched_tunable_scaling = 0
> > kernel.sched_latency_ns = 80000000
> 
> Why would this not be appropriate on other architectures?
For IBM System z is important to have it per default for RHEL 6 on System z. If Red Hat decides that it is important for other platforms, is fine for IBM System z as long as it is also default for RHEL on System z.
Please read past discussion:
Bug 589609 - kernel: Suboptimal vm dirty_ratio default value
Bug 590022 - kernel: Process scheduler tuning recommendation for s390
Bug 604046 - Profile for tuned missing

> > Additionally, deactivating the Fair-Sleepers feature improves performance on a
> > System z machine. To achieve this, set the following value in /etc/sysctl.conf
> > 
> > kernel.sched_features = 15834234
> 
> This sounds like a bug, not a configuration change. Please file this against
> the kernel.
Since there were some discussion, we would like to be sure this is the definitive and agreed within Red Hat approach.
Please read past discussion:
Bug 590022 - kernel: Process scheduler tuning recommendation for s390
 
> > False positive hung task reports
> > It is recommended to prevent false positive hung task reports (which are rare,
> > but might occur under very heavy overcommitment ratios). This feature can be
> > used, but to improve performance, deactivate it by default by setting the
> > following parameter in /etc/sysctl.conf:
> > 
> > kernel.hung_task_timeout_secs = 0
> 
> This is not a performance setting and will not be changed in a s390-specific
> file. As stated in another bug... if they feel this is an inappropiate setting
> (too much debugging info, etc), then it should be changed globally, not locally
> on s390.
The technical reasons for this discussion depends on the hypervisor - the RH6.0 discussion ended with an attempt to decide this in general. But it is a fact that the issue we described is hypervisor specific, so every architecture or actually every hypervisor environment has to decide this separate - in the system z case that is IBM for z/VM. And it is a performance item as it only becomes a problem in highly low performing environments due to high over commitment, and in those cases it can make bad cases even worse.
Since there were some discussion, we would like to be sure this is the definitive and agreed within Red Hat approach.
Please read past discussion:
Bug 590005 - kernel/sysctl: DETECT_HUNG_TASK default configuration may flood console due to false positives

Thank you for your support, looking forward how to address this issues for good so that RHEL 6 on System z are happy with the performance, which very critical/expensive in System z.
Gonzalo Muelas Serrano.

(In reply to comment #9)
> > > vm.dirty_ratio = 40
> > 
> > Why is this s390-specific? What is different about s390 that makes this
> > appropriate?
> For IBM System z is important to have it per default for RHEL 6 on System z.
> IBM System z neither can nor want to claim that we know how this affects other
> architectures, but we can do this sure for System z - that is the reason why we
> want it arch specific, to "avoid cross arch discussions, as it is too hardware
> related anyway".
> Please read past discussion:
> Bug 589609 - kernel: Suboptimal vm dirty_ratio default value
> Bug 604046 - Profile for tuned missing

OK.

> > > Scheduler
> > > To increase the average time a process runs continuously and also improve the
> > > cache utilization and server style workload throughput at minor latency cost it
> > > is recommended to set the following higher values in /etc/sysctl.conf.
> > > 
> > > kernel.sched_min_granularity_ns = 10000000
> > > kernel.sched_wakeup_granularity_ns = 15000000
> > > kernel.sched_tunable_scaling = 0
> > > kernel.sched_latency_ns = 80000000
> > 
> > Why would this not be appropriate on other architectures?
> For IBM System z is important to have it per default for RHEL 6 on System z. If
> Red Hat decides that it is important for other platforms, is fine for IBM
> System z as long as it is also default for RHEL on System z.
> Please read past discussion:
> Bug 589609 - kernel: Suboptimal vm dirty_ratio default value
> Bug 590022 - kernel: Process scheduler tuning recommendation for s390
> Bug 604046 - Profile for tuned missing

OK.

> > > Additionally, deactivating the Fair-Sleepers feature improves performance on a
> > > System z machine. To achieve this, set the following value in /etc/sysctl.conf
> > > 
> > > kernel.sched_features = 15834234
> > 
> > This sounds like a bug, not a configuration change. Please file this against
> > the kernel.
> Since there were some discussion, we would like to be sure this is the
> definitive and agreed within Red Hat approach.
> Please read past discussion:
> Bug 590022 - kernel: Process scheduler tuning recommendation for s390

I've read this and read the associated performance/tuned bug; this was actively argued against by upstream kernel maintainers, was *not* recommended by Shak as a performance fix, and was rejected as part of the tuned profile, given that it's not even an option upstream. So I don't think it's appropriate to set here.

> The technical reasons for this discussion depends on the hypervisor - the RH6.0
> discussion ended with an attempt to decide this in general. But it is a fact
> that the issue we described is hypervisor specific, so every architecture or
> actually every hypervisor environment has to decide this separate - in the
> system z case that is IBM for z/VM. And it is a performance item as it only
> becomes a problem in highly low performing environments due to high over
> commitment, and in those cases it can make bad cases even worse.

I don't really agree with this analysis - if your system is so overcommitted that you're hitting this, then any additional effort from having to log the messages is not going to be a deal breaker.

> Since there were some discussion, we would like to be sure this is the
> definitive and agreed within Red Hat approach.
> Please read past discussion:
> Bug 590005 - kernel/sysctl: DETECT_HUNG_TASK default configuration may flood
> console due to false positives

Right, I was heavily involved in this bug, which was originally asking for the same sysctl.conf change. I believe that having a more consistent default across our hypervisors is a better answer; it avoids having to document that different RHEL behaves differently with respect to the same thing.

Bill,

I have yet another convenient solution for sysctls which is a compromise between common and architecture specific settings. However it would require a minimal change in your initscripts package ;-)

The idea is simple, create a /etc/sysctl.conf file and a /etc/sysctl.conf.d/ directory that might contain additional files.  This approach is similar to modprobe.d, init.d, and many other programs.
For initscripts, you just have to:
1. Create and install the /etc/sysctl.d/ directory in the Makefile
2. Change your initscript and call sysctl to load every file found in the directory

Regarding to this discussion, the generic architecture-agnostic settings go into /etc/sysctl.conf.  The remaining specifics, for example, s390x-specific settings can then go into a separate sysctl file which is installed with the s390utils package.

In general, this approach help you and others to delegate responsibility of sysctl settings to particular architectures or even package maintainers. For example, the network related sysctl like rp_filter or ip_forward can then be delivered by the network-related packages.

Please let me know your thoughts on that.
Thanks in advance and kind regards,
Hendrik

FWIW I think there will be other packages that would profit from the introduction of /etc/sysctl.conf.d/

That is tracked in an entirely different bug already in bugzilla.

(And doesn't really apply as a solution to settings that shouldn't be changed.)

(In reply to comment #13)
> That is tracked in an entirely different bug already in bugzilla.

What is the bugzilla number?

Bug 574870.

I got told this is (or will be) implemented via sysctl now!

Please confirm that the implementation has been (will be) done as described in #11 and #12 (LTC BZ #13 and #14) or post any deviations that we understand the implementation. 
Thx

(In reply to comment #19)
> I got told this is (or will be) implemented via sysctl now!

No, it's not currently planned to support this specific request via sysctl.d.

(In reply to comment #20)
> (In reply to comment #19)
> > I got told this is (or will be) implemented via sysctl now!
> 
> No, it's not currently planned to support this specific request via sysctl.d.

My $0.02 and just for clarification: The intended implementation will be to add/change the /etc/sysctl.conf defaults for s390x as described in comment #17 by Bill which makes the sysctl.d change unnecessary.

Thanks & regards, Phil

Created attachment 464598 [details]
system z optimized sysctl settings per default

/etc/sysctl.conf change for s390x

http://git.fedorahosted.org/git/?p=initscripts.git;a=commitdiff;h=186bdd8c5d57c2273c46990e306e1a9aef9a4da2

This enhancement request was evaluated by the full Red Hat Enterprise Linux 
team for inclusion in a Red Hat Enterprise Linux minor release.   As a 
result of this evaluation, Red Hat has tentatively approved inclusion of 
this feature in the next Red Hat Enterprise Linux Update minor release.   
While it is a goal to include this enhancement in the next minor release 
of Red Hat Enterprise Linux, the enhancement is not yet committed for 
inclusion in the next minor release pending the next phase of actual 
code integration and successful Red Hat and partner testing.

------- Comment From mgrf.com 2011-04-05 01:47 EDT-------
This is verified on R 6.1 snapshots - set to verified    Thanks

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0647.html