Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 619812 - Please document all tuned profiles and provide use-cases
Please document all tuned profiles and provide use-cases
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: tuned (Show other bugs)
6.0
All Linux
low Severity high
: rc
: ---
Assigned To: Jan Vcelak
qe-baseos-daemons
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-07-30 11:31 EDT by Jiri Hladky
Modified: 2013-03-03 20:27 EST (History)
7 users (show)

See Also:
Fixed In Version: tuned-0.2.19-2.el6
Doc Type: Bug Fix
Doc Text:
Manual pages were updated. 'tuned' manpage newly contains available plugins description (cpu, disk, net). 'tuned-adm' manpage adds descriptions of new profiles and removes inaccuracies in old profiles descriptions.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-19 09:13:50 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Comparison between RHEL6run without tuned profile and with tuned profile server-powersave. (10.63 KB, text/plain)
2010-08-04 09:03 EDT, Kamil Kolakowski
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0581 normal SHIPPED_LIVE tuned bug fix and enhancement update 2011-05-18 13:56:52 EDT

  None (edit)
Description Jiri Hladky 2010-07-30 11:31:31 EDT
Description of problem:
Currently, there is no documentation coming with tuned package which would:
-list all profiles
-provide use cases for profiles
-will describe trade-offs (each profile brings not only benefits but has also some negative effects)

Version-Release number of selected component (if applicable):
tuned-0.2.11-6.el6

How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 2 Phil Knirsch 2010-07-30 11:39:11 EDT
man tuned-adm not enough?

Thanks & regards, Phil
Comment 3 RHEL Product and Program Management 2010-07-30 11:48:00 EDT
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **
Comment 4 Jiri Hladky 2010-07-30 12:21:19 EDT
Hi Phil,

I do apologize, I have overlooked man tuned-adm.

IMHO, some information is not up-to-date:
1)
=======================================================================
latency-performance:
"enables sysctl settings that improve the  latency  performance of your network IO."

This is wrong. latency-performance ONLY sets deadline scheduler. I think its probably only still around for legacy reasons, and anyone that was using it should probably be pointed to the throughput-performance profile instead (or just use no tuning profile at all).
========================================================================
2)
========================================================================
default:
This is the default powersaving profile. It is the lowest of the available profiles in regard to powersaving and only enables CPU and disk plugins of tuned.

Please correct me if I'm wrong but IMHO, default is the same as doing
service tuned stop
========================================================================

I would like to see some trade-offs documented as well. We have seen that server-powersave can tank IO operations on disks dramatically ( regression ~30 % ).

Thanks 
Jirka
Comment 5 Phil Knirsch 2010-08-02 13:02:05 EDT
(In reply to comment #4)
> Hi Phil,
> 
> I do apologize, I have overlooked man tuned-adm.
> 
> IMHO, some information is not up-to-date:
> 1)
> =======================================================================
> latency-performance:
> "enables sysctl settings that improve the  latency  performance of your network
> IO."
> 
> This is wrong. latency-performance ONLY sets deadline scheduler. I think its
> probably only still around for legacy reasons, and anyone that was using it
> should probably be pointed to the throughput-performance profile instead (or
> just use no tuning profile at all).

Ah yes, those were removed recently as the network stack is optimizing that automatically already. We can changed the docu for that part certainly.

> ========================================================================
> 2)
> ========================================================================
> default:
> This is the default powersaving profile. It is the lowest of the available
> profiles in regard to powersaving and only enables CPU and disk plugins of
> tuned.
> 
> Please correct me if I'm wrong but IMHO, default is the same as doing
> service tuned stop

Thats not correct.

The default powersaving profile is specified like this:

default
              This is the default powersaving profile. It is the lowest of the available profiles in regard to powersaving and only enables CPU and disk plugins of tuned.

So tuned-adm off is different from tuned-adm profile default

Hope that clears it up a bit.

> ========================================================================
> 
> I would like to see some trade-offs documented as well. We have seen that
> server-powersave can tank IO operations on disks dramatically ( regression ~30
> % ).
> 

Do you have more information regarding that with some data? That would be really helpful so we can extend the documentation. Also we haven't see such a drastic drop in performance so far, so it would be great to know how you measured this.

Thanks & regards, Phil
Comment 6 Jiri Hladky 2010-08-03 05:12:39 EDT
Hi Phil,

> Ah yes, those were removed recently as the network stack is optimizing that
> automatically already. We can changed the docu for that part certainly.

Great, thanks!


> > Please correct me if I'm wrong but IMHO, default is the same as doing
> > service tuned stop
> 
> Thats not correct.
> 
> The default powersaving profile is specified like this:
> 
> default
>               This is the default powersaving profile. It is the lowest of the
> available profiles in regard to powersaving and only enables CPU and disk
> plugins of tuned.
> 
> So tuned-adm off is different from tuned-adm profile default
> 
> Hope that clears it up a bit.

It's interesting. Could you provide here more details what the difference between tuned-adm off and tuned-adm profile default is?

I have checked the default profile as shipped with tuned 0.2.11-6.el6 and I cannot see anything set:

 [root@hp-bl495cg5-02 default]# grep -v "^#" * | sort -u
ktune.sysconfig:
ktune.sysconfig:ELEVATOR_TUNE_DEVS=""
ktune.sysconfig:SYSCTL_POST="/etc/sysctl.conf"
ktune.sysconfig:USE_KTUNE_D="yes"
tuned.conf:
tuned.conf:[CPUMonitor]
tuned.conf:[CPUTuning]
tuned.conf:[DiskMonitor]
tuned.conf:[DiskTuning]
tuned.conf:enabled=False
tuned.conf:[main]
tuned.conf:[NetMonitor]
tuned.conf:[NetTuning]



> 
> > ========================================================================
> > 
> > I would like to see some trade-offs documented as well. We have seen that
> > server-powersave can tank IO operations on disks dramatically ( regression ~30
> > % ).
> > 
> 
> Do you have more information regarding that with some data? That would be
> really helpful so we can extend the documentation. Also we haven't see such a
> drastic drop in performance so far, so it would be great to know how you
> measured this.

My colleague Kamil Kolakowski is looking into this. He will post here his results.

Thanks
Jirka
Comment 7 Jiri Hladky 2010-08-03 05:19:25 EDT
Hi Phil,

I just wanted to start tuned but it's failing:

==========================================================
[root@hp-bl495cg5-02 default]# tuned-adm profile default
Switching to profile 'default'
Applying ktune sysctl settings:
/etc/ktune.d/tunedadm.conf:                                [  OK  ]
Applying sysctl settings from /etc/sysctl.conf
Starting tuned: Traceback (most recent call last):
  File "/usr/sbin/tuned", line 92, in <module>
    tuned.init(TUNEDDIR, cfgfile, debug = debug)
  File "/usr/share/tuned/tuned.py", line 82, in init
    self.__initplugins__(path, "monitorplugins", self.mp)
  File "/usr/share/tuned/tuned.py", line 43, in __initplugins__
    exec _cmd
  File "<string>", line 1, in <module>
  File "/usr/share/tuned/monitorplugins/net.py", line 113, in <module>
    _plugin = NetMonitor()
  File "/usr/share/tuned/monitorplugins/net.py", line 41, in __init__
    max_speed = self.__calcspeed__( ethcard(d).get_max_speed() );
  File "/usr/share/tuned/tuned_nettool.py", line 221, in ethcard
    ethcard.list[interface] = Nettool(interface)
  File "/usr/share/tuned/tuned_nettool.py", line 40, in __init__
    self.update()
  File "/usr/share/tuned/tuned_nettool.py", line 197, in update
    self.supported_modes.append(1000, True)
TypeError: append() takes exactly one argument (2 given)
                                                           [FAILED]
=======================================================================
[root@hp-bl495cg5-02 default]# uname -a
Linux hp-bl495cg5-02.rhts.eng.bos.redhat.com 2.6.32-54.el6.x86_64 #1 SMP Wed Jul 28 00:06:54 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@hp-bl495cg5-02 default]# rpm -qi tuned
Name        : tuned                        Relocations: (not relocatable)
Version     : 0.2.11                            Vendor: Red Hat, Inc.
Release     : 6.el6                         Build Date: Wed 14 Jul 2010 11:54:18 AM EDT
Install Date: Tue 03 Aug 2010 05:04:39 AM EDT      Build Host: x86-003.build.bos.redhat.com

RHEL6 version: RHEL6.0-20100730.5_ftp-Server-x86_64

Other profiles are working fine:
[root@hp-bl495cg5-02 default]# tuned-adm profile throughput-performance
Switching to profile 'throughput-performance'
Applying ktune sysctl settings:
/etc/ktune.d/tunedadm.conf:                                [  OK  ]
Calling '/etc/ktune.d/tunedadm.sh start':                  [  OK  ]
Applying sysctl settings from /etc/sysctl.conf
Applying deadline elevator: sda sdb                        [  OK  ]


Should I open another BZ for this problem?

Thanks
Jirka
Comment 8 Jiri Hladky 2010-08-03 05:28:26 EDT
Hi Phil,

I have opened BZ 

https://bugzilla.redhat.com/show_bug.cgi?id=620686

for the problem described in Comment 7.

I think it's RHEL 6 blocker.

Thanks
Jirka
Comment 9 Phil Knirsch 2010-08-03 05:39:54 EDT
(In reply to comment #6)
> Hi Phil,
> 
> > Ah yes, those were removed recently as the network stack is optimizing that
> > automatically already. We can changed the docu for that part certainly.
> 
> Great, thanks!
> 
> 
> > > Please correct me if I'm wrong but IMHO, default is the same as doing
> > > service tuned stop
> > 
> > Thats not correct.
> > 
> > The default powersaving profile is specified like this:
> > 
> > default
> >               This is the default powersaving profile. It is the lowest of the
> > available profiles in regard to powersaving and only enables CPU and disk
> > plugins of tuned.
> > 
> > So tuned-adm off is different from tuned-adm profile default
> > 
> > Hope that clears it up a bit.
> 
> It's interesting. Could you provide here more details what the difference
> between tuned-adm off and tuned-adm profile default is?
> 
> I have checked the default profile as shipped with tuned 0.2.11-6.el6 and I
> cannot see anything set:
> 
>  [root@hp-bl495cg5-02 default]# grep -v "^#" * | sort -u
> ktune.sysconfig:
> ktune.sysconfig:ELEVATOR_TUNE_DEVS=""
> ktune.sysconfig:SYSCTL_POST="/etc/sysctl.conf"
> ktune.sysconfig:USE_KTUNE_D="yes"
> tuned.conf:
> tuned.conf:[CPUMonitor]
> tuned.conf:[CPUTuning]
> tuned.conf:[DiskMonitor]
> tuned.conf:[DiskTuning]
> tuned.conf:enabled=False
> tuned.conf:[main]
> tuned.conf:[NetMonitor]
> tuned.conf:[NetTuning]
> 
> 
> 

Ah, yes. By default all plugins from tuned are enabled, so empty sections for plugins basically still mean they are active and enabled. Thats the reason why DiskTuning has the

enabled=False

line in it.

Hope that makes it a bit more clear.


> > 
> > > ========================================================================
> > > 
> > > I would like to see some trade-offs documented as well. We have seen that
> > > server-powersave can tank IO operations on disks dramatically ( regression ~30
> > > % ).
> > > 
> > 
> > Do you have more information regarding that with some data? That would be
> > really helpful so we can extend the documentation. Also we haven't see such a
> > drastic drop in performance so far, so it would be great to know how you
> > measured this.
> 
> My colleague Kamil Kolakowski is looking into this. He will post here his
> results.

Great, thanks!

Regards, Phil
Comment 10 Phil Knirsch 2010-08-03 05:41:17 EDT
(In reply to comment #8)
> Hi Phil,
> 
> I have opened BZ 
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=620686
> 
> for the problem described in Comment 7.
> 
> I think it's RHEL 6 blocker.
> 

Already looked at it and found the problem and fixed it in git. Granted Devel ACK as well and will discuss blocker status for it this afternoon in our RHEL-6 call, but i agree that this should be a blocker as it prevents tuned from working on certain network cards which we already have found.

Thanks & regards, Phil
Comment 11 Jiri Hladky 2010-08-03 08:59:15 EDT
Hi Phil,

as discussed on the IRC could you please add following information to the man page?

[14:22] <jhladky> Phil, could you explain to me difference between tuned-adm profile default and tuned-adm off?
[14:23] <pknirsch> with tuned-adm off you disable both ktune and tuned completely (chkconfig off tuned; chkconfig off ktune)
[14:23] <pknirsch> but tuned has 2 of the tuning plugins activated
[14:23] <pknirsch> or rather, not deactivated
[14:24] <jhladky> What are these plugins doing?
[14:24] <pknirsch> CPU and disk tuning is enabled in tuned for the default profile

[14:25] <pknirsch> and CPU tuning uses PM-QOS when there is low cpu activity.
[14:27] <jhladky> google: Power Management Quality of Service (PM_QOS)
[14:27] <jhladky> Still not sure what PM-QOS does.
[14:27] <pknirsch> http://www.lesswatts.org/projects/power-qos/
[14:28] <pknirsch> you can basically improve CPU idle time with it
[14:28] <jhladky> wake up latencies, right?
[14:29] <pknirsch> aye
[14:29] <jhladky> I assume it works only with ondemand governor, right?
[14:29] <pknirsch> jup
[14:29] <pknirsch> which is default in rhel-6
[14:30] <jhladky> Could we add these inormation into the man page?
[14:30] <pknirsch> certainly. it's really good that someone outside does a review now, too.
[14:31] <pknirsch> we oftentimes miss the obvious things to put in there as we're working with it every day 

[14:25] <pknirsch> network tuning monitors network activity and changes the auto negotiation speed based on that

Summary: document the basic functionality of CPU, network and disk plugins. Please add notice that tuned-adm off differs from tuned-adm profile default.

Thanks a lot!
Jirka
Comment 12 Kamil Kolakowski 2010-08-04 09:00:12 EDT
Hi guys,

I'm replying to message about performance drop on disk IO operations when we use server-powersave tuned profile.

I probably used first time wrong input data for comparing, but now I have statistically correct data.

I run postmark test. First run without any tuned profiles second one with server-powersave tuned profile.
Postmark test is described here: https://wiki.test.redhat.com/Kernel/Performance/postmark_test_description
Postmark basically simulate mail server - tests performance operation on a lots of very small files.

Both runs on the same machine (dell-per210-01.lab.eng.brq.redhat.com) with the same version of RHEL (RHEL6.0-20100722.0). 

SUMMARY OF RESULTS:
I got stable results for both runs. So there is NO REGRESSION AT ALL.

Sorry Jiri, Phil for giving you wrong information about REGRESSION but somehow I used for comparison wrong inputs.

I will attach file with results. First two tables are for ext2, next two are ext3, ext4, xfs. Column O/H is % of comparison between run without profile and with server-powersave tuned profile.

Thanks a lot!
Comment 13 Kamil Kolakowski 2010-08-04 09:03:08 EDT
Created attachment 436525 [details]
Comparison between RHEL6run without tuned profile and with tuned profile server-powersave.

Comparison between RHEL6run without tuned profile and with tuned profile server-powersave.
Comment 14 Phil Knirsch 2010-08-05 08:32:01 EDT
Thanks for the info, Kamil, we'll include that in the next versions of tuned.

Incidentally i saw that this bug is still flagged for RHEL-6.0 but certainly doesn't count as a blocker, so i've moved it to 6.1 now so we can update the documentation then.

Thanks & regards, Phil
Comment 15 Jiri Hladky 2010-08-05 08:50:02 EDT
Hi Phil,

it's certainly not a blocker. Thanks for removing blocker flag.

I still think that changes to the documentation as discussed on IRC could make it into RHEL6.0. I will let you decide, based on your bandwidth.  

Thanks
Jirka
Comment 20 Jan Vcelak 2011-01-13 12:15:31 EST
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Manual pages were updated. 'tuned' manpage newly contains available plugins description (cpu, disk, net). 'tuned-adm' manpage adds descriptions of new profiles and removes inaccuracies in old profiles descriptions.
Comment 22 Kamil Kolakowski 2011-02-07 08:10:44 EST
Retested on

Linux ibm-x3650m3-01.lab.eng.brq.redhat.com 2.6.32-94.el6.x86_64 #1 SMP Tue Dec 28 21:55:53 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

with tuned version tuned-0.2.19-2.el6.noarch.

Closing this bug.
Comment 23 errata-xmlrpc 2011-05-19 09:13:50 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0581.html

Note You need to log in before you can comment on or make changes to this bug.