Bug 608432 - crypto: on-demand governor kills write performace, performance governor lowers read performance
crypto: on-demand governor kills write performace, performance governor lower...
Status: CLOSED NEXTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.0
All Linux
high Severity medium
: rc
: ---
Assigned To: Larry Woodman
Madper Xie
:
Depends On:
Blocks: 960070 1056239 846704 987208
  Show dependency treegraph
 
Reported: 2010-06-27 10:01 EDT by David Kovalsky
Modified: 2014-03-31 19:45 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 987208 (view as bug list)
Environment:
Last Closed: 2014-03-28 16:24:21 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/proc/cpuinfo (6.33 KB, text/plain)
2010-07-01 06:55 EDT, David Kovalsky
no flags Details

  None (edit)
Description David Kovalsky 2010-06-27 10:01:48 EDT
I was doing a quick read/write test for luks crypto devices to figure out the best partition for my home and ran into performance issues. 

Write test:
dd if=/dev/zero bs=64k oflag=direct of=/dev/mapper/${name}

Read test:
dd of=/dev/null bs=64k if=/dev/mapper/${name}


Write performace significantly suffered if the cpuspeed governor was set to on-demand. There was a slight performance degradation on read when the governor is set to performance. Quick stats:

Disk is Seagate ST9500420AS with LVM on top, tested on 1GB LVs. Speed is in MB/s. I did a sync + echo 3 > /proc/sys/vm/drop_caches before every run.


                wr (demand/perf)    rd (demand/perf)
(no encryption) 101/101              101/101
aes-xts-plain key-size=256 : 42.5/59.2            73.5/66.8
aes-xts-plain key-size=512 : 35.7/52.1            93.5/85.4
aes-cbc-essiv:sha256 key-size=128  44.9/62.3      72.3/70.8
aes-cbc-essiv:sha256 key-size=256  34.3/47.3      86.5/83.3

As you can see, on-demand governor kills 1/3 (!) of the write performance. I'm able to reproduce this across reboots in multiple runs. 

Attaching /proc/cpuinfo

kernel-2.6.32-37.el6.x86_64
Comment 2 RHEL Product and Program Management 2010-06-27 10:22:58 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 3 Anton Arapov 2010-06-28 08:52:37 EDT
Known issue,... but fair to leave this bug open or either create general one, for ondemand performance, I think.

There were an article, by the way: http://lwn.net/Articles/384132/

The problem is kernel related, so I changed the component.
Thanks, David.
Comment 4 Eric Sandeen 2010-06-30 16:30:35 EDT
This might be interesting to try, not sure which cpu you're on but this is only enabled for some intel processors by default:

commit 6b9d1a64ddd58ff8e402bf8cbb072ff9447c1025
Author: Rik van Riel <riel@redhat.com>
Date:   Mon May 17 22:36:08 2010 -0400

    [kernel] cpufreq: make the iowait-is-busy-time a sysfs tunable
    
    Message-id: <20100517183608.78b17c7a@annuminas.surriel.com>
    Patchwork-id: 25156
    O-Subject: [RHEL6 PATCH 9/9] cpufreq: make the iowait-is-busy-time a sysfs
        tunable
    Bugzilla: 585330
    RH-Acked-by: Larry Woodman <lwoodman@redhat.com>
    
    Fixes bug 585330
    
    From: Arjan van de Ven <arjan@linux.intel.com>
    Subject: [PATCH] cpufreq: make the iowait-is-busy-time a sysfs tunable
    
    Pavel Machek pointed out that not all CPUs have an efficient idle
    at high frequency. Specifically, older Intel and various AMD cpus
    would get a higher power usage when copying files from USB.
    
    Mike Chan pointed out that the same is true for various ARM chips
    as well.
    
    Thomas Renninger suggested to make this a sysfs tunable with a
    reasonable default.
    
    This patch adds a sysfs tunable for the new behavior, and uses
    a very simple function to determine a reasonable default, depending
    on the CPU vendor/type.
    
    Signed-off-by: Arjan van de Ven <arjan@linux.intel.com>
    Signed-off-by: Aristeu Rozanski <arozansk@redhat.com>

so:

# echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy

might be an interesting test.
Comment 5 David Kovalsky 2010-07-01 06:55:33 EDT
Created attachment 428274 [details]
/proc/cpuinfo

Attaching promised /proc/cpuinfo
Comment 6 David Kovalsky 2010-07-01 07:02:47 EDT
Hi Eric, 

tried with xts-plain-256, write. 

echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy

On demand:
dd if=/dev/zero bs=64k of=/dev/mapper/xts-plain-256 oflag=direct
1072689152 bytes (1.1 GB) copied, 27.0709 s, 39.6 MB/s

Performance:
1072689152 bytes (1.1 GB) copied, 18.6565 s, 57.5 MB/s


It is slightly less, but during my measurements the numbers varied by a few MB/s. It looks like a NOOP for my case. Is the patch in kernel-2.6.32-37.el6.x86_64?
Comment 7 Eric Sandeen 2010-07-01 11:51:31 EDT
(In reply to comment #6)
> Hi Eric, 
> 
> tried with xts-plain-256, write. 
> 
> echo 1 > /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy

...

> Is the patch in kernel-2.6.32-37.el6.x86_64?    

Since the sysfs file exists, yes it is.  :)

-Eric
Comment 8 RHEL Product and Program Management 2010-07-15 11:17:10 EDT
This issue has been proposed when we are only considering blocker
issues in the current Red Hat Enterprise Linux release. It has
been denied for the current Red Hat Enterprise Linux release.

** If you would still like this issue considered for the current
release, ask your support representative to file as a blocker on
your behalf. Otherwise ask that it be considered for the next
Red Hat Enterprise Linux release. **
Comment 9 RHEL Product and Program Management 2011-10-07 11:07:06 EDT
Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.
Comment 10 RHEL Product and Program Management 2012-12-14 02:41:34 EST
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Note You need to log in before you can comment on or make changes to this bug.