Bug 103305 - Performance issue with sync on RHEL 2.1
Summary: Performance issue with sync on RHEL 2.1
Alias: None
Product: Red Hat Enterprise Linux 2.1
Classification: Red Hat
Component: kernel   
(Show other bugs)
Version: 2.1
Hardware: i686
OS: Linux
Target Milestone: ---
Assignee: Larry Woodman
QA Contact: Brian Brock
Depends On:
TreeView+ depends on / blocked
Reported: 2003-08-28 18:27 UTC by Heather Conway
Modified: 2007-11-30 22:06 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2003-09-26 15:25:03 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

Description Heather Conway 2003-08-28 18:27:01 UTC
Description of problem:
When using the sync option to mount an ext3 filesystem on a LUN, the 
performance degrades to that of a floppy drive. 
Dell PE2650
RHEL 2.1 updated to v2.4.9-e.24
Qlogic QLA2340
QLogic driver v6.04.01 and BIOS v1.34
PowerPath v3.0.3 b065

Version-Release number of selected component (if applicable):
kernel-enterprise-2.4.9-e.24, kernel-smp-2.4.9-e.24, kernel-smp-2.4.9-e.16

How reproducible:
Every time

Steps to Reproduce:
An example with PowerPath:
- mount /dev/emcpowera1 /share
- rsync -rvP /boot /share => 6574740.86 bytes/sec
- rm -rf /share/boot
- mount -o remount,sync /dev/emcpowera1 /share
- mount => to verify that the sync option is enabled
- rsync -rvP /boot /share => 1122518.24 bytes/sec

An example WITHOUT PowerPath:
- mount /dev/sdd1 /share
- rsync -rvP /boot /share => 9204649.60 bytes/sec
- rm -rf /share/boot
- mount -o remount,sync /dev/sdd1 /share
- mount 
- rsync -rvP /boot /share => 1122518.24 bytes/sec
Actual results:

Expected results:

Additional info:

Comment 1 Heather Conway 2003-09-12 17:48:16 UTC
I ran the test again on a Dell 2600.  The following are the results I got:
With PowerPath:
 - mount /dev/emcpowerc1 /share
- rsync -rvP /boot /share => 1360727.20 bytes/sec
- rm -rf /share/boot
- mount -o remount,sync /dev/emcpowera1 /share
- mount 
- rsync -rvP /boot /share => 427.60 bytes/sec

Without PowerPath:
 - mount /dev/sdd1 /share
- rsync -rvP /boot /share => 140070.29 bytes/sec
- rm -rf /share/boot
- mount -o remount,sync /dev/sdd1 /share
- mount 
- rsync -rvP /boot /share => 155689.67 bytes/sec

Comment 2 Arjan van de Ven 2003-09-26 14:43:49 UTC
Just to be clear, this is the qlogic driver binary as Red Hat ships it, right?

Comment 3 Stephen Tweedie 2003-09-26 15:25:03 UTC
This is not a bug: synchronous IO is slow.

Instead of having IOs submitted in large, contiguous chunks to the disk --- both
to the journal and to the files themselves: mount -o sync forces each 4k write
to flush the data to disk, wait for that to complete, then flush the transaction
to the journal, wait for that to complete, then flush the commit block to the
journal, and wait for _that_ to complete.  That's 2 seeks and 3 pipeline stalls
per IO.

So the basic slowdown without EMC modules is entirely expected.  The huge
slowdown with PowerPath is something that EMC will need to debug: one
observation is that drivers with high throughput but high latencies will be
disproportionately affected by all forms of synchronous IO, so if powerpath is
using any form of deferred command completion then that could explain much of
the performance loss.

Note You need to log in before you can comment on or make changes to this bug.