Description of problem: When using the sync option to mount an ext3 filesystem on a LUN, the performance degrades to that of a floppy drive. Environment: Dell PE2650 RHEL 2.1 updated to v2.4.9-e.24 Qlogic QLA2340 QLogic driver v6.04.01 and BIOS v1.34 PowerPath v3.0.3 b065 Version-Release number of selected component (if applicable): kernel-enterprise-2.4.9-e.24, kernel-smp-2.4.9-e.24, kernel-smp-2.4.9-e.16 How reproducible: Every time Steps to Reproduce: An example with PowerPath: - mount /dev/emcpowera1 /share - rsync -rvP /boot /share => 6574740.86 bytes/sec - rm -rf /share/boot - mount -o remount,sync /dev/emcpowera1 /share - mount => to verify that the sync option is enabled - rsync -rvP /boot /share => 1122518.24 bytes/sec An example WITHOUT PowerPath: - mount /dev/sdd1 /share - rsync -rvP /boot /share => 9204649.60 bytes/sec - rm -rf /share/boot - mount -o remount,sync /dev/sdd1 /share - mount - rsync -rvP /boot /share => 1122518.24 bytes/sec Actual results: Expected results: Additional info:
I ran the test again on a Dell 2600. The following are the results I got: With PowerPath: - mount /dev/emcpowerc1 /share - rsync -rvP /boot /share => 1360727.20 bytes/sec - rm -rf /share/boot - mount -o remount,sync /dev/emcpowera1 /share - mount - rsync -rvP /boot /share => 427.60 bytes/sec Without PowerPath: - mount /dev/sdd1 /share - rsync -rvP /boot /share => 140070.29 bytes/sec - rm -rf /share/boot - mount -o remount,sync /dev/sdd1 /share - mount - rsync -rvP /boot /share => 155689.67 bytes/sec
Just to be clear, this is the qlogic driver binary as Red Hat ships it, right?
This is not a bug: synchronous IO is slow. Instead of having IOs submitted in large, contiguous chunks to the disk --- both to the journal and to the files themselves: mount -o sync forces each 4k write to flush the data to disk, wait for that to complete, then flush the transaction to the journal, wait for that to complete, then flush the commit block to the journal, and wait for _that_ to complete. That's 2 seeks and 3 pipeline stalls per IO. So the basic slowdown without EMC modules is entirely expected. The huge slowdown with PowerPath is something that EMC will need to debug: one observation is that drivers with high throughput but high latencies will be disproportionately affected by all forms of synchronous IO, so if powerpath is using any form of deferred command completion then that could explain much of the performance loss.