Red Hat Bugzilla – Bug 134338
Intolerable Disk I/O Performance under 64-bit VM: fix I/O buffers
Last modified: 2007-11-30 17:07:13 EST
From Bugzilla Helper:
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET
Description of problem:
Linux guests running under 64-bit z/VM that do a lot of I/O have
severe performance problems.
Up to 98% performance impact has been observed.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1.run many Linux guest under 64-bit z/VM
2.do heavy DASD I/O on multiple Linux systems
Actual Results: VM is virtually dead
VM memory management requires locality of I/O buffers.
Introducing this kind of locality fixes the problem.
Could you please give us some more detail about this bug? Ie: How
many linux guests are you running, what exactly do you mean by "heavy
DASD I/O", and also, could you please go into more detail about your
additional info comment about locality fixing the problem?
The problem occurs with any number of guests. Substantial disk I/O
load can be generated with IOZONE or TPCC. This is the problem:
The current design for I/O under z/VM is that all I/O has to be below
the 2GB bar. For larger z/VM guests like 64-bit Linux guests the
problem exists, that pages above the 2GB bar are used. In those cases
VM has to copy the pages below the 2GB bar in order to execute the
I/O. These additional copy steps are not considered to be a problem.
A problem occurs with the number of pages. If the total size exceeds
2GB storage, then z/VM starts paging into xstor and later to disk
(again through the 2GB area). In such situations z/VM is more or less
Well, this problem doesn't sound like it is a *Linux* problem. It
sounds like a problem with z/VM. This problem should be noted to the
zSeries developers, as I doubt there is much of anything RedHat can do
about it... :)
Martin came back with a satisfactory explanation why my counter
will not work. I dunno about RC, although the patch is enabled by a
kernel parameter, so it should not be too dangerous.
Created attachment 106643 [details]
dasd fixed buffers patch
Created attachment 106644 [details]
dasd fixed buffers patch
Modified in 2.6.9-1.751_EL. Requestor, please test and close.
Regarding comment #9, here's an explanation for posterity:
From: Martin Schwidefsky
> And finally, it seems to me that the block layer is perfectly capable
> of doing this bouncing for you, and it will do it with right GFP, too,
> so you do not have to lean on atomics so much. Please tell me why the
> following will not work:
Fixed I/O buffers are NOT bounce buffers. This is a common
we faced quited often already. The problem are the bounce buffers in
in the linux image running on z/VM. To ease the problem for z/VM linux has
to use as few I/O pages as possible. Then z/VM doesn't have to shuffle
from above the 2G bar to below the 2G bar all the time. If you define
buffers with q->bounce_pfn=0x80000000 linux will happily use up to 2G
I/O pages. This is not what we want. With the fixed I/O buffers patch
240*2 I/O pages per device (if the slab allocation doesn't fail). Which is
about 2MB per device.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.