Bug 67199 - 2.4.18-4,5: very slow 'sync' nfs writes (~50k/sec)
2.4.18-4,5: very slow 'sync' nfs writes (~50k/sec)
Status: CLOSED DUPLICATE of bug 67461
Product: Red Hat Linux
Classification: Retired
Component: kernel (Show other bugs)
7.3
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Ben LaHaise
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2002-06-20 16:53 EDT by Rex Dieter
Modified: 2014-01-21 17:48 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2002-07-22 12:30:35 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Rex Dieter 2002-06-20 16:53:28 EDT
kernel-2.4.18-4,kernel-2.4.18-5(errata) yields very slow nfs writes (~50K/sec) 
on mounts using the 'sync' option (and on mounts that do not specify either 
sync or async) against both a 2.4.18-5 and 2.2.19-6.2.16 (rh62) nfs server.  
Both nfs server and client are on fast ethernet switches.  I tried varying 
nfsvers=2,3 and rsize/wsize=1024,4096,8192 all with similar results.
Comment 1 Rex Dieter 2002-06-20 17:15:09 EDT
Sorry, ene exception, kernel-2.4.18-4 nfs client and kernel-2.2.19... nfs 
server does not appear to experience the same write performance degradation 
with the 'sync' option.
Comment 2 Rex Dieter 2002-06-27 10:18:44 EDT
FYI, all machines in question here are Intel network cards, tested with both 
the e100 and eepro100 network drivers.
Comment 3 Ben LaHaise 2002-07-04 14:19:40 EDT
Note that 2.4.18 kernels default to sync operation instead of async.  Does
adding the async option to /etc/exports on the server improve performance?
Comment 4 Rex Dieter 2002-07-04 15:31:14 EDT
mounting with the 'async' option yields ~3-4MB/sec writes.  

Now that you mention it, is async/sync an exports or mount option (or 
both)?  So far... I've only been manually using async/sync when using the 
mount command...
Comment 5 Ben LaHaise 2002-07-04 15:34:14 EDT
It is both.  Previous kernels would default to async exports and mounts by
default, but newer kernels default to sync.
Comment 6 Raphael Clifford 2002-07-09 14:44:17 EDT
There seems to be an important difference in nfs client speed (large slowdown)
between 2.4.18-4 and 2.4.18-5.  Are all the defaults the same between those
kernel versions.  The mount options I am using (to a Solaris nfs server) are  
rw,rsize=32768,wsize=32768,nfsvers=3,proto=tcp

I am also using an eepro100 NIC with the default redhat driver.  This nfs
problem has rendered the linux machines in our lab almost unusable.  I will try
playing with different options but the ones I am using now were fine with 2.4.18-4.
Comment 7 Brian Smith 2002-07-11 13:29:24 EDT
I think this may be related to bug 64921.

I'm using Red Hat 7.3 as a client to a Solaris 8 nfs server.  In kernel 2.4.18-3
everything worked fine, 2.4.18-4 would hang server and client, unless I put in
rsize,wsize of 8192 or less.  With 2.4.18-5, it doesn't hang client or server,
but i get a ratio of client rpc calls to retrans (in nfsstat) of about 10:1 so
it's _real_ slow.  Under the 2.4.18-4 setup it's like 100000:1, sync or async.

So something is still busted when using Solaris as a server, which wasn't busted
with the defaults in 2.4.18-3.
Comment 8 Rex Dieter 2002-07-11 15:25:20 EDT
Here's more data on write speeds on switched fast ethernet.  This was just a 
quick test, but highlights nicely the problem combination, sync on both server 
and client: 
Server	Client	Speed 
async	async	8.7MB/sec 
async	sync	6.8MB/sec 
sync	async	350K/sec 
sync	sync	50k/sec 
 
Is this really the kind of performance I should expect to get? (I hope not)
Comment 9 Seth Vidal 2002-07-18 14:03:11 EDT
2.4.18-5e fixes the slowness problems I was seeing. But sync performance does
suck pretty harshly.

Comment 10 Seth Vidal 2002-07-18 14:32:40 EDT
What changed in 2.4.18-5e from 2.4.18-5?

THe changelog does not appear to have been updated and I was wondering what the
fixes were.

thanks
-sv
Comment 11 Ben LaHaise 2002-07-18 14:43:10 EDT
5e fixed a problem in the eepro100 driver.
Comment 12 Ben LaHaise 2002-07-18 14:45:22 EDT
To keep people uptodate, yes we are aware of the slowness of sync/sync writes,
but the fix is not going to be ready for a while.
Comment 13 Joshua Baker-LePain 2002-07-18 16:11:24 EDT
If I have 2.4.18-5e on both ends, async/async performance is great reading or
writing.  If, however, the server is not 2.4.18-5e, then even async/async read
performance bites.  I've tried both almost vanilla 2.4.18 (patched with XFS 1.1
from SGI) on a 7.3 system as well 2.4.9-31 on a 7.1 system as the server.  In
both cases async/async read performance is under 2MB/s.  Write performance is
OK, but not stellar (~6.5MB/s).

Just trying to point out that this isn't just a problem of sync/sync
performance, which really shouldn't be blazing.
Comment 14 Ben LaHaise 2002-07-18 18:20:08 EDT
A performance issue that is fixed with 5e is a bug in the network driver, and
shouldn't be hoisted on top of the NFS problem.
Comment 15 Sean O'Connell 2002-07-19 17:49:21 EDT
I am seeing substantial improvements running 2.4.18-5e on my desktop (3c59x
network driver) for writes to a RH 7.2 server (2.4.9-31 kernel), but read speeds
are abysmal (however, the stock 2.4.18-5 is bad for both reads and writes).
Reads and writes to Solaris 2.7 servers are better with the 2.4.18-5e kernel.

The interaction with the 2.4.9-31 machines has me concerned the most (async
ameliorates that a bit, but not as much as I would like).

-Sean
Comment 16 Seth Vidal 2002-07-21 13:10:52 EDT
So I guess the question I have is if/when we might see an errata kernel that
fixes this. I ask b/c I'm trying to determine if I should push 2.4.18-5e to the
machines I maintain or if I should wait for the errata.

if its not too far out I'll wait for the errata but if its going to be a while....

thanks
Comment 17 Ben LaHaise 2002-07-22 11:41:46 EDT
The errata will be ready when it's ready.
Comment 18 Seth Vidal 2002-07-22 12:26:35 EDT
I was mostly interested if it was in QA's queue yet or not.

I wasn't trying to be snippy, just trying to avoid duplicating work for myself.
Comment 19 Ben LaHaise 2002-07-22 12:30:31 EDT
The next errata kernels could be out this week, or several weeks; I just can't
say for certain.
Comment 20 Rex Dieter 2002-08-07 12:48:22 EDT
I just tried one of the test kernels (2.4.18-7) as referenced in bug #67461, 
and write speeds are back!  I'm now getting at least >5MB/sec for any 
combination of async/sync. 

*** This bug has been marked as a duplicate of 67461 ***

Note You need to log in before you can comment on or make changes to this bug.