Bug 67199 - 2.4.18-4,5: very slow 'sync' nfs writes (~50k/sec)
Summary: 2.4.18-4,5: very slow 'sync' nfs writes (~50k/sec)
Keywords:
Status: CLOSED DUPLICATE of bug 67461
Alias: None
Product: Red Hat Linux
Classification: Retired
Component: kernel
Version: 7.3
Hardware: i386
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Ben LaHaise
QA Contact: Brian Brock
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2002-06-20 20:53 UTC by Rex Dieter
Modified: 2014-01-21 22:48 UTC (History)
5 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2002-07-22 16:30:35 UTC
Embargoed:


Attachments (Terms of Use)

Description Rex Dieter 2002-06-20 20:53:28 UTC
kernel-2.4.18-4,kernel-2.4.18-5(errata) yields very slow nfs writes (~50K/sec) 
on mounts using the 'sync' option (and on mounts that do not specify either 
sync or async) against both a 2.4.18-5 and 2.2.19-6.2.16 (rh62) nfs server.  
Both nfs server and client are on fast ethernet switches.  I tried varying 
nfsvers=2,3 and rsize/wsize=1024,4096,8192 all with similar results.

Comment 1 Rex Dieter 2002-06-20 21:15:09 UTC
Sorry, ene exception, kernel-2.4.18-4 nfs client and kernel-2.2.19... nfs 
server does not appear to experience the same write performance degradation 
with the 'sync' option.

Comment 2 Rex Dieter 2002-06-27 14:18:44 UTC
FYI, all machines in question here are Intel network cards, tested with both 
the e100 and eepro100 network drivers.

Comment 3 Ben LaHaise 2002-07-04 18:19:40 UTC
Note that 2.4.18 kernels default to sync operation instead of async.  Does
adding the async option to /etc/exports on the server improve performance?

Comment 4 Rex Dieter 2002-07-04 19:31:14 UTC
mounting with the 'async' option yields ~3-4MB/sec writes.  

Now that you mention it, is async/sync an exports or mount option (or 
both)?  So far... I've only been manually using async/sync when using the 
mount command...

Comment 5 Ben LaHaise 2002-07-04 19:34:14 UTC
It is both.  Previous kernels would default to async exports and mounts by
default, but newer kernels default to sync.

Comment 6 Raphael Clifford 2002-07-09 18:44:17 UTC
There seems to be an important difference in nfs client speed (large slowdown)
between 2.4.18-4 and 2.4.18-5.  Are all the defaults the same between those
kernel versions.  The mount options I am using (to a Solaris nfs server) are  
rw,rsize=32768,wsize=32768,nfsvers=3,proto=tcp

I am also using an eepro100 NIC with the default redhat driver.  This nfs
problem has rendered the linux machines in our lab almost unusable.  I will try
playing with different options but the ones I am using now were fine with 2.4.18-4.

Comment 7 Brian Smith 2002-07-11 17:29:24 UTC
I think this may be related to bug 64921.

I'm using Red Hat 7.3 as a client to a Solaris 8 nfs server.  In kernel 2.4.18-3
everything worked fine, 2.4.18-4 would hang server and client, unless I put in
rsize,wsize of 8192 or less.  With 2.4.18-5, it doesn't hang client or server,
but i get a ratio of client rpc calls to retrans (in nfsstat) of about 10:1 so
it's _real_ slow.  Under the 2.4.18-4 setup it's like 100000:1, sync or async.

So something is still busted when using Solaris as a server, which wasn't busted
with the defaults in 2.4.18-3.

Comment 8 Rex Dieter 2002-07-11 19:25:20 UTC
Here's more data on write speeds on switched fast ethernet.  This was just a 
quick test, but highlights nicely the problem combination, sync on both server 
and client: 
Server	Client	Speed 
async	async	8.7MB/sec 
async	sync	6.8MB/sec 
sync	async	350K/sec 
sync	sync	50k/sec 
 
Is this really the kind of performance I should expect to get? (I hope not)

Comment 9 Seth Vidal 2002-07-18 18:03:11 UTC
2.4.18-5e fixes the slowness problems I was seeing. But sync performance does
suck pretty harshly.



Comment 10 Seth Vidal 2002-07-18 18:32:40 UTC
What changed in 2.4.18-5e from 2.4.18-5?

THe changelog does not appear to have been updated and I was wondering what the
fixes were.

thanks
-sv


Comment 11 Ben LaHaise 2002-07-18 18:43:10 UTC
5e fixed a problem in the eepro100 driver.

Comment 12 Ben LaHaise 2002-07-18 18:45:22 UTC
To keep people uptodate, yes we are aware of the slowness of sync/sync writes,
but the fix is not going to be ready for a while.

Comment 13 Joshua Baker-LePain 2002-07-18 20:11:24 UTC
If I have 2.4.18-5e on both ends, async/async performance is great reading or
writing.  If, however, the server is not 2.4.18-5e, then even async/async read
performance bites.  I've tried both almost vanilla 2.4.18 (patched with XFS 1.1
from SGI) on a 7.3 system as well 2.4.9-31 on a 7.1 system as the server.  In
both cases async/async read performance is under 2MB/s.  Write performance is
OK, but not stellar (~6.5MB/s).

Just trying to point out that this isn't just a problem of sync/sync
performance, which really shouldn't be blazing.

Comment 14 Ben LaHaise 2002-07-18 22:20:08 UTC
A performance issue that is fixed with 5e is a bug in the network driver, and
shouldn't be hoisted on top of the NFS problem.

Comment 15 Sean O'Connell 2002-07-19 21:49:21 UTC
I am seeing substantial improvements running 2.4.18-5e on my desktop (3c59x
network driver) for writes to a RH 7.2 server (2.4.9-31 kernel), but read speeds
are abysmal (however, the stock 2.4.18-5 is bad for both reads and writes).
Reads and writes to Solaris 2.7 servers are better with the 2.4.18-5e kernel.

The interaction with the 2.4.9-31 machines has me concerned the most (async
ameliorates that a bit, but not as much as I would like).

-Sean

Comment 16 Seth Vidal 2002-07-21 17:10:52 UTC
So I guess the question I have is if/when we might see an errata kernel that
fixes this. I ask b/c I'm trying to determine if I should push 2.4.18-5e to the
machines I maintain or if I should wait for the errata.

if its not too far out I'll wait for the errata but if its going to be a while....

thanks


Comment 17 Ben LaHaise 2002-07-22 15:41:46 UTC
The errata will be ready when it's ready.

Comment 18 Seth Vidal 2002-07-22 16:26:35 UTC
I was mostly interested if it was in QA's queue yet or not.

I wasn't trying to be snippy, just trying to avoid duplicating work for myself.


Comment 19 Ben LaHaise 2002-07-22 16:30:31 UTC
The next errata kernels could be out this week, or several weeks; I just can't
say for certain.

Comment 20 Rex Dieter 2002-08-07 16:48:22 UTC
I just tried one of the test kernels (2.4.18-7) as referenced in bug #67461, 
and write speeds are back!  I'm now getting at least >5MB/sec for any 
combination of async/sync. 

*** This bug has been marked as a duplicate of 67461 ***


Note You need to log in before you can comment on or make changes to this bug.