Bug 173535

Summary: nfs server is slow
Product: [Fedora] Fedora Reporter: Erik A. Espinoza <phomey>
Component: kernelAssignee: Steve Dickson <steved>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Brian Brock <bbrock>
Severity: medium Docs Contact:
Priority: medium    
Version: 5CC: davej, jonstanley, wtogami
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: MassClosed
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-01-20 04:39:34 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Erik A. Espinoza 2005-11-18 00:00:14 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8) Gecko/20051107 Firefox/1.5

Description of problem:
I am getting abysmal speed when using kernel 2.6.14-1.1637_FC4 as an nfs server. Our test lab has two opteron systems, deployed with identical kickstart configs. They are connected via a Gig-E switch.

Version-Release number of selected component (if applicable):
kernel-smp-2.6.14-1.1637_FC4, nfs-utils-1.0.7-12.FC4

How reproducible:
Always

Steps to Reproduce:
1. Setup two hosts
2. Export directory on one, mount on the other
3. measure transfer rate with iptraf or 'iostat -x 5'
  

Actual Results:  I am getting about ~165 megabit/sec between two identical systems when using the latest fc4 kernel on a Gig-E switch.

Expected Results:  I expect to get approx ~900 megabit/sec between two identical systems on a Gig-E switch.

Additional info:

It appears that the issue is with the nfs server only. It also does not appear to be an issue with vanilla 2.6.14.2.

Using 2.6.14-1.1637_FC4smp as a client, I get about ~920 megabit/sec from an nfs server using vanilla 2.6.14.2.

Our vanilla 2.6.14.2 is built using the fc4 kernel config from /boot/config-2.6.14-1.1637_FC4smp.

Also it does not appear to be traced to the nfs patches in the Fedora Kernel. As a test I tried applying the two NFSD related patches to the vanilla 2.6.14.2 (linux-2.6-NFSD-ctlbits.patch, linux-2.6-NFSD-non-null-getxattr.patch) and kernel 2.6.14.2 was still able to get ~920 megabit/sec to a 2.6.14-1.1637_FC4smp client.

It appears that this issue could be related to some other patch that is applied to the Fedora kernel. I will continue testing and will provide more info as necessary.

Comment 1 Erik A. Espinoza 2005-11-23 19:39:10 UTC
After spending some time investigating I found that I could reproduce the slow
NFS problem on a Vanilla kernel by applying linux-2.6-default-elevator.patch to
the Vanilla 2.6.14.

After that I realized that cfq doesn't appear to be NFS aware. I have solved the
symptoms on our NFS Servers by appending "elevator=deadline".

Comment 2 Dave Jones 2005-11-28 20:24:03 UTC
a 5x decrease in speed due to the elevator is very suprising.
what disk hardware/setup do you have ?


Comment 3 Dave Jones 2005-11-28 20:44:23 UTC
also, are you using a specific test to generate those IO patterns ?


Comment 4 Erik A. Espinoza 2005-11-28 22:27:51 UTC
The systems were two machines as follows:
2x Dual Core AMD Opteron 275 2.2GHz
Supermicro H8DAR-T with IPMI (AOC-1UIPMI-B)
4GB PC3200 Reg ECC
4x250GB Seagate SATA (ST3250823AS
3Ware 9000 Series SATA Raid Controller
Raid 5 across all disks

Test data was 10g from urandom that was dd'd to a file called tst.dat.
# dd if=/dev/urandom bs=1024k count=10240 of=/export/tst.dat

The data was simply copied from the /export directory of the nfs server to
/var/tmp while I ran 'iostat -x 5' or 'iptraf'. We unmounted the nfs share from
the server between each test and used the default nfs parameters in the fstab.


Comment 5 Erik A. Espinoza 2005-11-28 22:28:31 UTC
By the way, this was FC4 x86_64. Not sure why this was changed in architecture
to am31.

Comment 6 Dave Jones 2005-12-10 07:38:09 UTC
I couldn't reproduce this, though I don't have a 3ware. I wonder if thats the
key that makes a difference here.


Comment 7 Dave Jones 2006-02-03 05:34:48 UTC
This is a mass-update to all currently open kernel bugs.

A new kernel update has been released (Version: 2.6.15-1.1830_FC4)
based upon a new upstream kernel release.

Please retest against this new kernel, as a large number of patches
go into each upstream release, possibly including changes that
may address this problem.

This bug has been placed in NEEDINFO_REPORTER state.
Due to the large volume of inactive bugs in bugzilla, if this bug is
still in this state in two weeks time, it will be closed.

Should this bug still be relevant after this period, the reporter
can reopen the bug at any time. Any other users on the Cc: list
of this bug can request that the bug be reopened by adding a
comment to the bug.

If this bug is a problem preventing you from installing the
release this version is filed against, please see bug 169613.

Thank you.


Comment 8 Erik A. Espinoza 2006-02-03 19:15:52 UTC
Current kernel version still needs elevator=deadline for decent nfs performance.
Without this the performance is still dirt slow. As per bug 178731, I believe
this is caused by the 3ware driver at some point.

Comment 9 Dave Jones 2006-09-17 02:13:11 UTC
[This comment added as part of a mass-update to all open FC4 kernel bugs]

FC4 has now transitioned to the Fedora legacy project, which will continue to
release security related updates for the kernel.  As this bug is not security
related, it is unlikely to be fixed in an update for FC4, and has been migrated
to FC5.

Please retest with Fedora Core 5.

Thank you.

Comment 10 Dave Jones 2006-10-16 18:20:02 UTC
A new kernel update has been released (Version: 2.6.18-1.2200.fc5)
based upon a new upstream kernel release.

Please retest against this new kernel, as a large number of patches
go into each upstream release, possibly including changes that
may address this problem.

This bug has been placed in NEEDINFO state.
Due to the large volume of inactive bugs in bugzilla, if this bug is
still in this state in two weeks time, it will be closed.

Should this bug still be relevant after this period, the reporter
can reopen the bug at any time. Any other users on the Cc: list
of this bug can request that the bug be reopened by adding a
comment to the bug.

In the last few updates, some users upgrading from FC4->FC5
have reported that installing a kernel update has left their
systems unbootable. If you have been affected by this problem
please check you only have one version of device-mapper & lvm2
installed.  See bug 207474 for further details.

If this bug is a problem preventing you from installing the
release this version is filed against, please see bug 169613.

If this bug has been fixed, but you are now experiencing a different
problem, please file a separate bug for the new problem.

Thank you.

Comment 11 Jon Stanley 2008-01-20 04:39:34 UTC
(this is a mass-close to kernel bugs in NEEDINFO state)

As indicated previously there has been no update on the progress of this bug
therefore I am closing it as INSUFFICIENT_DATA. Please re-open if the issue
still occurs for you and I will try to assist in its resolution. Thank you for
taking the time to report the initial bug.

If you believe that this bug was closed in error, please feel free to reopen
this bug.