Hide Forgot
Description of problem: On a 6.2 system, unpacking a large archive from an NFSv4 mounted directory from a 6.2 server, to the local disk takes seconds. cd'ing into an NFS mounted directory, even with that directory export from, and mounted to, the same server, takes 7.5 minutes. The same command performed on a 6.2 system, into an NFSv3 mounted directory, takes 1.5 minutes. Version-Release number of selected component (if applicable): nfs-utils & nfs-utils-libs 1.2.3.15.el6 How reproducible: 100% Steps to Reproduce: 1. mkdir /tmp/foo, export /tmp/foo to same server as /mnt/foo(rw,sync,no_wdelay) 2. fstab entry: canary:/scratch/foo /mnt/foo nfs rw,hard,intr,async 0 0 3. cd /tmp, time <unpack large archive> 4. cd /mnt/foo, time <unpack large archive> Actual results: Approximately 7 min 35 sec. Expected results: Approximately 1.5 minutes Additional info:
Since RHEL 6.3 External Beta has begun, and this bug remains unresolved, it has been rejected as it is not proposed as exception or blocker. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
This is still a show stopper. Redhat has never acknowledged this report nor looked at it. Note that I work for a US government agency, and this impacts us: we cannot migrate to 6.x from 5.x for home directory servers.
Apologies for not looking at this before, but: could you talk to support? That would help make progress. "cd'ing into an NFS mounted directory... takes 7.5 minutes." I'm taking that to mean "cd'ing to an NFS mounted directory and then unpacking a large archive in that directory... takes 7.5 minutes"? Assuming so: how many files and directories does the archive contain? And what kind of filesystem and disk is the export (/tmp/foo) stored on?
I tried using the support email address my manager was using on another issue - I sent that a couple of weeks ago, and got no response. I've tried emailing nfs-main, and it *fails*, with a "too many hops". Yes, I cd into the NFS mounted directory, and use my manager's unpack script to unpack the tar.gz file, from 28M to 92M. I get the time by running the unpack within the time command. I'm going to ext3; tried on ext4, and it's the same. The final straw is that on our RHEL test server, I'm exporting a directory, and mounting it *on* *the* *same* server, and get these results. find /mnt/foo/<unpacked tarfile directory> | wc -l says 4626. mark
Hi Mark, nfs-maint is an alias for developers. What you need to do is to contact Red Hat Support and work with them to debug your configuration if you are customer. Our support team is *really* good at this. Red Hat developers (who are on this bz and behind that email alias) work with support once they gather the data, debug common issues, etc. Please open up a formal support ticket if you have a Red Hat subscription. Thanks!
What you might be seeing is the impact of RHEL6 enabling write barriers - that makes your data safe in face of a power outage. That would explain the performance difference (your RHEL5 config is fast, but not power failure safe). Again, you need to work this with Red Hat support, not developers.
What I did: mkdir /scratch/foo /mnt/foo vi /etc/exports /scratch/foo <thisserver>(rw,sync,no_wdelay) /etc/nfsmount - all defaults, # RPCGSS security flavors # [none, sys, krb5, krb5i, krb5p ] # Sec=sys so no kerboros service nfs start mount <thisserver>:/scratch/foo /mnt/foo test 0: cd /tmp time unpack file.tar.gz test 1: cd /mnt/foo time unpack file.tar.gz Time for test 0 is approx 1.5 min Time for test 1 is approx 7.5 min Please identify any additional information required mark
You seem to be comparing untar something in /tmp to the performance over NFS. It will certainly be much slower over NFS, exact speeds and ratios depend on what is under /tmp and what kind of NFS you use. You still need to find out how to engage Red Hat support officially. If you can send me (rwheeler) details about your agency I will try to hook you up with the correct channel. Thanks!
No response on this for almost a year. I'm going to close it on the basis that the problem report was not an apples-to-apples comparison.
Pasting in Mark's reply that he sent via email: > No, nothing's changed. I AM comparing apples to apples. If my NFS export > server is 5.x, w/ rw,sync, it's 6-7 times faster untaring, while in an > NFS-mounted directory, than it is if the NFS export server is running 6.x > with *exactly* the same parms in exports. > > As far as I can tell, RH never even tested the bug I complained about. These sorts of problems take a fair bit of effort to chase down. Performance problems come down to numbers and you need to ensure that as many variables as possible are taken out of the equation. It's also heavily dependent on things like the server involved, latency between client and server, etc... Our engineering team relies heavily on our support organization to help us do that legwork. In comment #4 and comment #6, we asked whether you could open a case with our support group so they could help you do that. Almost one year later, and it appears that that was not done. If you do wish to do that at this time, then please refer them to this bug and they can reopen it.