Red Hat Bugzilla – Bug 619464
hang on existing systems when exporting NFS share to new systems
Last modified: 2011-12-08 10:36:10 EST
Description of problem:
My list of NFS exports has been gradually growing over the years. Right now, for example, my home directories are exported to around 800 hosts, even though only a relatively small subset of those will mount at the same time. I used to just add hosts to /etc/exports on the file server, and run "exportfs -r", and everything would be fine. New systems would be able to mount everything perfectly, and existing systems would not be affected by the additional export at all.
As my list of exports has grown, I've been noticing a problem. Now, when I run exportfs -r, there is an approximate 7-10 second hang on the systems that have already mounted the share, and then everything returns to normal. This doesn't happen *while* exportfs -r is running, but just after it exits. I figured that maybe exportfs was "unexporting"/re-exporting to hosts that already had the share in use which might have caused the problem, so I tried to manually add/remove hosts thinking that this would only affect those hosts, but it did not. Exporting to one new host still causes the hang on all existing hosts.
Since I have multiple exports to all of the hosts, adding one new host can hang things for a while. I can see that reducing the list of exports, or hosts would reduce the delay. What I am wondering is if there is a better way that I can add hosts without affecting connectivity to existing hosts?
The NFS server itself is pretty powerful -- dual quad core box, lots of memory, many NFS threads, exclusive NFS server, etc... Running 4.6, I have nfs-utils 1.0.6-84.EL4, and I can see that this has been upgraded to 1.0.6-93.EL4, but it doesn't sound like anything in the release notes would imply a fix in this area.
Version-Release number of selected component (if applicable):
export NFS directory to say, 800 hosts, then on one of those hosts, mount the directory, and run an "ls -R /export" on it. On the file server, export to one additional host with exportfs. After exportfs returns, the host doing the ls -R will hang for 10 seconds or so.
running 4.6 with kernel 2.6.9-89.0.23.EL
I sent a message on the linux nfs list, and it was suggested that the "-t" option in later versions of rpc.mountd might help. The -t option was not backported to rpc.mountd in 1.0.6-93.EL4. I doubt it will be in 4.9.
One minor correction -- given a large /etc/exports file where I export many filesystems to over 800 systems each, if I add a new export to only say, 1 system, then I use exportfs to manually export that export to one additional host (total 2 exports), I get the same hang on the original host that I was exporting to when I export to the other host. This leads me to believe that the sheer TOTAL size of the exports file has more to do with this problem than the size of the individual export list for one export.
Doing more experimentation, I can verify that it is indeed the length of the exports file that makes a difference... When exportfs runs, it updates /var/lib/nfs/etab. When it exists, rpc.mountd processes the new etab file, and during this time, existing exports are non-functional. The -t option in later versions of rpc.mountd would probably solve this. From the man page of a later rpc.mountd on -t: " This option specifies the number of worker threads that rpc.mountd spawns. The default is 1 thread, which is probably enough. More threads are usually only needed for NFS servers which need to handle mount storms of hundreds of NFS mounts in a few seconds, or when your DNS server is slow or unreliable. ".
If possible, it would be really interesting to see how RHEL5 or 6 behaves in your setup, and whether -t helps there.
Installing a recent nfs-utils from source would also be another way to test this.
It is tricky to test RHEL5 or 6 in the production environment, but I have a couple of other ideas. I tried to compile a later nfs-utils, but ran into a problem when there's a call for libblkid that doesn't exist in the older version of the lib.... I tried compiling the RHEL5 version from source RPM, but this would have required installing all the other dependencies... I tried to backport -t to RHEL4, and I did do it, but when I went to compile the source RPM, I realized that the patches are installed in order, and hence my patch which is based on the original mountd.c wouldn't apply cleanly... but I can fix that tomorrow.. I do have a new theory that the time involved in the transaction is because mountd is doing reverse lookup for each of the hostnames in the DNS. That's 13,000 DNS lookups, and it does this every single time you add or remove a single host via exportfs. On another server where I have 30,000 shares total, the delay is actually around 30 seconds! I didn't even know that before. I'm interested to see what happens if I convert exports over to use IP instead of name... 13,000 or 30,000 DNS lookups might take a while, and it appears that during this process, everything hangs... hmmm... will try tomorrow.
Ok. I don't get it. I converted all of /etc/exports to using IPs, and when exportfs is called, it reverse lookups every IP and converts back to name to place in /var/lib/nfs/etab! why!? There's no option to exportfs to tell it NOT to do that either. This, of course forces mountd to convert them back to IP again. ugh. I guess -t is the only way. :(