Description of Problem: I am observing various problems with rpc.mountd on my new NFS server. Sometimes it times out, and I have even managed to crash tne rpc.mountd process. I have a bit unusual setup here - big LVM partition with ~2200 users, each with his own entry in /etc/exports for the home directory. Most entries look like this: /export/home/aisa/<login> aisa.fi.muni.cz(rw,sync) nymfe??.fi.muni.cz(rw,sync) oreias??.fi.muni.cz(rw,sync) erinys??.fi.muni.cz(rw,sync) atys.fi.muni.cz(rw,sync) pyrrha.fi.muni.cz(rw,sync) (as you can see from the DNS zone fi.muni.cz, I have nymfe01-34, erinys01-20 and oreias01-25 hosts here). Some entries (about 5-10%) are different - I can send the whole /etc/exports on demand. Version-Release number of selected component (if applicable): nfs-utils-1.0.1-2 How Reproducible: 100% Steps to Reproduce: 1. create such long /etc/exports (and corresponding directories to be exported) 2. run exportfs -a (on my dual athlon MP 2000+ with local caching nameserver it takes about 30 seconds) 3. run showmount -e <server_name> Actual Results: showmount reports timeout, rpc.mountd crashes with no message in /var/log/messages Expected Results: The actual export list :-) Additional Information: When I start the rpc.mountd server without running exportfs -a, I get the correct results. After running exportfs -a the showmount -e crashes the server. Another situation is that after exportfs -a I get the timeout when I try to mount anything from the server (this time, however, the mountd keeps running and subsequent mount request succeeds almost immediately). More information (my /etc/exports, strace of the rpc.mountd, etc) available on request. The server is not (yet) in production use, so I can do few days of tests. However, I need to get this running in a week or so.
Well, I've straced rpc.mountd and I think I know what's going on: The showmount program decided the request took too long and closes the connection. When rpc.mountd finishes generating the export list and starts writing the response, it dies on SIGPIPE. This can lead to a DoS attacks even on shorter export lists - just make the showmount close the connection immediately after sending the request :-( -Yenya
I'd like to be able to reproduce both those two scenarios. It's easy enough to kill the showmount, but it would be useful to see your exports list too to see if there's anything specific in it which might make the problem worse and that I should test. You should be able to add it as an attachment to this bugzilla entry, or you can email it directly to me if you don't want it to be visible in a public bug.
I've send my /etc/exports to Stephen by mail. I have another observations. When I ignore SIGPIPE by running this: trap "" PIPE /usr/sbin/rpc.mountd I am not able to kill rpc.mountd. First "showmount -e" after "exportfs -a" still times out, but does not kill mountd. I am willing to accept timeouts for showmount -e, but not for mount requests (which can be quite bad when combined with (for example) autofs). It is possible that the fact that mount request does not kill mountd while showmount request does is because mount runs over UDP (so SIGPIPE is not delivered) and showmount over TCP.
I can't reproduce it yet, but the problem looks real if SIGPIPE is getting past the rpc code. I'm also not sure whether it's a low-level rpc problem or a nfs-utils problem yet; is your glibc fully uptodate?
Adding the code to ignore SIGPIPE in rpc.mountd is really easy. So is increasing the timeout for mount, but I'm not sure that that's the right answer for the "exportfs -a; mount" case. When you say that the next mount after an exportfs fails, could you possibly trap strace output from the rpc.mountd while it's processing that mount request and let me see that?
I've put up a modified 8.0 package which includes the SIGPIPE ignore, at http://people.redhat.com/sct/packages/nfs-utils/nfs-utils-1.0.1-2.1.i386.rpm Could you check if that fixes the problem with showmount killing the server? If it does, we still need to deal with showmount timing out, but that's a trivial change. I'd also still like to see the strace of mount itself timing out after an exportfs, as that sounds like a different problem entirely.
As for glibc: it is RedHat 8.0 and AFAIK there is no glibc in the updates/ directory for 8.0. Your RPM seems to fix the first problem (showmount/mount after exportfs killing the server with sigpipe). I am sending by e-mail two straces of Linux rpc.mountd. First one is strace -t -f -p of the mount request coming from Solaris (The Solaris mount(8) program retries the request when the mount-daemon times out). The other one is strace of the mount request coming from the Linux box (the Linux mount(8) program gives up after the first timeout. Both straces start _after_ the exportfs -a command has finished. Let me know if you want more info. And thanks for your support.
What's happening is that the exportfs is updating a shared NFS-wide state file, but mountd is only rereading that when it needs to, ie. on the subsequent mount. At that point, it needs to reread and revalidate the whole export table, and that in turn involves a large amount of gethostbyname and gethostbyaddr operations, communicating with DNS, ncsd and NIS. That simply takes a while. In fact, in your case it's taking over a minute. There's really nothing that can be done about this --- the NFS mount daemon *WILL* be unable to respond until it has set up the new exports, and it _will_ take a while to pull all of the host information across from the DNS server. So the question is really a timeout problem in mount(8) itself --- we probably ought to be more compatible with the Sun mount binary, and do our retries there correctly. How long does the Linux mount(8) take before it times out for you? And the Solaris one? There's a problem, because increasing the timeouts too long makes things hang for longer if you try to mount from an unresponsive server, so we can't increase the timeouts without limit. A workaround would be to poke the mountd into rereading its whole export table after the exportfs by trying a dummy mount request at that time.
> A workaround would be to poke the mountd into rereading its whole export table > after the exportfs by trying a dummy mount request at that time. Yes,this is what I am currently doing (by running "showmount -e localhost" after "exportfs -a"). However, I think it should be possible (via a run-time option to exportfs, for example) that exportfs kicks the mountd itself after changing the exports list. A better solution would be to make mountd to re-read the export list on background, while answering the mount requests using the old setup. It would of course need that it rereads the export list immediately after exportfs -a, and not later.
No, the old setup may have been modified for security reasons --- we are not allowed to use it after the change, as exports may have been removed for good reason.
sct wrote: > No, the old setup may have been modified for security reasons --- we are not > allowed to use it after the change, as exports may have been removed for good > reason. However, the removal should not be a time-expensive operation. Even when I try to exportfs -u client:/directory, both exportfs and then showmount -e takes _long_ time, probably both exportfs and then rpc.mountd are reading and DNS-querying all the entries of the export list. While I consider reasonable for rpc.mountd to re-read all the entries in the export list after "exportfs -a", I do not consider this reasonable when single entry is added or even deleted from the export list. Maybe each entry in xtab (or whatever) should carry a timestamp or serial # or whatever, and mountd have to re-read only entries with newer timestamp or bigger serial number. BTW, why the mount-daemon has to read (and DNS-translate) all the entries in the export-list? Why shouldn't it search for the appropriate line when queried? Samba does this, for example (no exportfs -a, smb.conf is re-read every time new client logs in). -Yenya
Anything new here? Should I read the source and try to rewrite the relevant parts of mountd/exportfs?
Removing security severity; This is not a security issue.
Created attachment 115800 [details] Patch posted to NFS list on 06/21/2005 There's patch for my optimalization of exportfs and rpc.mountd (attachment). There were some problems with exportfs and rpc.mountd for long export lists - see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=76643 I do optimalization as my bachelors thesis (Facuulty of informatics, Masaryk's university Brno, Czech Republic), under lead of Yenya Kasprzak. Both exportfs and rpc.mount build linked list of exports (shared functions in export.c). Every time they are inserting new export into list, they search for same export in list. I replaced linked list by hash table and functions export_add and export_lookup by functions hash_export_add and hash_export_lookup (export.c). Because some other functions required exportlist as linked list, hash table has some implementation modification im comparison with ordinary hash table. It also keeps exports in linked list and has pointer to head of the list. So there's no need of implementation function <for_all_in_hash_table>. Tomas Richter
Red Hat Linux is no longer supported by Red Hat, Inc. If you are still running Red Hat Linux, you are strongly advised to upgrade to a current Fedora Core release or Red Hat Enterprise Linux or comparable. Some information on which option may be right for you is available at http://www.redhat.com/rhel/migrate/redhatlinux/. Red Hat apologizes that these issues have not been resolved yet. We do want to make sure that no important bugs slip through the cracks. Please check if this issue is still present in a current Fedora Core release. If so, please change the product and version to match, and check the box indicating that the requested information has been provided. Note that any bug still open against Red Hat Linux on will be closed as 'CANTFIX' on September 30, 2006. Thanks again for your help.
The above Tomas Richter's patch improved things a lot, and even with current system (we use RHEL4 on the above NFS server), the problem is not so bad (running named locally helps, as rpc.mountd and exportfs do lots of redundant queries). So I agree this bug can be closed - at least it is not a problem for me now.
OK, closing.