Red Hat Bugzilla – Bug 76643
rpc.mountd problems w/ long export list
Last modified: 2007-03-26 23:58:01 EDT
Description of Problem:
I am observing various problems with rpc.mountd on my new NFS server.
Sometimes it times out, and I have even managed to crash tne rpc.mountd process.
I have a bit unusual setup here - big LVM partition with ~2200 users, each with
his own entry in /etc/exports for the home directory. Most entries look like this:
erinys??.fi.muni.cz(rw,sync) atys.fi.muni.cz(rw,sync) pyrrha.fi.muni.cz(rw,sync)
(as you can see from the DNS zone fi.muni.cz, I have nymfe01-34, erinys01-20 and
oreias01-25 hosts here).
Some entries (about 5-10%) are different - I can send the whole /etc/exports on
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. create such long /etc/exports (and corresponding directories to be exported)
2. run exportfs -a (on my dual athlon MP 2000+ with local caching nameserver it
takes about 30 seconds)
3. run showmount -e <server_name>
showmount reports timeout, rpc.mountd crashes with no message in /var/log/messages
The actual export list :-)
When I start the rpc.mountd server without running exportfs -a, I get the
correct results. After running exportfs -a the showmount -e crashes the server.
Another situation is that after exportfs -a I get the timeout when I try to
mount anything from the server (this time, however, the mountd keeps running
and subsequent mount request succeeds almost immediately).
More information (my /etc/exports, strace of the rpc.mountd, etc) available on
request. The server is not (yet) in production use, so I can do few days of
tests. However, I need to get this running in a week or so.
Well, I've straced rpc.mountd and I think I know what's going on: The showmount
program decided the request took too long and closes the connection. When
rpc.mountd finishes generating the export list and starts writing the response,
it dies on SIGPIPE.
This can lead to a DoS attacks even on shorter export lists - just make the
showmount close the connection immediately after sending the request :-(
I'd like to be able to reproduce both those two scenarios. It's easy enough to
kill the showmount, but it would be useful to see your exports list too to see
if there's anything specific in it which might make the problem worse and that I
You should be able to add it as an attachment to this bugzilla entry, or you can
email it directly to me if you don't want it to be visible in a public bug.
I've send my /etc/exports to Stephen by mail.
I have another observations. When I ignore SIGPIPE by running this:
trap "" PIPE
I am not able to kill rpc.mountd. First "showmount -e" after "exportfs -a" still
times out, but does not kill mountd. I am willing to accept timeouts for
showmount -e, but not for mount requests (which can be quite bad when combined
with (for example) autofs).
It is possible that the fact that mount request does not kill mountd
while showmount request does is because mount runs over UDP
(so SIGPIPE is not delivered) and showmount over TCP.
I can't reproduce it yet, but the problem looks real if SIGPIPE is getting past
the rpc code. I'm also not sure whether it's a low-level rpc problem or a
nfs-utils problem yet; is your glibc fully uptodate?
Adding the code to ignore SIGPIPE in rpc.mountd is really easy. So is
increasing the timeout for mount, but I'm not sure that that's the right answer
for the "exportfs -a; mount" case.
When you say that the next mount after an exportfs fails, could you possibly
trap strace output from the rpc.mountd while it's processing that mount request
and let me see that?
I've put up a modified 8.0 package which includes the SIGPIPE ignore, at
Could you check if that fixes the problem with showmount killing the server?
If it does, we still need to deal with showmount timing out, but that's a
trivial change. I'd also still like to see the strace of mount itself timing
out after an exportfs, as that sounds like a different problem entirely.
As for glibc: it is RedHat 8.0 and AFAIK there is no glibc in the updates/
directory for 8.0.
Your RPM seems to fix the first problem (showmount/mount after exportfs killing
the server with sigpipe).
I am sending by e-mail two straces of Linux rpc.mountd. First one is
strace -t -f -p of the mount request coming from Solaris (The Solaris
mount(8) program retries the request when the mount-daemon times out).
The other one is strace of the mount request coming from the Linux box
(the Linux mount(8) program gives up after the first timeout. Both
straces start _after_ the exportfs -a command has finished.
Let me know if you want more info. And thanks for your support.
What's happening is that the exportfs is updating a shared NFS-wide state file,
but mountd is only rereading that when it needs to, ie. on the subsequent mount.
At that point, it needs to reread and revalidate the whole export table, and
that in turn involves a large amount of gethostbyname and gethostbyaddr
operations, communicating with DNS, ncsd and NIS. That simply takes a while.
In fact, in your case it's taking over a minute.
There's really nothing that can be done about this --- the NFS mount daemon
*WILL* be unable to respond until it has set up the new exports, and it _will_
take a while to pull all of the host information across from the DNS server. So
the question is really a timeout problem in mount(8) itself --- we probably
ought to be more compatible with the Sun mount binary, and do our retries there
How long does the Linux mount(8) take before it times out for you? And the
Solaris one? There's a problem, because increasing the timeouts too long makes
things hang for longer if you try to mount from an unresponsive server, so we
can't increase the timeouts without limit.
A workaround would be to poke the mountd into rereading its whole export table
after the exportfs by trying a dummy mount request at that time.
> A workaround would be to poke the mountd into rereading its whole export table
> after the exportfs by trying a dummy mount request at that time.
Yes,this is what I am currently doing (by running "showmount -e localhost"
after "exportfs -a"). However, I think it should be possible (via a run-time
option to exportfs, for example) that exportfs kicks the mountd itself
after changing the exports list.
A better solution would be to make mountd to re-read the export list on
background, while answering the mount requests using the old setup. It would
of course need that it rereads the export list immediately after exportfs -a,
and not later.
No, the old setup may have been modified for security reasons --- we are not
allowed to use it after the change, as exports may have been removed for good
> No, the old setup may have been modified for security reasons --- we are not
> allowed to use it after the change, as exports may have been removed for good
However, the removal should not be a time-expensive operation. Even when I try
to exportfs -u client:/directory, both exportfs and then showmount -e takes
_long_ time, probably both exportfs and then rpc.mountd are reading and
DNS-querying all the entries of the export list.
While I consider reasonable for rpc.mountd to re-read all the entries in the
export list after "exportfs -a", I do not consider this reasonable when
single entry is added or even deleted from the export list. Maybe each entry
in xtab (or whatever) should carry a timestamp or serial # or whatever,
and mountd have to re-read only entries with newer timestamp or bigger serial
BTW, why the mount-daemon has to read (and DNS-translate) all the entries
in the export-list? Why shouldn't it search for the appropriate line
when queried? Samba does this, for example (no exportfs -a, smb.conf is re-read
every time new client logs in).
Anything new here? Should I read the source and try to rewrite the
relevant parts of mountd/exportfs?
Removing security severity; This is not a security issue.
Created attachment 115800 [details]
Patch posted to NFS list on 06/21/2005
There's patch for my optimalization of exportfs and rpc.mountd (attachment).
There were some problems with exportfs and rpc.mountd for long export lists -
I do optimalization as my bachelors thesis (Facuulty of informatics, Masaryk's
university Brno, Czech Republic), under lead of Yenya Kasprzak.
Both exportfs and rpc.mount build linked list of exports (shared functions in
export.c). Every time they are inserting new export into list, they search for
same export in list.
I replaced linked list by hash table and functions export_add and export_lookup
by functions hash_export_add and hash_export_lookup (export.c).
Because some other functions required exportlist as linked list, hash table has
some implementation modification im comparison with ordinary hash table. It
also keeps exports in linked list and has pointer to head of the list. So
there's no need of implementation function <for_all_in_hash_table>.
Red Hat Linux is no longer supported by Red Hat, Inc. If you are still
running Red Hat Linux, you are strongly advised to upgrade to a
current Fedora Core release or Red Hat Enterprise Linux or comparable.
Some information on which option may be right for you is available at
Red Hat apologizes that these issues have not been resolved yet. We do
want to make sure that no important bugs slip through the cracks.
Please check if this issue is still present in a current Fedora Core
release. If so, please change the product and version to match, and
check the box indicating that the requested information has been
provided. Note that any bug still open against Red Hat Linux on will be
closed as 'CANTFIX' on September 30, 2006. Thanks again for your help.
The above Tomas Richter's patch improved things a lot, and even with current
system (we use RHEL4 on the above NFS server), the problem is not so bad
(running named locally helps, as rpc.mountd and exportfs do lots of redundant
So I agree this bug can be closed - at least it is not a problem for me now.