Bug 76643 - rpc.mountd problems w/ long export list
rpc.mountd problems w/ long export list
Status: CLOSED CURRENTRELEASE
Product: Red Hat Linux
Classification: Retired
Component: nfs-utils (Show other bugs)
8.0
i386 Linux
high Severity medium
: ---
: ---
Assigned To: Steve Dickson
Ben Levenson
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2002-10-24 11:13 EDT by Jan "Yenya" Kasprzak
Modified: 2007-03-26 23:58 EDT (History)
5 users (show)

See Also:
Fixed In Version: RHEL 4
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-08-07 16:29:40 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch posted to NFS list on 06/21/2005 (12.17 KB, patch)
2005-06-22 03:14 EDT, Steve Dickson
no flags Details | Diff

  None (edit)
Description Jan "Yenya" Kasprzak 2002-10-24 11:13:31 EDT
Description of Problem:
I am observing various problems with rpc.mountd on my new NFS server.
Sometimes it times out, and I have even managed to crash tne rpc.mountd process.

I have a bit unusual setup here - big LVM partition with ~2200 users, each with
his own entry in /etc/exports for the home directory. Most entries look like this:

/export/home/aisa/<login>      aisa.fi.muni.cz(rw,sync)
nymfe??.fi.muni.cz(rw,sync) oreias??.fi.muni.cz(rw,sync)
erinys??.fi.muni.cz(rw,sync) atys.fi.muni.cz(rw,sync) pyrrha.fi.muni.cz(rw,sync)

(as you can see from the DNS zone fi.muni.cz, I have nymfe01-34, erinys01-20 and
oreias01-25 hosts here).

Some entries (about 5-10%) are different - I can send the whole /etc/exports on
demand.

Version-Release number of selected component (if applicable):
nfs-utils-1.0.1-2

How Reproducible:
100%

Steps to Reproduce:
1. create such long /etc/exports (and corresponding directories to be exported)
2. run exportfs -a (on my dual athlon MP 2000+ with local caching nameserver it
takes about 30 seconds)
3. run showmount -e <server_name>

Actual Results:
showmount reports timeout, rpc.mountd crashes with no message in /var/log/messages

Expected Results:
The actual export list :-)

Additional Information:
When I start the rpc.mountd server without running exportfs -a, I get the
correct results. After running exportfs -a the showmount -e crashes the server.

Another situation is that after exportfs -a I get the timeout when I try to
mount anything from the server (this time, however, the mountd keeps running
and subsequent mount request succeeds almost immediately).

More information (my /etc/exports, strace of the rpc.mountd, etc) available on
request. The server is not (yet) in production use, so I can do few days of
tests. However, I need to get this running in a week or so.
Comment 1 Jan "Yenya" Kasprzak 2002-10-24 11:17:01 EDT
Well, I've straced rpc.mountd and I think I know what's going on: The showmount
program decided the request took too long and closes the connection. When
rpc.mountd finishes generating the export list and starts writing the response,
it dies on SIGPIPE.

This can lead to a DoS attacks even on shorter export lists - just make the
showmount close the connection immediately after sending the request :-(

-Yenya
Comment 2 Stephen Tweedie 2002-10-25 10:17:43 EDT
I'd like to be able to reproduce both those two scenarios.  It's easy enough to
kill the showmount, but it would be useful to see your exports list too to see
if there's anything specific in it which might make the problem worse and that I
should test.

You should be able to add it as an attachment to this bugzilla entry, or you can
email it directly to me if you don't want it to be visible in a public bug.
Comment 3 Jan "Yenya" Kasprzak 2002-10-25 10:58:25 EDT
I've send my /etc/exports to Stephen by mail.

I have another observations. When I ignore SIGPIPE by running this:

trap "" PIPE
/usr/sbin/rpc.mountd

I am not able to kill rpc.mountd. First "showmount -e" after "exportfs -a" still
times out, but does not kill mountd. I am willing to accept timeouts for
showmount -e, but not for mount requests (which can be quite bad when combined
with (for example) autofs).

It is possible that the fact that mount request does not kill mountd
while showmount request does is because mount runs over UDP
(so SIGPIPE is not delivered) and showmount over TCP.
Comment 4 Stephen Tweedie 2002-11-07 11:16:03 EST
I can't reproduce it yet, but the problem looks real if SIGPIPE is getting past
the rpc code.  I'm also not sure whether it's a low-level rpc problem or a
nfs-utils problem yet; is your glibc fully uptodate?
Comment 5 Stephen Tweedie 2002-11-07 12:57:32 EST
Adding the code to ignore SIGPIPE in rpc.mountd is really easy.  So is
increasing the timeout for mount, but I'm not sure that that's the right answer
for the "exportfs -a; mount" case.

When you say that the next mount after an exportfs fails, could you possibly
trap strace output from the rpc.mountd while it's processing that mount request
and let me see that?
Comment 6 Stephen Tweedie 2002-11-08 07:39:23 EST
I've put up a modified 8.0 package which includes the SIGPIPE ignore, at 

  http://people.redhat.com/sct/packages/nfs-utils/nfs-utils-1.0.1-2.1.i386.rpm

Could you check if that fixes the problem with showmount killing the server?

If it does, we still need to deal with showmount timing out, but that's a
trivial change.  I'd also still like to see the strace of mount itself timing
out after an exportfs, as that sounds like a different problem entirely.
Comment 7 Jan "Yenya" Kasprzak 2002-11-11 10:58:31 EST
As for glibc: it is RedHat 8.0 and AFAIK there is no glibc in the updates/
directory for 8.0.

Your RPM seems to fix the first problem (showmount/mount after exportfs killing
the server with sigpipe).

I am sending by e-mail two straces of Linux rpc.mountd. First one is
strace -t -f -p of the mount request coming from Solaris (The Solaris
mount(8) program retries the request when the mount-daemon times out).
The other one is strace of the mount request coming from the Linux box
(the Linux mount(8) program gives up after the first timeout. Both
straces start _after_ the exportfs -a command has finished.

Let me know if you want more info. And thanks for your support.
Comment 8 Stephen Tweedie 2002-11-11 13:07:26 EST
What's happening is that the exportfs is updating a shared NFS-wide state file,
but mountd is only rereading that when it needs to, ie. on the subsequent mount.
 At that point, it needs to reread and revalidate the whole export table, and
that in turn involves a large amount of gethostbyname and gethostbyaddr
operations, communicating with DNS, ncsd and NIS.  That simply takes a while.

In fact, in your case it's taking over a minute.

There's really nothing that can be done about this --- the NFS mount daemon
*WILL* be unable to respond until it has set up the new exports, and it _will_
take a while to pull all of the host information across from the DNS server.  So
the question is really a timeout problem in mount(8) itself --- we probably
ought to be more compatible with the Sun mount binary, and do our retries there
correctly.

How long does the Linux mount(8) take before it times out for you?  And the
Solaris one?  There's a problem, because increasing the timeouts too long makes
things hang for longer if you try to mount from an unresponsive server, so we
can't increase the timeouts without limit.

A workaround would be to poke the mountd into rereading its whole export table
after the exportfs by trying a dummy mount request at that time.
Comment 9 Jan "Yenya" Kasprzak 2002-11-15 06:53:30 EST
> A workaround would be to poke the mountd into rereading its whole export table
> after the exportfs by trying a dummy mount request at that time.

Yes,this is what I am currently doing (by running "showmount -e localhost"
after "exportfs -a"). However, I think it should be possible (via a run-time
option to exportfs, for example) that exportfs kicks the mountd itself
after changing the exports list.

A better solution would be to make mountd to re-read the export list on
background, while answering the mount requests using the old setup. It would
of course need that it rereads the export list immediately after exportfs -a,
and not later.
Comment 10 Stephen Tweedie 2002-11-15 07:09:55 EST
No, the old setup may have been modified for security reasons --- we are not
allowed to use it after the change, as exports may have been removed for good
reason.
Comment 11 Jan "Yenya" Kasprzak 2003-01-27 11:57:04 EST
sct wrote:
> No, the old setup may have been modified for security reasons --- we are not
> allowed to use it after the change, as exports may have been removed for good
> reason.

However, the removal should not be a time-expensive operation. Even when I try
to exportfs -u client:/directory, both exportfs and then showmount -e takes
_long_ time, probably both exportfs and then rpc.mountd are reading and 
DNS-querying all the entries of the export list.

While I consider reasonable for rpc.mountd to re-read all the entries in the
export list after "exportfs -a", I do not consider this reasonable when
single entry is added or even deleted from the export list. Maybe each entry
in xtab (or whatever) should carry a timestamp or serial # or whatever,
and mountd have to re-read only entries with newer timestamp or bigger serial
number.

BTW, why the mount-daemon has to read (and DNS-translate) all the entries
in the export-list? Why shouldn't it search for the appropriate line
when queried? Samba does this, for example (no exportfs -a, smb.conf is re-read
every time new client logs in).

-Yenya

Comment 12 Jan "Yenya" Kasprzak 2003-07-15 12:17:32 EDT
Anything new here? Should I read the source and try to rewrite the
relevant parts of mountd/exportfs?
Comment 13 Josh Bressers 2004-06-18 17:48:22 EDT
Removing security severity; This is not a security issue.
Comment 14 Steve Dickson 2005-06-22 03:14:15 EDT
Created attachment 115800 [details]
Patch posted to NFS list on 06/21/2005 

There's patch for my optimalization of exportfs and rpc.mountd (attachment).

There were some problems with exportfs and rpc.mountd for long export lists -
see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=76643
I do optimalization as my bachelors thesis (Facuulty of informatics, Masaryk's
university Brno, Czech Republic), under lead of Yenya Kasprzak.

Both exportfs and rpc.mount build linked list of exports (shared functions in
export.c). Every time they are inserting new export into list, they search for
same export in list.
I replaced linked list by hash table and functions export_add and export_lookup
by functions hash_export_add and hash_export_lookup (export.c).
Because some other functions required exportlist as linked list, hash table has
some implementation modification im comparison with ordinary hash table. It
also keeps exports in linked list  and has pointer to head of the list. So
there's no need of implementation function <for_all_in_hash_table>.

Tomas Richter
Comment 15 Bill Nottingham 2006-08-07 15:54:28 EDT
Red Hat Linux is no longer supported by Red Hat, Inc. If you are still
running Red Hat Linux, you are strongly advised to upgrade to a
current Fedora Core release or Red Hat Enterprise Linux or comparable.
Some information on which option may be right for you is available at
http://www.redhat.com/rhel/migrate/redhatlinux/.

Red Hat apologizes that these issues have not been resolved yet. We do
want to make sure that no important bugs slip through the cracks.
Please check if this issue is still present in a current Fedora Core
release. If so, please change the product and version to match, and
check the box indicating that the requested information has been
provided. Note that any bug still open against Red Hat Linux on will be
closed as 'CANTFIX' on September 30, 2006. Thanks again for your help.
Comment 16 Jan "Yenya" Kasprzak 2006-08-07 16:08:29 EDT
The above Tomas Richter's patch improved things a lot, and even with current
system (we use RHEL4 on the above NFS server), the problem is not so bad
(running named locally helps, as rpc.mountd and exportfs do lots of redundant
queries).
So I agree this bug can be closed - at least it is not a problem for me now.
Comment 17 Bill Nottingham 2006-08-07 16:29:40 EDT
OK, closing.

Note You need to log in before you can comment on or make changes to this bug.