Bug 1125039

Summary: mount.nfs4: timeout, on clients after reboot of server.
Product: [Fedora] Fedora Reporter: g. artim <gartim>
Component: nfs-utilsAssignee: Steve Dickson <steved>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: bfields, jlayton, steved
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-06-29 21:52:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description g. artim 2014-07-30 23:32:10 UTC
Description of problem:

this morning I rebooted the NFS server while clients were active. After that I continued, even after reboot both the client and server, mount.NFS4: timeout

Version-Release number of selected component (if applicable):
+++++++++++
clients (2)
+++++++++++
3.15.6-200.fc20.x86_64 #1 SMP Fri Jul 18 02:36:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
nfs-utils-1.3.0-2.1.fc20.x86_64

++++++
server
++++++
3.13.9-200.fc20.x86_64 #1 SMP Fri Apr 4 12:13:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
nfs-utils-1.3.0-2.1.fc20.x86_64

How reproducible:
dont know, but its a huge file system of about 12TB/btrfs/lzo.

Steps to Reproduce:
1. above description (drop the nfs-server, will clients are rsync'ing)
2.
3.

Actual results:
timeout

Expected results:
Just mount the filesystem

Additional info:

I logged into the server and issued:
   exportfs -a
   exportfs -f
   exportfs -r
   exportfs -a

and in less then a second I could mount the filesystem.

not technically sure this is a bug or a procedural thing I've missed? but it had me pulling my hair out (and little of that)! Please advise if this is a bug or not? thanks!

Comment 1 J. Bruce Fields 2014-08-14 19:31:15 UTC
Sounds like there may be a bug, but:  I don't understand why you were trying to mount after the reboot?

When the server reboots while the clients are active, clients shouldn't have to do anything; just wait for the reboot (plus another 90 seconds for the NFSv4 grace period) and normal operation should resume.

What exactly was the sequence of events here, with about how much time between each step?

Comment 2 g. artim 2014-08-14 19:55:14 UTC
couple of things:
the filesystem is huge, 20TB, btrfs and takes a long time to mount on the nfsserver -- not sure why this is. I may switch back to xfs, seemed more stable under nfs.

order of this issue:
1) reboot nfsserver after yum update -y.
2) nfsserver slow booting waiting on mount, as above.
3) log in to clients and issue 'mount -a' and the mount times out.
   on some clients, both fedora and non-fedora, the nfs mount comes back as you said without a 'mount -a' needed.
4) issue the exportfs -a -f -r .
5) the problem clients 'mount -a' happens in less then a second.

caveat:
the clients that have the problem have been 'fedup' for quite a few releases, god bless me! the fedora client that works as you said, has had less upgrades and newer MB/CPU/etc, its *not* fedup free...I know fedup can be suspect.

Comment 3 g. artim 2014-08-14 19:57:34 UTC
forgot to say that i only tried the 'mount -a' on the clients *after* a clean mount on the nfsserver. g.)

Comment 4 J. Bruce Fields 2014-08-14 20:02:59 UTC
I doubt fedup is an issue, but the largish btrfs fs does make this somewhat of an unusual case.

I guess if I were to debug this my next steps would be:

  - reproduce the problem, and at step 3 above run "rpcdebug -m rpc -s cache", then get the contents of /proc/net/rpc/*/content; they may show that the server is trying to ask mountd about the problem export?

  - reproduce the problem and get a network trace showing all the traffic between the problem client and server.  Look in particular for rpc's that the client is sending to the server without getting any reply.

Comment 5 g. artim 2014-08-14 20:11:54 UTC
sounds like you're thinking its a network/rpc issue, no?

I also wonder if btrfs is doing something that keeps the fs from *really* being available. wish there was someway I could put the nfsserver in debug or maybe extra logging mode to see the errors/chatter...but I guess thats what your doing on the rpc, net trace suggestion. thanks much.

Comment 6 Fedora End Of Life 2015-05-29 12:31:24 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 7 Fedora End Of Life 2015-06-29 21:52:05 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.