Bug 1181708

Summary: sm-notify 1.3.1 timeout causes 15 minute boot hang when starting rpc-statd-notify.service
Product: [Fedora] Fedora Reporter: Justin M. Forbes <jforbes>
Component: nfs-utilsAssignee: Steve Dickson <steved>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: bfields, jik, jlayton, moneta.mace, steved, tkirby
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nfs-utils-1.3.1-5.0.fc21 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-20 21:01:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Client journal entry from sm-notify on boot none

Description Justin M. Forbes 2015-01-13 15:58:23 UTC
Created attachment 979660 [details]
Client journal entry from sm-notify on boot

Description of problem:
After the nfs-utils 1.3.1 update, my Fedora 21 client hangs for 15 minutes on boot waiting on rpc-statd-notify.service. This is caused by sm-notify timing out. This did not happen prior to the Jan 03 1.3.1 update.  Client is fully updated Fedora 21 with nfs-utils 1.3.1-4.1.fc21. Server is Fedora 20, also fully updated.

Version-Release number of selected component (if applicable):
1.3.1

How reproducible:
100% on this client, other client doesn't seem to have the issue.

Steps to Reproduce:
1. reboot system
2. watch hang


Actual results:
System hangs for 15 minutes until sm-notify times out

Expected results:
System boots in a reasonable time

Additional info:
Logs attached

Comment 1 japa-fi 2015-01-15 08:09:43 UTC
I have the same issue. Since updating to nfs-utils 1.3.1.-4.1.fc21
The boot sequence gets stuck to:
"A start job is running fo to notify nfs peers of a restart" for some 15 minutes.

Journalctl shows the system is trying to notify "mediaserver", which is my NAS, normally mounted by autofs.service


========== some entries from journalctl ==================
Jan 15 09:20:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Sending PMAP_GETPORT for 100024, 1, udp
Jan 15 09:20:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Added host mediaserver to notify list
Jan 15 09:20:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Host mediaserver due in 120 seconds
Jan 15 09:20:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Received packet...
Jan 15 09:20:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Received PMAP_GETPORT result: 0
Jan 15 09:20:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: No statd on host mediaserver
Jan 15 09:20:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Added host mediaserver to notify list
Jan 15 09:20:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Host mediaserver due in 120 seconds
Jan 15 09:21:52 Oma-t400.localdomain chronyd[696]: Selected source 77.86.199.2
Jan 15 09:22:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Sending PMAP_GETPORT for 100024, 1, udp
Jan 15 09:22:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Added host mediaserver to notify list
Jan 15 09:22:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Host mediaserver due in 120 seconds
Jan 15 09:22:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Received packet...
Jan 15 09:22:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Received PMAP_GETPORT result: 0
Jan 15 09:22:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: No statd on host mediaserver
Jan 15 09:22:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Added host mediaserver to notify list
Jan 15 09:22:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Host mediaserver due in 120 seconds
Jan 15 09:24:45 Oma-t400.localdomain sshd[1281]: fatal: Access denied for user jani by PAM account configuration [preauth]
Jan 15 09:24:51 Oma-t400.localdomain sm-notify[1182]: sm-notify: Unable to notify mediaserver, giving up

========== some entries from journalctl ==================


In addition, nfs-utils 1.3.1.-4.1.fc21 seems to add /etc/systemd/system/remote-fs.target.wants/nfs-client.target and
/etc/systemd/system/multi-user.target.wants/nfs-client.target

the nfs-client.target then causes these problems to occur. 

Noted from the boot.log:
[ INFO ] PNFS blkmaping enablement. is not active.
[ DEPENDESC ] Dependency failed for pNFS block layout mapping daemon.
Perhaps the above is the reason why the notify fails.


The long boot wait time and issue can be worked around by removing
/etc/systemd/system/remote-fs.target.wants/nfs-client.target
/etc/systemd/system/multi-user.target.wants/nfs-client.target

(why are those files there anyway if the service was not active to start with?)

See also: https://bugzilla.redhat.com/show_bug.cgi?id=1182461

Comment 2 Fedora Update System 2015-01-15 11:40:13 UTC
nfs-utils-1.3.1-5.0.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/nfs-utils-1.3.1-5.0.fc21

Comment 3 Fedora Update System 2015-01-17 05:38:55 UTC
Package nfs-utils-1.3.1-5.0.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing nfs-utils-1.3.1-5.0.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-0749/nfs-utils-1.3.1-5.0.fc21
then log in and leave karma (feedback).

Comment 4 japa-fi 2015-01-17 08:12:10 UTC
Grabbed the package from Koji and installed it. Fixed the boot delay for me.

Comment 5 Jonathan Kamens 2015-01-19 04:28:11 UTC
Another possible manifestation of this same issue is that "yum update nfs-utils" hangs for a similar amount of time. At least, I assume it was a similar amount of time; I didn't actually wait 15 minutes to see if it would unhang, because I hadn't yet discovered this bug when my yum update hung, so rather I determined that "systemctl restart nfs-client.target" was hung and I killed it so the yum update would proceed.

Comment 6 trev 2015-01-19 11:26:55 UTC
Had the same issue - 16min 6.938s rpc-statd-notify.service .
nfs-utils-1.3.1-5.0.fc21  fixed it.

Comment 7 Fedora Update System 2015-01-20 21:01:10 UTC
nfs-utils-1.3.1-5.0.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.