Bug 1372817

Summary: nfs server start fail: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
Product: Red Hat Enterprise Linux 7 Reporter: Thom Carlin <tcarlin>
Component: nfs-utilsAssignee: Steve Dickson <steved>
Status: CLOSED WORKSFORME QA Contact: Yongcheng Yang <yoyang>
Severity: medium Docs Contact:
Priority: urgent    
Version: 7.2CC: bcodding, bfields, brian, brian.murrell, eguan, extras-qa, gansalmon, info, itamar, jiyin, jlayton, jonathan, kernel-maint, lnie, madhu.chinakonda, mchehab, posta, steved, tcarlin, yoyang
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1233005 Environment:
Last Closed: 2016-09-05 17:40:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1233005    
Bug Blocks:    

Description Thom Carlin 2016-09-02 19:03:45 UTC 
+++ This bug was initially created as a clone of Bug #1233005 +++

Description of problem:

[root@nec-em11 ~]# mkdir /exportdir
[root@nec-em11 ~]# vim /etc/exports
[root@nec-em11 ~]# service nfs start
Redirecting to /bin/systemctl start  nfs.service
Job for nfs-server.service failed. See "systemctl status nfs-server.service" and "journalctl -xe" for details.
[root@nec-em11 ~]# service nfs status
Redirecting to /bin/systemctl status  nfs.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2015-06-17 23:09:16 EDT; 9s ago
  Process: 1784 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE)
  Process: 1779 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 1784 (code=exited, status=1/FAILURE)

Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: Starting NFS server and services...
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com rpc.nfsd[1784]: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com rpc.nfsd[1784]: rpc.nfsd: unable to set any sockets for nfsd
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: nfs-server.service: main process exited, code=exited, status=1/FAILURE
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: Failed to start NFS server and services.
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: Unit nfs-server.service entered failed state.
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: nfs-server.service failed.


Version-Release number of selected component (if applicable):
Fedora 22

How reproducible:
always

Steps to Reproduce:
1. config a export dir in /etc/exports
2. systemctl start nfs-server
3.

Actual results:
[root@nec-em11 ~]# service nfs status
Redirecting to /bin/systemctl status  nfs.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2015-06-17 23:09:16 EDT; 9s ago
  Process: 1784 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE)
  Process: 1779 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 1784 (code=exited, status=1/FAILURE)

Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: Starting NFS server and services...
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com rpc.nfsd[1784]: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com rpc.nfsd[1784]: rpc.nfsd: unable to set any sockets for nfsd
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: nfs-server.service: main process exited, code=exited, status=1/FAILURE
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: Failed to start NFS server and services.
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: Unit nfs-server.service entered failed state.
Jun 17 23:09:16 nec-em11.rhts.eng.bos.redhat.com systemd[1]: nfs-server.service failed.


Expected results:
start ok

Additional info:

--- Additional comment from Yin.JianHong on 2015-06-17 23:28:15 EDT ---

seems because the rpcbind service not start.
  nfs-utils/systemd should auto start the rpcbind service.

--- Additional comment from Yin.JianHong on 2015-06-17 23:36:30 EDT ---

Got it:

[root@nec-em11 ~]# cat /usr/lib/systemd/system/nfs-server.service
[Unit]
Description=NFS server and services
DefaultDependencies=no    #<<<---- this line cause the problem.
Requires= network.target proc-fs-nfsd.mount rpcbind.target

--- Additional comment from Roman C on 2015-06-24 08:20:14 EDT ---

Can confirm. I need to start rbcbind by hand.

--- Additional comment from Fedora Update System on 2015-06-26 11:19:14 EDT ---

nfs-utils-1.3.2-9.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/nfs-utils-1.3.2-9.fc22

--- Additional comment from Fedora Update System on 2015-06-27 08:41:41 EDT ---

Package nfs-utils-1.3.2-9.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing nfs-utils-1.3.2-9.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-10837/nfs-utils-1.3.2-9.fc22
then log in and leave karma (feedback).

--- Additional comment from lnie on 2015-06-29 01:58:53 EDT ---

nfs-utils-1.3.2-9.fc22 works

--- Additional comment from Fedora Update System on 2015-07-02 13:08:36 EDT ---

nfs-utils-1.3.2-9.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

--- Additional comment from Brian J. Murrell on 2015-07-22 08:35:41 EDT ---

I have nfs-utils-1.3.2-9.fc22.x86_64 and I still get failures to start NFS on system boot.

--- Additional comment from mario on 2016-03-20 17:32:08 EDT ---

Fedora 22, nfs-utils-1:1.3.3-6.fc22.i686 such that 'service nfs start' always fails.
Downgrade to nfs-utils-1:1.3.2-4.fc22.i686 fixes it.

--- Additional comment from mario on 2016-03-20 17:37:32 EDT ---

Correction (missing rc3): nfs-utils-1:1.3.3-6.rc3.fc22.i686.
Also: nfs-utils.x86_64-1:1.3.3-6.rc3.fc23 works fine on a Fedora 23 machine.

--- Additional comment from Steve Dickson on 2016-03-21 16:27:43 EDT ---

(In reply to mario from comment #9)
> Fedora 22, nfs-utils-1:1.3.3-6.fc22.i686 such that 'service nfs start'
> always fails.
> Downgrade to nfs-utils-1:1.3.2-4.fc22.i686 fixes it.

I just update f22 to latest upstream release which hopefully 
takes care of this problem
  https://bodhi.fedoraproject.org/updates/FEDORA-2016-b4b42c9157

--- Additional comment from mario on 2016-03-21 18:15:40 EDT ---

(In reply to Steve Dickson from comment #11)
> I just update f22 to latest upstream release which hopefully 
> takes care of this problem
>   https://bodhi.fedoraproject.org/updates/FEDORA-2016-b4b42c9157

at link, I see some update for f23, not for f22, am I wrong?

During QCI 1.0 testing, QE saw this in stage...
um list nfs-utils
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Installed Packages
nfs-utils.x86_64                                                                         1:1.3.0-0.21.el7_2                                                                          @rhel-7-server-rpms

Can we get the newer nfs-utils version (>= 1.3.2) to RHEL 7 to resolve this?

Workaround [As listed above]:
1) systemctl restart rpcbind
2) systemctl restart nfs-server
Comment 3 Yongcheng Yang 2016-09-05 03:00:58 UTC 
Hi Thom, please help to supply the detailed failure logs. Thanks in advance.

As we have never meet this issue during rhel7 testing.

[root@hp-dl585g7-04 ~]# mkdir /export_test /mnt/mnt_test
[root@hp-dl585g7-04 ~]# echo "/export_test *(rw,no_root_squash)" > /etc/exports
[root@hp-dl585g7-04 ~]# service nfs start
Redirecting to /bin/systemctl start  nfs.service
[root@hp-dl585g7-04 ~]# service nfs status
Redirecting to /bin/systemctl status  nfs.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
   Active: active (exited) since Sun 2016-09-04 22:59:23 EDT; 12s ago
  Process: 29326 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
  Process: 29322 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
 Main PID: 29326 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

Sep 04 22:59:23 hp-dl585g7-04.rhts.eng.pek2.redhat.com systemd[1]: Starting NFS server and services...
Sep 04 22:59:23 hp-dl585g7-04.rhts.eng.pek2.redhat.com systemd[1]: Started NFS server and services.
[root@hp-dl585g7-04 ~]# mount $HOSTNAME:/export_test/ /mnt/mnt_test/
[root@hp-dl585g7-04 ~]# nfsstat -m
/mnt/mnt_test from hp-dl585g7-04.rhts.eng.pek2.redhat.com:/export_test
 Flags:	rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.73.4.155,local_lock=none,addr=10.73.4.155

[root@hp-dl585g7-04 ~]# umount /mnt/mnt_test/
[root@hp-dl585g7-04 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.3 Beta (Maipo)
[root@hp-dl585g7-04 ~]#
Comment 4 Thom Carlin 2016-09-05 17:40:19 UTC 
Did not reproduce in production so closing until consistent reproducer.  At that point, please reopen with steps from https://bugzilla.redhat.com/show_bug.cgi?id=1372817#c3 along with sosreport