Bug 1979816 - update to 2.5.4 broke nfs-mountd
Summary: update to 2.5.4 broke nfs-mountd
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: nfs-utils
Version: 33
Hardware: All
OS: All
unspecified
urgent
Target Milestone: ---
Assignee: Steve Dickson
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-07-07 07:30 UTC by customercare
Modified: 2021-10-09 00:27 UTC (History)
6 users (show)

Fixed In Version: nfs-utils-2.5.4-2.rc3.fc33 nfs-utils-2.5.4-2.rc3.fc34 nfs-utils-2.5.4-2.rc3.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-24 20:32:38 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description customercare 2021-07-07 07:30:31 UTC
Bug state set to urgent according to condition "update broke productionsystems"


Version-Release number of selected component (if applicable):

2.5.4

Description of problem:

The following update:

021-07-07T04:01:45+0200 SUBDEBUG Upgrade: libnfsidmap-1:2.5.4-0.fc33.x86_64
2021-07-07T04:01:45+0200 SUBDEBUG Upgrade: nfs-utils-1:2.5.4-0.fc33.x86_64
2021-07-07T04:01:46+0200 SUBDEBUG Upgraded: nfs-utils-1:2.5.3-2.fc33.x86_64
2021-07-07T04:01:46+0200 SUBDEBUG Upgraded: libnfsidmap-1:2.5.3-2.fc33.x86_64


broke nfs:

# systemctl status nfs-mountd
● nfs-mountd.service - NFS Mount Daemon
     Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static)
     Active: failed (Result: exit-code) since Wed 2021-07-07 09:20:22 CEST; 9s ago
    Process: 4925 ExecStart=/usr/sbin/rpc.mountd (code=exited, status=0/SUCCESS)
   Main PID: 4928 (code=exited, status=1/FAILURE)
        CPU: 43ms

Jul 07 09:20:22 XXX systemd[1]: Starting NFS Mount Daemon...
Jul 07 09:20:22 xxx rpc.mountd[4928]: Unable to watch /proc/fs/nfsd/clients: No such file or directory
Jul 07 09:20:22 XXX systemd[1]: Started NFS Mount Daemon.
Jul 07 09:20:22 xxx systemd[1]: nfs-mountd.service: Main process exited, code=exited, status=1/FAILURE
Jul 07 09:20:22 XXX systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.


Server could no longer mount shares on this device. 

Instantly after the downgrade to 2.5.3 nfs started to work again, as mountd started successfully.

Comment 1 Steve Dickson 2021-07-31 14:58:45 UTC
Hmm... it appears the proc-fs-nfsd.mount service was not run or failed
which is called by  nfs-mountd.service

How reproducible is this.. because it seems to be
working every else

Comment 2 customercare 2021-08-03 10:17:18 UTC
It's 100 reproduceable as thsi test from 5 minutes ago shows:

[root@XXXXX ~]# dnf update nfs-utils
Letzte Prüfung auf abgelaufene Metadaten: vor 0:12:10 am Di 03 Aug 2021 12:01:09 CEST.
Abhängigkeiten sind aufgelöst.
================================================================================================================================================================================================================================================================================
 Package                                                           Architecture                                                   Version                                                                 Repository                                                       Size
================================================================================================================================================================================================================================================================================
Aktualisieren:
 nfs-utils                                                         x86_64                                                         1:2.5.4-0.fc33                                                          updates                                                         421 k

Transaktionsübersicht
================================================================================================================================================================================================================================================================================
Aktualisieren  1 Paket

Gesamte Downloadgröße: 421 k
Ist dies in Ordnung? [j/N]: j
Pakete werden heruntergeladen:
nfs-utils-2.5.4-0.fc33.x86_64.rpm                                                                                                                                                                                                               2.5 MB/s | 421 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Gesamt                                                                                                                                                                                                                                          1.2 MB/s | 421 kB     00:00     
Transaktionsüberprüfung wird ausgeführt
Transaktionsprüfung war erfolgreich.
Transaktion wird getestet
Transaktionstest war erfolgreich.
Transaktion wird ausgeführt
  Vorbereitung läuft    :                                                                                                                                                                                                                                                   1/1 
  Ausgeführtes Scriptlet: nfs-utils-1:2.5.4-0.fc33.x86_64                                                                                                                                                                                                                   1/1 
  Ausgeführtes Scriptlet: nfs-utils-1:2.5.4-0.fc33.x86_64                                                                                                                                                                                                                   1/2 
  Aktualisieren         : nfs-utils-1:2.5.4-0.fc33.x86_64                                                                                                                                                                                                                   1/2 
  Ausgeführtes Scriptlet: nfs-utils-1:2.5.4-0.fc33.x86_64                                                                                                                                                                                                                   1/2 
  Ausgeführtes Scriptlet: nfs-utils-1:2.5.3-2.fc33.x86_64                                                                                                                                                                                                                   2/2 
  Aufräumen             : nfs-utils-1:2.5.3-2.fc33.x86_64                                                                                                                                                                                                                   2/2 
  Ausgeführtes Scriptlet: nfs-utils-1:2.5.3-2.fc33.x86_64                                                                                                                                                                                                                   2/2 
  Überprüfung läuft     : nfs-utils-1:2.5.4-0.fc33.x86_64                                                                                                                                                                                                                   1/2 
  Überprüfung läuft     : nfs-utils-1:2.5.3-2.fc33.x86_64                                                                                                                                                                                                                   2/2 

Aktualisiert:
  nfs-utils-1:2.5.4-0.fc33.x86_64                                                                                                                                                                                                                                               

Fertig.
[root@XXXXX ~]# systemctl restart nfs
Failed to restart nfs.service: Unit nfs.service not found.
[root@XXXXX ~]# systemctl restart nfs-mountd.service 
[root@XXXXX ~]# systemctl status nfs-mountd.service 
● nfs-mountd.service - NFS Mount Daemon
     Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static)
     Active: failed (thawing) (Result: exit-code) since Tue 2021-08-03 12:13:37 CEST; 5s ago
    Process: 519291 ExecStart=/usr/sbin/rpc.mountd (code=exited, status=0/SUCCESS)
   Main PID: 519294 (code=exited, status=1/FAILURE)
        CPU: 44ms

Aug 03 12:13:37 XXXXX systemd[1]: Starting NFS Mount Daemon...
Aug 03 12:13:37 XXXXX rpc.mountd[519294]: Unable to watch /proc/fs/nfsd/clients: No such file or directory
Aug 03 12:13:37 XXXXX systemd[1]: Started NFS Mount Daemon.
Aug 03 12:13:37 XXXXX systemd[1]: nfs-mountd.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 12:13:37 XXXXX systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.

downgrading to 2.5.3 and the services starts again.


who creates /proc/fs/nfsd/clients? 

Is it something the kernel needs to supply?

Comment 3 J. Bruce Fields 2021-08-03 16:04:28 UTC
What's the output of rpc.nfsd?  And is the "nfsd" filesystem mounted at /proc/fs/nfsd/?

Yes, that directory is created by the kernel (if the nfsd filesystem is mounted); /proc/fs/nfsd/clients was added to kernel 5.3.

For compatibility with older kernels, rpc.mountd should be falling back gracefully when it doesn't find /proc/fs/nfsd/clients, rather than exiting with an error.  That sounds like an rpc.mountd bug.

Until that's fixed, either a kernel upgrade or an nfs-utils downgrade should work around this problem for anyone hitting it.

Comment 4 customercare 2021-08-03 16:18:21 UTC
a) 


with : 2.5.4

# systemctl status nfs-mountd.service 
● nfs-mountd.service - NFS Mount Daemon
     Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static)
     Active: failed (thawing) (Result: exit-code) since Tue 2021-08-03 18:12:26 CEST; 2min 20s ago
    Process: 669483 ExecStart=/usr/sbin/rpc.mountd (code=exited, status=0/SUCCESS)
   Main PID: 669486 (code=exited, status=1/FAILURE)
        CPU: 46ms

Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: Starting NFS Mount Daemon...
Aug 03 18:12:26 s120.resellerdesktop.de rpc.mountd[669486]: Unable to watch /proc/fs/nfsd/clients: No such file or directory
Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: Started NFS Mount Daemon.
Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: nfs-mountd.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.

# rpc.nfsd -d
rpc.nfsd: knfsd is currently up


b) 

no, kernel to old.

c) then pls inform upstream. 


d) fine for me, as long as we don't have a security problem in mountd

Comment 5 Sam James 2021-08-14 23:52:48 UTC
This just came up in Gentoo as we upgraded to nfs-utils 2.5.4. A user running an older kernel (4.19.198 in this case) found that they couldn't start NFS, manifesting in the same way as this bug (see https://bugs.gentoo.org/808183).

This commit seems to be the cause: https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=b5aa8378e1a100898d8e0dd628eb704c68667027.

Noting the errors given above (and in the linked bug):
>Aug 03 18:12:26 s120.resellerdesktop.de rpc.mountd[669486]: Unable to watch /proc/fs/nfsd/clients: No such file or directory

/proc/fs/nfsd/clients was introduced in kernel 5.3 according to https://man7.org/linux/man-pages/man8/nfsdclnts.8.html.

Is there a roadmap for how to handle older kernels in nfs-utils here?

Comment 6 Sam James 2021-08-14 23:57:02 UTC
(In reply to sam from comment #5)
> [...]
> This commit seems to be the cause:
> https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;
> h=b5aa8378e1a100898d8e0dd628eb704c68667027.
> 

I meant https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=edeb381593d65a4519c71568c969d3e2937ad911, of course!

Comment 7 Steve Dickson 2021-09-21 18:29:48 UTC
I am not seeing this problem.... 

kernel: 4.18.16-300.fc29.x86_64
nfs-utils: nfs-utils-2.5.4-2.rc2.fc35.x86_64

# systemctl start nfs-utils
# systemctl status nfs-utils
● nfs-utils.service - NFS server and client services
   Loaded: loaded (/usr/lib/systemd/system/nfs-utils.service; static; vendor preset: disabled)
   Active: active (exited) since Tue 2021-09-21 14:27:42 EDT; 47s ago
  Process: 990 ExecStart=/bin/true (code=exited, status=0/SUCCESS)
 Main PID: 990 (code=exited, status=0/SUCCESS)

Sep 21 14:27:42 steved-f29 systemd[1]: Starting NFS server and client services...
Sep 21 14:27:42 steved-f29 systemd[1]: Started NFS server and client services.

What am I missing?

Comment 8 customercare 2021-09-21 23:15:54 UTC
Try : 5.1.20-300.fc30.x86_64

Thats the kernel running at the time, the bug emerged.

Comment 9 customercare 2021-09-21 23:21:33 UTC
This the version causing trouble: nfs-utils-1:2.5.4-0.fc33.x86_64

yours is a newer version, which could have solved the issue. How knows.

Comment 10 Steve Dickson 2021-09-22 14:20:22 UTC
(In reply to customercare from comment #9)
> This the version causing trouble: nfs-utils-1:2.5.4-0.fc33.x86_64
> 
> yours is a newer version, which could have solved the issue. How knows.

Well, I spun up a Fedora 30 vm with a kernel of kernel-5.0.9-301.fc30.x86_64
and installed nfs-utils-2.5.4-0.fc33.x86_64... Added '/home *(rw)' to
/etc/exports... and the server came up just fine... 

# rpm -q nfs-utils
nfs-utils-2.5.4-0.fc33.x86_64
# uname -r
5.0.9-301.fc30.x86_64
# cat /etc/exports
/home *(rw)

systemctl start nfs-server
systemctl status nfs-server
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled)
   Active: active (exited) since Wed 2021-09-22 10:10:08 EDT; 2min 19s ago
  Process: 1941 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
  Process: 1942 ExecStart=/usr/sbin/rpc.nfsd (code=exited, status=0/SUCCESS)
  Process: 1961 ExecStart=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi (code=exited, status=0/SUCCESS)
 Main PID: 1961 (code=exited, status=0/SUCCESS)

Sep 22 10:10:08 steved-f30.fs.lab.eng.bos.redhat.com systemd[1]: Starting NFS server and services...
Sep 22 10:10:08 steved-f30.fs.lab.eng.bos.redhat.com systemd[1]: Started NFS server and services.

So I'm a bit stumped...  Maybe it is an SElinux issue? I do have SELINUX=enforcing in /etc/selinux/config

Comment 11 Steve Dickson 2021-09-22 14:33:17 UTC
Never mind... I see the problem now... 

# rpc.mountd -F -d all
rpc.mountd: Unable to watch /proc/fs/nfsd/clients: No such file or directory

Comment 12 Fedora Update System 2021-09-23 18:26:55 UTC
FEDORA-2021-21c3211d85 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-21c3211d85

Comment 13 Fedora Update System 2021-09-23 18:26:57 UTC
FEDORA-2021-bc48d66739 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-bc48d66739

Comment 14 Fedora Update System 2021-09-23 18:26:59 UTC
FEDORA-2021-7e9ae4b0fe has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-7e9ae4b0fe

Comment 15 Fedora Update System 2021-09-23 19:42:00 UTC
FEDORA-2021-21c3211d85 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-21c3211d85`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-21c3211d85

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 16 Fedora Update System 2021-09-23 20:11:58 UTC
FEDORA-2021-7e9ae4b0fe has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-7e9ae4b0fe`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-7e9ae4b0fe

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2021-09-24 02:52:17 UTC
FEDORA-2021-bc48d66739 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-bc48d66739`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-bc48d66739

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2021-09-24 20:32:38 UTC
FEDORA-2021-21c3211d85 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 19 Fedora Update System 2021-09-24 20:33:06 UTC
FEDORA-2021-bc48d66739 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 20 customercare 2021-09-25 14:57:59 UTC
confirmed: works.

Comment 21 Fedora Update System 2021-10-09 00:27:07 UTC
FEDORA-2021-7e9ae4b0fe has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.